Unmanned aerial vehicle image positioning precision improvement method and system under GNSS denial condition

Abstract

The present application relates to the technical field of unmanned aerial vehicle positioning, and particularly relates to a GNSS denial condition unmanned aerial vehicle image positioning precision improvement method and system, target image data and reference image data are collected; the target image data and the reference image data are pretreated in turn, and for the pretreated data, the feature vectors of the image data are extracted by separating the RGB channels, the approximate position of the target image data on the reference image data is obtained based on the similarity of the feature vectors between the target image data and the reference image data; the corresponding adjacent unmanned aerial vehicle images are extracted according to the sequence image strip relationship in the target image data and are composed into a sequence image strip group; the features are extracted and matched based on the sequence image strip group and through a pre-trained deep learning neural network model, and the corresponding relationship between the target image data and the reference image data is constructed to obtain the precise geographic coordinates of the unmanned aerial vehicle image. The present application can improve the positioning precision of the unmanned aerial vehicle image and has strong robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle positioning, and particularly relates to a method and system for improving positioning accuracy of unmanned aerial vehicle images under GNSS denial conditions. BACKGROUND

[0002] Unmanned aerial vehicles have the advantages of being able to adapt to complex terrain environments and fly at high speed, and thus have been successfully applied to many fields such as agriculture, firefighting, express delivery, field search and rescue, location tracking and military operations. High-precision positioning information of the unmanned aerial vehicle comes from a global navigation satellite system (GNSS). In complex environments such as valleys, forests, deserts, water areas and urban buildings, and in the case of human interference, the airborne GNSS receiver is prone to problems such as signal weakening, interference and multipath effect, resulting in unreliable positioning of the unmanned aerial vehicle. When the unmanned aerial vehicle is in the above-mentioned harsh environments, high-precision positioning information cannot be obtained by relying on the GNSS, and the positioning information contained in the images taken by the unmanned aerial vehicle is also inaccurate. When surveying and mapping using the images of the unmanned aerial vehicle, the images of the unmanned aerial vehicle are required to have high-precision positioning information. Therefore, under the GNSS denial condition, the positioning method of the unmanned aerial vehicle cannot meet the actual demand, and external data needs to be used to improve the precision of the images of the unmanned aerial vehicle.

[0003] Currently, many scholars have studied the positioning of UAVs under GNSS denial conditions. For example, there is a method of using conditional generative adversarial neural network cGAN to convert aerial images into a representation similar to a street map to realize visual self-positioning of the UAV. The median positioning error of this method is 22.7 m, and the median error of the entire data set is about 40 m, so the positioning error is large. For example, there is a method based on a similarity function similar to mutual information, which uses a series of geographic images to splice into a map as a positioning reference. This method is suitable for positioning in textured urban environments, but it is less robust in areas with weak texture. At the same time, there is a method of using histogram of oriented gradient (HOG) features to register the UAV frame and satellite image, and then using a particle filter algorithm to position the UAV. This method is not very robust and requires buildings, roads, and other texture features to be clear. For example, there is a method of using feature recognition to identify landmark buildings to determine the positioning information of the UAV. This method requires pre-installing the images of buildings in the task area to achieve feature extraction and UAV image positioning, which is not conducive to rapid positioning of UAV images. For example, there is a method based on convolutional neural network (CNN) and using existing satellite images to train a deep learning model. The UAV image and the geographic reference satellite image are transmitted and matched through a convolutional neural network composed of VGG16 neural network layers, and the average positioning error is less than 8 meters. This method requires pre-training of the model using similar images in the task area, and it is not very robust for other scenarios. SUMMARY

[0004] The present application provides a method and system for improving the positioning accuracy of UAV images under GNSS denial conditions, which solves the problems of weak robustness and large error in existing UAV image positioning under GNSS denial conditions.

[0005] According to the design scheme provided by the present application, on the one hand, a method for improving the positioning accuracy of UAV images under GNSS denial conditions is provided, which comprises:

[0006] Collecting target image data for simulating UAV images taken under GNSS denial conditions and reference image data for UAV image positioning reference, wherein the UAV image without positioning information is used as the target image data, and the satellite image with positioning information is used as the reference image data;

[0007] The target image data and the reference image data are preprocessed in sequence, and the feature vectors of the image data are extracted by separating the RGB channels based on the preprocessed data. The approximate position of the target image data on the reference image data is obtained based on the similarity of the feature vectors between the target image data and the reference image data.

[0008] According to the approximate position of the target image data on the reference image data, corresponding adjacent unmanned aerial vehicle images are extracted according to the sequence image strip relationship in the target image data, and a sequence image strip group is formed;

[0009] Based on the sequence image strip group and through a pre-trained deep learning neural network model, features between the target image data and the reference image data are extracted and matched, and a corresponding relationship between the target image data and the reference image data is constructed according to a matching result, so that accurate geographic coordinates corresponding to unmanned aerial vehicle image pixel coordinates are obtained through the corresponding relationship.

[0010] As the GNSS rejection condition unmanned aerial vehicle image positioning precision improvement method of the application, further, the target image data and the reference image data are sequentially preprocessed, including:

[0011] First, the target image data and the reference image data are respectively encoded by longitude and latitude, and the encoded image data is cut into blocks according to a preset cutting rule;

[0012] Then, the cut image data is renamed, reordered and scaled according to the longitude and latitude encoding.

[0013] As the GNSS rejection condition unmanned aerial vehicle image positioning precision improvement method of the application, further, according to the sequence image strip relationship in the target image data, corresponding adjacent unmanned aerial vehicle images are extracted, including:

[0014] First, the unmanned aerial vehicle image with the highest similarity to the reference image data is selected as the strip center image;

[0015] Then, each N image before and after the strip center image is selected from the sequence image of the target image data in turn to form a sequence image strip group with a heading overlap degree and containing adjacent unmanned aerial vehicle images.

[0016] As the GNSS rejection condition unmanned aerial vehicle image positioning precision improvement method of the application, further, the feature extraction and matching between the target image data and the reference image data are performed through a pre-trained deep learning neural network model, including:

[0017] First, the SuperPoint model is used to extract feature points from the sequence image strip group and the corresponding reference image;

[0018] Then, the SuperGlue network model is used to match the extracted feature points and output.

[0019] As the GNSS rejection condition under the unmanned aerial vehicle image positioning precision improvement method of the application, further, the SuperPoint model comprises: a feature encoder for spatial mapping of input image features, a key point decoder for extracting spatial feature interest points and a descriptor decoder for extracting spatial feature interest point descriptors, wherein the feature encoder is composed of a convolution layer, a Max-Pooling layer and a nonlinear activation layer, the key point decoder is composed of a point-by-point convolution layer, a Softmax activation function and a tensor transformation function, and the descriptor decoder is composed of a convolution layer, an interpolation function and L2 normalization.

[0020] As the GNSS rejection condition under the unmanned aerial vehicle image positioning precision improvement method of the application, further, the SuperGlue network model comprises: an attention GNN for generating a feature matching vector and an optimal matching layer for obtaining an optimal matching and outputting, wherein the attention GNN is composed of a key point encoder for generating a feature vector and a graph attention mechanism for generating a feature matching vector.

[0021] As the GNSS rejection condition under the unmanned aerial vehicle image positioning precision improvement method of the application, further, the feature matching between the target image data and the reference image data also contains:

[0022] For the extracted feature points, random sample consistency is used to purify the feature points to eliminate the mis-matching feature points, wherein the random sample consistency uses a homography matrix as an estimation model, first, a group of feature points are randomly extracted from the paired feature points of the target image data and the reference image data as a minimum sample, an initial homography matrix used as an initial state of the estimation model is established according to the correspondence relationship of the extracted feature points, the remaining feature point data is brought into the estimation model, and reliable feature points are extracted based on the estimation model and through iteration.

[0023] As the GNSS rejection condition under the unmanned aerial vehicle image positioning precision improvement method of the application, further, the reliable feature points are extracted based on the estimation model and through iteration, comprising:

[0024] Firstly, the feature points are divided into inliers and outliers;

[0025] Then, the errors between all feature point pairs between the target image data and the reference image data are compared with a preset error threshold, the feature points with errors less than the preset error threshold are taken as inliers, and the feature points with errors not less than the preset error threshold are taken as outliers, the inliers meeting the geometric constraint condition are taken as reliable feature points based on the estimation model, and the outliers not meeting the geometric constraint condition are taken as mis-matching points and eliminated.

[0026] As the unmanned aerial vehicle image positioning precision improving method under the GNSS denial condition of the application, further, the corresponding relationship between the target image data and the reference image data is constructed according to the matching result, so as to obtain the accurate geographic coordinates corresponding to the pixel coordinates of the unmanned aerial vehicle image by the corresponding relationship, comprising:

[0027] Firstly, the affine transformation matrix is constructed according to the corresponding relationship between the feature matching points between the target image data and the reference image data;

[0028] Then, the geographic information coordinates of the feature points in the satellite image are mapped into the pixel coordinates of the unmanned aerial vehicle image by using the affine transformation matrix and the corresponding relationship between the feature points of the target image data and the reference image data, so as to obtain the accurate geographic coordinates of the corresponding feature points of the unmanned aerial vehicle image.

[0029] Further, the application also provides a GNSS denial condition unmanned aerial vehicle image positioning precision improving system, comprising: a data collection module, a data processing module, a rough estimation module and an image matching module, wherein,

[0030] The data collection module is used for collecting the target image data for simulating the unmanned aerial vehicle image shot under the GNSS denial condition and the reference image data for positioning the unmanned aerial vehicle image, wherein the unmanned aerial vehicle image without positioning information is used as the target image data, and the satellite image with positioning information is used as the reference image data;

[0031] The data processing module is used for sequentially pre-processing the target image data and the reference image data, and extracting the feature vectors of the image data by separating the RGB channels for the pre-processed data, and obtaining the rough position of the target image data on the reference image data based on the similarity of the feature vectors between the target image data and the reference image data;

[0032] The rough estimation module is used for extracting the corresponding adjacent unmanned aerial vehicle images and forming a sequence image strip group according to the sequence image strip relationship in the target image data for the rough position of the target image data on the reference image data;

[0033] The image matching module is used for extracting and matching the features between the target image data and the reference image data based on the sequence image strip group and by the pre-trained deep learning neural network model, and constructing the corresponding relationship between the target image data and the reference image data according to the matching result, so as to obtain the accurate geographic coordinates corresponding to the pixel coordinates of the unmanned aerial vehicle image by the corresponding relationship.

[0034] The application has the following beneficial effects:

[0035] The application adopts a similarity cosine image retrieval positioning method to screen out single unmanned aerial vehicle images with higher similarity, and then screens out unmanned aerial vehicle short strip sequence images for assisting rough position retrieval according to the relationship characteristics among the unmanned aerial vehicle sequence images, so as to realize rough position estimation of the unmanned aerial vehicle images on high-precision satellite images; deep learning image registration network is used for high-precision feature point extraction and matching, so as to improve the unmanned aerial vehicle image positioning precision, which has strong robustness in actual application; further, the feature point results obtained by virtual control field can be used to obtain the homography matrix of the unmanned aerial vehicle image and the satellite image, and then high-precision positioning of the unmanned aerial vehicle image is realized. Experimental data show that the error of the application can be controlled within 2 pixels, and when the satellite reference image with a resolution of 0.30 m / pix is used, the error of the unmanned aerial vehicle positioning method is controlled to be 0.356 m, which further shows the feasibility of the unmanned aerial vehicle image positioning scheme. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The figure is a schematic diagram of the unmanned aerial vehicle image positioning precision improvement process under the GNSS denial condition in the embodiment.

[0037] Figure 2 The figure is a schematic diagram of the unmanned aerial vehicle image positioning principle in the embodiment.

[0038] Figure 3 The figure is a schematic diagram of the cosine similarity in the embodiment.

[0039] Figure 4 The figure is a schematic diagram of the coplanar relationship of adjacent images in the embodiment.

[0040] Figure 5 The figure is a schematic diagram of sequence image strip extraction in the embodiment.

[0041] Figure 6 The figure is a schematic diagram of the SuperPoint model in the embodiment.

[0042] Figure 7 The figure is a schematic diagram of the SuperGlue network in the embodiment.

[0043] Figure 8 The figure is a schematic diagram of the unmanned aerial vehicle aerial image data in the embodiment.

[0044] Figure 9 The figure is a schematic diagram of the matching results of the similarity cosine before and after image denoising in the embodiment.

[0045] Figure 10 The figure is a schematic diagram of image rough position matching in the embodiment.

[0046] Figure 11 The figure is a schematic diagram of the matching results of the unmanned aerial vehicle image and the commercial satellite image in the embodiment.

[0047] Figure 12 The RANSAC rejection control point matching error result comparison diagram in the embodiment. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the drawings and technical solutions.

[0049] The unmanned aerial vehicle is widely used in multiple industry fields, and the unmanned aerial vehicle image is required to have high-precision positioning information when being used for surveying and mapping. When the GNSS positioning environment is complex, the unmanned aerial vehicle positioning information will be inaccurate. Therefore, the embodiment of the present application provides a method for improving the positioning accuracy of unmanned aerial vehicle images under GNSS denial conditions, as shown in Figure 1 The method comprises the following steps:

[0050] S101, collect target image data for simulating unmanned aerial vehicle images taken under GNSS denial conditions and reference image data for unmanned aerial vehicle image positioning, wherein the unmanned aerial vehicle image without positioning information is used as the target image data, and the satellite image with positioning information is used as the reference image data.

[0051] By using the unmanned aerial vehicle image without positioning information to simulate the image taken by the unmanned aerial vehicle under the GNSS denial condition, the commercial satellite image can be used as the reference image of the virtual control field.

[0052] S102, sequentially pre-process the target image data and the reference image data, and based on the pre-processed data, extract the feature vector of the image data by separating the RGB channel, and obtain the approximate position of the target image data on the reference image data based on the similarity of the feature vectors between the target image data and the reference image data.

[0053] Specifically, the target image data and the reference image data are sequentially pre-processed, which can be designed to comprise the following contents:

[0054] Firstly, the target image data and the reference image data are respectively encoded with longitude and latitude, and the encoded image data is cut into blocks according to the preset cutting rule;

[0055] Then, the cut image data is renamed, reordered and normalized according to the longitude and latitude encoding.

[0056] In order to solve the problems of the differences in resolution and size between the satellite image and the unmanned aerial vehicle image in the actual production process, and the limitation of the input image size of the deep learning network and computer hardware performance, a geographic coding method is used to process cross-scale and cross-view heterogeneous remote sensing images.

[0057] The satellite image and the high-resolution unmanned aerial vehicle image are encoded by longitude and latitude, and are cut according to certain rules, so that the scale of the processed data can be effectively reduced and the processing efficiency can be improved. The cut images are renamed and reordered according to the geographic coding, so that the processed data is in order and has a unified coordinate system, which facilitates subsequent feature point extraction and matching. On this basis, the unmanned aerial vehicle image is taken as a candidate image, and is sequentially matched with the query image, and the feature points are extracted, so that the problem of the difference in resolution and size between the satellite image and the unmanned aerial vehicle image can be effectively solved, and the limitation of the input image size of the deep learning network and the computer hardware performance can be adapted. Through the above steps, the unmanned aerial vehicle image with high-precision positioning information can be obtained, and the high-precision geographic coordinates can be quickly obtained by using the pixel coordinates of the unmanned aerial vehicle image, which not only improves the positioning accuracy of the unmanned aerial vehicle image, but also effectively processes the cross-scale and cross-view heterogeneous remote sensing images, and has good practicability and expansibility.

[0058] S103, according to the approximate position of the target image data on the reference image data, corresponding adjacent unmanned aerial vehicle images are extracted according to the sequence image flight strip relationship in the target image data and are composed into a sequence image flight strip group.

[0059] Specifically, according to the sequence image flight strip relationship in the target image data, corresponding adjacent unmanned aerial vehicle images are extracted, which can be designed to include:

[0060] First, the unmanned aerial vehicle image with the highest similarity to the reference image data is selected as the flight strip center image.

[0061] Then, each N image before and after the flight strip center image is selected from the sequence image of the target image data to form a sequence image flight strip group with heading overlap and containing adjacent unmanned aerial vehicle images.

[0062] In GNSS-denied environments, the focus is often on a small range of areas, so the satellite image coverage range can be relatively small. In the embodiments of the present case, the UAV image is compared with the satellite image with positioning information, and the similarity between them is found through feature extraction and registration. The cosine similarity-based algorithm can be used to roughly search the position of the sequence of UAV images on the satellite image: Cosine similarity is a method for measuring the similarity of two vectors in a vector space. It calculates the cosine of the angle between the two vectors to achieve the similarity. If the angle between the two vectors is close to 0 degrees, their cosine similarity is high, indicating that the similarity of the two images is high. Conversely, if the angle between the two vectors is close to 90 degrees, their cosine similarity is low, indicating that the similarity of the two images is low. Based on cosine similarity, the relative position of the UAV image on the satellite image can be quickly searched, and the limitations of deep learning network and computer hardware performance on input image size are avoided. In addition, since the cosine similarity calculation is simple and efficient, this method has the advantages of high efficiency and speed, and is easy to deploy lightweight, with less hardware pressure, so it has high practicability and expansibility.

[0063] At the same time, comparative experiments are conducted on various traditional image retrieval methods, such as mean hash, interpolation hash, perceptual hash, wavelet hash, and cosine similarity detection-based methods. The cosine similarity detection method has good effect on heterogeneous remote sensing images. Therefore, in the image preprocessing stage, the cosine similarity detection method is used. Cosine similarity uses the cosine of the angle between two vectors to measure the similarity of two images, which can be specifically represented as:

[0064]

[0065] In the above formula, A and B are feature vectors, iA and iB are elements in feature vectors A and B. θ is the angle between feature vectors A and B in the vector space. As shown in Figure 3 The cosine similarity between the reference satellite image and the sequence of UAV images can be calculated as a feature to evaluate their similarity, so as to achieve the probability position prediction.

[0066] The UAV will take a series of images along a certain flight strip during flight, and these images have strong correlation. The positional relationship between adjacent images can be solved by relative orientation, as shown in Figure 4 Therefore, in the embodiments of the present case, the flight strip relationship between UAV images can be used to roughly search the UAV images. In the short flight strip-based UAV rough position calculation, the flight strip relationship can be used to combine the UAV images together, so as to more effectively estimate the rough position of the images and improve the calculation efficiency.

[0067] Specifically, the UAV sequence images are registered with the reference satellite images, and the cosine similarity between each pair of images is calculated, then the UAV image with the highest cosine similarity is selected as the center image of the strip, and the next 2 images before and after the center image are selected from the sequence images, a total of 5 images, forming a sequence image short strip with a heading overlap, as shown in Figure 5 The short strip can be calculated as a whole separately with the satellite images.

[0068] wherein, Figure 4 S1S2 is the photographic center, B is the base line, (X1, Y1, Z1), (X2, Y2, Z2) represent the coordinates of p and p' in the selected image space auxiliary coordinate system respectively:

[0069]

[0070] After arrangement:

[0071]

[0072] The rotation matrices of the left and right images in the respective reference coordinate systems are R1 and R2 respectively, and the coordinate transformation relationship with the photograph space coordinate system is:

[0073]

[0074] After arrangement:

[0075]

[0076] Accordingly, it can be considered that the sequence image of the strip group has a rough corresponding relationship with the satellite image. The strip group is taken as a whole, and the 5 images in the strip group are precisely registered, as shown in Figure 5

[0077] S104, based on the sequence image strip group and through the pre-trained deep learning neural network model, the features between the target image data and the reference image data are extracted and matched, and the corresponding relationship between the target image data and the reference image data is constructed according to the matching result, so as to obtain the accurate geographic coordinates corresponding to the pixel coordinates of the UAV image through the corresponding relationship.

[0078] Wherein, the feature extraction and matching between the target image data and the reference image data through the pre-trained deep learning neural network model can include:

[0079] Firstly, the pre-trained SuperPoint model is used to extract feature points from the sequence image strip group and the corresponding reference image;

[0080] Then, the pre-trained SuperGlue network model is used to match the extracted feature points and output.​

[0081] Specifically, the SuperPoint model comprises a feature encoder for spatial mapping of input image features, a keypoint decoder for extracting spatial feature interest points, and a descriptor decoder for extracting spatial feature interest point descriptors, wherein the feature encoder is composed of a convolutional layer, a Max-Pooling layer and a nonlinear activation layer, the keypoint decoder is composed of a point-wise convolutional layer, a Softmax activation function and a tensor transformation function, and the descriptor decoder is composed of a convolutional layer, an interpolation function and L2 normalization.

[0082] SuperPoint is a fully convolutional neural network framework, in the embodiment of the application, a lightweight convolutional neural network (CNN) is used to detect key points in an image, and the network result is mainly composed of three parts, namely a feature encoder, a keypoint decoder and a descriptor decoder, as shown in FIG. 1. Figure 6 When processing a full-size image, the dimension of the input image is first processed and reduced by a shared feature encoder, and the shared feature encoder is a lightweight fully convolutional neural network modified from a VGG-style and is composed of a convolutional layer, a Max-Pooling layer and a nonlinear activation layer. The image size is changed to one-eighth of the output through three Max-Pooling layers. After this step, the architecture is divided into two independent decoding parts, the feature decoder is responsible for interest point detection of the reduced dimension image, and is composed of a point-wise convolutional layer, a Softmax activation function and a tensor transformation function; the descriptor decoder is composed of a convolutional layer, an interpolation function and L2 normalization, and is responsible for generating interest point descriptors of the reduced dimension image. Then, a loss function is established based on the corresponding feature points and the corresponding feature point descriptors, and the final loss function is the sum of the loss of the feature point detector and the loss of the descriptor, as shown in formula 6, wherein is the loss related to the feature points, is the loss related to the feature vector, is the feature point extracted by the three-direction convolutional network, and is the feature point loss function before and after processing the same image, λ is a weight parameter, is the loss function of the corresponding feature point descriptor, and the loss function is:

[0083]

[0084] The SuperGlue network model comprises an attention GNN for generating feature matching vectors and an optimal matching layer for obtaining optimal matching and outputting, wherein the attention GNN is composed of a keypoint encoder for generating feature vectors and a graph attention mechanism for generating feature matching vectors.

[0085] Feature matching is completed using a SuperGlue network, which consists of two main modules, namely an attention GNN (Graph Neural Network) module and an optimal matching module. In image matching, learning feature matching is regarded as finding a partial assignment between two sets of local features. By solving a linear assignment problem, the assignment problem is converted into an optimal transport problem for solving, and GNN is used to predict the optimization cost function. Inspired by the Transformer, the use of intra-image and inter-image attention to use the spatial relationship of key points and their visual appearance strengthens the predicted assignment structure and can well handle occluded and repeated key points. The processing flowchart of the entire algorithm is shown in Figure 7 The two frameworks are attention GNN and optimal matching layer, wherein the attention GNN is divided into a key point encoder and a graph attention mechanism, the key point encoder is used to generate a feature vector, and the graph attention mechanism is used to generate a feature matching vector; the optimal matching layer is used to obtain an optimal matching score matrix and output a matching result.

[0086] Further, the feature matching between the target image data and the reference image data also includes:

[0087] For the extracted feature points, random sample consensus is used to purify the feature points to eliminate mis-matched feature points, wherein the random sample consensus uses a homography matrix as an estimation model. First, a set of feature points is randomly extracted from the paired feature points of the target image data and the reference image data as a minimum sample, an initial homography matrix used as an estimation model initial state is established according to the correspondence relationship of the extracted feature points, the remaining feature point data is brought into the estimation model, and reliable feature points are extracted based on the estimation model and through iteration.

[0088] Among them, the reliable feature points extracted based on the estimation model and through iteration can be designed to include:

[0089] First, the feature points are divided into inliers and outliers;

[0090] Then, the errors between all feature point pairs between the target image data and the reference image data are compared with a preset error threshold, the feature points with errors less than the preset error threshold are regarded as inliers, and the feature points with errors not less than the preset error threshold are regarded as outliers. Based on the estimation model, the optimal homography matrix is obtained, at the same time, the inliers that meet the geometric constraint condition are regarded as reliable feature points, and the outliers that do not meet the geometric constraint condition are regarded as mis-matched points and are eliminated.

[0091] In the process of matching image features of different sources, the algorithm combining SuperPoint and SuperGlue can achieve good image feature matching, but due to the differences in view angle, scale and affine deformation between UAV images and satellite images, there are often individual mis-matching of feature points, which affects the subsequent correction of UAV images by taking feature points as control points. In the embodiment of the case, the RANSAC algorithm can be used to remove the mis-matching feature points, which can improve the estimation accuracy of the homography matrix between images, and further improve the accuracy of image registration, and has the ability to accurately estimate model parameters in the presence of a large number of outlier feature matching point pairs.

[0092] Among them, the RANSAC algorithm is used for feature point purification, and the homography matrix between images established according to the relationship between the corresponding feature points of the two images is used as the estimation model. The homography matrix is a vector related to rotation, translation and plane parameters, and the homography matrix formula can be expressed as:

[0093]

[0094] Among them, u1, v1, u2, v2 respectively represent the pixel coordinates of the feature points on different images. By expanding the matrix of formula 3-1, three equations can be obtained. By bringing the third equation into the first two, the following formula can be obtained:

[0095]

[0096]

[0097] The feature points extracted by SuperPoint are substituted into the model, and the RANSAC algorithm randomly selects a group of feature point pairs from the paired feature points as the minimum sample to calculate the initial homography matrix model. Then the remaining feature point data is brought into the model and iterative operation is performed to verify the reliability of the feature point pair data. Under normal circumstances, the error of all matching feature points is compared with the pre-set threshold value, and the feature points are divided into inliers and outliers. The points with an error less than the threshold value are regarded as inliers, and the optimal homography matrix is obtained at the same time. The inliers that meet the geometric constraints are retained as correctly matched feature points, and the outliers that do not meet the conditions are removed as mis-matching points. In this way, reliable feature points can be effectively extracted, and mis-matching feature points can be removed, thereby improving the accuracy and reliability of feature point matching.

[0098] After extracting and purifying the feature matching points, the corresponding relationship between the feature matching points of the two images can be used to construct an affine transformation matrix. This matrix is a general linear transformation matrix that can realize rotation, scaling and translation transformation between images. The affine matrix can be expressed as:

[0099]

[0100] h 11 , h 12 , h 21 , h 22 are scale, rotation variables between images, t x and t y are translation variables.

[0101] By using the affine transformation matrix and the mapping relationship between the feature points of the unmanned aerial vehicle image and the satellite image, the geographic information coordinates of the feature points in the satellite image are mapped into the pixel coordinates of the unmanned aerial vehicle image, so that the geographic information coordinates of the corresponding unmanned aerial vehicle image feature points are obtained, and a virtual control field capable of improving the positioning accuracy of the unmanned aerial vehicle is initially formed.

[0102] Further, based on the above method, the embodiment of the present application also provides a system for improving the positioning accuracy of unmanned aerial vehicle images under GNSS denial conditions, comprising a data collection module, a data processing module, a rough estimation module and an image matching module, wherein,

[0103] The data collection module is used to collect target image data for simulating unmanned aerial vehicle images taken under GNSS denial conditions and reference image data for positioning reference of the unmanned aerial vehicle images, wherein the unmanned aerial vehicle images without positioning information are used as the target image data, and the satellite images with positioning information are used as the reference image data.

[0104] The data processing module is used to sequentially pre-process the target image data and the reference image data, and based on the rough position of the target image data on the reference image data, the feature vectors of the image data are extracted by separating the RGB channels, and the rough position of the target image data on the reference image data is obtained based on the similarity of the feature vectors between the target image data and the reference image data.

[0105] The rough estimation module is used to extract corresponding adjacent unmanned aerial vehicle images according to the sequence image strip relationship in the target image data based on the rough position of the target image data on the reference image data, and to form a sequence image strip group.

[0106] The image matching module is used to extract and match the features between the target image data and the reference image data based on the sequence image strip group and through the pre-trained deep learning neural network model, and to construct the corresponding relationship between the target image data and the reference image data according to the matching result, so as to obtain the accurate geographic coordinates corresponding to the pixel coordinates of the unmanned aerial vehicle image through the corresponding relationship.

[0107] To verify the effectiveness of the scheme, the following experimental data are further explained:

[0108] In the satellite image data set, the satellite images used are from commercial satellites. The original images are 19 levels, the image ground resolution is 0.60m / pix, and the size is 2480pix*2823pix. These images have high ground resolution and can provide more detailed information. In the actual experiment, according to the specific model and efficiency requirements, these images are processed, and the size normalization is performed. In the experiment, the input size of these images is normalized to 1280pix*960pix. This normalization helps to reduce the amount of data processed, thereby improving the efficiency of the experiment.

[0109] During the size normalization process, no adjustment or processing is performed on any properties of the image quality, color scheme, brightness, white balance, contrast, and shadow, or any form of image quality improvement or color adjustment. Through size normalization processing and preservation of the original image features, these satellite images can be processed more efficiently while preserving all important features of the images.

[0110] In the UAV image data set, the main source is the DJI M300 UAV camera. The camera parameters of these images are shown in Table 1.

[0111] As shown in Figure 8 , these UAV image data were taken in the Chinese Mountaineering area and a certain area in Zhengzhou, China, with rich geographical environmental characteristics, including cities, residential areas, lakes, vegetation, and other elements. These image data not only have good representativeness, but also provide a basis for comparing and analyzing different regions. Among them, the UAV image data of the Chinese Mountaineering area has a total of 607 images, taken on June 22, 2022, with a ground resolution of 9.91 centimeters per pixel; the UAV image data of a certain area in Zhengzhou, China, has a total of 512 images, taken on September 26, 2021, with a ground resolution of 2.94 centimeters per pixel.

[0112] Table 1 Camera Parameters (DJI P1 Lens)

[0113]

[0114] In the experiment, the computer used is Lenovo Y9000P notebook, the graphics card is NVIDIA GeForce RTX 3060 6GB, the CPU is Intel Core i7-11900H@2.50GHz, and the memory is 32GB DDR4 3200MHz. The specific configuration parameters are shown in Table 2.

[0115] Table 2 Experimental Environment

[0116]

[0117] As Figure 9 shown in the analysis of the similarity cosine matching results before and after image denoising, it can be found that the denoising processing has a significant improvement effect on the overall calculation result, which can effectively reduce the calculation error and is beneficial to more accurate determination of the similarity threshold, so as to more accurately screen the matching result. On the other hand, from the perspective of fitting effect, the image retrieval estimation result ranking before and after denoising does not change. In the embodiment of the case, the image retrieval based on similarity is robust to whether the denoising processing is performed, and even if there is noise in the original image, the result ranking can still be accurately estimated.

[0118] Through experimental comparison, it can be found that the method of calculating image similarity by using similarity cosine has higher accuracy and stronger robustness when processing large-format heterogeneous remote sensing images. In addition, compared with the deep learning method, it does not need pre-training and supervision, has higher efficiency and faster matching speed.

[0119] As Figure 10 shown, the left image is the reference satellite image, the 10 images shown in a are the sequence of unmanned aerial vehicle images that should be matched, and the red box image is the normally estimated image. The blue line in b is the sequence of unmanned aerial vehicle image results. By comparing the images circled by the red and blue lines, it can be found that the short strip position estimation based on the internal relationship between the sequence of unmanned aerial vehicle images can effectively extract the unmatched images, thereby effectively supplementing the image matching results based on the similarity cosine. In addition, by extracting the sequence of unmanned aerial vehicle image strips, the accuracy of the position estimation can be improved.

[0120] In practical applications, the resolution of unmanned aerial vehicle images and satellite images often differs greatly, which brings challenges to image matching. In order to better perform cross-resolution image matching, commercial satellite image data is used in the experiment. The size of the two images is 1280x960 pixels, the resolution of the left image is 0.60 meters / pixel, and the resolution of the right image is 2.39 meters / pixel.

[0121] A variety of feature matching methods with rotation and scale invariance are used for comparison test, including traditional sift feature extraction algorithm, suft feature extraction algorithm, ORB feature extraction algorithm, FLANN fast nearest neighbor algorithm, and currently popular deep learning algorithms CMM-Net network model and SuperPoint and SuperGlue combined network model for experiment. The results are as follows Figure 11As shown, through comparative tests, it can be found that these methods can all realize cross-resolution image matching, however, the number of feature matching points of the traditional matching methods of sift, suft and ORB algorithm all exceeds the algorithm combined with CMM-Net, SuperPoint and SuperGlue of deep learning, as shown in Table 3, the traditional algorithm has certain advantages in the number of feature points, but the feature matching effect is poor, and there are phenomena of few feature matching points and mis-matching, the main reason is that the shallow feature descriptor of the neighborhood gradient of the traditional algorithm cannot adapt to the nonlinear distortion between the heterogeneous images, and cannot meet the requirement of deep feature matching, therefore, it can be seen that the traditional matching algorithm cannot realize the feature matching between the unmanned aerial vehicle image and the satellite image.

[0122] Table 3 Feature point matching results of unmanned aerial vehicle image and commercial satellite image

[0123]

[0124] The algorithms combined with CMM-Net, SuperPoint and SuperGlue of deep learning all extract sufficient and uniformly distributed feature points in two images, and can realize efficient and accurate cross-resolution image matching. Among them, the number of feature points extracted by CMM-Net is more than that of the algorithm combined with SuperPoint and SuperGlue, but from the number of successfully matched feature points, the algorithm combined with SuperPoint and SuperGlue can extract more abstract feature points and construct deep descriptors under different light, different scales and extreme view angle differences, even in the case of about 4 times difference in resolution, still can obtain key points with strong robustness and stability. Therefore, in this case, the algorithm combined with SuperPoint and SuperGlue is applied to the feature matching of unmanned aerial vehicle image and satellite image, which can provide higher precision of matching feature point information.

[0125] The unmanned aerial vehicle cut candidate image and the satellite cut query image are used as the test data of this group, and the feature matching results are compared and analyzed, and the feature extraction results as shown in Figure 12 are obtained. A large number of feature points appear in the building corner points and the vegetation coverage area, and in the vegetation area, due to the complex texture, the vegetation area in the heterogeneous images often has time difference, and the texture features presented by the images are inconsistent, it is difficult to realize high-precision matching effect of pixel-level feature area. In order to eliminate the feature matching error points with low precision, the RANSAC algorithm is used to process the matching results. After removing the error points, the control point matching feature is as shown in Figure 12The features of the vegetation coverage area have lower matching accuracy, and a large number of false matching points are removed. The control points are concentrated in the areas with obvious texture information such as building corners. By removing the coarse matching feature points through the algorithm, high-precision feature points with robustness can be extracted. Through the experimental comparison, it can be seen that the RANSAC algorithm is used to optimize the error of the control points matched by the virtual control field, and most of the false matching points without obvious features are removed, and the matching points with obvious features are retained. Therefore, the RANSAC algorithm shows robustness when processing the experimental data in this group, and has high feature matching accuracy.

[0126] In the positioning improvement analysis based on the virtual control field, 12 feature points in the region were selected as checkpoints, and the feature point coordinates of the satellite reference image provided by the virtual control field were compared with the orthographic image of the unmanned aerial vehicle generated in the region. The results are shown in Table 0:

[0127] Table 4 Comparison of unmanned aerial vehicle positioning accuracy error (m)

[0128]

[0129] First, high-resolution satellite images can obtain more abundant ground information, which can improve the accuracy of the unmanned aerial vehicle positioning. The satellite reference image used in the experiment has a resolution of 0.30 m / pix, which can ensure the quality and accuracy of the image; second, the positioning improvement method based on the virtual control field has high accuracy and stability, and the error is within 0.356 m; in addition, the virtual control field strategy can effectively improve the accuracy of the unmanned aerial vehicle positioning, and the overall error is within 2 pixels, and the accuracy is higher than that of the ORB algorithm.

[0130] In summary, the registration scheme in the present case has high accuracy and stability, and the virtual control field strategy can effectively improve the accuracy of the unmanned aerial vehicle positioning, and can achieve the expected experimental results.

[0131] Unless specifically stated otherwise, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0132] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between each embodiment can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0133] The units and method steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described above in functional generalities. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation does not exceed the scope of the present application.

[0134] Those skilled in the art can understand that all or part of the steps in the above method can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, etc. Alternatively, all or part of the steps of the above embodiments can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiments can be implemented in the form of hardware or in the form of a software function module. The present application is not limited to any specific form of combination of hardware and software.

[0135] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, which are used to illustrate the technical solutions of the present application, rather than limiting them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for improving the positioning accuracy of UAV images under GNSS denial conditions, characterized in that, Include: Collect target image data for simulating GNSS denial conditions and reference image data for UAV image positioning. UAV images without positioning information are used as target image data, and satellite images with positioning information are used as reference image data. The target image data and the reference image data are preprocessed sequentially. For the preprocessed data, the feature vector of the image data is extracted by separating the RGB channels. The approximate position of the target image data on the reference image data is obtained based on the similarity of the feature vectors between the target image data and the reference image data. Based on the approximate location of the target image data on the reference image data, corresponding adjacent UAV images are extracted according to the flight strip relationship of the sequential images in the target image data and formed into a sequential image flight strip group. The extraction of corresponding adjacent UAV images according to the flight strip relationship of the sequential images in the target image data includes: first, selecting the UAV image with the highest similarity to the reference image data as the center image of the flight strip; then, sequentially selecting N images before and after the center image of the flight strip from the sequential images of the target image data to form a sequential image flight strip group with flight direction overlap and containing adjacent UAV images. Based on the flight strip group of sequential images, the features between the target image data and the reference image data are extracted and matched using a pre-trained deep learning neural network model. The correspondence between the target image data and the reference image data is constructed based on the matching results, so as to obtain the accurate geographic coordinates corresponding to the pixel coordinates of the UAV image through the correspondence.

2. The method for improving UAV image positioning accuracy under GNSS denial conditions according to claim 1, characterized in that, The target image data and the reference image data are preprocessed sequentially, including: First, the target image data and the reference image data are encoded with latitude and longitude respectively, and the encoded image data is then segmented according to a preset cutting rule. Then, the image data after being segmented is renamed, reordered, and scaled according to latitude and longitude encoding.

3. The method for improving UAV image positioning accuracy under GNSS denial conditions according to claim 1, characterized in that, Feature extraction and matching between target image data and reference image data is performed using a pre-trained deep learning neural network model, including: First, feature points are extracted from the sequential image flight strip group and the corresponding reference image using the pre-trained SuperPoint model; Then, the extracted feature points are matched and output using a pre-trained SuperGlue network model.

4. The method for improving UAV image positioning accuracy under GNSS denial conditions according to claim 3, characterized in that, The SuperPoint model includes: a feature encoder for spatial mapping of input image features, a keypoint decoder for extracting spatial interest points, and a descriptor decoder for extracting spatial interest point descriptors. The feature encoder consists of a convolutional layer, a Max-Pooling layer, and a non-linear activation layer. The keypoint decoder consists of a pointwise convolutional layer, a Softmax activation function, and a tensor transformation function. The descriptor decoder consists of a convolutional layer, an interpolation function, and L2 normalization.

5. The method for improving UAV image positioning accuracy under GNSS denial conditions according to claim 3, characterized in that, The SuperGlue network model includes: An attention-based GNN is used to generate feature matching vectors, and an optimal matching layer is used to obtain and output the optimal match. The attention-based GNN consists of a keypoint encoder for generating feature vectors and a graph attention mechanism for generating feature matching vectors.

6. The method for improving UAV image positioning accuracy under GNSS denial conditions according to claim 3, characterized in that, Feature matching between target image data and reference image data also includes: For the extracted feature points, random sample consistency is used to purify the feature points and eliminate mismatched feature points. The random sample consistency uses the homography matrix as the estimation model. First, a set of feature points is randomly selected from the paired feature points of the target image data and the reference image data as the minimum sample. Based on the correspondence of the extracted feature points, an initial homography matrix is ​​established between the images to serve as the initial state of the estimation model. The remaining randomly selected feature point data is then brought into the estimation model. Based on the estimation model, reliable feature points are extracted through iteration.

7. The method for improving UAV image positioning accuracy under GNSS denial conditions according to claim 6, characterized in that, Reliable feature points are extracted based on an estimation model and through iteration, including: First, the feature points are divided into interior points and exterior points; Then, the error between all feature point pairs between the target image data and the reference image data is compared with a preset error threshold. Feature points with errors less than the preset error threshold are designated as inliers, and feature points with errors not less than the preset error threshold are designated as outliers. While obtaining the optimal homography matrix based on the estimation model, inliers that satisfy the geometric constraints are designated as reliable feature points, and outliers that do not satisfy the geometric constraints are designated as mismatched points and removed.

8. The method for improving UAV image positioning accuracy under GNSS denial conditions according to claim 1, characterized in that, Based on the matching results, a correspondence is constructed between the target image data and the reference image data to obtain the precise geographic coordinates corresponding to the pixel coordinates of the UAV image. This includes: First, an affine transformation matrix is ​​constructed based on the correspondence between feature matching points between the target image data and the reference image data; Then, by using the affine transformation matrix and the correspondence between the feature points of the target image data and the reference image data, the geographic coordinates of the feature points in the satellite image are mapped to the pixel coordinates of the UAV image to obtain the accurate geographic coordinates of the corresponding UAV image feature points.

9. A system for improving the positioning accuracy of UAV images under GNSS denial conditions, characterized in that, It includes: a data collection module, a data processing module, a preliminary estimation module, and an image matching module. The data collection module is used to collect target image data for simulating GNSS denial conditions and reference image data for UAV image positioning. UAV images without positioning information are used as target image data, and satellite images with positioning information are used as reference image data. The data processing module is used to preprocess the target image data and the reference image data in sequence, and extract the feature vector of the image data by separating the RGB channels for the preprocessed data. Based on the similarity of the feature vectors between the target image data and the reference image data, the approximate position of the target image data on the reference image data is obtained. The approximate estimation module is used to extract corresponding adjacent UAV images and form a sequence image flight strip group based on the approximate position of the target image data on the reference image data. The extraction of corresponding adjacent UAV images based on the sequence image flight strip relationship in the target image data includes: first, selecting the UAV image with the highest similarity to the reference image data as the center image of the flight strip; then, sequentially selecting N images before and after the center image of the flight strip from the sequence images of the target image data to form a sequence image flight strip group with flight direction overlap and containing adjacent UAV images. The image matching module is used to extract and match features between target image data and reference image data based on sequential image flight strip groups and a pre-trained deep learning neural network model. Based on the matching results, a correspondence between the target image data and the reference image data is constructed to obtain the precise geographic coordinates corresponding to the pixel coordinates of the UAV image through the correspondence.

Images