Unmanned aerial vehicle geolocation method for image feature matching of on-site security
By constructing an encoder-decoder module that combines an expanded image set with a locality-sensitive hashing algorithm and a visual converter, the problems of high computational demands and information loss in UAV image processing are solved, achieving fast, accurate visual positioning and high recall.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2023-12-20
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies require significant computation and take a long time to process high-resolution, information-rich drone images, making them difficult to deploy on devices with limited computing power. Furthermore, non-uniform cropping operations may result in the loss of important information and lead to incorrect matching relationships.
An expanded image set is constructed, feature vectors are extracted using first and second image feature extractors, features are generated and projected using first and second geometric layout description vector extractors, image retrieval is performed using a locality-sensitive hashing algorithm, the image attention requirement of non-uniform cropping is eliminated, and detailed features are extracted using an encoder-decoder module of a single visual converter.
It achieves fast and accurate visual positioning on devices with limited computing power, improves robustness and recall, and meets the requirements for real-time positioning.
Smart Images
Figure CN117853952B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of visual positioning technology, and more particularly to a UAV geolocation method based on image feature matching for near-site security. Background Technology
[0002] Visual positioning is a technology that uses computer vision algorithms and sensor information to accurately locate and track targets in three-dimensional space. It plays a crucial role in fields such as autonomous driving and intelligent navigation for drones. The near-ground security technology system is a cutting-edge interdisciplinary field operating in near-ground space. In this system, low-altitude security drones refer to drones operating in the core near-ground security area (0-1000m above ground). This visual positioning technology can be used for autonomous positioning and navigation of drones in extreme environments such as those without satellite positioning signals. Traditional image geolocation technologies mainly rely on SIFT and SURF feature extraction algorithms. These methods achieve positioning by extracting feature points from images or videos and matching them with feature points in reference images. However, in complex environments, traditional methods face many challenges, such as changes in lighting and image blurring, leading to lower accuracy and stability in positioning.
[0003] Currently, image-based geolocation methods primarily rely on the extraction and matching of image feature vectors. This approach first utilizes convolutional neural networks (CNNs) to extract image features. Subsequently, the extracted feature vectors are combined with learnable location codes and input into a visual transducer for further feature extraction. This method fully leverages the advantages of visual transducers in extracting global image features while also taking full advantage of convolutional neural networks in extracting local image features. Therefore, it can comprehensively and accurately capture various image features. However, when processing high-resolution, information-rich UAV images, this method requires significant computational memory and time, making it difficult to deploy on devices with limited computing power, such as UAVs.
[0004] In addition, some lightweight methods use cropping operations to reduce the processing time required by the network. This method can remove redundant parts in images from different viewpoints, accelerate image matching, and improve the overall recall rate of cross-view geolocation tasks. However, when performing UAV geolocation tasks, since most of the ground lacks obvious features for the visual converter to extract, the generated attention map may be too smooth or too coarse. In this case, if the non-uniform cropping operation in the above algorithm is performed, it may cause the loss of important information in the original image and further lead to a large number of incorrect matching relationships. Summary of the Invention
[0005] To avoid the shortcomings of existing technologies, this application provides a UAV geolocation method for image feature matching for local security, which solves the problems in existing technologies that require a large amount of computing memory and a long computing time when processing UAV images with high resolution and rich information, making it difficult to deploy on UAVs and other devices with limited computing power; and that non-uniform cropping operations may lead to the loss of important information in the original image, which may further cause a large number of incorrect matching relationships.
[0006] According to embodiments of this disclosure, a UAV geolocation method based on image feature matching for near-site security is provided, the method comprising:
[0007] An extended image set is constructed based on the UAV image set and the satellite image set; wherein, the extended image set includes the UAV extended image set and the satellite extended image set;
[0008] The first image feature extractor is used to extract features from the UAV extended image set to obtain a first feature vector, and the second image feature extractor is used to extract features from the satellite extended image set to obtain a second feature vector;
[0009] A first geometric layout description vector extractor is trained using the first feature vector to generate a first geometric layout description vector. A second geometric layout description vector extractor is trained using the second feature vector to generate a second geometric layout description vector. The first geometric layout description vector and the second geometric layout description vector are then projected onto each other.
[0010] The first feature vector and the first geometric layout description vector after projection processing are fused to obtain the first image feature, and the second feature vector and the second geometric layout description vector after projection processing are fused to obtain the second image feature.
[0011] A retrieval function is constructed based on the first image features and the second image features;
[0012] The aforementioned retrieval function is used to perform image retrieval on UAV images to obtain satellite image similarity ranking, and the target satellite image is output.
[0013] Furthermore, the step of constructing an expanded image set based on the UAV image set and the satellite image set includes:
[0014] By reading drone images collected during actual drone operation The UAV image set constitutes ;
[0015] Construct a corresponding latitude and longitude set based on the latitude and longitude of each location in the UAV image set. ;in, Representing the image of the drone Longitude and latitude;
[0016] Based on the latitude and longitude set in the satellite imagery of the UAV's operating area Cropping out satellite images corresponding to latitude and longitude positions. And construct the satellite image set The initial image set is comprised of the UAV image set and the satellite image set. , , .
[0017] The UAV image set and the satellite image set are transformed separately to obtain the transformed UAV image set. and the transformed satellite image set ;
[0018] repeat k The transformation processing operation described above yields the UAV augmented image set. and the satellite extended image set ;
[0019] According to the UAV expanded image set and the satellite extended image set The expanded image set constitutes .
[0020] Furthermore, the transformation process includes at least random rotation, translation, scaling, mirroring, cropping, and color dithering.
[0021] Furthermore, the first geometric layout description vector extractor includes a channel-side max pooling layer, a feature embedding layer 1, a convolutional layer, an L× standard Transformer encoder layer, and a feature embedding layer 2 connected in sequence, and the channel-side max pooling layer and the convolutional layer are also connected by position encoding.
[0022] Furthermore, the dimensions of both the first geometric layout description vector and the second geometric layout description vector after projection processing are 1. .
[0023] Furthermore, the step of constructing a retrieval function based on the first image features and the second image features includes:
[0024] The retrieval function is constructed using the Locality Sensitive Hashing algorithm based on all the first image features and all the second image features.
[0025] Furthermore, the step of using the retrieval function to perform image retrieval on UAV images to obtain satellite image similarity ranking and outputting the target satellite image includes:
[0026] The image retrieval module uses the retrieval function to perform image retrieval on any of the UAV images, obtains multiple satellite images in the satellite image set that are similar to the UAV images, and sorts them from highest to lowest according to similarity to obtain satellite image similarity ranking;
[0027] Obtain the latitude and longitude coordinates of the satellite image with the highest similarity and output it as the location of the UAV.
[0028] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:
[0029] In the embodiments of this disclosure, the UAV geolocation method based on image feature matching for near-site security, on the one hand, eliminates the requirement for image attention maps due to non-uniform cropping by using a first geometric layout description vector extractor and a second geometric layout description vector extractor, ensuring effectiveness and robustness under various positioning conditions, thereby achieving accurate visual positioning and enhancing robustness. On the other hand, the Locality-Sensitive Hashing (LS Hash) algorithm is used to accelerate image retrieval, and a certain strategy is used to short-circuit the image retrieval module, thus meeting the speed requirements of real-time positioning. The first and second geometric layout description vectors are extracted using only the encoder-decoder module of a single visual converter, and at the same time, convolutional neural networks can be used to extract detailed features of the image with low operating costs, thereby achieving better recall. Attached Figure Description
[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0031] Figure 1 The diagram illustrates the steps of an image feature matching-based UAV geolocation method for near-site security in an exemplary embodiment of this disclosure.
[0032] Figure 2 A flowchart illustrating a UAV geolocation method for image feature matching for near-site security in an exemplary embodiment of this disclosure is shown.
[0033] Figure 3This diagram illustrates the structure of the extractor for the first geometric layout description vector in an exemplary embodiment of this disclosure.
[0034] Figure 4 The diagram illustrates the input drone image, satellite image, and matched satellite image used in a simulation experiment in an exemplary embodiment of this disclosure. Detailed Implementation
[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0036] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.
[0037] This example implementation first provides a UAV geolocation method based on image feature matching for near-site security. (Reference) Figure 1 As shown, the UAV geolocation method for image feature matching for local security may include steps S101 to S106.
[0038] Step S101: Construct an extended image set based on the UAV image set and the satellite image set; wherein, the extended image set includes the UAV extended image set and the satellite extended image set;
[0039] Step S102: Use the first image feature extractor to extract features from the UAV extended image set to obtain a first feature vector; use the second image feature extractor to extract features from the satellite extended image set to obtain a second feature vector.
[0040] Step S103: Train a first geometric layout description vector extractor using the first feature vector to generate a first geometric layout description vector; train a second geometric layout description vector extractor using the second feature vector to generate a second geometric layout description vector; and perform projection processing on the first geometric layout description vector and the second geometric layout description vector respectively.
[0041] Step S104: Perform feature fusion on the first feature vector and the first geometric layout description vector after projection processing to obtain the first image feature; perform feature fusion on the second feature vector and the second geometric layout description vector after projection processing to obtain the second image feature.
[0042] Step S105: Construct a retrieval function based on the first image features and the second image features;
[0043] Step S106: Use the retrieval function to perform image retrieval on the UAV image to obtain satellite image similarity ranking, and output the target satellite image.
[0044] The UAV geolocation method based on image feature matching for near-site security addresses two main issues. First, by using a first and second geometric layout description vector extractor, the requirements of non-uniform cropping on the image attention map are eliminated, ensuring effectiveness and robustness under various positioning conditions and achieving accurate visual positioning with enhanced robustness. Second, a Locality-Sensitive Hashing (LS Hash) algorithm is used to accelerate image retrieval, and a short-circuit operation is employed to the image retrieval module, thus meeting the speed requirements of real-time positioning. The first and second geometric layout description vectors are extracted using only the encoder-decoder module of a single visual converter, and convolutional neural networks can be used to extract detailed image features with low runtime costs, resulting in better recall.
[0045] Below, we will refer to Figures 1 to 4 The steps of the UAV geolocation method for image feature matching for local security described in this example embodiment will be explained in more detail.
[0046] In one embodiment, such as Figure 2 As shown, the model used in this application includes an image feature extraction module, an image geometric layout description vector extraction module, and a feature vector fusion and matching module.
[0047] In step S101, firstly, this application reads images of the drone collected during the actual operation of the drone. The drone image set The latitude and longitude coordinates of each drone image are used to construct a set of latitude and longitude coordinates representing the current real location. ,in Representative drone image The longitude and latitude were then used to assemble satellite images of the UAV's operating area based on the longitude and latitude. Cropping out satellite images corresponding to latitude and longitude positions. And construct a satellite image set At this point, the initial image set required for training the algorithm can be obtained. Any image , The transformed drone image is obtained by randomly rotating, translating, scaling, mirroring, cropping, and color dithering the image. and the transformed satellite image set Repeat this process Second drone image and satellite images Extended image set of drones and satellite extended image set The initial image set All images are converted into corresponding augmented image sets to obtain the initial image set. Expanded image set .
[0048] In step S102, the expanded image set is used. Supervised training of two residual network-based image feature extractors and (i.e., the first image feature extractor) Second image feature extractor ), respectively used to extract drone image sets and satellite image set Based on the characteristics, the first feature vector is obtained. Second eigenvector For ease of subsequent explanation, let , .
[0049] In step S103, the first feature vector of the output is extracted using the image feature module. Second eigenvector Training two image geometry layout description vector extractors based on vision converters and (i.e., the first geometric layout description vector extractor) Second geometric layout description vector extractor Each image geometry description vector extractor generates a geometry description vector. and (i.e., the first geometric layout description vector) Second geometric layout description vector Then, it is projected into a vector. and (i.e., the first geometric layout description vector after projection processing) and the second geometric layout description vector after projection processing Both vectors have dimensions of 1. .
[0050] In steps S104 and S105, the image geometric layout description vector is... , and image feature vectors , Feature fusion is performed to obtain the first image features. Second image features Features of all drone images and all satellite images , The symbol represents performing the Frobenius inner product over two vectors, where .
[0051] For the generated Perform operation Counterfactual learning vectors for generating image geometric layout description vectors This operation will cause The elements in maintain and Given the same dimensions, the random selection is based on a uniform distribution. At this point, a loss function is introduced. The loss function is calculated from a specific perspective. Image features after fusion Learning image features with counterfactual methods The distance in the feature space is penalized using a counterfactual learning technique. And encourage Capturing more specific geometric layout cues in images will, during training, enable... As small as possible, of which In addition, we introduce a loss function. The overall training results are constrained using a triplet soft-boundary loss function, where... and , This represents the distance between fused image features at the same location from different viewpoints in the feature space. This represents the distance between fused image features from different viewpoints and positions in the feature space, which will be set during training. To minimize it as much as possible, the overall loss function is: In subsequent training, the overall loss function will be further optimized. .
[0052] Use trained , , and Extract image set Features of all images These features are then used to construct a retrieval function using the Locality-sensitive hashing (LS Hash) algorithm. .
[0053] In step S106, after completing the above training and construction process, the image retrieval module uses the image provided by the UAV as the input image to filter out all similar satellite images from the feature retrieval set. For any input image... The image retrieval module uses retrieval functions. Complete the image retrieval process, that is ,in For image set All of the above Similar images are selected and sorted from highest to lowest similarity. The center of these satellite images represents the possible location where the input image was captured, and the latitude and longitude coordinates of the satellite image with the highest similarity are used as the result.
[0054] During the drone's flight, it continuously takes pictures of the currently visible area and performs image retrieval operations, at which point the drone's current latitude and longitude can be determined.
[0055] In one embodiment, the extractor of the first geometric layout description vector Extractor of the second geometric layout description vector The structures are the same, only the parameters differ. The structure of the vector extractor described by the first geometric layout is as follows: Figure 3 As shown, the first geometric layout describes the vector extractor as including a channel-side max pooling layer, a feature embedding layer 1, a convolutional layer, an L× standard Transformer encoder layer, and a feature embedding layer 2 connected in sequence. The channel-side max pooling layer and the convolutional layer are also connected by position encoding.
[0056] The effectiveness of this application can be further illustrated by the following simulation experiments.
[0057] Simulation conditions
[0058] This application describes a simulation experiment conducted using Python software and the PyTorch deep learning framework on a CPU consisting of an Intel(R) Core(TM) i7-10700F CPU, 16GB of memory, an NVIDIA GeForce RTX 1660 SUPER GPU, and a Windows 10 operating system. The dataset used in the simulation was a self-collected dataset.
[0059] Simulation content
[0060] The experiment used real-world footage captured by a drone in flight. The model had already been trained using other data, and the corresponding satellite image set already contained a large number of satellite view images for matching. As the drone flew, the corresponding satellite view image could be obtained through matching, and its actual location could be marked. Figure 4 On the satellite map on the left.
[0061] Depend on Figure 4 As can be seen, the results obtained from visual positioning are basically consistent with the actual flight trajectory of the UAV, thus proving that this application can effectively locate the UAV. During the test, the frame rate of the video captured by the UAV was 30 FPS. After the features were transformed into a retrieval function by the Locality Sensitive Hashing algorithm, the speed of network extraction and matching of corresponding satellite maps can be greatly improved. The matched latitude and longitude can be displayed on the system within 1 / 30 of a second, which is sufficient to obtain the location required for UAV operation.
[0062] The quantitative metric for this unmanned positioning system is recall (Recall@1, 5, 10, 1%), as shown in Table 1. This application can achieve a higher recall rate than the methods mentioned above on a self-built dataset.
[0063]
[0064] The UAV geolocation method based on image feature matching for near-site security, as described above, exhibits enhanced system robustness. The image geometric layout description vector extraction module eliminates the requirement for image attention maps due to non-uniform cropping in existing methods, ensuring effectiveness and robustness under various positioning conditions and achieving accurate visual positioning. Continuous positioning is fast and accurate. The LS-Hash method accelerates image retrieval, and a short-circuit operation is implemented in the image retrieval module using a specific strategy, thus meeting the speed requirements of real-time positioning. The geometric layout description vector is extracted using only the encoder-decoder module of a single visual transducer. Furthermore, it achieves better recall by using convolutional neural networks to extract detailed image features at a cost close to that of existing algorithms with a visual encoder as the backbone network, while maintaining a similar operational cost.
[0065] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.
[0066] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.
[0067] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.
Claims
1. A UAV geolocation method based on image feature matching for near-site security, characterized in that, The method includes: An extended image set is constructed based on the UAV image set and the satellite image set; wherein, the extended image set includes the UAV extended image set and the satellite extended image set; The first image feature extractor is used to extract features from the UAV extended image set to obtain a first feature vector, and the second image feature extractor is used to extract features from the satellite extended image set to obtain a second feature vector; A first geometric layout description vector extractor is trained using the first feature vector to generate a first geometric layout description vector. A second geometric layout description vector extractor is trained using the second feature vector to generate a second geometric layout description vector. The first geometric layout description vector and the second geometric layout description vector are then projected onto each other. The first feature vector and the first geometric layout description vector after projection processing are fused to obtain the first image feature, and the second feature vector and the second geometric layout description vector after projection processing are fused to obtain the second image feature. A retrieval function is constructed based on the first image features and the second image features; The aforementioned retrieval function is used to perform image retrieval on UAV images to obtain satellite image similarity ranking, and the target satellite image is output.
2. The UAV geolocation method for image feature matching for near-site security according to claim 1, characterized in that, The step of constructing an expanded image set based on the UAV image set and satellite image set includes: By reading drone images captured during actual drone operation x i The UAV image set X = {x1, x2, ..., x} constitutes the image set X = {x1, x2, ..., x} n }; Based on the latitude and longitude of each location in the UAV image set, construct the corresponding latitude and longitude set Pos_info = {(lon1,lat1),(lon2,lat2),…,(lon1,lat1),(lon2,lat2),…,(lon2,lat2),…} n ,lat n )}; where, (lon i ,lat i ) represents the image x of the drone. i Longitude and latitude; Based on the latitude and longitude set Pos_info, the satellite image y corresponding to the latitude and longitude position is cropped from the satellite image of the UAV's operating area. i And construct the satellite image set Y = {y1, y2, ..., y...} n }; where the UAV image set and the satellite image set constitute the initial image set T = {X, Y}, x i ∈X, y i ∈Y; The UAV image set and the satellite image set are transformed respectively to obtain the transformed UAV image set x'. i and the transformed satellite image set y' i ; Repeat the transformation process k times to obtain the UAV augmented image set D. i ={x i ,x' i1 ,x' i2 ,…,x' ik } and the satellite augmented image set S i ={y i ,y' i1 ,y' i2 ,…,y' ik }; According to the UAV extended image set D i ={x i ,x' i1 ,x' i2 ,…,x' ik } and the satellite augmented image set S i ={y i ,y' i1 ,y' i2 ,…,y' ik The expanded image set T' = {(D1,S1),(D2,S2),…,(D...} constitutes the expanded image set T' = {(D1,S1),(D2,S2),…,(D...}}. n ,S n )}.
3. The UAV geolocation method for image feature matching for near-site security according to claim 2, characterized in that, The transformation process includes at least random rotation, translation, scaling, mirroring, cropping, and color dithering.
4. The UAV geolocation method for image feature matching for near-site security according to claim 3, characterized in that, The first geometric layout description vector extractor includes a channel-side max pooling layer, a feature embedding layer 1, a convolutional layer, an L× standard Transformer encoder layer, and a feature embedding layer 2 connected in sequence. The channel-side max pooling layer and the convolutional layer are also connected by position encoding.
5. The UAV geolocation method for image feature matching for near-site security according to claim 4, characterized in that, The dimensions of both the first geometric layout description vector and the second geometric layout description vector after projection processing are (H×W) / 2.
6. The UAV geolocation method for image feature matching for near-site security according to claim 5, characterized in that, The step of constructing a retrieval function based on the first image features and the second image features includes: The retrieval function is constructed using the Locality Sensitive Hashing algorithm based on all the first image features and all the second image features.
7. The UAV geolocation method for image feature matching for near-site security according to claim 6, characterized in that, The step of performing image retrieval on UAV images using the aforementioned retrieval function to obtain satellite image similarity ranking and outputting the target satellite image includes: The image retrieval module uses the retrieval function to perform image retrieval on any of the UAV images, obtains multiple satellite images in the satellite image set that are similar to the UAV images, and sorts them from highest to lowest according to similarity to obtain satellite image similarity ranking; Obtain the latitude and longitude coordinates of the satellite image with the highest similarity and output it as the location of the UAV.