Keypoint representation matching and unmanned aerial vehicle positioning method

Abstract

The application provides a key point representation matching and unmanned aerial vehicle positioning method, a dense prediction task feature extraction network Swin Descriptors is designed, local key point semantic features containing global space perception in an image are extracted, a robust matching relationship is obtained, and the accurate position of an unmanned aerial vehicle is solved based on a multi-view geometry theory, so that accurate and reliable unmanned aerial vehicle positioning is realized under extreme conditions such as denial.The method provided by the application can obtain absolute geographic position information of the unmanned aerial vehicle only by relying on a visual sensor when satellite navigation such as GPS is completely invalid, and has great application value in denial environments such as canyons and building blockades.

Description

Technical Field

[0001] This invention relates to the field of autonomous positioning of unmanned aerial vehicles (UAVs), specifically a key point representation matching and UAV positioning method. Background Technology

[0002] Positioning is fundamental for drones to perform various functions, determining their location and guiding their flight. Traditional positioning methods, including satellite navigation, radio positioning and navigation, and inertial navigation, suffer from susceptibility to interference and deception, and difficulty in calibrating accumulated errors over long flight times, limiting their applications. Visual sensors, on the other hand, offer advantages such as small size, low cost, immunity to interference and deception, no accumulated errors, and high stealth, and their images can capture sufficient information to represent a location. Therefore, using vision as the primary information source for drone positioning and environmental exploration has become an effective solution for many applications.

[0003] Existing visual positioning technologies can be broadly categorized into two types: structure-based and semantic matching. Structure-based methods, such as Structure-for-Motion (SfM) and Simultaneous Localization and Mapping (SLAM), solve camera motion models to obtain pose changes by constructing co-view relationships between matching points in images. Without additional sensors, they can only acquire relative pose, and are prone to feature matching failures under extreme lighting and viewpoint changes, leading to positioning failures. Furthermore, the increased computational load due to the increased number of global optimization keyframes in large scenes limits their application. Semantic feature matching-based positioning methods match one or more images taken at an unknown location with a set of geotagged reference images, obtaining global geographic coordinates. The development of deep learning algorithms has also provided promising solutions for semantic feature matching-based positioning methods. However, they still suffer from matching failures due to significant differences in appearance between heterogeneous images caused by weather, time, and viewpoint variations, and can only return the nearest neighbor location information from the database, resulting in low accuracy. Summary of the Invention

[0004] To overcome the problems of existing visual positioning technologies, such as feature descriptions being greatly affected by changes in environmental appearance leading to matching errors and positioning failures, and heterogeneous image semantic matching positioning methods failing to obtain precise location information, as well as the inability to obtain absolute location information in multi-view stereo geometry calculations, this invention proposes a keypoint representation matching and UAV positioning method. By designing a dense prediction task feature extraction network—Swin Descriptors—it extracts semantic features of local keypoints containing global spatial awareness from images, obtains robust matching relationships, and calculates the precise pose of the UAV based on multi-view geometry theory, achieving accurate and reliable UAV positioning under extreme conditions such as rejection.

[0005] The technical solution of this invention is as follows:

[0006] The key point representation matching and UAV localization method includes the following steps:

[0007] Step 1: Extract global features from the current UAV aerial image and all satellite tiles in the satellite image database; based on the global features, use Euclidean distance as the criterion, and retrieve the top N satellite tile images in the satellite image database that are closest to the features of the current aerial image as candidate images;

[0008] Step 2: Input the current aerial image and the N satellite tile images retrieved in Step 1 into the local feature extraction network to extract local key points from the current aerial image and the N candidate images, and use descriptors to represent all key points; the local feature extraction network adopts the Swin Descriptros network architecture based on Swin Transformer.

[0009] Step 3: Perform FLANN nearest neighbor search on the key points of the aerial image and N candidate images to obtain all matching point pairs between the aerial image and the candidate images; then perform filtering based on the dynamic adaptive distance constraint algorithm to coarsely purify the matching point pairs; then obtain the final key point matching relationship between the aerial image and each candidate image based on RANSAC geometric cross-validation; calculate the average pixel distance of all final key point matching pairs between the aerial image and each candidate image, and obtain the top M candidate images with the smallest average pixel distance as the final matching images;

[0010] Step 4: Filter the matching point pairs between the M final matching images obtained in Step 3 and the current aerial image, removing many-to-one and one-to-many matching relationships and retaining one-to-one matching pairs; use the final matching relationship between the current aerial image and the M final matching images and the geographical information of the M final matching images to solve the precise location information of the UAV corresponding to the current aerial image based on the PnP method.

[0011] Furthermore, in step 1, the process of extracting global features includes two parts: local feature extraction and feature aggregation.

[0012] The local feature extraction is as follows: the image is input into a convolutional neural network to obtain a feature map of size W×H×D, which is used as a local feature descriptor x of size N×D; W is the width of the feature map, H is the height of the feature map, N = W×H represents the number of local features, and D is the dimension of the feature map, i.e. the dimension of the local features.

[0013] The feature aggregation is as follows: Local feature descriptors x of size N×D are linearly transformed using a 1×1×D×K convolution kernel, and then subjected to soft-max to obtain an N×K soft-assignment result. Simultaneously, the local feature descriptors x of size N×D are fed into the VLAD core and clustered to obtain K D-dimensional cluster center vectors. The N×K soft-assignment is used to allocate the weights of the feature residuals to the cluster centers, and the weighted summation and normalization according to the cluster centers yield a K×D vector as the final global descriptor of the image.

[0014] Furthermore, in step 1, during the extraction of global features, a twin neural network architecture based on NetVLAD is used to extract global features from aerial and satellite images: two convolutional neural networks are used to extract local features from aerial and satellite tiles respectively, and then all weight parameters of the feature aggregation part are shared; and a spatial attention module is added to the local feature extraction part to focus on the regions that contribute more to the task.

[0015] Furthermore, in step 1, during the extraction of global features, a weighted soft-margin ranking loss function is used as the loss function during network training.

[0016] Furthermore, in step 1, the retrieval and matching process is as follows: the global features of the current UAV aerial image and all database satellite tiles are represented as a feature vector respectively; then the Euclidean distance between the aerial image and the feature vectors of all satellite tiles is calculated, and the retrieval is performed based on the Euclidean distance, returning the N database satellite images that are closest to the feature distance of the aerial image as candidate images.

[0017] Furthermore, in step 2, the Swin Descriptors network architecture consists of an encoder, a bottleneck layer, a decoder, a key point selection module, and a key point feature description module; the main network is responsible for extracting feature maps and consists of an encoder, a bottleneck layer, and a decoder, based on a continuous Swin Transformer Block design; the Swin Transformer Block consists of two consecutive blocks, each block consisting of a LayerNorm, a multi-head self-attention module W-MSA, a sliding window multi-head attention module SW-MSA, residual connections, and two MLP layers.

[0018] Furthermore, the specific steps of step 2 are as follows:

[0019] Step 2-1: Key point extraction: Extract features from the aerial image and candidate image respectively based on Swin Descriptors; let the input image be I with size w×h, and the output feature map be M=m(I), M∈F w×h×n Its number of channels is n; in the feature space Fw×h×n The algorithm employs a selection strategy that maximizes both the channel direction and the local plane of the high-dimensional feature map to initially select key points. The formula is as follows:

[0020]

[0021] In the formula, D z Let D be the eigenvalue of the z-th layer, and D z ∈F w×h ; D z The eigenvalue at point (x,y); for the key point P(x,y) to be selected, first perform channel selection, choosing the channel z with the largest response value from the n channel feature maps, and obtain the feature map D on the corresponding channel. z Next, verify. Is it the largest in a local 3×3 neighborhood? If both conditions are met, it is a key feature point.

[0022] Step 2-2: Keypoint Feature Description: Extract the vector (1×n) at (x,y) on the feature map M(w×h×n) and perform L2 normalization to obtain the feature descriptor.

[0023]

[0024] In the formula, v xy =M xy ,v∈F n .

[0025] Furthermore, the specific steps of step 3 are as follows:

[0026] Step 3-1: Based on the FLANN algorithm, search for all matching pairs between the aerial image and each candidate image to be filtered, and obtain the first matching point pair and the second closest matching point pair with the Euclidean distance of the feature descriptors;

[0027] Step 3-2: Dynamically Adaptive Distance Constraint Filtering of Matching Points: First, calculate the average difference between the distances of the first and second matching point pairs found in the previous step:

[0028]

[0029] The filtering criteria are: the distance of the first matching point pair is less than the difference between the distance of the second matching point pair and the mean distance difference;

[0030] Step 3-3: Further filter matching pairs based on the RANSAC geometric constraint algorithm to obtain the final key point matching relationship between the aerial image and each candidate image;

[0031] Steps 3-4: Calculate the average pixel distance between the aerial image and each candidate image for the final matching pair, and obtain the M candidate images with the smallest average distance as the final matching images.

[0032] Furthermore, based on the above method, the present invention also proposes a computer-readable storage medium and a computer system.

[0033] One of the computer-readable storage media stores computer-executable instructions, which, when executed, are used to implement the above-described method.

[0034] A computer system includes: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the above-described method.

[0035] Beneficial effects

[0036] The key point representation matching and UAV localization method proposed in this invention has advantages over existing technologies in the following aspects:

[0037] 1. This invention designs an attention twin network to extract global features that ignore appearance interference such as lighting and season between heterogeneous images and contain more critical information in the matching and localization task. It also designs a weighted soft contrast loss function to gradually distinguish near and far images in geometric and feature space, generating image feature representations that are closer in feature distance ranking and image geometric distance ranking.

[0038] 2. This invention designs a keypoint extraction and representation module—Swin Descriptors—based on the self-attention Swin Transformer, addressing the contextual information loss problem caused by the receptive field limitation of CNNs. It restricts attention to a window, constructs feature pyramids of different scales, and increases information interaction between different windows through cross-window links. This results in more robust and reliable extraction of localization-related invariant features under extreme weather conditions and viewpoint changes.

[0039] 3. This invention combines semantic and geometric perception to achieve 2D->3D pose calculation based on depth features. It extracts high-level semantic features using Swing Descriptors and learns robust matching relationships between features. Based on these matching relationships and known satellite image geographic coordinates, it calculates the geometric motion of the UAV-mounted camera relative to the world coordinate system. Compared to existing depth-feature-based image matching and localization algorithms that can only return the geographic coordinates of the nearest neighbor satellite images in the database, this algorithm can obtain more accurate absolute geographic location information.

[0040] The method proposed in this invention can obtain the absolute geographical location information of a UAV solely by visual sensors when satellite navigation such as GPS fails completely, and has great application value in denial environments such as canyons and buildings.

[0041] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0042] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0043] Figure 1 This is a flowchart of the UAV visual positioning method based on deep semantic feature matching and multi-view geometric pose calculation according to the present invention.

[0044] Figure 2 This is a flowchart of the 2D to 3D pose calculation algorithm, with gray flowchart boxes representing known information. Specifically, the GPS coordinates of the satellite image to be matched are known information, while the coordinates of the matching points in the satellite image and the aerial image are obtained by the Swin Descriptors local key point feature matching algorithm in this method.

[0045] Figure 3 This is a diagram of the global feature extraction twin network architecture.

[0046] Figure 4 This is the entity graph of the NetVLAD module in global feature extraction.

[0047] Figure 5 This is a schematic diagram of the spatial attention module in global feature extraction.

[0048] Figure 6 It is the Swin Descriptors network architecture for image local key point extraction, feature description, and matching algorithms.

[0049] Figure 7 This is a detailed diagram of the Swin Transformer Block in the Swin Descriptors network architecture. Detailed Implementation

[0050] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0051] The keypoint representation matching and UAV localization method proposed in this invention mainly involves UAV visual localization and heterogeneous image semantic matching and localization. This method utilizes the hierarchical structure and self-attention mechanism of the Swing Transformer to extract salient keypoint invariant semantic features that are unaffected by scale or appearance changes within the same geographical location. Simultaneously, it considers the geometric constraints in the motion model of the camera mounted on the UAV, calculating the precise pose rather than returning existing geographical coordinates from the database. Furthermore, this method fully considers the semantic and geometric information in the visual localization problem, detecting salient features that do not change with weather, season, etc., effectively improving the robustness and localization accuracy of the algorithm under rejection conditions in dynamic, long-term, and other extreme environments.

[0052] like Figure 1 As shown, the main steps include:

[0053] Step 1: To improve positioning speed, first extract the global features of the current UAV aerial image and all satellite tiles in the database (the database contains all satellite tiles in an area slightly larger than the UAV flight mission). Then, based on the global features, use Euclidean distance as the criterion to retrieve the top 50 satellite tile images in the database that are closest to the features of the current aerial image as candidate images.

[0054] Specifically, it includes:

[0055] Step 1.1: Global feature extraction: This mainly consists of two parts: local feature extraction and feature aggregation.

[0056] The image is input into the convolutional neural network (ResNet101) to obtain a feature map of size W×H×D (W is the width of the feature map, H is the height of the feature map, and D is the dimension of the feature map), which is a local feature descriptor x of size N×D (N = W×H, representing the number of local features, and D is the dimension of the local features).

[0057] In feature aggregation, the N×D local feature descriptor x is linearly transformed by a 1×1×D×K convolution kernel and then softmax to obtain an N×K soft-assignment result; simultaneously, the N×D local feature descriptor x enters the VLAD core and is clustered to obtain K D-dimensional cluster center vectors.

[0058] Finally, using N×K soft-assignment, the features are assigned weights to the residuals at the cluster centers. Then, a weighted summation and normalization are performed according to the cluster centers to obtain a K×D vector as the final global descriptor of the image, which can be represented as:

[0059]

[0060] In the formula, ck w represents the cluster center k b k c k All of these are learnable parameters. x i (j) and c k (j) represents the j-th dimension of the i-th descriptor and the k-th cluster center.

[0061] Furthermore, as an improvement to NetVLAD, this embodiment considers the potential for significant appearance differences between aerial images and satellite tiles representing the same geographical location due to variations in time and season, which could lead to matching errors. This embodiment further designs a Siamese neural network architecture based on NetVLAD to extract global features from aerial and satellite images. Specifically, two convolutional neural networks are used to extract local features from the aerial images and satellite tiles respectively, and then all weight parameters of the feature aggregation part are shared. A spatial attention module is added to the local feature extraction part (after ResNet101) to focus on regions that contribute more to the task. The spatial attention module processing can be represented as follows:

[0062]

[0063] A(F origin )=σ(f 7×7 ([AvgPool(F origin MaxPool(F) origin )]))

[0064] In the above formula, This indicates that the corresponding elements are multiplied one by one, and the feature map F output by ResNet is... origin First, it is sent to the spatial attention module to generate a feature map A(F) with spatial attention weights. origin Multiplying the two elements by their coordinates yields a new feature map F. global A(F) origin Feature map F origin First, average pooling (AvgPool()) and max pooling (MaxPool()) are performed separately, then concatenated and subjected to a 7×7 convolution f. 7×7 It is obtained by using the sigmoid activation function σ().

[0065] Furthermore, to improve the convergence efficiency of network training during global feature extraction in this step and to make the network more adaptable to difficult samples, the existing triplet loss function that divides positive and negative samples in half is abandoned. Instead, a weighted soft-margin ranking loss function is designed, where negative samples at different distances from positive samples have different impacts on training, thereby improving the accuracy of global matching, as follows:

[0066]

[0067] In the formula, d pos and d neg γ represents the distance between the positive and negative samples and their corresponding anchor points, respectively. In this embodiment, the anchor point represents an aerial image from one iteration. Positive samples are satellite tiles within 50m of the aerial image, and negative samples are samples beyond 50m. γ is a hyperparameter, set to 2 in this embodiment.

[0068] Step 1-2: Global Feature Matching: After the previous step, the current UAV aerial image and all database satellite tiles are represented as feature vectors. Then, the Euclidean distance between the feature vectors of the aerial image and all satellite tiles is calculated. Based on the Euclidean distance, the Faiss built-in function IndexFlat2 is used to achieve efficient and reliable retrieval, returning the 50 database satellite images with the closest feature distance to the aerial image as candidate images, which are the 50 satellite images initially selected that are geographically close to the aerial image.

[0069] Step 2: Input the aerial image and the Top 50 satellite images into the local feature extraction network to perform key point selection and representation. That is, extract the local key points of the current aerial image and the candidate 50 satellite tile images, and use descriptors to represent all key points. These local key points can represent more detailed local information, and cross-validation based on local information can help to obtain more accurate location information.

[0070] This section specifically designs the Swin Descriptross network architecture based on Swin Transformer to achieve local key point feature extraction and representation.

[0071] The network architecture consists of an encoder, a bottleneck layer, a decoder, a keypoint selection module, and a keypoint feature description module. The main network is responsible for extracting feature maps and consists of an encoder, a bottleneck layer, and a decoder. It is based on a continuous SwinTransformer Block design and uses skip connections to better perceive the contextual information of local features. The SwinTransformer Block consists of two consecutive blocks. Each block consists of a LayerNorm (LN), a multi-head self-attention module (W-MSA), a sliding window multi-head attention module (SW-MSA), residual connections, and two MLP layers. This can be expressed by the following formula:

[0072]

[0073]

[0074]

[0075]

[0076] in, and z l Representing SW-MSA and l th The output of the MLP.

[0077] The multi-head self-attention module (W-MSA) can be represented as:

[0078]

[0079] in, M represents the query, key, and value matrix in a Transformer. 2 d and d represent the number of blocks and the dimension of the key or value in a window, respectively. The value of B is taken from the bias matrix.

[0080] The local keypoint features represented by this network include global attention, which focuses on features that remain unchanged in appearance due to changes in lighting, viewpoint, etc. at the same location, enabling robust matching and localization under drastic lighting changes and different seasons.

[0081] Step 2-1: Key point extraction: Extract features from the aerial image and candidate image respectively based on Swin Descriptors. Assume the input image is I with size w×h, and the output feature map is M=m(I), M∈F. w×h×n Its number of channels is n. In the feature space F w×h×n The algorithm employs a selection strategy that maximizes both the channel direction and the local plane of the high-dimensional feature map to initially select key points. The formula is as follows:

[0082] ( D z (Mid-local maximum)

[0083] In the formula, D z Let D be the eigenvalue of the z-th layer, and D z ∈F w×h ; D z The eigenvalue at point (x,y). For the keypoint P(x,y) to be selected, channel selection is first performed, that is, selecting the channel z with the largest response value from the n channel feature maps, resulting in the feature map D for that channel. z Next, verify. Is it the largest within a local 3×3 neighborhood? A point that satisfies both conditions is a key feature point.

[0084] Step 2-2: Keypoint Feature Description: Extract the vector (1×n) at (x,y) on the feature map M(w×h×n) and perform L2 normalization to obtain the feature descriptor.

[0085]

[0086] In the formula, v xy =M xy ,v∈F n Since the feature map size is smaller than the original image size, the SIFT algorithm is used to obtain precise sub-pixel-level keypoint locations through local interpolation. The final interpolated keypoint descriptor... It is an n-dimensional vector, and then matching is performed based on the Euclidean space distance of the key point descriptors.

[0087] Step 3: After extracting keypoints and obtaining corresponding feature descriptors, firstly, perform a FLANN nearest neighbor search on the keypoints of the aerial image and 50 candidate images to obtain all matching point pairs between the aerial image and the candidate images. Then, a dynamic adaptive distance constraint algorithm is used for filtering, removing matching pairs where the first distance is greater than the second distance and the average distance difference. This process is then used to coarsely refine the matching point pairs, filtering out mismatches with large differences. Finally, RANSAC geometric cross-validation is used to obtain the final keypoint matches between the aerial image and the 50 satellite tile images. The average pixel distance between all final matching point pairs between the aerial image and each candidate image is calculated. The candidate images are then rearranged in ascending order of distance, and the 5 satellite tiles with the smallest average distance to the matching point pairs of the aerial image are returned.

[0088] Specifically, the following steps are included:

[0089] Step 3-1: Based on the FLANN algorithm, search for all matching pairs between the aerial image and each candidate image to be screened, and obtain the first matching point pair with the closest Euclidean distance of the feature descriptors and the second closest matching point pair.

[0090] Step 3-2: Dynamically Adaptive Distance Constraint Filtering of Matching Points: First, calculate the average difference between the distances of the first and second matching point pairs found in the previous step:

[0091]

[0092] The filtering criteria are: the distance of the first matching point pair is less than the difference between the distance of the second matching point pair and the mean distance difference;

[0093] Step 3-3: Further filter matching pairs based on the RANSAC geometric constraint algorithm to obtain the final key point matching relationship between the aerial image and each satellite image in the Top 50.

[0094] Steps 3-4: Calculate the average pixel distance between the aerial image and each of the Top 50 candidate images for the final matching pair, and sort them by size. Return the 5 candidate images with the smallest average distance as the final matching images.

[0095] Step 4: Filter the matching point pairs between the 5 satellite tiles rearranged in the previous step and the current aerial image, eliminating many-to-one and one-to-many matching relationships, and retaining one-to-one matching pairs. Finally, using the final matching relationship between the current aerial image and the 5 satellite tiles, and the geographic information of the satellite tiles, the precise location information of the UAV corresponding to the current aerial image is solved based on the PnP (Perspective-n-Point) principle. That is, the pose change of all retained matching points is calculated based on PnP theory, and the translation vector of the camera coordinate system relative to the world coordinate system is the absolute geographic location of the current UAV.

[0096] Specifically, the following steps are included:

[0097] Step 4-1: For the aerial image and the 5 matching images returned in step 3, filter all one-to-many and many-to-one matching pairs in the final matching relationship generated in step 3, and retain the one-to-one matching relationship.

[0098] Step 4-2: 2D->3D Pose Calculation: After obtaining the matching point pairs, the precise current position of the UAV is calculated based on the multi-view geometric relationship. Given the GPS coordinates of the five satellite images to be matched and the pixel coordinates of the matching points, the GPS coordinates of the matching points in the satellite images can be calculated, thus obtaining the GPS coordinates of the matching points corresponding to the aerial images. Also, given the pixel coordinates of the matching points corresponding to the current aerial images of the UAV, based on the PnP (Perpendicular-n-Point) principle, the motion model of the coordinate system of the UAV's camera relative to the world coordinate system can be obtained, which is the camera pose. This camera pose is used as the UAV pose, and the translation amount in the pose is the precise current position of the UAV.

[0099] Specifically, to solve for the optimal camera pose, a nonlinear least squares problem regarding the reprojection error is constructed. The corresponding errors of the n matching points are summed to find the optimal camera pose that minimizes the total error. The reprojection error refers to the difference between the projected coordinates of the matching point in the pixel coordinate system and the projected coordinates in the pixel coordinate system calculated from the camera pose and the corresponding 3D point positions. It is expressed as follows:

[0100]

[0101] In the formula, p i These are the point coordinates in pixel coordinates. These represent the coordinates of the matching point in the corresponding world coordinate system. Both are different representations of the same point in different coordinate systems. The reprojected coordinates in the pixel coordinate system are obtained by calculating the camera's pose from the coordinates in the world coordinate system. Ideally, the reprojected coordinates should coincide; therefore, minimizing the reprojection error yields an estimate of the optimal camera pose T.

[0102] The UAV visual localization method based on deep semantic feature matching and multi-view geometric pose calculation adopted in this invention includes three major modules: global feature extraction and coarse matching, local key point feature extraction and matching, and multi-view geometric pose calculation of matching points; among which,

[0103] The global feature extraction and coarse matching module extracts global features from the current UAV aerial image and global features from the database satellite image, and returns the 50 satellite images whose features are closest to the aerial image features in Euclidean distance.

[0104] The local keypoint feature extraction and matching module extracts local keypoint features from the aerial image and 50 satellite images respectively, and obtains the final matching pairs after multiple screenings. The candidate images are then rearranged according to the average pixel distance of the final matching pairs, and the five nearest neighbor satellite images and their keypoint matching relationships are returned.

[0105] The matching point multi-view geometric pose calculation module filters the matching relationship between the aerial image and the final 5 satellite images, retains the one-to-one matching point pairs, performs 2D->3D pose calculation based on PnP, obtains the relative motion of the camera in the world coordinate system, and returns the final current absolute geographical location information of the UAV.

[0106] The method of this invention can obtain the absolute geographical location information of a drone by relying solely on a visual sensor when GPS is completely unavailable, and has great application value in denial environments such as canyons and buildings.

[0107] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A key point representation matching and UAV localization method, characterized in that: Includes the following steps: Step 1: Extract global features from the current drone aerial image and all satellite tiles in the satellite image database; Based on global features, Euclidean distance is used as the criterion to retrieve the top N satellite tile images in the satellite image database that are closest to the features of the current aerial image as candidate images. The process of extracting global features includes two parts: local feature extraction and feature aggregation. The local feature extraction is as follows: the image is input into a convolutional neural network to obtain a feature map of size W×H×D, which is used as a local feature descriptor x of size N×D; W is the width of the feature map, H is the height of the feature map, N=W×H represents the number of local features, and D is the dimension of the feature map, i.e. the dimension of the local features. The feature aggregation is as follows: Local feature descriptors x of size N×D are linearly transformed using a 1×1×D×K convolution kernel, and then subjected to softmax to obtain an N×K soft-assignment result; simultaneously, the local feature descriptors x of size N×D are fed into the VLAD core and clustered to obtain K D-dimensional cluster center vectors; the N×K soft-assignment is used to allocate the weights of the feature residuals to the cluster centers, and the weighted summation and normalization according to the cluster centers yield a K×D vector as the final global descriptor of the image; In the process of extracting global features, a Siamese neural network architecture based on NetVLAD is used to extract global features from aerial images and satellite images: two convolutional neural networks are used to extract local features from aerial images and satellite tiles respectively, and then all weight parameters of the feature aggregation part are shared; and a spatial attention module is added to the local feature extraction part to focus on the regions that contribute more to the task. Step 2: Input the current aerial image and the N satellite tile images retrieved in Step 1 into the local feature extraction network to extract local key points from the current aerial image and the N candidate images, and use descriptors to represent all key points; the local feature extraction network adopts the Swin Descriptros network architecture based on Swin Transformer. The Swin Descriptors network architecture consists of an encoder, a bottleneck layer, a decoder, a key point selection module, and a key point feature description module. The main network is responsible for extracting feature maps and consists of an encoder, a bottleneck layer, and a decoder. It is based on a continuous Swin Transformer Block design. The Swin Transformer Block consists of two consecutive blocks. Each block consists of a LayerNorm, a multi-head self-attention module W-MSA, a sliding window multi-head attention module SW-MSA, residual connections, and two MLP layers. Step 3: Perform FLANN nearest neighbor search on the key points of the aerial image and N candidate images to obtain all matching point pairs between the aerial image and the candidate images; then perform filtering based on the dynamic adaptive distance constraint algorithm to coarsely purify the matching point pairs; then obtain the final key point matching relationship between the aerial image and each candidate image based on RANSAC geometric cross-validation; calculate the average pixel distance of all final key point matching pairs between the aerial image and each candidate image, and obtain the top M candidate images with the smallest average pixel distance as the final matching images; Step 4: Filter the matching point pairs between the M final matching images obtained in Step 3 and the current aerial image, removing many-to-one and one-to-many matching relationships and retaining one-to-one matching pairs; use the final matching relationship between the current aerial image and the M final matching images and the geographical information of the M final matching images to solve the precise location information of the UAV corresponding to the current aerial image based on the PnP method.

2. The key point representation matching and UAV localization method according to claim 1, characterized in that: In step 1, during the extraction of global features, the weighted soft-margin ranking loss function is used as the loss function during network training.

3. The key point representation matching and UAV localization method according to claim 1, characterized in that: In step 1, the retrieval and matching process is as follows: the global features of the current UAV aerial image and all database satellite tiles are represented as a feature vector respectively; then the Euclidean distance between the aerial image and the feature vectors of all satellite tiles is calculated, and the retrieval is performed based on the Euclidean distance, returning the N database satellite images that are closest to the feature distance of the aerial image as candidate images.

4. The key point representation matching and UAV localization method according to claim 1, characterized in that: The specific steps of step 2 are as follows: Step 2-1: Key point extraction: Extract features from the aerial image and candidate image respectively based on Swin Descriptors; let the input image be I, with size w×h, and the output feature map. Its number of channels is n; in the feature space The algorithm employs a selection strategy that maximizes both the channel direction and the local plane of the high-dimensional feature map to initially select key points. The formula is as follows: In the formula, Let be the eigenvalue of the z-th layer, and ; for The eigenvalues ​​at the point (x, y); Key points for selection First, channel selection is performed. From the n channel feature maps, the channel z with the largest response value is selected, and the feature map of the corresponding channel is obtained. Next, verify. Is it the largest in a local 3×3 neighborhood? If both conditions are met, it is a key feature point. Step 2-2: Key Point Feature Description: In the feature map Extract the vector at (x,y) Then, L2 normalization is performed to obtain the feature descriptor. : In the formula, .

5. The key point representation matching and UAV localization method according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3-1: Based on the FLANN algorithm, search for all matching pairs between the aerial image and each candidate image to be filtered, and obtain the first matching point pair and the second closest matching point pair with the Euclidean distance of the feature descriptors; Step 3-2: Dynamically Adaptive Distance Constraint Filtering of Matching Points: First, calculate the average difference between the distances of the first and second matching point pairs found in the previous step: The filtering criteria are: the distance of the first matching point pair is less than the difference between the distance of the second matching point pair and the mean distance difference; Step 3-3: Further filter matching pairs based on the RANSAC geometric constraint algorithm to obtain the final key point matching relationship between the aerial image and each candidate image; Steps 3-4: Calculate the average pixel distance between the aerial image and each candidate image for the final matching pair, and obtain the M candidate images with the smallest average distance as the final matching images.

6. A computer-readable storage medium, characterized in that: The device stores computer-executable instructions, which, when executed, are used to implement the method described in any one of claims 1 to 5.

7. A computer system, characterized in that: include: One or more processors, a computer-readable storage medium, for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 5.

Images