UAV positioning method based on multi-level feature fusion

Abstract

The present invention discloses a method for unmanned aerial vehicle (UAV) positioning based on multi-level feature fusion. A UAV is flown in the air, and air-to-ground perspective photos are taken by the UAV as UAV photos. A dataset is constructed using the UAV photos. A pyramid self-attention network with conditional position encoding is used to process the dataset and extract image feature maps. The extracted image feature maps are cross-layer fused to obtain a feature heat map, and the heat value distribution in the fused feature heat map is used to determine the GPS of the UAV image. The present invention fully utilizes and enhances hierarchical features of different scales to generate feature heat maps. The method has a wide range of applications and can be combined with most hierarchical networks. While reducing the amount of computation, it also improves subsequent positioning performance, facilitating accurate positioning of UAVs when GNSS signals are lost.

Description

Technical Field

[0001] The present invention relates to a UAV positioning method belonging to the fields of image processing, deep learning technology, and geographic positioning, and in particular to a UAV positioning method based on multi-level feature fusion. Background Art

[0002] Drone technology is rapidly developing, with numerous drones being used in applications such as aerial photography and mapping, precision delivery, disaster monitoring, and military reconnaissance. The core of these applications lies in their autonomous positioning capabilities and autonomous navigation and obstacle avoidance capabilities to complete specific tasks. The core of drones' autonomous positioning capabilities relies on the Global Positioning System (GPS) and onboard inertial navigation units. The strength of the GPS signal received by a drone directly impacts its positioning performance. However, in real-world scenarios, drones often operate in environments characterized by strong interference, numerous obstacles, and numerous uncertainties. GPS signals can be affected by external obstructions, electromagnetic interference, and inclement weather. These factors can disrupt the GPS signal. In extreme cases, the GPS signal can be completely lost, rendering the drone unable to operate normally and, in this case, in a GPS-denied environment. How to enable drones to overcome these positioning challenges and achieve a certain degree of autonomous positioning in such environments has become a research challenge.

[0003] To address the problem of autonomous drone positioning in GPS-denied environments, visual algorithms are currently being used. These algorithms primarily rely on image retrieval methods. The general process involves cropping a satellite map of the query area to generate a satellite image library of that area. Each satellite image in the library uses the GPS of the center point as the image's geographic location. A trained neural network model is then used to extract features from the current drone-viewed image and all images in the satellite image library to generate a feature vector. The distance between the feature vector of the query drone image and the feature vectors in the entire image library is calculated. The satellite image in the satellite image library with the feature vector closest to the drone image's feature distance is used as the matching image of the drone image from the satellite view. The GPS of this satellite image can then be used as the GPS of the drone image to achieve drone positioning.

[0004] Because the drone positioning method based on image retrieval needs to crop and store the satellite map of the query area, the granularity of the cropping will affect the final positioning error and the pressure on the storage end. If you want to achieve accurate positioning through image retrieval, you need to crop the query area according to a smaller pixel interval, which greatly increases the number of photos in the gallery, causing huge storage pressure on the storage end and greatly increasing the time it takes for the subsequent model to extract features and compare image feature distances. In the actual positioning process, the cropping operation on the satellite map cannot be done at a very small pixel interval. This is a balance between positioning accuracy, storage pressure and calculation time. The current drone positioning method based on image retrieval can only achieve approximate positioning and cannot obtain a more accurate result. Summary of the Invention

[0005] In order to solve the technical problems in the background technology, the present invention proposes a drone positioning method based on multi-level feature fusion. In order to enable the drone to still have autonomous positioning capabilities in a GPS-denied environment and to be able to work normally when the GPS signal is affected in various complex working environments, feature extraction is performed on drone images and satellite map images through a pyramid self-attention feature extraction network with conditional position coding, and then the features of the drone and satellite images are fused and enhanced through a multi-level fusion module. Finally, the output heat map is interpolated to the size of the satellite image, and the final positioning result is given by the heat value distribution of the heat map.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] Step 1: Fly the drone in the air and use the drone to take air-to-ground perspective photos as drone photos;

[0008] Step 2: Build a dataset using drone photos;

[0009] Step 3: Use the pyramid self-attention network with conditional position encoding to process the dataset and extract image feature maps;

[0010] Step 4: Cross-level fusion of the extracted image feature map is performed to obtain a feature heat map, and the heat value distribution in the fused feature heat map is used to determine the GPS of the drone image.

[0011] Specifically, step 1 is as follows: when the drone is flying in the air, the gimbal camera on the drone is used to take photos of the ground at a vertical ground perspective on routes at different flight altitudes to obtain photos of the ground perspective. Each photo is recorded with the GPS information at the time of shooting through the drone's own GPS.

[0012] The step 2 is specifically as follows:

[0013] After removing abnormal drone photos, the GPS information contained in each retained drone photo is used to extract the image of the same GPS location in the satellite map and crop it to obtain a satellite map picture. The satellite map picture and its corresponding drone photo are combined to construct a drone-satellite map image pair. In this way, all drone-satellite map image pairs are produced to form a dataset.

[0014] In step 3:

[0015] After preprocessing, the drone photos and satellite map images of each drone-satellite map pair in the dataset are fed into their respective conditional position encoding pyramid self-attention networks for feature extraction.

[0016] After entering the pyramid self-attention network with conditional position encoding, drone photos and satellite map images go through four consecutive stages. In each stage, the image is first compressed by the encoding convolutional layer using the patch embedding operation. The features are then fed into an attention mechanism encoder module, the Transformer Encoder. The features are then conditionally position-encoded using the position encoding generator (PEG). Finally, the features are fed into different numbers of attention mechanism encoder modules. After four stages, image feature maps of different sizes are generated for each image. The corresponding image feature maps obtained in the four stages for drone photos and satellite map images are saved.

[0017] The number of attention mechanism encoder modules Transformer Encoder in different stages of PCPVT is different, 2 in the first stage, 2 in the second stage, 5 in the third stage, and 2 in the fourth stage.

[0018] The step 4 is specifically as follows:

[0019] 4.1. The feature maps of drone photos and satellite map images saved in list form are fused through group convolution:

[0020] First, the image feature map obtained from the drone photo in the third stage is used as the template feature map for dimensionality reduction. The template feature map dimension is then reduced to the same dimension as the image feature map obtained from the satellite map image in the three stages. Then, the template feature map is fused with the image feature map obtained from the corresponding stages of the satellite map image in the form of group convolution to obtain feature heat maps of three different scales. The feature maps of the three scales are then sent to the feature pyramid network for processing to obtain a heat map of a larger scale.

[0021] 4.2. Interpolate the heat map to the same size as the satellite map image, determine the heat value in the interpolated heat map, and find the pixel position with the largest heat value in the heat map as the positioning position of the drone image on the satellite map.

[0022] Specifically, the step 4.2) is to use the geographic information of the pixel position with the largest heat value to calculate the positioning position of the drone image, which is used as the current positioning position of the drone.

[0023] In the solution of the present invention, a drone is flown in the air, and air-to-ground perspective photos are taken by the drone as drone photos; a dataset is constructed using the drone photos; the dataset is processed using a pyramid self-attention network with conditional position encoding to extract image feature maps; the extracted image feature maps are cross-layer fused to obtain a feature heat map, and the heat value distribution in the fused feature heat map is used to determine the GPS of the drone image.

[0024] The beneficial effects of the present invention are:

[0025] The present invention fully utilizes the hierarchical features of different scales generated by the feature extraction network by designing a multi-level feature fusion network structure, thereby enhancing the generated feature heat map. This network structure has a wide range of applications and can be combined with most hierarchical structure networks, thereby reducing the amount of computation while improving the subsequent positioning effect. Compared with the current mainstream positioning method based on image retrieval, the content of the present invention does not require the generation of a satellite map library with the area to be queried, which puts less pressure on the local storage device. It does not require the retrieval of the satellite image closest to the drone image to be queried from the satellite library to be queried, thereby reducing the inference time of geolocation and optimizing the positioning efficiency. Ultimately, the current location of the drone image to be queried is directly given on the satellite map. Compared with traditional methods, the positioning accuracy is higher, which is beneficial for drones to use pure visual algorithms to achieve higher-precision geolocation effects when the GNSS signal is lost. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the process of the present invention;

[0027] Figure 2 This is the overall network architecture diagram of the present invention;

[0028] Figure 3 Schematic diagram of the multi-level feature fusion module in the present invention. DETAILED DESCRIPTION

[0029] The following is a detailed description of the multi-level feature fusion UAV positioning method proposed in the present invention in conjunction with the accompanying drawings.

[0030] like Figure 1As shown, the embodiments of the present invention are as follows:

[0031] Step 1: The drone flies in the air in real time, and uses the drone to take air-to-ground perspective photos as drone photos;

[0032] When the drone is flying in the air in real time, the gimbal camera on the drone is used to shoot ground perspectives at a vertical ground perspective at different flight altitudes of 80m, 90m and 100m. The ground perspective photos are used as air-to-ground perspective photos. Each photo records the GPS information at the time of shooting through the drone's built-in GPS.

[0033] Use a drone to shoot air-to-ground images at the planned waypoints. The specific method is as follows:

[0034] Use the drone to fly a pre-planned route. Set a fixed waypoint on the route. When the drone reaches the waypoint, use the drone's gimbal camera to take photos perpendicular to the ground. Use this method to take photos of the ground from the drone's perspective on multiple routes.

[0035] Step 2: Build a dataset using drone photos;

[0036] After removing abnormal drone photos, the GPS information contained in each retained drone photo is used to extract a preset fixed-size image of the same GPS location in the satellite map and crop it to obtain a satellite map image. The satellite map image and its corresponding drone photo are combined to construct a drone-satellite map image pair. All retained air-to-ground perspective photos are processed in this way to produce a dataset of all drone-satellite map image pairs for subsequent network training.

[0037] Step 2 begins with a preliminary image screening, removing low-quality drone photos such as those that are overexposed, underexposed, or out of focus. Next, we use the GPS information embedded in the drone images and combine it with satellite maps to create a dataset. The GPS in the drone image is the GPS of the image's center point. Based on this GPS information, we capture a certain range of satellite map images on the corresponding satellite map as the corresponding drone image for training.

[0038] The satellite map can be a Google satellite map.

[0039] Step 3: Use the pyramid self-attention network with conditional position encoding to process the dataset and extract the image feature map;

[0040] like Figure 2As shown in the figure, the drone photos and satellite map images of each pair of drone-satellite map images in the dataset are preprocessed and then sent to their respective conditional position encoding pyramid self-attention networks for feature extraction;

[0041] After entering the pyramid self-attention network with conditional position encoding, drone photos and satellite map images go through four consecutive stages. In each stage, the features are first compressed by the encoding convolution layer, then sent to an attention mechanism encoder, and then conditionally encoded using the position encoding generator PEG. Finally, they enter the Transformer Encoder module with different numbers of attention mechanism encoders. After four stages, image feature maps of different sizes are generated. The image feature maps obtained in the four stages for drone photos and satellite map images are saved in list form for subsequent operations.

[0042] In step 3, the pyramid self-attention network with conditional position coding (PCPVT) is used to extract image features for the image feature information of drone images and satellite map images. The specific method is as follows:

[0043] The drone image and satellite map image are fed into the pyramid self-attention network with conditional position encoding for feature extraction. Before feeding into the pyramid self-attention network with conditional position encoding, the pixel size of the drone image is adjusted to 176×176 and the pixel size of the satellite image is adjusted to 400×400. Data augmentation is then performed on the image. When performing data manipulation on the image, a new coordinate (X new , Y new ) as the new center coordinate of the image. Then use (X new , Y new ) is used as the center to crop an image of random size, and the cropped image must contain the center position of the original satellite image. The drone image and satellite map image are then fed into a pyramid self-attention network with conditional position encoding.

[0044] Because images captured from drone and satellite perspectives differ significantly in style, such as lighting and clarity, these two perspectives can be considered heterogeneous, necessitating feature extraction for each. The drone image and satellite map image are fed into two conditional position-encoded pyramid self-attention networks for feature extraction. The conditional position-encoded pyramid self-attention network has four stages, and only the feature maps extracted from the first three stages are utilized in this invention. After the 176×176 drone image is fed into the conditional position-encoded pyramid self-attention network, it undergoes a coding convolutional layer for size reduction before passing through the first stage. It then passes through an attention mechanism encoder, where features are encoded using a positional encoding generator. The features are then fed into several attention mechanism encoders: two layers in the first stage, two layers in the second, and five layers in the third. The size and dimensionality of the feature images output from each stage vary. As the stages increase, the size of the image features decreases while the dimensionality increases. The image feature dimensions and sizes generated by the drone image feature extraction branch after the three stages of the feature extraction network are: 64×44×44, 128×22×22, and 320×11×11.

[0045] Similarly, satellite map images will also go through the three stages of the feature extraction network, and finally produce feature maps with sizes of 64×100×100, 128×50×50, and 320×25×25.

[0046] Step 4: Figure 3 As shown in the figure, the extracted image feature map is fused across levels to obtain a feature heat map. The heat value distribution in the fused feature heat map is used to determine the GPS of the drone image, thus realizing drone positioning with multi-level feature fusion.

[0047] 4.1. The feature maps of drone photos and satellite map images saved in list form are fused through group convolution:

[0048] First, the image feature map obtained from the drone photo in the third stage is used as the template feature map for dimensionality reduction. The template feature map dimension is then reduced to the same dimension as the image feature map obtained from the satellite map image at each level in the three stages. Then, the template feature map is fused with the image feature map obtained from the corresponding stages of the satellite map image in the form of group convolution to obtain feature heat maps of three different scales. The feature maps of the three sizes are then sent to the feature pyramid network for processing to obtain a larger-scale heat map.

[0049] In the specific implementation step 4.1, the extracted features are fused across levels, as follows:

[0050] The 320×11×11 feature map generated in the third stage of the drone image extraction branch is subjected to dimensionality reduction to the same dimensions as the feature maps generated in the three stages of the satellite map image feature extraction branch: 64, 128, and 320, respectively. This dimensionality reduction is accomplished using 1×1 convolution operations, which facilitates subsequent feature fusion.

[0051] The dimensions of the three features after dimensionality reduction are 64×11×11, 128×11×11, and 320×11×11. They are named Z1, Z2, and Z3.

[0052] The three feature maps generated by the satellite map image are named X1, X2 and X3 respectively.

[0053] A cross-level fusion operation is performed on the drone image features and satellite map image features. The drone features Z1, Z2, and Z3 generated after dimensionality reduction are used as convolution kernels to perform grouped convolution operations on the satellite map features X1, x2, and X3, respectively. Three fused feature maps C1, C2, and C3 are generated. The feature sizes of C1, C2, and C3 are 1×100×100, 1×50×50, and 1×25×25, respectively.

[0054] The three feature maps C1, C2 and C3 are sent to the feature pyramid network for feature enhancement, C3 is upsampled by 2 times and added to C2 to obtain C2′, and then C2′ is upsampled by 2 times and added to C3 to obtain the final output feature map F.

[0055] When adding the upsampled feature map in the feature pyramid network, a convolution operation with a size of 1×1 and a stride of 1 is used to perform a convolution operation on the added feature map.

[0056] 4.2. Interpolate the heat map to the same size as the satellite map image. Determine the heat value in the interpolated heat map and find the pixel position with the largest heat value in the heat map. This will be used as the location of the drone image on the satellite map.

[0057] Step 4.2) is specifically as follows: since the satellite map already contains geographic information, such as latitude and longitude information, the geographic information of the pixel position with the largest heat value is used to calculate the location of the drone image to be queried, which is used as the current location of the drone.

[0058] The final step is to use the heat value distribution in the fused feature heat map obtained in step 4.2 to determine the GPS of the drone image. The details are as follows:

[0059] The feature map F output after multi-level feature fusion is interpolated, and the size of the interpolated feature heat map is restored to the size of the satellite map. Finally, the relative position of the drone image in the satellite map image is determined by the distribution of heat values ​​in the heat map. Since the GPS properties of the satellite map are known in advance, the relative position of the drone on the satellite map image can be used to calculate the specific GPS information of the drone for positioning.

[0060] In summary, the UAV visual positioning method based on the feature fusion algorithm provided in the embodiment of the present application performs multiple-level feature extraction on the UAV satellite image by using the pyramid self-attention network with conditional position coding, and then fuses and enhances the feature images of the UAV satellite at different scales through the multi-level fusion module to output a feature heat map. After the feature heat map is interpolated, the relative position of the UAV image on the satellite map image is determined by the distribution of the heat values ​​in the heat map, and finally the GPS information of the UAV is calculated. Compared with the traditional image retrieval geolocation method, the present invention eliminates the need for the satellite image library establishment and the search process in the satellite image library, and adopts the feature fusion method to directly provide the positioning result of the UAV to be queried in the satellite image. This positioning method has low computational complexity, fast positioning speed, and more accurate results, and can be used for UAV self-positioning in a GNSS-denied environment.

Claims

1. A method for UAV positioning based on multi-level feature fusion, characterized in that: The method comprises the following steps: Step 1: Fly the drone in the air and use the drone to take air-to-ground perspective photos as drone photos; Step 2: Build a dataset using drone photos; Step 3: Use the pyramid self-attention network with conditional position encoding to process the dataset and extract image feature maps; Step 4: Cross-level fusion of the extracted image feature maps to obtain a feature heat map, and use the heat value distribution in the fused feature heat map to determine the GPS of the drone image; The step 4 is specifically as follows: 4.

1. The feature maps of drone photos and satellite map images are fused by group convolution: First, the image feature map obtained from the drone photo in the third stage is used as the template feature map for dimensionality reduction. The template feature map dimension is then reduced to the same dimension as the image feature map obtained from the satellite map image in the three stages. Then, the template feature map is fused with the image feature map obtained from the corresponding stages of the satellite map image in the form of group convolution to obtain feature heat maps of three different scales. The feature maps of the three scales are then sent to the feature pyramid network for processing to obtain a heat map of a larger scale. 4.

2. Interpolate the heat map to the same size as the satellite map image, determine the heat value in the interpolated heat map, and find the pixel position with the largest heat value in the heat map as the positioning position of the drone image on the satellite map.

2. The method for positioning an unmanned aerial vehicle based on multi-level feature fusion according to claim 1, characterized in that: Specifically, step 1 is as follows: when the drone is flying in the air, the gimbal camera on the drone is used to take photos of the ground at a vertical ground perspective on routes at different flight altitudes to obtain photos of the ground perspective. Each photo is recorded with the GPS information at the time of shooting through the drone's own GPS.

3. The method for positioning a UAV based on multi-level feature fusion according to claim 1, characterized in that: Step 2 specifically includes: after removing abnormal photos from the drone photos, using the GPS information contained in each retained drone photo to extract the image of the same GPS location in the satellite map and cropping it to obtain a satellite map picture, combining the satellite map picture with its corresponding drone photo to construct a drone-satellite map picture pair, and in this way, all drone-satellite map picture pairs are produced to form a dataset.

4. The method for positioning a UAV based on multi-level feature fusion according to claim 1, characterized in that: In step 3, the drone photos and satellite map images of each pair of drone-satellite map images in the data set are pre-processed and then sent to their respective conditional position encoded pyramid self-attention networks for feature extraction; after entering the conditional position encoded pyramid self-attention network, the drone photos and satellite map images undergo four consecutive stages, each of which is first compressed by the encoding convolution layer operation, then sent to an attention mechanism encoder module, and then conditionally encoded using the position encoding generator, and finally enters a different number of attention mechanism encoder modules; After the four stages, image feature maps of different sizes are generated respectively, and the image feature maps corresponding to the four stages of the drone photos and satellite map images are saved.

5. The method for positioning a UAV based on multi-level feature fusion according to claim 1, characterized in that: Specifically, step 4.2 is to calculate the location of the drone image using the geographic information of the pixel position with the largest heat value, which is used as the current location of the drone.

Images