A Visual Localization Method for Unmanned Aerial Vehicles Based on Multi-Prediction Graph Fusion Network

Abstract

This invention discloses a UAV visual localization method based on a multi-prediction graph fusion network. The method utilizes UAV images taken from above within a search area, extracts positioning information from satellite images, and marks the UAV's position. The UAV images and satellite images are then fed into a neural network for processing to obtain the UAV's position within the satellite images. Pre-obtained UAV images, satellite images, and the UAV's position region within the satellite images are input into the neural network for training. Real-time UAV images captured during flight are then input into the trained feature extraction network for prediction processing to obtain the UAV's positioning information. This invention employs a Siamese network, introduces a novel loss calculation during training, and addresses the issues of missing positional information and multi-scale problems through a feature pyramid structure and a multi-prediction graph fusion network.

Description

Technical Field

[0001] This invention belongs to the field of image processing and deep learning technology and relates to a visual image processing method, particularly a UAV visual localization method based on a multi-prediction graph fusion network. Background Technology

[0002] In recent years, the use of drones has become increasingly widespread in both civilian and military fields. However, the problem of autonomous drone movement in GPS-denied environments has not yet been effectively solved. After transmission over thousands of kilometers, interference with satellite signals is easily achieved. Cross-view geolocation aims to match an image from one perspective with an image from another perspective representing the same geographical target. Applications of drone-satellite image matching can be broadly categorized into two types: drone-line-of-view target localization and drone navigation. For example, drone images can be used to match satellite images of the same geographical location. Satellite images generally contain accurate GPS information, allowing for real-time drone positioning and thus enabling drone navigation without the need for GPS devices.

[0003] Currently, most cross-view geolocation is achieved through image retrieval, which can only provide a rough location. Furthermore, due to differences in perspective, even if the image matching is correct, image retrieval methods still introduce significant distance errors. This is especially true for drone aerial photography, where the actual location of the drone and the location of the photographed object are inherently difficult to determine due to the perspective.

[0004] Based on this, VIGOR also proposed adding X and Y bias terms based on the localization after image retrieval to achieve a more accurate method. However, this method is quite difficult for the network to learn. Summary of the Invention

[0005] To address the problems existing in the background technology and to achieve visual positioning of UAVs, this invention proposes a UAV visual positioning method based on a multi-prediction map fusion network. This method solves the problems of lost location information and poor ability to handle multi-scale issues. By introducing a pyramid structure to restore the feature map to a larger size, the loss of location information is reduced. Then, through a multi-prediction map fusion network, different prediction results are weighted and fused together. The position of the UAV in the satellite image is calculated based on the final output prediction map.

[0006] To achieve the above objectives, the technical solution of this application includes the following steps:

[0007] Step 1: Use a drone to take aerial photos of the known search area from a vertical perspective. At the same time, capture satellite images of the search area and the positioning information in the satellite images. The drone's position is marked in the satellite images.

[0008] After step 1 and before step 2, the drone image and satellite image are scaled proportionally. The size of the drone image is set to 128*128*3, and the size of the satellite image is set to 400*400*3.

[0009] In step 1, latitude and longitude information is extracted from the upper left and lower right corners of the satellite image as positioning information.

[0010] Step 2: Input the drone images and satellite images of the search area into the neural network for processing to obtain the drone's position in the satellite images;

[0011] Step 3: Input the pre-obtained drone images, satellite images, and the location region of the drone in the satellite images into the neural network of Step 2 for training;

[0012] Step 4: In the scenario to be located, the drone image with unknown location information collected during the drone's real-time flight is input into the trained feature extraction network for prediction processing, and the output is the drone's location information at the time of image collection, thus realizing drone visual positioning.

[0013] Step 1 specifically involves setting the gimbal to vertically downwards, allowing the drone to use its built-in camera to take pictures of the ground in the search area from a height of over 80m.

[0014] Step 2 specifically involves:

[0015] Step 2.1: Input the UAV images and satellite images of the search area into their respective feature extraction networks, and obtain the final feature map through two-stage feature extraction processing in the feature extraction network;

[0016] Step 2.2: The feature maps output by the feature extraction network from the UAV images and satellite images are fused at different scales by the multi-prediction map fusion network to obtain the final prediction map. The final prediction map is then post-processed to obtain the position of the UAV in the satellite image when the UAV images were acquired.

[0017] Step 2.1 specifically involves: inputting the drone images and satellite images into their respective feature extraction networks to obtain their respective final feature maps. Each feature extraction network includes a conditionally located pyramid attention network and a feature pyramid network. First, preliminary feature extraction is performed through the conditionally located pyramid attention network to obtain a preliminary feature map. Then, the feature pyramid network is used to further extract features from the preliminary feature map to obtain the final feature map. The scale of the generated final feature map is four times larger than that of the original preliminary feature map. Furthermore, the conditionally located pyramid attention network and the feature pyramid network in the feature extraction networks for drone images and satellite images do not share weights.

[0018] The conditionally located pyramid attention network includes a sequential 4x downsampling and two consecutive 2x downsampling operations, and the feature pyramid network includes two consecutive 2x upsampling operations. The input image is processed by the conditionally located pyramid attention network through the 4x downsampling and two consecutive 2x downsampling operations to obtain a preliminary feature map. The preliminary feature map is then processed by a 1*1 convolution operation and input into the feature pyramid network, and then processed by two consecutive 2x upsampling operations to obtain the final feature map.

[0019] The feature map obtained after the first 2x upsampling operation in the feature pyramid network and the feature map obtained after the first 2x downsampling operation in the pyramid attention network with conditional position encoding are added together and then input into the second 2x upsampling operation of the feature pyramid network. The feature map obtained after the second 2x upsampling operation in the feature pyramid network and the feature map obtained after the fourth downsampling operation in the pyramid attention network with conditional position encoding are added together and used as the final feature map of the feature extraction network.

[0020] Step 2.2 specifically includes:

[0021] First, in the multi-prediction map fusion network, the feature maps output by the last three operations of the feature extraction network of the UAV image are compared with the final feature map output by the final stage of the feature extraction network of the satellite image to obtain three prediction maps. The three prediction maps are then weighted and added together using their respective learnable parameters to generate a final prediction map.

[0022] Then, the nearest neighbor interpolation method is used to restore the final predicted map to the same size as the satellite image to obtain the restored map. The restored map is then normalized to obtain the heat map. The position of the pixel with the largest pixel value in the heat map is mapped to the satellite image to obtain the position of the drone in the satellite image when the drone image was collected.

[0023] In step 3, during training, the Hanning loss is used to calculate the loss of the neural network. The losses of the prediction maps at different scales output by the neural network are combined for forward propagation, while backpropagation is also performed to update the network parameters, thereby training the neural network.

[0024] In step 3, Hanning weights are used to calculate the loss of the three prediction maps and the final prediction map to obtain four losses. The four losses are then combined and backpropagated to update the network parameters of the feature extraction network.

[0025] In step 4, the position of the UAV in the satellite image is first obtained through prediction processing in the neural network. Then, the positioning information of the UAV in the satellite image is combined with the positioning information in the satellite image to perform interpolation calculation to obtain the positioning information of the UAV in the satellite image. This information is used as the positioning information of the UAV when the UAV image is acquired, thus realizing the UAV's visual positioning.

[0026] The network of this invention generates four prediction maps of the same scale after similarity calculation and weighted fusion, and then uses Hanning weights to calculate the loss of different feature maps respectively.

[0027] This invention incorporates Hanning loss and uses Hanning weights to assign weights to the image regions corresponding to positive samples. In practical applications, a hyperparameter Center-R is set to adjust the range of positive samples.

[0028] The loss calculation is specifically performed according to the following formula:

[0029]

[0030] Where L represents the loss of a single prediction graph, y i The label parameter represents pixel i, with 1 for positive and 0 for negative; p i W represents the probability that pixel i is predicted to be of the positive class; i This represents the weight of pixel i; i represents the pixel number.

[0031] Furthermore, a square region is constructed centered on the drone's position in the satellite image, with the added hyperparameter Center-R being of variable length, to represent the drone's position region in the satellite image. This position region is used as the positive sample, and the remaining regions outside the position region are used as negative samples. During each loss calculation, the weights W of all pixels in the positive samples are... i The sum is set to 1, and the weights W of all pixels in the negative sample are set to 1. i The sum of these remains 1.

[0032] Since the number of negative samples is much greater than the number of positive samples, this invention introduces a hyperparameter, negativeweight, which can be used to adjust the weight of negative samples. This processing enables drone image localization to obtain accurate location information, avoiding the problems of inaccurate localization caused by an excessively large positive sample range and the difficulties in network training caused by an excessively small positive sample range.

[0033] This invention utilizes Siamese networks, employing a feature pyramid structure and a multi-prediction graph fusion network to address the issues of missing location information and multi-scale problems. Furthermore, it introduces a novel loss calculation method during training and develops a completely new loss function, the Hanning loss.

[0034] This invention first requires feature extraction from UAV images and satellite images. Then, the similarity between UAV feature maps at different scales and satellite feature maps is calculated to obtain different prediction results. Finally, the different prediction results are weighted and fused to obtain the final result.

[0035] The beneficial effects of this invention are:

[0036] This invention proposes a simple and effective model for UAV localization in denied environments.

[0037] This invention allows for the fusion of different learned information from satellite and UAV imagery, calculated based on similarity, through a weighted approach. This significantly enhances the model's ability to solve multi-scale problems. Furthermore, the invention addresses the issue of inaccurate localization caused by the low resolution of the output heatmap by introducing an advanced pyramid structure. Finally, the invention incorporates Hanning weights, enabling the model to focus more on the center position of the target region, thereby improving localization accuracy.

[0038] The method of this invention achieved excellent results on the UL14 dataset, with positioning accuracies of 17.83%, 32.35%, and 55.35% within 3 meters, 5 meters, and 10 meters, respectively. This represents a significant improvement over previous models and lays a solid foundation for the implementation of purely vision-based UAV positioning and navigation technology.

[0039] This invention solves the problem of location information loss caused by feature map compression during the localization process, and improves the network's ability to solve multi-scale problems, thereby enhancing the network's localization performance. Attached Figure Description

[0040] Figure 1 This is a simplified flowchart illustrating the method of the present invention.

[0041] Figure 2 This is a flowchart of the method of the present invention.

[0042] Figure 3 This refers to the label setting method in the present invention.

[0043] Figure 4 This is a graph of the Hanning window function in the method of the present invention.

[0044] Figure 5 This diagram illustrates the positioning performance of the method of the present invention.

[0045] Figure 6 This is a process diagram of an embodiment of the method of the present invention. Detailed Implementation

[0046] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent; it is understandable that certain well-known structures and their descriptions may be omitted in the drawings for those skilled in the art.

[0047] like Figure 1 and Figure 6 The embodiments and implementation process of the present invention are shown below:

[0048] Step 1: Use a drone to take aerial photos of the known search area from a vertical perspective. At the same time, capture satellite images of the search area and the positioning information in the satellite images. The location of the drone when the images were taken is marked in the satellite images, which serves as the drone's position in the satellite images.

[0049] In practice, satellite images containing the search area can be captured from the satellite based on the search area flown by the drone, and the drone's position can be marked in the images, with a dot marking the drone's position in the satellite image.

[0050] After step 1 and before step 2, the drone images and satellite images are scaled proportionally, with the drone images set to 128*128*3 and the satellite images set to 400*400*3.

[0051] Step 1 involves setting the gimbal to point vertically downwards, and then using the built-in camera to take pictures of the ground in the search area from a height of over 80m.

[0052] The specific process is as follows: Under good conditions, the drone's built-in camera is used to take pictures of the ground. During the flight, the drone's gimbal is always kept vertically downward. After that, the data stored in the drone is extracted and saved.

[0053] Step 2: Input the drone images and satellite images of the search area into the neural network for processing to obtain the drone's position in the satellite images;

[0054] Step 2.1: Input the UAV images and satellite images of the search area into their respective feature extraction networks, and obtain the final feature map through two-stage feature extraction processing in the feature extraction network;

[0055] Step 2.1 specifically involves inputting the drone images and satellite images into their respective feature extraction networks to obtain their final feature maps. Each feature extraction network includes a conditional location encoding pyramid attention network and a feature pyramid network. First, the conditional location encoding pyramid attention network performs preliminary feature extraction to obtain preliminary feature maps. The feature map size output by the satellite image after preliminary feature extraction is 25*25*320, and the feature map size output by the drone image after preliminary feature extraction is 8*8*320. Then, the feature pyramid network is used to further extract features from the preliminary feature maps obtained after preliminary feature extraction to obtain the final feature maps. The scale of the generated final feature maps is four times larger than that of the original preliminary feature maps, and the loss of location information caused by the compression of the preliminary feature maps is reduced.

[0056] In addition, different stages of the pyramid attention network through conditional location encoding will generate different feature maps, which will also be saved for subsequent network fusion.

[0057] In the feature extraction networks for drone and satellite images, the conditional location encoding pyramid attention network and the feature pyramid network do not share weights.

[0058] The Conditional Location Encoding Pyramid Attention Network includes a sequential 4x downsampling and two consecutive 2x downsampling operations, while the Feature Pyramid Network includes two consecutive 2x upsampling operations. The input image is processed by the Conditional Location Encoding Pyramid Attention Network through the 4x downsampling and two consecutive 2x downsampling operations to obtain a preliminary feature map. The preliminary feature map is then processed by a 1x1 convolution operation and input into the Feature Pyramid Network, where two consecutive 2x upsampling operations are performed to obtain the final feature map. Specifically, the feature map obtained after the second 2x downsampling operation in the Conditional Location Encoding Pyramid Attention Network is then processed by a 1x1 convolution operation before being input into the Feature Pyramid Network.

[0059] The feature map obtained after the first 2x upsampling operation in the feature pyramid network and the feature map obtained after the first 2x downsampling operation in the pyramid attention network with conditional position encoding are added together and then input into the second 2x upsampling operation of the feature pyramid network. The feature map obtained after the second 2x upsampling operation in the feature pyramid network and the feature map obtained after the fourth downsampling operation in the pyramid attention network with conditional position encoding are added together and used as the final feature map of the feature extraction network.

[0060] In practice, the feature maps obtained after processing by three 2x upsampling operations in the feature pyramid network are feature maps of different scales, namely 32*32*64, 16*16*64, and 8*8*64.

[0061] The pyramid attention network, after conditional location encoding, will generate three feature maps of different scales at different stages of extraction. The number of channels in the feature maps will be adjusted by convolution operation with 1*1 kernels, which will be used for subsequent feature extraction by the pyramid network.

[0062] In practice, the extraction at different stages of the pyramid attention network after conditional location encoding will generate preliminary satellite feature maps of 100*100*64, 50*50*128, and 25*25*320, respectively denoted as "satellite features". Figure 1 "Satellite characteristics" Figure 2 "Satellite characteristics" Figure 3 "And preliminary UAV feature maps of 32*32*64, 16*16*128, and 8*8×320, respectively denoted as "UAV features". Figure 1 "Unmanned Aerial Vehicle Characteristics" Figure 2 "Unmanned Aerial Vehicle Characteristics" Figure 3 ".

[0063] Further processing of the drone feature maps involves adjusting the number of channels in the six feature maps using a 1x1 convolution kernel. The number of channels in all feature maps is adjusted to 64. Among these, the "drone feature map"... Figure 3 "After processing with a 1*1 convolution kernel, the "drone features" are obtained." Figure 4 "Satellite features" Figure 3 "Satellite features are obtained after processing with a 1x1 convolution kernel." Figure 4 Then use, for example Figure 2 The feature pyramid network in the middle first analyzes the 8*8*64 "drone features" Figure 4 "An upsampling operation is performed to obtain a 16*16*64 feature map, which is then convolved with the 'drone feature map' after a 1*1 convolution operation." Figure 2 "After fusion, upsampling is performed to obtain 32*32*64 "UAV features" Figure 5 Then, “drone characteristics” Figure 5 "Upsampling is performed to obtain 32*32*64 feature map pairs, which are then convolved with the 'drone features' obtained after a 1*1 convolution operation." Figure 1 "The fusion process yields a 32*32*64 "UAV feature" Figure 6 Ultimately, three feature maps of different scales were obtained: 32*32*64, 16*16*64, and 8*8*64, which are the "drone features". Figure 4 "Unmanned Aerial Vehicle Characteristics" Figure 5"Unmanned Aerial Vehicle Characteristics" Figure 6 Similarly, the features extracted from the satellite image are processed to obtain only the final feature map, with a scale of 100*100*64 for "satellite features". Figure 6 ".

[0064] Step 2.2: The feature maps output by the feature extraction network from the UAV images and satellite images are fused at different scales by the multi-prediction map fusion network to obtain the final prediction map as the final output. The final prediction map is then post-processed to obtain the position of the UAV in the satellite image when the UAV images were acquired.

[0065] First, in the multi-prediction map fusion network, the feature maps with scales of 32*32*64, 16*16*64, and 8*8*64 output from the last three operations of the feature extraction network for the drone image are compared with the final feature map with a scale of 100*100*64 output from the final stage of the feature extraction network for the satellite image to obtain three prediction maps. These three prediction maps are then weighted and summed using their respective learnable parameters, resulting in a weighted fusion to generate a final prediction map with a scale of 101*101*1. Figure 2 The "multi-prediction graph fusion network" region is shown in the figure.

[0066] Then, nearest neighbor interpolation is used to reconstruct the final predicted image to a size of 400*400*1, consistent with the satellite image, to obtain a reconstructed image. This reconstructed image is then normalized to obtain a heatmap. Mapping the position of the pixel with the highest pixel value in the heatmap to the satellite image yields the location of the drone in the satellite image at the time of image acquisition. Figure 2 As shown in the rightmost figure.

[0067] Figure 2 The images labeled A, B, and C represent three different feature maps.

[0068] The final feature map output by the feature extraction network after satellite imagery is 100*100*64 pixels.

[0069] The last three stages of the drone image processing through the feature extraction network refer to the three double upsampling operations of the feature pyramid network.

[0070] Specifically, the similarity calculation in step 3 is performed using a grouped convolution method.

[0071] Step 3: Based on the location of the drone in the satellite image, establish the location region of the drone in the satellite image. Input the pre-obtained drone image, satellite image, and location region of the drone in the satellite image into the neural network of Step 2 for training.

[0072] In step 3, during training, Hanning loss is used to calculate the loss of the neural network. The losses of the prediction maps at different scales in the output of the neural network are combined for forward propagation, thereby introducing multiple loss calculation. At the same time, backpropagation is performed to update the network parameters, thus realizing the training of the neural network.

[0073] Specifically, Hanning weights were used to calculate the loss of the three predicted images and the final predicted image, resulting in four losses. These four losses were then combined to supervise the target of multiple losses and perform backpropagation to update the network parameters of the feature extraction network in order to optimize training.

[0074] The four feature maps for loss calculation are as follows: Figure 2 The numbers A, B, C, and D are given.

[0075] In the specific implementation, a hyperparameter, Center-R, is set to adjust the range of positive samples for loss calculation. The loss calculation is performed according to the following formula:

[0076]

[0077] Where L represents the loss of a single prediction graph, L i Indicates... y i The label parameter represents pixel i, with 1 for positive and 0 for negative; p i W represents the probability that pixel i is predicted to be of the positive class; i This represents the weight of pixel i; i represents the pixel number.

[0078] In practice, a square is created within the positive sample coverage area of ​​the label. The side length of the square is set by the hyperparameter Center-R, which was found to be 33 in experiments. The actual location of the drone in the satellite image is taken as the center of the square, and the side length is set to 33. The area covered by this square is the positive sample, and the remaining area is the negative sample.

[0079] like Figure 3 As shown, the first image is a drone photo, and the second is a satellite image. The rectangle in the satellite image represents the area captured by the drone (the target area). The satellite image is 400*400 pixels in size, therefore the label size is also 400*400 pixels. A label is then assigned to each pixel, as follows: Figure 3 As shown in the third image, a 33*33 rectangle is set with the target location as the center. The pixels within the rectangle are uniformly set as positive samples, and the rest of the area is set as negative samples.

[0080] By assigning different weights to different positive and negative samples, and keeping the sum of the weights of positive samples at 1, the weights of negative samples are also kept at 1, thus balancing the weights of positive and negative samples.

[0081] In practice, negative samples are calculated as: 1 / (number of negative samples) * negative weight (each negative sample has a consistent weight; the sum of the negative sample weights equals the negative weight). Positive samples are weighted using a Hanning weight, with positive samples closer to the center receiving a higher weight, and the sum of all positive sample weights is 1. Finally, all samples are normalized so that the sum of the positive and negative sample weights is 1.

[0082] The weights of positive samples are calculated using a Hanning window. The following formula is the formula for the Hanning window function:

[0083]

[0084] Figure 4 The Hanning window function is used, where the values ​​on both sides of the image eventually tend to 0. In practice, M is set to 35. If the length of the Hanning window is set to 33, equal to the side length of the positive sample, then the weights of the positive sample edges will be set to 0, which is detrimental to network training. Finally, the generated Hanning window is normalized so that the final sum of weights is 1.

[0085] Figure 3 The fourth image shows the generated Hanning window weight map, with the highest weight in the central region, decreasing progressively. In practical use, this weight assignment method ensures that the sum of the weights of all positive samples is 1. Finally, all samples are normalized so that the sum of the weights of positive and negative samples is 1.

[0086] Step 4: In the scenario to be located, the drone image, which collects unknown location information during real-time flight in an unknown area, is input into the trained feature extraction network for prediction processing. The output is the drone's location information at the time the drone image was collected, thus realizing drone visual positioning.

[0087] Step 4 involves first obtaining the drone's position in the satellite image through prediction processing in a neural network, and then performing interpolation calculations using the latitude and longitude information extracted from the upper left and lower right corners of the satellite image to obtain the drone's positioning information in the satellite image. This information serves as the drone's positioning information during image acquisition, thus achieving visual positioning of the drone.

[0088] In summary, the method provided in this application, by introducing a feature pyramid network and inventing a multi-prediction map fusion network, presents a novel approach to UAV visual localization. This method can directly output the UAV's latitude and longitude information by locating the UAV image in a satellite image. It considers both spatial and multi-scale information, and the introduction of multi-loss calculation further reduces training difficulty. This invention treats the UAV localization task as a combination of target tracking and semantic segmentation, where spatial information in the feature map is crucial for UAV localization. Using a pyramid attention network with conditional position encoding as the backbone, combined with the pyramid structure, the resolution of the feature map can be magnified to the size of the original image, thereby reducing loss from the source. Furthermore, the multi-scale capability of the model is improved through the multi-prediction map fusion network, especially on larger-scale satellite images. Increasing the coverage of positive samples during training can reduce the difficulty of model training, but it also introduces the problem of inaccurate localization. By assigning different weights (Hanning weights) to different positive samples, the accuracy of localization can be maintained while increasing the number of positive samples. Figure 5 The image shows a demonstration of the positioning effect of this invention. The number in the upper left corner represents the distance between the model's predicted position and the actual position, in meters.

Claims

1. A visual localization method for unmanned aerial vehicles (UAVs) based on a multi-prediction graph fusion network, characterized in that, The method includes the following steps: Step 1: Use a drone to take aerial photos of the known search area from a vertical perspective. At the same time, capture satellite images of the search area and the positioning information in the satellite images. The drone's position is marked in the satellite images. Step 2: Input the drone images and satellite images of the search area into the neural network for processing to obtain the drone's position in the satellite images; Step 2 specifically involves: Step 2.1: Input the UAV images and satellite images of the search area into their respective feature extraction networks, and obtain the final feature map through two-stage feature extraction processing in the feature extraction network; Step 2.1 specifically involves: inputting the drone images and satellite images into their respective feature extraction networks to obtain their final feature maps. Each feature extraction network includes a conditionally located pyramid attention network and a feature pyramid network. First, preliminary feature extraction is performed through the conditionally located pyramid attention network to obtain a preliminary feature map. Then, the feature pyramid network is used to further extract features from the preliminary feature map to obtain the final feature map. The scale of the generated final feature map is four times larger than that of the original preliminary feature map. Furthermore, the conditionally located pyramid attention network and the feature pyramid network in the feature extraction networks for drone images and satellite images do not share weights. Step 2.2: The feature maps output by the feature extraction network from the UAV images and satellite images are fused at different scales by the multi-prediction map fusion network to obtain the final prediction map. The final prediction map is then post-processed to obtain the position of the UAV in the satellite image when the UAV images were acquired. Step 2.2 specifically includes: First, in the multi-prediction map fusion network, the feature maps output by the last three operations of the feature extraction network of the UAV image are compared with the final feature map output by the final stage of the feature extraction network of the satellite image to obtain three prediction maps. The three prediction maps are then weighted and added together using their respective learnable parameters to generate a final prediction map. Then, the nearest neighbor interpolation method is used to restore the final predicted map to the same size as the satellite image to obtain the restored map. The restored map is then normalized to obtain the heat map. The position of the pixel with the largest pixel value in the heat map is mapped to the satellite image to obtain the position of the drone in the satellite image when the drone image was collected. Step 3: Input the pre-obtained drone images, satellite images, and the location region of the drone in the satellite images into the neural network of Step 2 for training; Step 4: In the scenario to be located, the drone image with unknown location information collected during the drone's real-time flight is input into the trained feature extraction network for prediction processing, and the output is the drone's location information at the time of image collection, thus realizing drone visual positioning.

2. The UAV visual localization method based on a multi-prediction graph fusion network as described in claim 1, characterized in that, Step 1 specifically involves setting the gimbal to vertically downwards, allowing the drone to use its built-in camera to take pictures of the ground in the search area from a height of over 80m.

3. The UAV visual localization method based on a multi-prediction graph fusion network as described in claim 1, characterized in that, The conditionally located pyramid attention network includes a sequential 4x downsampling and two consecutive 2x downsampling operations. The feature pyramid network includes two consecutive 2x upsampling operations. The input image is processed by the conditionally located pyramid attention network to obtain a preliminary feature map. The preliminary feature map is then processed by a 1x1 convolution operation and input into the feature pyramid network, and then processed by two consecutive 2x upsampling operations to obtain the final feature map. in The feature map obtained after the first 2x upsampling operation in the feature pyramid network and the feature map obtained after the first 2x downsampling operation in the pyramid attention network with conditional position encoding are added together and then input into the second 2x upsampling operation of the feature pyramid network. The feature map obtained after the second 2x upsampling operation in the feature pyramid network and the feature map obtained after the fourth downsampling operation in the pyramid attention network with conditional position encoding are added together and used as the final feature map of the feature extraction network.

4. The UAV visual localization method based on a multi-prediction graph fusion network as described in claim 1, characterized in that, In step 3, during training, the Hanning loss is used to calculate the loss of the neural network. The losses of the prediction maps at different scales output by the neural network are combined for forward propagation, while backpropagation is also performed to update the network parameters, thereby training the neural network.

5. The UAV visual localization method based on a multi-prediction graph fusion network as described in claim 4, characterized in that, In step 3, Hanning weights are used to calculate the loss of the three prediction maps and the final prediction map to obtain four losses. The four losses are then combined and backpropagated to update the network parameters of the feature extraction network.

6. The UAV visual localization method based on a multi-prediction graph fusion network as described in claim 1, characterized in that, In step 4, the position of the UAV in the satellite image is first obtained through prediction processing in the neural network. Then, the positioning information of the UAV in the satellite image is combined with the positioning information in the satellite image to perform interpolation calculation to obtain the positioning information of the UAV in the satellite image. This information is used as the positioning information of the UAV when the UAV image is acquired, thus realizing the UAV's visual positioning.

7. The UAV visual localization method based on a multi-prediction graph fusion network as described in claim 5, characterized in that, The loss calculation is specifically performed according to the following formula: ； Where L represents loss, y i The label parameter represents pixel i, with 1 for positive and 0 for negative; p i W represents the probability that pixel i is predicted to be of the positive class; i This represents the weight of pixel i; i represents the pixel index. Furthermore, a square region is constructed centered on the drone's position in the satellite image, with the added hyperparameter Center-R being of variable length, to represent the drone's position region in the satellite image. This position region is treated as positive samples, while the remaining regions are treated as negative samples. During loss calculation, the weights W of all pixels in the positive samples are used... i The sum is set to 1, and the weights W of all pixels in the negative sample are set to 1. i The sum of these remains 1.

Images