Rapid positioning method for aerial images based on semantic segmentation

Abstract

The application provides a kind of based on semantic segmentation aerial image fast positioning method, including the following steps: semantic segmentation stage: the object of feature significant such as building and road in aerial image data set is manually labeled, then select semantic segmentation model to adjust and train to convergence, complete the semantic segmentation of unmanned aerial vehicle aerial image;Fast search stage: satellite remote sensing image is labeled according to the same method and generates remote sensing label map, then with minimum normalized total error as the basis, the segmented aerial image is searched on remote sensing label map, and the approximate area of aerial image on remote sensing image is determined;Precise matching stage: feature point matching algorithm based on feature point is used to match the aerial image with the remote sensing image in the determined area, and precise positioning is realized.The application greatly simplifies the complexity of matching process, improves the positioning efficiency, and is also suitable for aerial images containing rotation changes.

Description

Technical Field

[0001] This invention relates to the fields of image matching, deep learning, and computer vision, and specifically to a method for rapid localization of aerial images based on semantic segmentation. Background Technology

[0002] With the continuous development of technology, drones are playing an important role in many application fields. Drone aerial image positioning technology is a technique that uses aerial images to match satellite remote sensing images with location information in order to achieve aerial image positioning. This technology does not rely on satellite positioning signals and can obtain the drone's position information solely based on visual images, making it of great research and practical value.

[0003] Because remote sensing images cover a very wide area, have complex scenes, and contain a large amount of feature information, their resolution is much greater than that of drone aerial images. If drone aerial images are directly used to match remote sensing images, a large number of mismatches and redundant calculations will occur, resulting in low matching accuracy and efficiency.

[0004] To address the aforementioned matching challenges, researchers have conducted extensive studies. Zheng Mingguo, in his doctoral dissertation "Research on Image Matching Algorithms for UAV Visual-Assisted Localization," proposed a coarse localization method based on global feature fusion, followed by adaptive thresholding and point-by-point pairing using the principal directionality of local features, and finally, precise localization using the ORSA algorithm, although its real-time performance was poor. Luo Rui, in his dissertation "Research on Aerial Image Localization Based on Image Matching," proposed a coarse matching algorithm based on regional features, using normalized coefficient correlation as a similarity metric function. This algorithm performs coarse localization on preprocessed remote sensing tile images, improving matching efficiency, but it is not suitable for aerial images with rotational transformations. Currently, supported by massive amounts of data and powerful computing capabilities, deep learning algorithms, with their excellent nonlinear learning capabilities, have become a commonly used method for image matching. Liao Mingzhe, in his dissertation "Research on Remote Sensing Image Matching Methods Based on Deep Learning," used RF-Net to generate image feature descriptors before matching, transforming image matching into a classification problem, which makes it difficult to accurately determine the location information of aerial images.

[0005] Studies have shown that the above methods have different problems and limitations in aerial image matching and localization. Aerial images and remote sensing images differ greatly in quality due to various factors such as shooting time, angle, and cloud / fog interference. However, regardless of changes in the external environment, roads, buildings, bridges, and water flows in the images always possess relatively stable features. Therefore, a deep learning-based semantic segmentation model can be constructed to first simplify the image into a five-value image containing only roads, buildings, bridges, water flows, and the background, before performing subsequent matching.

[0006] In summary, to address the issues of slow matching speed and inaccurate positioning in UAV aerial image matching and localization, there is an urgent need for a rapid aerial image localization method based on semantic segmentation to achieve fast and accurate determination of the location information of aerial images. Summary of the Invention

[0007] The purpose of this invention is to provide a method for rapid localization of aerial images based on semantic segmentation. This method first simplifies the image into a five-value image containing only five types of pixel values: roads, buildings, bridges, water flow, and background, using a semantic segmentation algorithm for rapid searching, and then uses a feature point matching algorithm for accurate localization.

[0008] The technical solution to achieve the purpose of this invention is as follows:

[0009] A fast localization method for aerial images based on semantic segmentation includes the following steps:

[0010] Semantic segmentation stage: Objects with significant features, such as buildings and roads, in the aerial image dataset are manually labeled, and then a semantic segmentation model is selected, adjusted, and trained until convergence to complete the semantic segmentation of the drone aerial images;

[0011] Rapid search phase: The satellite remote sensing images are labeled using the same method to generate a remote sensing label map. Then, based on the minimum normalized total error, the segmented aerial images are quickly searched on the remote sensing label map to determine the approximate area of ​​the aerial images on the remote sensing images.

[0012] Precise matching stage: A feature-point-based image matching algorithm is used to match the aerial image with the remote sensing image of the defined area to achieve precise positioning.

[0013] The significant advantages of this invention compared to existing technologies are:

[0014] This paper innovatively applies a deep learning-based semantic segmentation algorithm to the matching of aerial images and satellite remote sensing images, which solves the problem of slow matching of large images and improves the efficiency of large image matching. The semantic segmentation algorithm simplifies the representation of aerial images, reducing complex RGB images to grayscale images containing only five grayscale values, thus reducing the complexity of subsequent calculations. By using the mean error and the error in the number of grayscale values, the location of the aerial image on the satellite remote sensing image can be quickly found. Attached Figure Description

[0015] Figure 1 The flowchart shows a fast localization method for aerial images based on semantic segmentation.

[0016] Figure 2 Aerial images taken by drones;

[0017] Figure 3 Labels for the completed drone aerial images;

[0018] Figure 4 Satellite image of a certain region;

[0019] Figure 5 The satellite image after annotation;

[0020] Figure 6 The selected region map for the fast search phase;

[0021] Figure 7 To achieve an accurate matching graph for the SIFT algorithm. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0023] This embodiment of a method for fast localization of aerial images based on semantic segmentation includes the following steps:

[0024] Step 1, Semantic Segmentation Stage: Objects with significant features, such as buildings and roads, in the aerial image dataset are manually labeled. Then, a semantic segmentation model is selected, adjusted, and trained until convergence, completing the semantic segmentation of the drone aerial images. This specifically includes the following steps:

[0025] 1.1 Collect aerial images or video frames taken by the drone as aerial image data. The image size is standardized to 608*608 pixels through cropping or scaling. Use LabelMe software to label four types of distinctive objects in the images—roads, buildings, bridges, and water flows—and their backgrounds. Labels are denoted as [_background_, building, road, river, bridge], representing background, buildings, roads, rivers, and bridges respectively. After labeling, each image generates a JSON file with the same name. Then, run the labelme2voc script to batch convert the labeled files to VOC format. Drone aerial images are shown below. Figure 2 As shown, the tags for the converted drone aerial images are as follows: Figure 3 As shown, different grayscale values ​​represent different objects.

[0026] 1.2. The Baidu PaddlePaddl deep learning framework is used for training and inference. Baidu PaddlePaddl provides a complete image segmentation development kit. Simply select the appropriate image segmentation model and adjust its configuration file to achieve the entire image segmentation application process from training to prediction and inference. The code can be found at https: / / github.com / PaddlePaddle / PaddleSeg / tree / release / 2.5. This invention selects the Deeplabv3p deep neural network as the semantic segmentation model to achieve pixel-level classification of aerial images. Its backbone network structure is ResNet50. To ensure the semantic segmentation model matches the dataset format annotated in step 1.1, the corresponding parameters in the configuration file are adjusted: num_class is changed to 5, bach_size to 4, iters to 60000, crop_size to [608, 608], learning_rate to 0.01, loss to CrossEntryLoss, and optimize_type to sgd. Other parameters remain unchanged by default.

[0027] 1.3. Using the adjusted semantic segmentation model Deeplabv3p from step 1.2, train the dataset from step 1.1. Train for 100 epochs until the model loss is essentially constant, then stop training and save the generated model weight file. The semantic segmentation model can then be used to segment the input aerial image by loading the trained model weight file, obtaining the segmented image, such as... Figure 3 As shown.

[0028] Step 2, Rapid Search Stage: The satellite remote sensing images are labeled using the same method to generate a remote sensing label map. Then, based on the minimum normalized total error, the segmented aerial images are rapidly searched on the remote sensing label map to determine the approximate area of ​​the aerial images within the remote sensing image. The specific steps are as follows:

[0029] 2.1 Label the satellite remote sensing images using the same labeling method as in step 1.1 to generate a remote sensing label map. The satellite remote sensing image used in this invention is a Level 19 satellite image of a certain region, such as… Figure 4 As shown, with a resolution of 6000*3600, the generated remote sensing label map is as follows. Figure 5 As shown.

[0030] The segmented aerial images and remote sensing label maps are first converted into grayscale images to reduce computational complexity. At this point, the grayscale values ​​are [0, 14, 38, 75, 113], where grayscale value 0 represents the background, grayscale value 14 represents buildings, grayscale value 38 represents water flow, grayscale value 75 represents bridges, and grayscale value 113 represents roads. The converted grayscale aerial images are then slid across the converted grayscale remote sensing map in the form of a window. Each time the map is slid, the mean grayscale value and the number of each grayscale value for the corresponding area are calculated.

[0031] 2.2 Calculate the absolute value of the mean error between the grayscale aerial image and the grayscale remote sensing label image, expressed as follows:

[0032] Mean_error = |Mean (UAV) -Mean (Temp) |

[0033] Where UVA represents aerial imagery, Temp represents remote sensing labeled imagery, and Mean represents mean.

[0034] 2.3 Calculate the sum of the absolute errors in the number of grayscale values ​​between the grayscale aerial image and the grayscale remote sensing label image:

[0035]

[0036] Where i represents each grayscale value, UVA i Temp represents the number of pixels with grayscale value i in a grayscale aerial image. i This represents the number of pixels with a grayscale value of i in a grayscale remote sensing label image.

[0037] 2.4 The total error between grayscale aerial images and grayscale remote sensing labeled images is the sum of the normalized mean error and the normalized absolute error:

[0038] Total_error=norm(Mean_error)+norm(Num_error)

[0039] Here, norm() represents the normalization operation.

[0040] 2.5 First, determine whether the grayscale remote sensing label image contains all grayscale values ​​of the grayscale aerial image. If not, skip that region. If it does contain them, select the region with the smallest total error between the grayscale aerial image and the grayscale remote sensing label image as the region to be precisely matched in the fast search phase. Figure 6 As shown, the white box represents the area selected during the fast search phase.

[0041] Step 3, Precise Matching Stage: Using a feature-point-based image matching algorithm, the aerial image is matched with the remote sensing image of the defined area to achieve precise positioning.

[0042] 3.1. Crop the area to be precisely matched as determined in step 2.5 from the original satellite remote sensing image. To ensure that the aerial image is completely contained within the cropped matching area, the cropping area can be appropriately enlarged.

[0043] 3.2. Using the Scale Invariant Feature Transform (SIFT) matching algorithm based on feature points, key feature points are calculated in both the aerial image and the remote sensing image of the defined region. Similar key feature points in the two images are then matched. After successful matching, perspective transformation is used to determine the specific location of the aerial image within the remote sensing image. For example... Figure 7 As shown, the connecting line represents a pair of key feature points successfully matched by the SIFT algorithm, and the quadrilateral box represents the area where the aerial image is finally accurately matched on the remote sensing image.

[0044] 3.3. Based on the correspondence between pixel coordinates and latitude and longitude in the remote sensing image, the precise location information of the aerial image is determined.

Claims

1. A semantic segmentation-based aerial image fast positioning method, characterized in that, The method comprises the following steps: The semantic segmentation stage: manually label the objects with prominent building and road features in the aerial image dataset, then select and adjust a semantic segmentation model and train it to convergence to complete the semantic segmentation of the unmanned aerial vehicle aerial image; The fast search stage: label the satellite remote sensing image in the same way and generate a remote sensing label map, then perform fast search of the segmented aerial image on the remote sensing label map based on the minimum normalized total error to determine the area of the aerial image on the remote sensing image; the specific steps are as follows: 2.1, the segmented aerial image and the remote sensing label map are first converted into gray images to reduce the calculation complexity, then the converted gray aerial image is slid in the form of a window on the converted gray remote sensing, and the gray mean value and the number of each gray value of the corresponding area of the gray remote sensing label map are calculated each time the window is slid; 2.2, the absolute value of the mean error of the gray aerial image and the gray remote sensing label image is calculated; 2.3, the absolute error sum of the number of each gray value of the gray aerial image and the gray remote sensing label image is calculated; 2.4, the total error between the gray aerial image and the gray remote sensing label image is the sum of the normalized mean error and the normalized absolute error sum; 2.5, based on the minimum total error between the gray aerial image and the gray remote sensing label image, the area with the minimum total error is selected as the area to be precisely matched determined in the fast search stage; The precise matching stage: using the feature point based image matching algorithm to match the feature points of the aerial image and the remote sensing image in the determined area to realize precise positioning.

2. The method of claim 1, wherein, The semantic segmentation stage, the specific steps are as follows: 1.1, select the image or video frame taken by the unmanned aerial vehicle as the aerial image data, use the labelme software to label the feature objects and their backgrounds in the image, after labeling, each image gets a json file with the same name, then run the labelme2voc script file to convert the labeled file into voc format; 1.2, select the deep neural network Deeplabv3p as the semantic segmentation model to realize the pixel-level classification of the aerial image, the main network structure of which is Resnet50, and adjust the class name and number, learning rate parameters to match the format of the data set labeled in step 1.1; 1.3, use the semantic segmentation model in step 1.2 to train the data set in step 1.1, save the corresponding weight file after training to convergence, and the semantic segmentation model can segment the input aerial image by loading the trained weight to get the segmented image.

3. The method of claim 1, wherein, The absolute value of the mean error of the gray aerial image and the gray remote sensing label image is calculated: Mean_error = |Mean (UAV) - Mean (Temp) | Wherein, UVA represents the aerial image, Temp represents the remote sensing label image, and Mean represents the mean value.

4. The method of claim 1, wherein, The absolute error sum of the number of each gray value of the gray aerial image and the gray remote sensing label image is calculated: where i represents each gray value, UVA i represents the number of pixels with gray value i in the aerial image, Temp i represents the number of pixels with gray value i in the remote sensing label image.

5. The method of claim 1, wherein, The total error between the gray aerial image and the gray remote sensing label image is the sum of the normalized mean error and the normalized absolute error sum: Total_error = norm(Mean_error) + norm(Num_error) Wherein, norm() represents the normalization operation, Mean_error represents the absolute value of the mean error of the gray aerial image and the gray remote sensing label image, and Num_error represents the absolute error sum of the number of each gray value of the gray aerial image and the gray remote sensing label image.

6. The method of claim 1, wherein, The specific steps of the accurate matching stage are as follows: 3.1, cutting out the region to be accurately matched in the original satellite remote sensing image; 3.2, using the scale-invariant feature transform matching algorithm based on feature points to accurately match the feature points of the aerial image and the remote sensing image in the determined region, and determining the specific position of the aerial image in the remote sensing image; 3.3, according to the corresponding relationship between the pixel point coordinates and the latitude and longitude of the remote sensing image, the accurate position information of the aerial image is located.

Images