Unmanned aerial vehicle absolute vision positioning method based on semantic segmentation and template matching

Abstract

The invention provides an unmanned aerial vehicle absolute vision positioning method based on semantic segmentation and template matching. Semantic segmentation is carried out on an unmanned aerial vehicle vertical high-angle image and a satellite remote sensing image, a semantic graph of the unmanned aerial vehicle high-angle image is used as a template image, and a semantic graph of the satellite remote sensing image is used as a reference image to carry out label and distance template matching. The point with the highest similarity score after template matching is the positioning point of the unmanned aerial vehicle. According to the method, a LoveDA data set is used for training a semantic segmentation network, and an unmanned aerial vehicle positioning test set is made by itself. Compared with other semantic segmentation data sets, the selection of the data set label category is more suitable for unmanned aerial vehicle positioning. Compared with other geographic positioning methods based on deep learning, the method provided by the invention has the advantages of modularization, high matching precision and strong generalization ability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image matching method based on semantic segmentation and template matching. Background Art

[0002] With the development of drone technology, the application fields of drones are becoming more and more extensive, from the initial military applications to search and rescue, firefighting, mapping, precision agriculture, aerial photography, express delivery and entertainment applications. Modern drones are equipped with advanced intelligent devices and can autonomously navigate or perform tasks according to high-level instructions from humans. Even if the drone signal is interfered or lost, they can return to the base autonomously. This level of functionality requires strong self-positioning capabilities to effectively guide the flight path of the drone.

[0003] Most of the work in the field of drone positioning uses GNSS combined with inertial navigation systems for position estimation within a sensor fusion framework. While this approach can solve most drone positioning problems in daily life, it also presents many risks. Since GNSS works by measuring the flight time of radio signals, any obstacles or interference between the drone and the transmitting satellite will cause problems. When GNSS cannot be used due to interference or satellite damage, only inertial navigation systems can be used for navigation, and the drift (accumulation of errors) of inertial navigation systems will increase exponentially. The most worrying thing is signal spoofing, especially in military defense and conflict, where such attacks will have catastrophic consequences. Therefore, a visual positioning algorithm is needed that can self-position without relying on external signals.

[0004] Traditional template matching methods and feature matching methods cannot meet the differences in visual representation of UAV images and satellite remote sensing images. With the continuous development and evolution of deep learning algorithms, feature extraction methods based on deep learning have better generalization capabilities. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a UAV positioning method based on semantic segmentation and template matching, which can provide higher matching accuracy and has strong generalization ability of the model.

[0006] The technical solution adopted by the present invention is: a method for absolute visual positioning of a UAV based on semantic segmentation and template matching, comprising the following steps:

[0007] Step 1: Use the LoveDA dataset to perform transfer learning on the DeepLabv3+ semantic segmentation network CITYSCAPES pre-trained model;

[0008] Step 2: Input the satellite remote sensing images and the aerial images taken by drones into the DeepLabv3+ semantic segmentation network to obtain their respective semantic maps;

[0009] Step 3: splice the satellite remote sensing semantic image output by the semantic segmentation network as a reference image, and rotate and scale the drone semantic image as a template image;

[0010] Step 4: Use the template image generated in the previous step and the reference image to perform label and distance template matching, and use the correlation results of the label and distance template matching to locate the final position of the drone.

[0011] Furthermore, in step 1, the LoveDA dataset is divided into a training set and a test set, 90% of which is used for training and 10% of which is used as a validation set. The training iterations are 200 rounds, the training batch size is 3, the learning rate is 0.01, and 50% of random horizontal flips are selected for image enhancement. The parameters of the round with the highest average intersection-over-union ratio on the training set are used as the neural network parameters. The test set is tested using a dataset taken at different times and locations near Nanjing University of Science and Technology. The location of the test data does not overlap with the training set and the validation set.

[0012] Furthermore, in step 2, the satellite remote sensing image and the drone aerial image are input into the DeepLabv3+ semantic segmentation network to obtain their respective semantic maps, which specifically includes the following steps:

[0013] Step 1: Crop the remote sensing satellite image into small blocks with a sliding window of 1024*1024 resolution at a 50% repetition rate, adjust the resolution of 1024*1024 to 512*512 and feed it into the trained Deeplabv3+ semantic segmentation network to generate the remote sensing image semantic map. Finally, upsample the 512*512 resolution semantic map to the original image size.

[0014] Step 2: For the image taken by the drone camera perpendicular to the ground, the resolution is adjusted to 512*512 and sent to the trained Deeplabv3+ semantic segmentation network to generate the drone semantic map.

[0015] Furthermore, in step 3, images with a height of 25% to 75% of the width in each remote sensing image semantic map are spliced ​​in the order in which the sliding window is scanned, to form an image with the same size as the original remote sensing satellite image as a reference image.

[0016] The drone semantic map is rotated according to the onboard compass data to ensure rotation invariance. The reference image is obtained from a remote sensing image, so the east, west, south, and north in the image are known. The drone image is rotated according to the real-time onboard compass data to ensure that the east, west, south, and north in the image are the same as the reference image. The drone semantic map is scaled according to the onboard altimeter data to ensure scale invariance. The scaling factor required to scale the image to the scale of the reference image is proportional to the drone height. The scale of the drone is scaled according to the proportional relationship to ensure scale invariance. The scaled drone semantic map is used as the template image.

[0017] Furthermore, in step 4, create a two-dimensional matrix layer equal to the number of semantic segmentation label categories, and assign each type of label to different layers with the same weight, that is, the label category marked in each layer of the matrix is ​​set to 1 and the other places are defaulted to 0, and the standard error square sum correlation matching is performed for each layer in the reference image and the template image. Finally, the correlation of each layer is superimposed to obtain the comprehensive correlation, and the highest point of the comprehensive correlation is used as the positioning coordinate of the drone.

[0018] Standard error squared and correlation:

[0019] Comprehensive correlation: R(x, y) = R class0 (x,y)+R class1 (x,y)+...+R classn (x, y)

[0020] Visualize the matching results and represent the comprehensive correlation with a heat map. The redder the color, the higher the comprehensive correlation. The place with the highest comprehensive correlation is the location of the drone. The range of the drone image in the remote sensing satellite image is marked with a white box. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the process of the present invention.

[0022] Figure 2 Schematic diagram of sliding window cropping and splicing.

[0023] Figure 3 It is the overlay image of the reference image and the original remote sensing image.

[0024] Figure 4 This is the principle diagram of label same-distance template matching.

[0025] Figure 5 These are some drone-viewing images of the test set.

[0026] Figure 6 This is a visualization diagram of the positioning results of the present invention.

[0027] Figure 7 This is a visualization diagram of the positioning results of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with the accompanying drawings.

[0029] The technical solution adopted by the present invention is: an absolute visual positioning method for unmanned aerial vehicles based on semantic segmentation and template matching.

[0030] Figure 1 This is a schematic diagram of the absolute visual positioning method for drones based on semantic segmentation and template matching. It includes the following steps:

[0031] Step 1, use the LoveDA dataset to perform transfer learning on the DeepLabv3+ semantic segmentation network CITYSCAPES pre-trained model. When using this dataset, we convert the labeled masks to background-0, building-1, road-2, water-3, tree-4, agriculture-5, no data-255. Among them, no data-255 is not involved in training and verification. Only the city dataset in this dataset is used for training and verification.

[0032] The LoveDA dataset is divided into a training set and a test set, 90% of which is used for training and 10% of which is used as a validation set. The training is repeated for 200 rounds, the training batch size is 3, the learning rate is 0.01, and 50% of the image enhancement is random horizontal flipping. The parameters of the round with the highest average intersection-over-union ratio on the training set are used as the neural network parameters.

[0033] Step 2: Input the collected satellite remote sensing images and drone aerial images into the DeepLabv3+ semantic segmentation network to obtain their respective semantic maps.

[0034] Figure 2 Schematic diagram of sliding window cropping and splicing, where 1 represents sliding window cropping with 0% repetition rate, and 2 represents sliding window cropping with 50% repetition rate.

[0035] The remote sensing satellite image is cropped into small blocks with a sliding window of 1024*1024 resolution at a 50% repetition rate, and the resolution of 1024*1024 is adjusted to 512*512 and sent to the trained Deeplabv3+ semantic segmentation network to generate the remote sensing image semantic map. Finally, the 512*512 resolution semantic map is upsampled to the original image size.

[0036] For the images taken by the drone camera perpendicular to the ground, the resolution is adjusted to 512*512 and sent to the semantic segmentation network to generate the drone semantic map.

[0037] Step 3: For the satellite remote sensing semantics output by the semantic segmentation network, 25% to 75% of the height and width of each remote sensing image semantic map are spliced ​​in the order of the sliding window to form an image of the same size as the original remote sensing satellite image as a reference image.

[0038] Figure 3 It is the overlay image of the reference image and the original remote sensing image.

[0039] The drone semantic map is rotated according to the onboard compass data to ensure rotation invariance. The reference image is obtained from a remote sensing image, so the east, west, south, and north in the image are known. The drone image is rotated according to the real-time onboard compass data to ensure that the east, west, south, and north in the image are the same as the reference image. The drone semantic map is scaled according to the onboard altimeter data to ensure scale invariance. The scaling factor required to scale the image to the scale of the reference image is proportional to the drone height. The scale of the drone is scaled according to the proportional relationship to ensure scale invariance. The scaled drone semantic map is used as the template image.

[0040] Step 4: Use the template image generated in the previous step and the reference image to perform label and distance template matching, and use the correlation result of the label and distance template matching to locate the final position of the drone.

[0041] Figure 4 This is the principle diagram of label same-distance template matching.

[0042] Create a two-dimensional matrix layer with the same number of semantic segmentation label categories, and assign each type of label to different layers with the same weight, that is, set the label category marked in each layer of the matrix to 1 and the default value to 0 elsewhere. Match the standard error square sum correlation of each layer in the reference image and the template image. Finally, superimpose the correlation of each layer to obtain the comprehensive correlation, and use the highest point of the comprehensive correlation as the positioning coordinate of the drone.

[0043] The reason for this is that when calculating the standard squared error and correlation of semantic segmentation image labels, the labels represented by numbers will affect each other's Euclidean distance. Figure 4 For example, when a 3-category semantic image is directly matched, the correlation between the template image with all labels as 0 and the area with all labels as 1 in the reference image will be better than the correlation between the area with all labels as 2 in the reference image. This is semantically unreasonable, and the content of the label should not be semantically distinguished by distance. So we need a template matching method for semantic images. When matching labels, as long as the same label is not encountered, the correlation is equally bad, and different Euclidean distances cannot be generated due to different numerical distances of the labels. Therefore, each label needs to be projected into another dimension for matching so that they do not interfere with each other. Figure 4The 0:, 1:, 2: layers in represent template matching for three different labels respectively.

[0044] Standard error squared and correlation:

[0045] Comprehensive correlation: R(x, y) = R class0 (x,y)+R class1 (x,y)+...+R classn (x, y)

[0046] Figure 5 Some drone-viewing images of the test set.

[0047] Visualize the matching results and represent the correlation with a heat map. The redder the color, the higher the correlation. The location with the highest correlation is the location of the drone. Overlay the correlation heat map onto the remote sensing satellite image. The range of the drone image in the remote sensing satellite image is marked with a white box.

[0048] Figure 6 This is a visualization diagram of the positioning results of the present invention.

[0049] Figure 7 This is a visualization diagram of the positioning results of the present invention.

[0050] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. A method for absolute visual positioning of a UAV based on semantic segmentation and template matching, comprising the following steps: Step 1: Use the LoveDA dataset to train on the DeepLabv3+ semantic segmentation network CITYSCAPES pre-trained model; Step 2: Input the satellite remote sensing images and the aerial images taken by drones into the DeepLabv3+ semantic segmentation network to obtain their respective semantic maps; Step 3: splice the satellite remote sensing semantic image output by the semantic segmentation network as a reference image, and rotate and scale the drone semantic image as a template image; Step 4: Use the template image generated in the previous step and the reference image to perform label-distance template matching, and use the correlation results of label-distance template matching to locate the final position of the drone.

2. The method for absolute visual positioning of a UAV based on semantic segmentation and template matching according to claim 1, characterized in that: The specific process of step 1 is as follows: the LoveDA dataset is divided into a training set and a test set, 90% of which is used for training and 10% of which is used as a validation set. The training is repeated for 200 rounds, the training batch size is 3, the learning rate is 0.01, and 50% of random horizontal flips are selected for image enhancement. The parameters of the round with the highest average intersection-over-union ratio on the training set are used as the neural network parameters.

3. The method for absolute visual positioning of a UAV based on semantic segmentation and template matching according to claim 1, characterized in that: The specific process of step 2 is: Step 1: Crop the remote sensing satellite image into small blocks with a sliding window of 1024*1024 resolution and a repetition rate of 50%, adjust the resolution of 1024*1024 to 512*512, and feed it into the semantic segmentation network to generate a remote sensing image semantic map. Step 2: For the image taken by the drone camera perpendicular to the ground, the resolution is adjusted to 512*512 and sent to the semantic segmentation network to generate the drone semantic map.

4. The method for absolute visual positioning of a UAV based on semantic segmentation and template matching according to claim 1, characterized in that: The specific process of step 3 is: Step 1: Take images with a height of 25% to 75% of the width in the remote sensing image semantic map and stitch them together to form a reference image with the same size as the original remote sensing satellite image. Step 2: Rotate the drone semantic map according to the onboard compass data to ensure rotation invariance, and scale the drone semantic map according to the onboard altimeter data to ensure scale invariance. The drone semantic map is rotated and scaled to form a template image.

5. The method for absolute visual positioning of a UAV based on semantic segmentation and template matching according to claim 1, characterized in that: The specific process of step 4 is as follows: create layers equal to the number of semantic segmentation label categories, assign each type of label to different layers with the same weight, and match the standard error square sum correlation for each layer. Finally, superimpose the correlation of each layer to obtain the comprehensive correlation, and use the highest correlation point as the positioning coordinate of the drone. Standard error squared and correlation: Comprehensive correlation: R(x, y) = R class0 (x,y)+R class1 (x,y)+...+R classn (x, y).

Images