A method for UAV ground target positioning based on heterogeneous image matching

Abstract

The present invention relates to a method for positioning a ground target of an unmanned aerial vehicle based on heterogeneous image matching, comprising the following steps: S1, according to the flight mission of the unmanned aerial vehicle, a high-resolution remote sensing satellite image of its flight area is obtained from the Internet; S2, the unmanned aerial vehicle is equipped with a visible light camera, a carpet search is performed in the flight area, the camera pitch angle is adjusted to keep it orthophotoed to the target, and the orthophoto image of the target is taken; S3, the aerial image is processed; S4, a sliding window search is performed on the satellite base map to obtain a plurality of base map blocks, and the base map block closest to the aerial image is searched in all base map blocks; S5, according to the matching pair obtained after heterogeneous image matching between the aerial image and the closest base map block, the homography matrix M of the two images is calculated by using OpenCV, M maps the center point of the aerial image to the satellite base map, and the center point of the aerial image is obtained, which is also the latitude and longitude of the target center. The method of the present invention has stronger anti-interference ability, is more adaptable to the field environment, and can perform ground target positioning of the unmanned aerial vehicle without relying on GPS.

Description

Technical Field

[0001] The present invention belongs to the field of ground target positioning, and more specifically, relates to a method for positioning ground targets of unmanned aerial vehicles based on heterogeneous image matching. Background Art

[0002] UAVs have unique advantages and flexibility, and are responsible for various application scenarios such as battlefield, rescue reconnaissance and target monitoring. The positioning function of UAVs for ground targets is the prerequisite for achieving these tasks. Existing positioning technology mainly relies on optoelectronic reconnaissance platforms, which need to be equipped with laser rangefinders. Through the optoelectronic reconnaissance platform, the azimuth, distance and other information of the target relative to the optoelectronic platform can be obtained, and then combined with the position and attitude information of the UAV itself, the longitude and latitude of the target are solved through the target positioning equation (such as the three-point positioning method). Since the UAV's own position information is required, when it is in an environment with weak GPS signals or under electromagnetic interference, the UAV's own position cannot be accurately known, and the accuracy of target positioning will be greatly impaired; and the above positioning method requires the UAV to be equipped with a laser rangefinder, which is costly. Therefore, a UAV ground target positioning method with stronger anti-interference ability, better adaptability to field environments, and independent of GPS is needed. Summary of the invention

[0003] In view of the problem that the existing UAV ground target positioning method cannot obtain its own accurate position in the absence of GPS signal, thus affecting the target positioning accuracy, the present invention proposes a UAV ground target positioning method based on heterogeneous image matching. This method does not rely on GPS and can complete ground target positioning without the UAV's own position information.

[0004] To achieve the above purpose, the technical solution of the present invention is as follows:

[0005] A method for locating ground targets of unmanned aerial vehicles based on heterogeneous image matching includes the following steps:

[0006] S1. According to the UAV flight mission, high-resolution remote sensing satellite images of the flight area are obtained from the Internet. Each pixel on the satellite image has accurate latitude and longitude information and is used as the satellite base map for image matching.

[0007] S2. The drone is equipped with a visible light camera and conducts a carpet search in the flight area. If a target is found, the drone is controlled to fly directly above the center of the target, and the camera pitch angle is adjusted to keep it facing the target, and an orthophoto of the target is taken;

[0008] S3, processing aerial images, first scale the aerial image to make its spatial resolution consistent with that of the satellite image, then rotate the aerial image according to the heading angle to keep the image direction true north. At this time, the orientation of all objects in the aerial image is consistent with that of the satellite image. Figure 1To;

[0009] S4, performing a sliding window search on the satellite base map, with the window size set to the size of the processed aerial image and the overlap rate set to 60% or more, obtaining a number of base map blocks, and searching for the base map block closest to the aerial image among all the base map blocks;

[0010] S5. Based on the matching pairs obtained after heterogeneous image matching between the aerial image and the closest base map block, the findHomography module in OpenCV is used to calculate the homography matrix M of the two images. M maps the center point of the aerial image to the satellite base map. Each point of the satellite base map carries longitude and latitude information, so the center point of the aerial image is also the longitude and latitude of the target center.

[0011] Furthermore, in step S1, the remote sensing satellite images have levels 1-18, which are derived from map software (Google Maps, Baidu Maps, Amap and Tencent Maps, etc.), and the highest level 18 satellite base map is selected, with a spatial resolution of more than 0.5 meters.

[0012] Furthermore, the specific search strategy in step S4 is: perform heterogeneous image matching between the aerial image and any base image block, and each matching will obtain a number of matching pairs between the two images, and the base image block with the largest number of matching pairs is the closest base image block.

[0013] Furthermore, the heterogeneous image matching process of the aerial image and any base image block is as follows:

[0014] S41. Use the deep learning model D2-Net to extract features from the aerial images, and use D2-Net to extract features from all base image blocks one by one;

[0015] S42. Use a K-nearest neighbor search algorithm to roughly match the features of the aerial image with any base image block to obtain a number of matching pairs;

[0016] S43. Refine matching pairs using dynamic adaptive constraints;

[0017] S44. Use RANSAC algorithm to further eliminate mismatched pairs;

[0018] S45. Finally, the base image block with the most matching pairs is taken as the closest base image block.

[0019] Furthermore, the specific process of the rough matching in step S42 is: use the K nearest neighbor algorithm to perform rough matching on the feature vectors of the two images, set K=2, and obtain N matching pairs, each of which contains the first matching point dis closest to the Euclidean distance. j and the second closest matching point dis′ j .

[0020] Furthermore, the specific process of purifying the matching pairs in step S43 is: purifying the matching pairs using the dynamic adaptive Euclidean distance constraint condition, and calculating the average of the distance difference between the first matching point and the second matching point in all matching pairs:

[0021]

[0022] For each matching pair to be screened, the purification condition is that the first distance is less than the difference between the second distance and the average distance difference avgdis, and the formula is:

[0023] dis j <dis′ j -avgdis

[0024] dis j Indicates the distance value to the first nearest matching point, dis′ j Indicates the distance value of the second closest matching point. Matching pairs that do not satisfy this formula are deleted, and matching pairs that satisfy the formula are retained.

[0025] Furthermore, the specific process of using the RANSAC algorithm to eliminate mismatched pairs in step S44 is as follows:

[0026] S441. randomly select a number of matching pair samples from the purified matching pairs, and fit a model P with the number of matching pair samples;

[0027] S442. Calculate the error between the remaining matching pairs and the model P. If the error is less than the threshold, it is considered to be an internal point. If the error is greater than the threshold, it is considered to be an external point.

[0028] S443. The above process is called an iteration. After r iterations, the result with the largest number of in-game points is the final result. At this time, all the out-game points calculated are mismatched pairs and can be directly eliminated.

[0029] Furthermore, the number of matching pair samples in step S441 is less than 10 pairs.

[0030] The UAV ground target positioning method based on heterogeneous image matching of the present invention has the following advantages:

[0031] (1) The present invention can achieve high-precision positioning of ground targets without the help of GPS and has strong anti-electromagnetic interference capability.

[0032] (2) Compared with the existing positioning methods based on laser ranging and attitude measurement, the present invention does not require complex positioning devices or other auxiliary sensors, is low-cost, and only requires a camera to be mounted on the drone to use image information to locate ground targets.

[0033] (3) The present invention combines drones with deep learning technology and can work normally in complex environments such as deserts and deep mountains where manual search is difficult. It is simple to operate and easy to implement, and can effectively replace large-scale ground searches.

[0034] In short, the method of the present invention has stronger anti-interference ability, is more adaptable to the field environment, and can locate the ground target of the UAV without relying on GPS. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The principle flow chart of the method for locating ground targets of unmanned aerial vehicles based on heterogeneous image matching of the present invention;

[0036] Figure 2 The principle diagram for matching aerial images with heterogeneous images of any base image block;

[0037] Figure 3 The aerial image and base map blocks with the fewest matching pairs;

[0038] Figure 4 The aerial image and base map blocks with the most matching pairs.

[0039] In the picture, 1-aerial photo, 11-aerial photo Figure 1 (It is the same picture as the aerial photo, but in a different direction), 2-satellite base image block, 21-base image block 1, 22-base image block 2, 3-the two endpoints of the line are matched pairs. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and implementation methods. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the implementation methods of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] like Figure 1 As shown, the present invention provides a method for locating ground targets of unmanned aerial vehicles based on heterogeneous image matching, comprising the following steps:

[0042] S1. According to the UAV flight mission, high-resolution remote sensing satellite images of the flight area are obtained from the Internet. Each pixel on the satellite image has accurate latitude and longitude information and is used as the satellite base map for image matching.

[0043] In step S1, the remote sensing map has levels 1-18, which are derived from map software (Google Maps, Baidu Maps, Amap and Tencent Maps, etc.). The highest level 18 satellite base map is selected with a spatial resolution of more than 0.5 meters.

[0044] S2. The drone is equipped with a visible light / infrared camera and conducts a carpet search in a certain area. If a target is found, the drone is controlled to fly directly above the center of the target, and the camera pitch angle is adjusted to keep it facing the target, and an orthophoto of the target is taken.

[0045] S3, processing aerial images, first scale the aerial image to make its spatial resolution consistent with that of the satellite image, then rotate the aerial image according to the heading angle to keep the image direction true north. At this time, the orientation of all objects in the aerial image is consistent with that of the satellite image. Figure 1 To.

[0046] S4. Perform a sliding window search on the satellite base map, set the window size to the size of the processed aerial image, and set the overlap rate to 60% or more to obtain a number of base map blocks. Search for the base map block that is closest to the aerial image among all base map blocks.

[0047] The specific search strategy in step S4 is: perform heterogeneous image matching between the aerial image and any base map block. Each match will obtain several matching pairs between the two images. The base map block with the largest number of matching pairs is the closest base map block. Since the ground range involved in the satellite base map is much larger than the ground range of the aerial image, and the size of the processed aerial image is often much smaller than the satellite image, the direct matching error between the two is large. The sliding window method allows the aerial image to be matched with the base map block of the same ground range each time, which effectively improves the accuracy of the matching.

[0048] like Figure 2 As shown, the heterogeneous image matching process of the aerial image and any base image block in the search strategy is: S41. Use the deep learning model D2-Net to extract features from the aerial image, and use D2-Net to extract features from all base image blocks one by one;

[0049] In detail, the feature extraction method in this patent uses the deep learning model D2-Net. The traditional feature extraction method is to first detect key points (keypoint detection) and then extract descriptors (feature description) for the key points, that is, the detect-then-describe mode. Finally, an image can obtain several key points and their corresponding descriptors, that is, key points and their corresponding n-dimensional feature vectors. The commonly used SIFT and SURF algorithms belong to the category of first detection and then description. Although the features extracted by the traditional method are scale-invariant, their expression ability is limited when facing heterogeneous images with large differences in illumination, season, and band. D2-Net is a learning key point detection and description algorithm. It can extract key points and descriptors end-to-end by training a CNN network. This method of extracting key points and descriptors at the same time is called describe-and-detect. D2-Net first uses CNN to calculate the feature map of the input image, then calculates the descriptor by slicing the feature map, and selects the local maximum value in the feature map as the key point. The key points and descriptors in D2-Net are extracted by CNN, which carry high-level semantic information and can achieve better results on heterogeneous images with large differences. After using D2-Net to extract features from the aerial image and the base image blocks respectively, the key points of the two images are obtained, and each key point is represented by an n-dimensional feature vector.

[0050] S42. Use a K-nearest neighbor search algorithm to roughly match the features of the aerial image with any base image block to obtain a number of matching pairs;

[0051] The specific process of coarse matching is as follows: use the K nearest neighbor search algorithm to roughly match the feature vectors of the two images, set K = 2, and obtain N matching pairs, each of which contains the first matching point dis with the closest Euclidean distance j and the second closest matching point dis′ j .

[0052] Theoretically, any number K>2 can be taken, but K=2 is generally taken because only the two nearest points are used.

[0053] In this implementation, first, n feature vectors (n1, n2...nn) are extracted from the aerial image, and m feature vectors (m1, m2,...mm) are extracted from the base image block, each of which represents a pixel in the original image. For n1, K vectors with the closest Euclidean distance to the n1 vector will be found from the m feature vectors, K = 2, which means that each time only the first matching point closest to the n1 vector and the second matching point next closest need to be found; each feature vector in n1...nn can find a corresponding point in (m1, m2...mm), so there are n pairs of matching pairs; for example, if the first matching point of the n1 vector is m3 and the second matching point is m9, then (n1, m3) is considered to be a matching pair.

[0054] After using K nearest neighbors, each point in (n1...nn) can find a matching pair. These matching pairs are not all good, so dynamic adaptive Euclidean distance constraints are used to purify the matching pairs. Because the constraints used involve the first matching point and the second closest matching point, K=2.

[0055] S43. Use dynamic adaptive constraints to purify matching pairs. The specific process is as follows:

[0056] The matching pairs are purified using the dynamic adaptive Euclidean distance constraint condition, and the average of the distance difference between the first matching point and the second matching point in all matching pairs is calculated, as shown in formula (1):

[0057]

[0058] For each matching pair to be screened, the purification condition is that the first distance is less than the difference between the second distance and the average distance difference avgdis, as shown in formula (2):

[0059] dis j <dis′ j -avgdis (2)

[0060] dis j Indicates the distance value to the first nearest matching point, dis′ j Indicates the distance value of the second closest matching point. For example, the n1 feature vector of the aerial image has its first matching point as the m3 vector of the base image block. The Euclidean distance between n1 and m3 is dis j ; Its second matching point is the m9 vector, then the Euclidean distance between n1 and m9 is dis′ j , the matching pairs that do not satisfy this formula are deleted, and the matching pairs that satisfy the formula are retained; the RANSAC algorithm is used on the purified matching pairs to eliminate the wrong matching pairs, and higher quality matching pairs are obtained, and the longitude and latitude calculation will be more accurate.

[0061] S44. Use RANSAC algorithm to further eliminate mismatched pairs;

[0062] The RANSAC algorithm is a commonly used algorithm in the field of computer vision. Assume that a set of data has "inside points" and "outside points". The distribution of inside points conforms to a certain mathematical model, and the outside points are data that cannot adapt to the model. In this specific implementation, the process of RANSAC algorithm to eliminate mismatched pairs is as follows:

[0063] S441. Randomly extract a number of matching pair samples from the purified matching pairs, and use these matching pair samples to fit a model P, specifically a 3×3 homography matrix;

[0064] S442. Calculate the error between the remaining matching pairs and the model P. If the error is less than the threshold, it is considered to be an internal point. If the error is greater than the threshold, it is considered to be an external point.

[0065] S443. The above process is called an iteration. After r iterations, the result with the largest number of in-game points is the final result. At this time, all the out-game points calculated are mismatched pairs and can be directly eliminated.

[0066] The number of matching pairs of samples in step S441 is less than 10. In this embodiment, 4 matching pairs of samples are selected.

[0067] Generally, after the initial matching of an aerial photo and a satellite base map block, that is, k nearest neighbors, there may be dozens to hundreds of matching pairs, and then some are eliminated by dynamic conditions, and some are eliminated by RANSAC, leaving only a dozen to dozens of matching pairs. The remaining matching pairs are used to calculate the homography matrix. To explain more specifically, suppose that there are 100 matching pairs between an aerial photo and a base map block before using the RANSAC algorithm, then the RANSAC algorithm will randomly select a few matching pairs from these 100 matching pairs to fit a model P, and then calculate the error between the remaining matching pairs and the P model, and separate the in-game points and out-game points. Suppose that after repeating this for n times, 30 pairs of out-game points are calculated, then they are eliminated, and the remaining 70 matching pairs are considered to be high-quality matching pairs, so that the longitude and latitude calculated are more accurate.

[0068] S45. Finally, the base image block with the most matching pairs is selected as the closest base image block;

[0069] like Figure 3 , 4 As shown, the left side of these two pictures is the aerial photography after rotation and scaling Figure 1 , Figure 3 On the right is a base map block 1. Each line in the middle represents a matching pair. This line connects a point in the left image with a point in the right image, indicating that the two points are in the same position in the actual environment. Figure 4On the right is a base image block 2. The more matching pairs there are, the more similar the two images are. The goal is to find the base image block that is most similar to the aerial image from among several base image blocks. An aerial image will be matched with several base image blocks. Figure 1 There are 10 matching pairs with "base map block 1", 20 matching pairs with "base map block 3" (not shown), and 25 matching pairs with "base map block 2". Base map block 2 has the most matching pairs. Figure 1 The two images are most similar to "Bottom Block 2", which is an aerial photo. Figure 1 The closest basemap tile.

[0070] S5. Based on the matching pairs obtained after heterogeneous image matching between the aerial image and the closest base map block, the findHomography module in OpenCV is used to calculate the homography matrix M of the two images. M can map the center point of the aerial image to the satellite base map. Each point of the satellite base map has longitude and latitude information, thereby obtaining the longitude and latitude of the center point of the aerial image, which is also the center of the target.

Claims

1. A method for UAV ground target positioning based on heterogeneous image matching, characterized in that: The steps include: S1. According to the UAV flight mission, high-resolution remote sensing satellite images of the flight area are obtained from the Internet. Each pixel on the satellite image has accurate latitude and longitude information and is used as the satellite base map for image matching. S2. The drone is equipped with a visible light camera and conducts a carpet search in the flight area. If a target is found, the drone is controlled to fly directly above the center of the target, and the camera pitch angle is adjusted to keep it facing the target, and an orthophoto of the target is taken; S3, processing the aerial image, first scaling the aerial image so that its spatial resolution is consistent with that of the satellite image, then rotating the aerial image according to the heading angle so that the image direction remains due north, at which point the orientation of all objects in the aerial image is consistent with that of the satellite image; S4, performing a sliding window search on the satellite base map, with the window size set to the size of the processed aerial image and the overlap rate set to 60% or more, obtaining a number of base map blocks, and searching for the base map block closest to the aerial image among all the base map blocks; S5. Based on the matching pair obtained by heterogeneous image matching between the aerial image and the closest base map block, the findHomography module in OpenCV is used to calculate the homography matrix M of the two images. M maps the center point of the aerial image to the satellite base map. Each point of the satellite base map has longitude and latitude information. The center point of the aerial image is also the longitude and latitude of the target center. The specific search strategy in step S4 is: perform heterogeneous image matching between the aerial image and any base image block, and each matching will obtain a number of matching pairs between the two images, and the base image block with the largest number of matching pairs is the closest base image block; The heterogeneous image matching process between the aerial image and any base image block is as follows: S41. Use the deep learning model D2-Net to extract features from the aerial images, and use D2-Net to extract features from all base image blocks one by one; S42. Use a K-nearest neighbor search algorithm to roughly match the features of the aerial image with any base image block to obtain a number of matching pairs; S43. Refine matching pairs using dynamic adaptive constraints; S44. Use RANSAC algorithm to further eliminate mismatched pairs; S45. Finally, the base image block with the most matching pairs is selected as the closest base image block; The specific process of the rough matching in step S42 is: use the K nearest neighbor algorithm to perform rough matching on the feature vectors of the two images, set K=2, and obtain N matching pairs, each of which contains the first matching point dis closest to the Euclidean distance. j and the second closest matching point dis′ j ; The specific process of purifying the matching pairs in step S43 is: purifying the matching pairs using the dynamic adaptive Euclidean distance constraint condition, and calculating the average of the distance difference between the first matching point and the second matching point in all matching pairs: For each matching pair to be screened, the purification condition is that the first distance is less than the difference between the second distance and the average distance difference avgdis, and the formula is: dis j <dis′ j -avgdis dis j Indicates the distance value to the first nearest matching point, dis′ j Indicates the distance value of the second closest matching point. Matching pairs that do not satisfy this formula are deleted, and matching pairs that satisfy the formula are retained.

2. The method for locating ground targets of unmanned aerial vehicles based on heterogeneous image matching according to claim 1 is characterized in that: In step S1, the remote sensing satellite images have levels 1-18, which are derived from map software, and the highest level 18 satellite base map is selected, with a spatial resolution of more than 0.5 meters.

3. The method for locating ground targets of unmanned aerial vehicles based on heterogeneous image matching according to claim 1 is characterized in that: The specific process of removing mismatched pairs in step S44 is as follows: S441. randomly select a number of matching pair samples from the purified matching pairs, and fit a model P with the number of matching pair samples; S442. Calculate the error between the remaining matching pairs and the model P. If the error is less than the threshold, it is considered to be an internal point. If the error is greater than the threshold, it is considered to be an external point. S443. The above process is called an iteration. After r iterations, the result with the largest number of in-game points is the final result. At this time, all the out-game points calculated are mismatched pairs and can be directly eliminated.

4. The method for locating ground targets of unmanned aerial vehicles based on heterogeneous image matching according to claim 3 is characterized in that: The number of matching pair samples in step S441 is less than 10 pairs.

Images