A three-dimensional surveying and mapping system mounted on a lightweight unmanned aerial vehicle

Abstract

A three-dimensional surveying system carried on a lightweight unmanned aerial vehicle, comprising the following steps: 1. image preprocessing; 2. image splicing using SIFT to obtain a large target area image; 3. local positioning and landing point selection based on the spliced image; and 4. depth recovery of the descending area using a seed growing method based on a homography matrix to determine whether it is suitable for landing. Through the above steps, the unmanned aerial vehicle can have a wide range of surveying, autonomous local navigation positioning and certain three-dimensional reconstruction capabilities only by relying on a monocular camera. The system has the advantages of strong adaptability, good robustness, wide application range and diverse uses. It can be used for surveying and reconstruction in a specified area, and can also be carried on a small unmanned aerial vehicle to enable it to land safely and autonomously in an emergency, thereby preventing accidents and protecting the lives and property of the public and users.

Description

(I) TECHNICAL FIELD

[0001] The present application relates to a three-dimensional mapping system mounted on a lightweight unmanned aerial vehicle, belonging to the field of computer vision and digital image processing. It has broad application prospects in the fields of geological exploration, regional planning and unmanned aerial vehicle software. (II) BACKGROUND

[0002] The SIFT feature point description was first published by David Lowe in the International Conference on Computer Vision in 1999, and then improved by Lowe in IJCV in 2004. As of August 2014, the number of citations of this paper alone has reached more than 25,000 times. Due to its superior stability to rotation, scale and brightness changes, view changes, and high speed and expansibility, it is widely used.

[0003] Image stitching technology emerged at the beginning of feature description. Its purpose is to stitch multiple overlapping images of the same scene into a larger image, which is of great significance in medical imaging, computer vision, military target automatic identification and other fields. It usually includes the following steps: feature extraction, image registration, homography matrix calculation and image deformation combination.

[0004] Three-dimensional reconstruction technology is one of the important directions of computer vision development, and is widely used in many fields.

[0005] Based on image stitching and three-dimensional reconstruction technology, this project attempts to design a mapping system mounted on a lightweight unmanned aerial vehicle. Based on image stitching, three-dimensional reconstruction is performed, so that the system has good functionality. (III)SUMMARY

[0006] (1) The purpose of the present application

[0007] With the continuous development of science and technology, the market share of unmanned aerial vehicle is also increasing year by year. With the continuous expansion of its scale, the unmanned aerial vehicle industry has begun to extend from the military field to the civil field, and many lightweight unmanned aerial vehicles equipped with only monocular cameras have been born. So far, the unmanned aerial vehicle used in regional exploration and other work generally has the characteristics of large size, high price, and carrying valuable and precise equipment, and can only be used in professional remote sensing companies and related scientific research institutions, which cannot meet the needs of the public. Therefore, the present application designs a surveying and mapping system that can be carried on a lightweight unmanned aerial vehicle, hoping to realize the surveying and mapping of a large area and local positioning by only using a monocular camera through image stitching and the depth recovery coefficient of the descending sequence, and can automatically select a landing site and judge the terrain features at the time of landing to complete a safe landing. In addition, due to the characteristics of the system, it can be carried on all models of unmanned aerial vehicles on the market, and can rely on the camera to automatically scan and select a landing site to complete a safe landing when the unmanned aerial vehicle loses the remote control signal and the network, thereby reducing property loss.

[0008] (2) Technical scheme

[0009] The three-dimensional surveying and mapping system carried on the lightweight unmanned aerial vehicle of the present application has the following specific method steps:

[0010] Step one: image preprocessing; for the subsequent steps, first, the incoming image is processed by using high-pass filtering technology to sharpen its edges, providing a basis for the subsequent feature point matching;

[0011] High-pass filtering can sharpen the edges and effectively improve the number and quality of feature points of the image;

[0012] Step two: use SIFT feature points to stitch the input image, combine image fusion and transformation matrix normalization technology to correct image distortion and improve its accuracy, so as to obtain a panoramic image in a certain area.

[0013] In step two, the method for "combining image fusion and transformation matrix normalization technology to correct image distortion" is as follows: after obtaining the stitching result of two images, due to the difference in brightness of the images, there will be obvious seams at the stitching part, which will change the SIFT descriptor of the feature points at the edge, and the objectivity and availability of the image will be greatly reduced. Therefore, after obtaining the stitching of two images, we use image fusion technology to count the RGB values of the overlapping area, and then perform normalized fusion on the area, thereby effectively improving the image quality at the seam.

[0014] Wherein, in step two, the image distortion is corrected by image fusion and transformation matrix normalization, etc. The method is as follows: due to the continuity of image stitching, each image is transformed based on the perspective of the previous image: Thus, the perspective of the first image determines the perspective of the entire stitched image. If the first image has an angle skew, it will greatly affect the stitching quality of the entire image. In this embodiment, the transformation matrix of each image is normalized, and after obtaining all the transformation matrices, each transformation matrix is corrected by using the mean value, and then the image stitching is performed based on the corrected transformation matrix, thereby improving the perspective dependence of the image on a single image and improving the robustness of the system

[0015] Step three: after obtaining the panoramic image, the panoramic image is used as the basis for subsequent navigation, and the current camera image of the UAV and the large image are used to realize the positioning of the UAV in the local area.

[0016] Wherein, in step three, the current camera image of the UAV and the large image are used to realize the positioning of the UAV in the local area. The method is as follows:

[0017] First, the incoming image is processed to sharpen the edges and increase the matching characteristics of the feature points. Then, the feature matching is used to count the feature point distribution of the current camera image and the large image. According to the number of feature points in each image, different methods such as quadrilateral fitting and circular fitting are selected for fitting. Then, the centroid coordinates are calculated and used to replace the current coordinates of the UAV to realize the local positioning of the UAV in the large scene.

[0018] Step four: in the large image, the UAV selects a landing site, then hovers above the landing site to perform depth recovery based on the homography matrix, and performs curved surface fitting to construct the depth information of the region, and then judges whether the terrain is suitable for the UAV landing, and then completes the entire mapping process.

[0019] Wherein, in step four, after hovering above the landing site, the image depth recovery based on the descending sequence is performed. The method is as follows: first, after judging that the UAV is correctly hovering above the target, the first image is taken. Then the UAV starts to descend, and when it descends to a certain height, the second image is taken. The three-dimensional reconstruction of the landing area is performed by using the front and rear two images and the height information of the descent, and the depth information constructed is combined to judge whether the region is suitable for landing

[0020] In the step four, the "hovering above the landing site using seed growing method based on homography matrix for depth recovery", the homography matrix calculation process is as follows: dividing the camera and the bottom surface into several parallel planes (assuming the number is M), then the plane equation is

[0021] nx+d k = 0 (k = 1, 2,..., M)

[0022] Where dk is the distance of the kth plane to the coordinate origin, in the case of assuming plane dk is known, then the homography matrix induced by the kth plane exists between the two images:

[0023]

[0024] Then the homography matrix recovery between the two images is performed for each point, and the correlation is calculated in the gray window, and the extreme value is taken to determine the maximum possible plane of the point, which is taken as the test value, and finally the corresponding relationship of the two images is obtained:

[0025] x'2(k) = H b H k (x'1-T)(k = 0...m)

[0026] 5. In the step four, the "hovering above the landing site using seed growing method based on homography matrix for depth recovery", the seed growing method process is as follows: selecting the extreme point, that is, the reliable point near the unreliable point, as the growth point of the seed, then judging whether the adjacent pixels meet the following constraint conditions

[0027] (1) Uniqueness: the pixels that have been judged should not be judged again, and the seed points with high E scores should be grown first to ensure that the depth has high reliability.

[0028] (2) Difference: when the unimodality of the correlation curve is small, the seed point cannot be included, so the standard deviation of the curve should be greater than the threshold.

[0029] (3) Correlation: if the highest zncc score is low, it is likely that the depth information is wrong. Therefore, the highest correlation score should also be greater than the threshold.

[0030] (4) Neighborhood difference: the terrain targeted by the algorithm has continuity, and the depth of the seed point and the neighborhood pixels should be limited within a certain range.

[0031] If the final condition is met, it will be added to the seed growing queue as a new seed point, and finally the seed growing is completed, and the extreme points that cannot accurately estimate the depth are eliminated

[0032] Through the above steps, the unmanned plane can complete area mapping, local positioning and autonomous landing and other functions by relying on only monocular camera, and has good stability.

[0033] (3) Compared with the prior art, the advantages of the present application are:

[0034] Unlike conventional remote sensing mapping using unmanned planes, the system can be carried on any unmanned plane, without relying on precision instruments such as depth cameras and laser radars to complete basic tasks. Since the instrument requirement is very low, the weight of the unmanned plane can be greatly reduced, so that it can be widely used in civil fields such as geological exploration and field exploration, and provide terrain guidance and mapping for users without GPS.

[0035] The present application deals with the landing scheme of the unmanned plane in the case of losing connection, through image stitching technology, so that the unmanned plane can automatically select the best landing site within a certain range, and estimate the safety of the landing site, realize autonomous and safe landing, thereby greatly reducing the danger of the unmanned plane in the case of losing connection and the possible property loss caused by unexpected situations (Four) Description of Drawings

[0036] Figure 1 The present application is a general flowchart

[0037] Figure 2 The present application is a flowchart of the stitching method.

[0038] Figure 3 The present application is a three-dimensional reconstruction flowchart.

[0039] x, y, x', y' are pixel point coordinates before and after transformation

[0040] R 00 , R 01 , R 10 , R 11 , R 20 , R 21 are homographic matrix transformation parameters

[0041] T x , T y are homographic matrix rotation parameters

[0042] H is a homographic matrix

[0043] is the average homographic matrix

[0044] d k is the distance from the kth plane to the coordinate origin

[0045] I2 kH is the homography of I2 to I1 via the kth plane

[0046] z is the plane height

[0047] E u,v is the estimate of the seed best match degree at (u,v)

[0048] η is the correction factor

[0049] is the local extremum of the curve associated with pixel (u,v)

[0050] δ s is the standard deviation curve

[0051] W is the half width of the best peak (V) DETAILED DESCRIPTION

[0052] In order to better understand the technical solutions of the present application, the embodiments of the present application are further described below in combination with the accompanying drawings.

[0053] The present application is generally divided into image stitching and three-dimensional reconstruction parts, and the task flow is as shown in Figure 1

[0054] Step one: the unmanned aerial vehicle cruises in the designated area:

[0055] After receiving the instruction, the unmanned aerial vehicle performs S-shaped cruising according to the specified step length, and simultaneously transmits the images into stitching

[0056] Step two: obtain the regional bird's-eye view through the image stitching technology:

[0057] After the cruising is completed, the regional bird's-eye view is obtained by using the following image stitching technology according to the images taken by the cruising

[0058] Step three: realize local positioning by using the large image and the current camera screen:

[0059] After the large image is obtained, the matching of the current camera screen and the large image and the polygon fitting are performed, so that the coordinates of the unmanned aerial vehicle in the large image are obtained, and the local positioning is realized

[0060] Step four: the unmanned aerial vehicle moves to the landing area:

[0061] The large stitched image is processed to obtain the coordinates of the alternative landing site, and the unmanned aerial vehicle starts to move towards it

[0062] Step five: the unmanned aerial vehicle descends and performs three-dimensional reconstruction on the landing area:

[0063] ​The UAV starts to descend over the landing site, takes two photos before and after the descent, starts to reconstruct the current area in three dimensions, and outputs the depth recovery information of the current terrain

[0064] Step six: complete landing

[0065] When the depth information meets the determination condition of flat terrain, the system determines that the area can be used for safe landing, and the UAV lands.

[0066] The image stitching flowchart of the application is shown in Figure 1 The image stitching part of the three-dimensional surveying and mapping system carried by the lightweight UAV has the following specific implementation steps:

[0067] Step one: image preprocessing

[0068] First, high-pass filtering is used to improve image matching performance. The image is convolved with a high-pass filter template to achieve the purpose of sharpening the edge

[0069] Step two: SIFT feature point picking

[0070] Through four steps of scale space extreme value detection, feature point positioning, key point direction assignment and feature generation

[0071] Step three: feature point matching

[0072] Using BFF-matcher for brute force matching, the best feature point pair is obtained

[0073] Step four: homography matrix calculation

[0074] Using the obtained feature point pair, substitute into the coordinate transformation equation:

[0075] And use RANSAC algorithm to correct it, and calculate the best homographic transformation matrix

[0076] Step five: homographic matrix normalization

[0077] After calculating the homographic matrix of each image, the average value of the homographic matrix is calculated

[0078]

[0079] Then the relative homographic matrix is calculated using the following formula

[0080]

[0081] Finally, multiply all images by the corresponding homographic matrix and merge to get the normalized image

[0082] Step six: image fusion

[0083] After each splicing, the overlapping area pixel coordinates are counted, pixel values are fused in a manner of RGB value weighting of the point in the former and latter two images, so that a fusion image is obtained, and thus the splicing is completed

[0084] The three-dimensional reconstruction part of the application has a flow chart as shown in the figure Figure 3 The three-dimensional reconstruction part of the application has a flow chart as shown in the figure

[0085] Step one: restore depth through homography matrix

[0086] For descending sequence images I1 and I2, the motion between the two images is (R, T), the camera internal parameter is K and is constant. A series of parallel planes (assuming the number of planes is M) are used to divide the terrain as shown in the figure, and the optical axis is perpendicular to the ground due to the vertical descent of the lander. The direction of the plane (normal line) is known as n, and the plane equation is

[0087] nx+d k = 0 (k = 1, 2,..., M)

[0088] Wherein n is the plane direction, d k is the distance from the kth plane to the coordinate origin (the camera coordinate system at the position of image I1), and d k is also known in the case of assuming the plane. Under the assumption condition, there is a homography matrix induced by the kth plane between image I1 and I2:

[0089]

[0090] Assume is the homographic inverse transformation of I2 to the image at the same position as I1 via the kth plane, a certain point A (x, y, z) in space is on the kth plane, a1 is the image point of A in image I1, and a2 is the image point of A in image I1. Since A is on the plane k, only the image point pair (a1, a2) determined by the homographic transformation of the plane passing through A is the actual matching point pair, which should be most similar on the image, and the extremum should be obtained by calculating the correlation through the gray window. Therefore, for the transformed image induced by the series of planes, the most relevant plane corresponding to the most relevant image in the local window of a1 determines the depth of the space point A corresponding to a1. By processing each point in the common field of view of I1 and I2 in the same way, the depth information of the space point corresponding to each pixel point in the common field of view can be restored.

[0091] In the actual application process, n = (0, 0, 1) TAt this time d represents the height value of the spatial point in the reference coordinate system. Each height layer corresponds to a homography matrix with only one variable d, so the homography matrix and the height are in one-to-one correspondence. Let the upper reference image be I'1, the lower image to be registered be I'2, and m be the number of height layers. The algorithm needs to give the maximum height value z max and z min and the number of height layers m. The height value corresponding to each layer is:

[0092] z k =z min +k*(z max -z min ) / m(k=0...m)

[0093] According to formula (2.2), when the height is zk, the homography matrix between the planar images I1 and I2(k) is

[0094] x2(k)=H k x1(k=0...m)

[0095] The corresponding relationship of the images I1' and I2' can be obtained as:

[0096] x'2(k)=H b H k (x′1-T)(k=0...m)

[0097] Further, the homography matrix restores the depth information

[0098] Step two: seed growing method is used for growing;

[0099] Using the index

[0100]

[0101] Evaluating whether the depth of the first part of the seed points is reliable. The meanings of each letter are as described above.

[0102] According to the E u,v sorting, appropriate seed points are selected, and seed growing is performed according to the principles of uniqueness, difference, correlation, and domain difference

[0103] Step three: surface fitting;

[0104] The points that cannot be restored after the seed growing method are repaired by repeatedly using, for example, quadratic surface fitting, plane fitting, and b-spline fitting, and finally the target image is obtained.

Claims

1. A three-dimensional mapping algorithm for a lightweight unmanned aerial vehicle, comprising: The specific method steps are as follows: Step one: image preprocessing; first, the incoming image is processed using high-pass filtering technology to sharpen its edges, providing a basis for subsequent feature point matching; Step two: use SIFT feature points to splice the input image, combine image fusion and transformation matrix normalization to correct image distortion and improve accuracy to obtain a panoramic image in a certain area; Step three: after obtaining the panoramic image, it is used as the basis for subsequent navigation, relying on the current camera image of the unmanned aerial vehicle and the large image to realize the positioning of the unmanned aerial vehicle in the local area; Step four: in the large image, the unmanned aerial vehicle will select the landing site, and then guide the unmanned aerial vehicle to move according to the navigation information obtained in step three, so that the unmanned aerial vehicle hovers above the landing site; Step five: after hovering above the landing site, the unmanned aerial vehicle will perform image depth recovery based on the descending sequence, and perform surface fitting to construct the depth information of the site, and then judge whether the site is suitable for the unmanned aerial vehicle to land, and then complete the entire mapping process.

2. The three-dimensional mapping algorithm for a lightweight unmanned aerial vehicle according to claim 1, wherein: In step two, the method for "combining image fusion and transformation matrix normalization to correct image distortion" is as follows: after splicing two images, image fusion technology is used to count the RGB values of the overlapping area, and then the area is normalized and fused, effectively improving the image quality at the seam.

3. The three-dimensional mapping algorithm for a lightweight unmanned aerial vehicle according to claim 1, wherein: In step two, the method for "combining image fusion and transformation matrix normalization to correct image distortion" is as follows: using transformation matrix normalization, the transformation matrix of each image is counted, and after obtaining all the transformation matrices, the mean value is used to correct each transformation matrix, and then the image splicing is performed based on this.

4. The three-dimensional mapping algorithm for a lightweight unmanned aerial vehicle according to claim 1, wherein: In step three, the positioning method process is as follows: first, the incoming image is processed to sharpen its edges and increase the matching characteristics of the feature points, and then the feature points in the current camera image and the large image after splicing are counted using feature matching; according to the number of feature points in each image, select quadrilateral fitting or circular fitting for fitting, then calculate the centroid coordinates and replace the current coordinates of the unmanned aerial vehicle to realize local positioning of the unmanned aerial vehicle in the large scene.

5. The three-dimensional mapping algorithm for a lightweight unmanned aerial vehicle according to claim 1, wherein: In step five, the implementation process of "after completing hovering above the landing site, the unmanned aerial vehicle will perform image depth recovery based on the descending sequence" is as follows: first, after judging that the unmanned aerial vehicle is correctly hovering above the target, the first image is taken; then the unmanned aerial vehicle starts to descend, and when it descends to a certain height, the second image is taken, and the three-dimensional reconstruction of the landing area is performed using the front and rear two images and the height information of the descent, and the depth information constructed is combined to judge whether the site is suitable for landing.

6. The three-dimensional mapping algorithm for a lightweight unmanned aerial vehicle according to claim 1, wherein: In step five, the recovery process of "the unmanned aerial vehicle will perform image depth recovery based on the descending sequence" mainly relies on the homography matrix, and the calculation process of the homography matrix is as follows: Divide the camera and the bottom surface into several parallel planes, then the plane equation is nx+d k = 0 (k = 1, 2,..., M) where dk is the distance from the kth plane to the coordinate origin, the number of planes is M, Given dk, the homography induced by the kth plane exists between the two images: wherein H k is the homography matrix of the kth plane, K is the camera intrinsic matrix, R is the rotation matrix during the UAV captures the first photo to the second photo, t is the translation vector, mainly represents the UAV descent height, n is the plane normal vector, and d is the distance from the plane to the camera center. Then the homography between the two images is recovered for each point, and the correlation is calculated in the gray window. The extreme value is taken to determine the maximum possible plane of the point, which is taken as the test value. Finally, the corresponding relationship between the two images is obtained as: x'2(n) = H b H k (x1'(n) - T)(k = 0...M) Where x2'(n) represents the output coordinates of the nth point, H b With H k Let x1'(n) represent the homography matrix between the current plane and the k-th plane, T represent the bias, and x1'(n) represent the input coordinates of the n-th point.

Images