Map construction method based on images shot by unmanned aerial vehicle

Abstract

The embodiment of the invention provides a map construction method based on images shot by an unmanned aerial vehicle, and the method is characterized in that the method comprises the following steps: 1, obtaining all to-be-spliced images shot by the unmanned aerial vehicle; 2, preprocessing the image shot by the unmanned aerial vehicle, and dividing the image into a plurality of strips; step 3, splicing the images in the strips; and step 4, splicing the spliced strips again to obtain a map. In order to solve the problems that map splicing has high requirements on a memory and accumulative errors exist when multi-map splicing is used for calculating a transformation matrix, the method for automatically dividing strips through image geographic information to realize batch processing is adopted, so that the requirements on the memory are reduced. And a partition splicing algorithm fusing local transformation information and global transformation information is provided based on SIFT and RANSAC algorithms, so that the problem that accumulated errors exist when a transformation matrix is calculated by multi-graph splicing is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a map construction method based on images taken by an unmanned aerial vehicle. Background Art

[0002] UAV imaging detection has been widely used in many fields, including smart cities, emergency rescue, lake supervision, etc. Due to the limitation of the field of view of the detection system carried by UAV, it is difficult to obtain wide-area images. Therefore, the stitching of UAV images has received great attention.

[0003] However, due to the high resolution of drone images, the computer memory requirements are also high, and the stitching effect is difficult to completely surpass the traditional method. In general, the research on the stitching of drone images using deep learning methods is not rich enough and is still in its infancy.

[0004] Considering that the map generation system needs to process a large amount of image data, the memory requirement is high, and there is a problem of cumulative error in the calculation of the transformation matrix when stitching multiple images, this paper proposes a method to automatically divide the image into strips using the geographic information of the image to achieve batch processing in order to reduce the memory requirement, and proposes a partition stitching algorithm based on SIFT and RANSAC algorithms that fuses local transformation information and global transformation information. The proposed method first divides the image into multiple strips according to the geographic information it carries, and performs stitching within each strip; then, each strip after stitching is partitioned, and adjacent strips are spliced ​​according to the corresponding partitions; finally, the partition transformation is used to fuse the global transformation to obtain the final full-image stitching result. Summary of the invention

[0005] The present invention provides a method for constructing a map based on images taken by an unmanned aerial vehicle, characterized in that it comprises the following steps:

[0006] Step 1, obtain all the drone images to be stitched;

[0007] Step 2, preprocessing the image taken by the drone and dividing it into a number of strips;

[0008] Step 3, stitching the images in the strip;

[0009] Step 4, stitch the stitched strips again to obtain a map.

[0010] The steps of step 2 are:

[0011] Step 201, parsing the EXIF ​​information in the image taken by the drone to obtain the corresponding longitude and latitude coordinates, drone gimbal attitude angle information, camera focal length and relative height;

[0012] Step 202, based on the UAV geometric imaging model, inversely calculate the spatial resolution corresponding to each pixel in the image taken by the UAV;

[0013] Step 203, based on the yaw angle information during the drone image acquisition, the drone image is rotated and the geographic coordinates are transformed, and the drone image is uniformly transformed into a Mercator projection coordinate system;

[0014] Step 204 : dividing the image captured by the drone into a plurality of strips according to the adjacency relationship of the trajectory points and the directional consistency relationship of the connection lines of the trajectory points.

[0015] The steps of step 3 are:

[0016] Step 301, determining the overlapping area between the single strip stitching image pairs according to the longitude and latitude coordinates obtained in step 2;

[0017] Step 302, using SIFT to extract and match features of the overlapping areas;

[0018] Step 303, using the RANSAC algorithm to eliminate mismatching points and obtain the final matching pair;

[0019] Step 304, calculating a transformation matrix between two images based on the matching pair;

[0020] Step 305, processing the overlapping area by weighted fusion method to complete the image stitching of a single strip;

[0021] Step 306, repeating steps 301-305 to obtain the image stitching results of all strips.

[0022] The transformation matrix calculation method is:

[0023] If a single strip needs to stitch 2k images, the kth image is used as the reference image, denoted by H ij represents the transformation matrix from the i-th image to the j-th image, then the transformation matrix from the i-th image to the reference image can be recorded as H ik , H ik It can be calculated according to the following formula

[0024]

[0025] The specific calculation method of step 305 is:

[0026] The point P(x, y) in the stitched image is the point in the overlapping area of ​​the two original images P1 and P2 before stitching. Its projection coordinates in the original image are P1(x1, y1) and P2(x2, y2), respectively. The distance from P1(x1, y1) to the center point in the original image P1 is d1, and the distance from P2(x1, y1) to the center point in the original image P2 is d2. The weight calculation formula is:

[0027] σ2=1-σ1;

[0028] The pixel value calculation formula of the spliced ​​image point P(x,y) is:

[0029]

[0030] The specific steps of step 4 are:

[0031] Step 401, rough matching between strips: firstly, roughly align the two strips A and B according to their longitude and latitude coordinates;

[0032] Step 402, based on step 401, the two strips A and B are divided into a number of image regions according to the flight direction of the drone during acquisition, the length of each region is three times the width of the strip image, the overlap between two adjacent regions is 50%, and a single matching relationship between each sub-region of strip A and each sub-region of strip B is established;

[0033] Step 403, feature point extraction: using SIFT algorithm to extract feature points from each region;

[0034] Step 404, local homography matrix estimation: According to the pairing relationship between sub-regions A and B, the corresponding sub-regions are matched with feature points, and the homography matrix between each pair of sub-regions is calculated respectively. The homography matrix of the i-th pair of regions is denoted as H i

[0035] Step 405, global homography matrix estimation: Use the feature points of all sub-regions to perform feature point matching, and obtain the global homography matrix between strips A and B based on the matching relationship, denoted as H all ;

[0036] Step 406, homography matrix fusion: the global homography parameters and the homography parameters of each sub-region are fused to obtain a modified local homography matrix.

[0037] Step 407, image generation, using the fused homography matrix, determine the corresponding relationship between the pixels on the image to be spliced, and calculate and assign the grayscale value of the final spliced ​​pixel based on step 305.

[0038] The present invention aims at the problem that map splicing has high memory requirements and there are cumulative errors in the calculation of transformation matrices for multi-image splicing. The method of automatically dividing image geographic information into strips to realize batch processing is used to reduce the memory requirements. Based on SIFT and RANSAC algorithms, a partition splicing algorithm that fuses local transformation information and global transformation information is proposed to avoid the problem of cumulative errors in the calculation of transformation matrices for multi-image splicing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the pre-processing strip;

[0040] Figure 2 This is a schematic diagram of image stitching in Example 1.

[0041] Figure 3 Schematic diagram of image stitching in Example 2, DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0043] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0044] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0045] It should also be understood that, although the present application has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present application that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0046] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0047] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to determine the true intent based on the user's historical operations and to avoid unnecessary or redundant details that make the present application unclear. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but are merely used as the basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0048] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0049] The embodiment of the present invention provides a method for constructing a map based on images taken by a drone.

[0050] Step 1: Input all images to be stitched;

[0051] Step 2: Perform image preprocessing;

[0052] Step 3: stitch the images in the strip;

[0053] Step 4: Re-stitch the stitched strips and generate the final map.

[0054] like Figure 1 As shown, the steps of image preprocessing in step 2 are:

[0055] Step 201 parses the EXIF ​​information in the drone image to obtain the corresponding longitude and latitude coordinates, drone gimbal attitude angle information, camera focal length and relative height.

[0056] Step 202: The UAV geometric imaging model is used to inversely calculate the spatial resolution corresponding to each pixel in the image;

[0057] Step 203 performs a rotation operation and a geographic coordinate transformation on the image according to the yaw angle information when the drone image is collected, and uniformly transforms the image into a Mercator projection coordinate system;

[0058] Step 204 divides the track points into multiple strips according to the adjacency relationship of the track points and the directional consistency relationship of the lines connecting the track points.

[0059] The steps for intra-strip splicing in step 3 are:

[0060] Step 301 determines the overlapping area between the stitched image pairs according to the pre-processed longitude and latitude information;

[0061] Step 302 uses SIFT to extract and match features of the overlapping areas;

[0062] Step 303 uses the RANSAC algorithm to eliminate mismatching points and obtain the final matching pair;

[0063] Step 304 calculates the transformation matrix between the two images based on the matching pairs;

[0064] Assuming that a strip needs to be stitched with 2k images, the kth image is taken as the reference image, denoted by H ijrepresents the transformation matrix from the i-th image to the j-th image, then the transformation matrix from the i-th image to the reference image can be recorded as H ik , H ik It can be calculated according to the following formula

[0065]

[0066] Step 305 processes the overlapping area by a weighted fusion method to obtain a splicing result of a single strip.

[0067] Suppose point P(x, y) in the stitched image is the point in the overlapping area of ​​the two original images P1 and P2 before stitching, and its projection coordinates in the original image are P1(x1, y1) and P2(x2, y2) respectively, where the distance from P1(x1, y1) to the center point in the original image P1 is d1, and the distance from P2(x1, y1) to the center point in the original image P2 is d2. The weight calculation formula is shown in formula (2):

[0068]

[0069] The pixel value of the stitched image point P(x, y) is calculated by formula (3):

[0070]

[0071] The steps for splicing between strips in step 4 are:

[0072] Step 401: Coarse matching between strips: First, the two strips A and B are roughly aligned according to their longitude and latitude information;

[0073] Step 402: Sub-region division: Based on the rough alignment, the two strips A and B are divided into several image regions according to the flight direction of the drone during acquisition. The length of each region is three times the width of the strip image, and the overlap between two adjacent regions is 50%. At the same time, a single matching relationship between each sub-region of strip A and each sub-region of strip B is established;

[0074] Step 403: feature point extraction: extract feature points from each region using SIFT algorithm;

[0075] Step 404: local homography matrix estimation: according to the pairing relationship between sub-regions A and B, feature matching is performed on the corresponding sub-regions, and the homography matrix between each pair of sub-regions is calculated respectively. The homography matrix of the i-th pair of regions is denoted as H i ;

[0076] Step 405: Global homography matrix estimation: Use the feature points of all sub-regions to perform feature point matching, and obtain the global homography matrix between strips A and B based on the matching relationship, denoted as H all

[0077] Step 406: homography matrix fusion: the global homography parameters and the homography parameters of each sub-region are fused to obtain a modified local homography matrix, so as to achieve the purpose of eliminating local inconsistencies and maintaining global consistency.

[0078] Step 407, image generation, using the fused homography matrix, determine the corresponding relationship between the pixels on the image to be spliced, and calculate and assign the grayscale value of the final spliced ​​pixel based on step 305.

[0079] Figure 2 and Figure 3 Two splicing effect diagrams of the embodiments of the present invention are respectively given, indicating that the splicing effect of the present invention is better.

[0080] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A method for constructing a map based on images taken by an unmanned aerial vehicle, characterized in that: The following steps are included: Step 1, obtain all the drone images to be stitched; Step 2, preprocessing the image taken by the drone and dividing it into a number of strips; Step 3, stitching the images in the strip; Step 4, stitch the stitched strips again to obtain a map.

2. A method for constructing a map based on images taken by an unmanned aerial vehicle according to claim 1, characterized in that: The steps of step 2 are: Step 201, parsing the EXIF ​​information in the image taken by the drone to obtain the corresponding longitude and latitude coordinates, drone gimbal attitude angle information, camera focal length and relative height; Step 202, based on the UAV geometric imaging model, inversely calculate the spatial resolution corresponding to each pixel in the image taken by the UAV; Step 203, based on the yaw angle information during the drone image acquisition, the drone image is rotated and the geographic coordinates are transformed, and the drone image is uniformly transformed into a Mercator projection coordinate system; Step 204 : dividing the image captured by the drone into a plurality of strips according to the adjacency relationship of the trajectory points and the directional consistency relationship of the connection lines of the trajectory points.

3. A method for constructing a map based on images taken by an unmanned aerial vehicle according to claim 2, characterized in that: The steps of step 3 are: Step 301, determining the overlapping area between the single strip stitching image pairs according to the longitude and latitude coordinates obtained in step 2; Step 302, using SIFT to extract and match features of the overlapping areas; Step 303, using the RANSAC algorithm to eliminate mismatching points and obtain the final matching pair; Step 304, calculating a transformation matrix between two images based on the matching pair; Step 305, processing the overlapping area by weighted fusion method to complete the image stitching of a single strip; Step 306, repeating steps 301-305 to obtain the image stitching results of all strips.

4. The method for constructing a map based on images taken by an unmanned aerial vehicle according to claim 3, characterized in that: The transformation matrix calculation method is: If a single strip needs to stitch 2k images, the kth image is used as the reference image, denoted by H ij represents the transformation matrix from the i-th image to the j-th image, then the transformation matrix from the i-th image to the reference image can be recorded as H ik , H ik It can be calculated according to the following formula 5. The method for constructing a map based on images taken by an unmanned aerial vehicle according to claim 3, characterized in that: The specific calculation method of step 305 is: The point P(x, y) in the stitched image is the point in the overlapping area of ​​the two original images P1 and P2 before stitching. Its projection coordinates in the original image are P1(x1, y1) and P2(x2, y2), respectively. The distance from P1(x1, y1) to the center point in the original image P1 is d1, and the distance from P2(x1, y1) to the center point in the original image P2 is d2. The weight calculation formula is: The pixel value calculation formula of the spliced ​​image point P(x,y) is:

6. The method for constructing a map based on images taken by an unmanned aerial vehicle according to claim 3, characterized in that: The specific steps of step 4 are: Step 401, rough matching between strips: firstly, roughly align the two strips A and B according to their longitude and latitude coordinates; Step 402: Based on step 401, the two strips A and B are divided into a number of image regions according to the flight direction of the drone during acquisition. The length of each region is three times the width of the strip image, and the overlap between two adjacent regions is 50%. At the same time, a single matching relationship between each sub-region of strip A and each sub-region of strip B is established; Step 403, feature point extraction: using SIFT algorithm to extract feature points from each region; Step 404, local homography matrix estimation: According to the pairing relationship between sub-regions A and B, the corresponding sub-regions are matched with feature points, and the homography matrix between each pair of sub-regions is calculated respectively. The homography matrix of the i-th pair of regions is denoted as H i Step 405, global homography matrix estimation: Use the feature points of all sub-regions to perform feature point matching, and obtain the global homography matrix between strips A and B based on the matching relationship, denoted as H all ; Step 406, homography matrix fusion: fusing the global homography parameters with the homography parameters of each sub-region to obtain a modified local homography matrix; Step 407, image generation, using the fused homography matrix, determine the corresponding relationship between the pixels on the image to be spliced, and calculate and assign the grayscale value of the final spliced ​​pixel based on step 305.

Images