A multi-view global map stitching method based on calibration plate

Abstract

The present invention provides a multi-view global map stitching method based on a calibration plate. The method comprises: acquiring images with overlapping areas through three cameras in different directions and performing distortion correction; performing HSV interval conversion on the background of a specific rectangular calibration plate containing three different markers in the image, performing binarization and closing operation to obtain a mask, performing edge detection using the Canny operator to extract the calibration plate contour, and then performing quadrilateral fitting on the contour to obtain the coordinates of the four corner points of the calibration plate; calculating the first-order difference of the coordinates of each corner point based on the sum of the horizontal and vertical coordinates of the corner points, with the smallest sum being the upper left corner point and the largest sum being the lower right corner point, performing perspective transformation; identifying each corner point using the features of the calibration plate, matching the identified corner points accordingly, and dividing and stitching using a seam line to complete the stitching.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle navigation and positioning technology, and in particular to a method for rapidly stitching multi-objective global maps based on a calibration plate. Background Art

[0002] In large spaces and relatively flat ground, images acquired by non-vertical cameras cannot be stitched together or automatically constructed into a global visual map using traditional image stitching methods. Currently, the following technologies are available: An improved visual simultaneous positioning and map creation algorithm proposed by Li Yuehua improves positioning accuracy, but map creation is time-consuming. Wang Q et al. manually selected four reference points to transform the image perspective into an orthographic projection view to complete monocular global visual map construction. This method lacks adaptability and the field of view obtained by a single visual sensor is small. Therefore, an accurate map is an urgent problem to be solved in drone navigation and positioning technology.

[0003] Therefore, the present invention provides a multi-objective global map stitching method based on a calibration plate to address the shortcomings of the existing technology and to solve or alleviate one or more of the above problems. Summary of the Invention

[0004] To address the problems of stitching images acquired in a non-vertical manner and automatically constructing visual maps, the present invention provides a fast multi-view global map stitching method based on a calibration plate. The method is highly effective for stitching non-vertical bird's-eye views and has a stitching speed superior to traditional image stitching algorithms. The method integrates image geographic target positioning technology, homography matrix image transformation, feature point matching and precise registration, and other computational methods.

[0005] The technical solutions of the present invention are as follows:

[0006] A multi-view global map stitching method based on a calibration plate includes the following steps:

[0007] S1: Acquire images with overlapping areas through three cameras in different directions and perform distortion correction;

[0008] S2: Perform HSV interval conversion on the background of a specific rectangular calibration plate containing three different markers in the image, perform binarization and closing operation to obtain a mask, use the Canny operator to perform edge detection on it to extract the calibration plate contour, and then perform quadrilateral fitting on the contour to obtain the coordinates of the four corner points of the calibration plate;

[0009] S3: Based on the sum of the horizontal and vertical coordinates of the corner points, the smallest sum is the upper left corner point, and the largest sum is the lower right corner point. Calculate the first-order difference of the coordinates of each corner point. The smallest difference is the upper right corner point, and the largest difference is the lower left corner point. Perform perspective transformation.

[0010] S4: Use the features of the calibration plate to identify each corner point, match the identified corner points accordingly, and divide and splice them using stitching lines.

[0011] In the above scheme, in order to solve the problem of automatically constructing a visual map by stitching images acquired in a non-vertical manner, first, the image is acquired, distortion correction is performed, and errors are reduced. The calibration plate of the present invention is square, and the canny operator is used for edge detection to extract the calibration plate contour, and then the contour is quadrilateral fitted to obtain the coordinates of the four corner points of the calibration plate. The corner points are identified using the markers in the calibration plate, and then the images are precisely aligned. The stitching is divided and stitched using a stitching line method, and the stitching effect is observed.

[0012] Among them, step S1 specifically completes image distortion correction based on OpenCV software in the Pycharm development environment through Zhang Zhengyou camera calibration method.

[0013] In the above solution, the camera distortion principle is mainly used for distortion correction, which effectively reduces errors and lays a good foundation for the accuracy of subsequent map stitching.

[0014] In step S1, the camera distortion is expressed as follows:

[0015]

[0016] Among them, (x, y) represents the original image coordinates obtained by the camera without distortion correction, (x d ,y d ) represents the coordinates of the image after eliminating radial distortion and tangential distortion, r represents (x d ,y d ) to the origin of the coordinate system, k1, k1, k3 represent radial distortion parameters, and p1, p2 represent tangential distortion parameters.

[0017] Wherein, the step S2 includes the following steps:

[0018] S21: Convert the original image from RGB mode to HSV mode, obtain the HSV value in the mouse monitoring function, expand the HSV value to a certain extent, and perform binarization on the image;

[0019] S22: performing a closing operation on the obtained binary image, first dilating and then corroding, eliminating the holes in the binary image, and obtaining a calibration mask;

[0020] S23: Use the Canny operator to perform edge detection on the calibration plate, use the contour extraction algorithm to obtain the calibration plate contour, and then fit the contour into a quadrilateral, with the vertices of the quadrilateral being the corner points of the calibration plate.

[0021] In the above scheme, the HSV color space is more intuitive and has better effect than the color synthesized by the three color components of the RGB color space. The Canny algorithm can remove noise in the source image and smooth the image using the Gaussian function.

[0022] Wherein, in step S22, the two-dimensional Gaussian function is as follows:

[0023]

[0024] Where σ is the standard deviation of the Gaussian filter function.

[0025] Wherein, the step S3 includes the following steps:

[0026] S31: Calculate the first-order difference of the coordinates of each corner point based on the sum of the horizontal and vertical coordinates of the corner points. The point with the smallest sum is the upper left corner point, and the point with the largest sum is the lower right corner point. The point with the smallest difference is the upper right corner point, and the point with the largest difference is the lower left corner point.

[0027] S32: Based on the extracted corner points and side lengths of the calibration plate, the target point coordinates are calculated to obtain a set of four pairs of matching coordinates, and the image can be adaptively perspective transformed into an orthographic projection view.

[0028] In the above solution, the image is converted into an orthographic projection view through image adaptive projection transformation, which can improve the accuracy of image stitching.

[0029] Wherein, in step S32, the image adaptive perspective transformation formula is as follows:

[0030]

[0031]

[0032]

[0033] According to the transformation of formula (5), we can get:

[0034]

[0035] Among them, let the coordinates of the point O in the original image to be perspective transformed be (x o ,y o ), the coordinates of point D corresponding to point O after perspective transformation are (x d ,y d ), the homography matrix is ​​H o2d .

[0036] Wherein, the step S4 includes the following steps:

[0037] S41: Select the upper left corner as the origin in the orthographic projection view, and use the side length of the calibration plate in the orthographic projection view as the side length to frame the ROI (Region of Interest) of the calibration plate. This can greatly narrow the detection range and reduce interference.

[0038] S42: Three marker shapes are designed in the calibration plate to identify the direction. The ROI is binarized using a grayscale threshold and then the contour of the marker shape is extracted using a contour extraction algorithm. The vertices of the contour are obtained through contour approximation. The shape is determined based on the contour vertices. If the number of vertices is 3, it is a triangle, 4 is a rectangle, and 0 is a circle. The spatial moment of the contour is calculated to obtain the centroid of the shape, which is the center coordinate of the marker.

[0039] S43: Calculate the Euclidean distance between the centroid coordinates of the identified shape and the coordinates of the corner points of the calibration plate. The corner point with the smallest Euclidean distance to the centroid coordinates of a certain shape is named after the shape. After distinguishing three of the corner points according to the shapes in the calibration plate, the remaining corner point is also identified.

[0040] S44: matching the marked corner points accordingly to complete image stitching, and dividing and stitching the images using stitching lines.

[0041] In the above scheme, three markers are set to speed up the recognition of corner points and effectively improve the speed of map splicing. In addition, the use of seam lines to divide the splicing can avoid ghosting in image fusion and improve real-time performance.

[0042] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0043] This method improves the effectiveness of stitching non-vertical bird's-eye view images, has a faster stitching speed than traditional image stitching algorithms, is low-cost, and has small global visual map errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the process of the present invention

[0045] Figure 2 Schematic diagram of the process of obtaining the corner points of the calibration plate

[0046] Figure 3 Schematic diagram of the process of adaptive perspective change of the image

[0047] Figure 4 Schematic diagram of the process of identifying corner points and dividing and splicing DETAILED DESCRIPTION

[0048] The accompanying drawings are for illustrative purposes only;

[0049] In order to better illustrate this example, some parts in the drawings may be omitted, enlarged or reduced, and may not represent the size of the actual product.

[0050] The technical solution of the present invention is further described below with reference to the accompanying drawings and examples.

[0051] Example 1

[0052] like Figure 1 As shown in FIG, a method for rapidly assembling a multi-view global map based on a calibration plate includes the following steps:

[0053] S1: Acquire images with overlapping areas through three cameras in different directions and perform distortion correction;

[0054] S2: Perform HSV interval conversion on the background of a specific rectangular calibration plate containing three different markers in the image, perform binarization and closing operation to obtain a mask, use the Canny operator to perform edge detection on it to extract the calibration plate contour, and then perform quadrilateral fitting on the contour to obtain the coordinates of the four corner points of the calibration plate;

[0055] S3: Based on the sum of the horizontal and vertical coordinates of the corner points, the smallest sum is the upper left corner point, and the largest sum is the lower right corner point. Calculate the first-order difference of the coordinates of each corner point. The smallest difference is the upper right corner point, and the largest difference is the lower left corner point. Perform perspective transformation.

[0056] S4: Use the features of the calibration plate to identify each corner point, match the identified corner points accordingly, and divide and splice them using stitching lines.

[0057] In the specific implementation process, in order to solve the problem of automatically constructing a visual map by stitching images acquired in a non-vertical manner, first, the image is acquired, distortion correction is performed, and errors are reduced. The calibration plate of the present invention is square, and the canny operator is used for edge detection to extract the calibration plate contour, and then the contour is fitted with a quadrilateral to obtain the coordinates of the four corner points of the calibration plate. The corner points are identified using the markers in the calibration plate, and then the images are precisely aligned. The stitching is divided and spliced ​​using a suture line method, and the splicing effect is observed.

[0058] Example 2

[0059] More specifically, based on Example 1, each step is further explained.

[0060] More specifically, step S1 is to complete the image distortion correction based on OpenCV software in the Pycharm development environment through Zhang Zhengyou camera calibration method.

[0061] During the specific implementation process, the camera distortion principle is mainly used for distortion correction, which effectively reduces errors and lays a good foundation for the accuracy of subsequent maps.

[0062] More specifically, in step S1, the expression of camera distortion is as follows

[0063]

[0064] Among them, (x, y) represents the original image coordinates obtained by the camera without distortion correction, (x d ,y d ) represents the coordinates of the image after eliminating radial distortion and tangential distortion, r represents (x d ,y d ) to the origin of the coordinate system, k1, k1, k3 represent radial distortion parameters, and p1, p2 represent tangential distortion parameters.

[0065] More specifically, Figure 2 As shown, step S2 specifically includes the following steps:

[0066] S21: Convert the original image from RGB mode to HSV mode, obtain the HSV value in the mouse monitoring function, expand the HSV value to a certain extent, and perform binarization on the image;

[0067] S22: performing a closing operation on the obtained binary image, first dilating and then corroding, eliminating the holes in the binary image, and obtaining a calibration mask;

[0068] S23: Use the Canny operator to perform edge detection on the calibration plate, use the contour extraction algorithm to obtain the calibration plate contour, and then fit the contour into a quadrilateral, with the vertices of the quadrilateral being the corner points of the calibration plate.

[0069] In the specific implementation process, the HSV color space is more intuitive and effective in expressing colors than the color synthesized by the three color components of the RGB color space. The canny algorithm can remove noise in the source image and smooth the image using the Gaussian function.

[0070] More specifically, in step S23, the canny operator smoothes the image using a Gaussian function. The two-dimensional Gaussian function is as follows:

[0071]

[0072] Where σ is the standard deviation of the Gaussian filter function.

[0073] More specifically, Figure 3 As shown, step S3 specifically includes the following steps:

[0074] S31: Calculate the first-order difference of the coordinates of each corner point based on the sum of the horizontal and vertical coordinates of the corner points. The point with the smallest sum is the upper left corner point, and the point with the largest sum is the lower right corner point. The point with the smallest difference is the upper right corner point, and the point with the largest difference is the lower left corner point.

[0075] S32: Based on the extracted corner points and side lengths of the calibration plate, the target point coordinates are calculated to obtain a set of four pairs of matching coordinates, and the image can be adaptively perspective transformed into an orthographic projection view.

[0076] In a specific implementation process, the image is converted into an orthographic projection view through image adaptive projection transformation, which can improve the accuracy of image stitching.

[0077] More specifically, in step S32, the image adaptive perspective transformation formula is as follows:

[0078]

[0079]

[0080]

[0081] According to the transformation of formula (5), we can get:

[0082]

[0083] Among them, let the coordinates of the point O in the original image to be perspective transformed be (x o ,y o ), the coordinates of point D corresponding to point O after perspective transformation are (x d ,y d ), the homography matrix is ​​H o2d .

[0084] More specifically, Figure 4 As shown, step S4 specifically includes the following steps:

[0085] S41: Select the upper left corner as the origin in the orthographic projection view, and use the side length of the calibration plate in the orthographic projection view as the side length to frame the ROI (Region of Interest) of the calibration plate. This can greatly narrow the detection range and reduce interference.

[0086] S42: Three marker shapes are designed in the calibration plate to identify the direction. The ROI is binarized using a grayscale threshold and then the contour of the marker shape is extracted using a contour extraction algorithm. The vertices of the contour are obtained through contour approximation. The shape is determined based on the contour vertices. If the number of vertices is 3, it is a triangle, 4 is a rectangle, and 0 is a circle. The spatial moment of the contour is calculated to obtain the centroid of the shape, which is the center coordinate of the marker.

[0087] S43: Calculate the Euclidean distance between the centroid coordinates of the identified shape and the coordinates of the corner points of the calibration plate. The corner point with the smallest Euclidean distance to the centroid coordinates of a certain shape is named after the shape. After distinguishing three of the corner points according to the shapes in the calibration plate, the remaining corner point is also identified.

[0088] S44: matching the marked corner points accordingly to complete image stitching, and dividing and stitching the images using stitching lines.

[0089] During the specific implementation process, three markers were set up to speed up the recognition of corner points, effectively improving the speed of map splicing. The use of stitching lines to divide the splicing can avoid ghosting in image fusion and improve real-time performance.

[0090] The above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A multi-view global map stitching method based on a calibration plate, characterized in that: The following steps are involved: S1: Acquire images with overlapping areas through three cameras in different directions and perform distortion correction; S2: Perform HSV interval conversion on the background of a specific rectangular calibration plate containing three different markers in the image, perform binarization and closing operation to obtain a mask, use the Canny operator to perform edge detection on it to extract the calibration plate contour, and then perform quadrilateral fitting on the contour to obtain the coordinates of the four corner points of the calibration plate; S3: Based on the sum of the horizontal and vertical coordinates of the corner points, the smallest sum is the upper left corner point, and the largest sum is the lower right corner point. Calculate the first-order difference of the coordinates of each corner point. The smallest difference is the upper right corner point, and the largest difference is the lower left corner point. Perform perspective transformation. S4: Use the features of the calibration plate to identify each corner point, match the identified corner points accordingly, and divide and splice them using stitching lines. The detailed steps of S4 are as follows: S41: In the orthographic projection view, select the upper left corner as the origin, and use the side length of the calibration plate in the orthographic projection view as the side length to frame the ROI (Region of Interest) of the calibration plate. ROI (Region of Interest) refers to a specific area in the image that needs to be focused on. The purpose of selecting ROI is to reduce the amount of data that needs to be processed and concentrate resources on processing the part of interest. This can greatly narrow the detection range and reduce interference. S42: Three marker shapes are designed in the calibration plate to identify the direction. The ROI is binarized using a grayscale threshold and then the contour of the marker shape is extracted using a contour extraction algorithm. The vertices of the contour are obtained through contour approximation. The shape is determined based on the contour vertices. If the number of vertices is 3, it is a triangle, 4 is a rectangle, and 0 is a circle. The spatial moment of the contour is calculated to obtain the centroid of the shape, which is the center coordinate of the marker. S43: Calculate the Euclidean distance between the centroid coordinates of the identified shape and the coordinates of the corner points of the calibration plate. The corner point with the smallest Euclidean distance to the centroid coordinates of a certain shape is named after the shape. After three of the corner points are distinguished according to the shapes in the calibration plate, the remaining corner point is also identified. S44: matching the marked corner points accordingly to complete image stitching, and dividing and stitching the images using stitching lines.

2. The multi-view global map stitching method based on the calibration plate according to claim 1 is characterized in that: Specifically, S1 completes image distortion correction based on OpenCV software in the Pycharm development environment through Zhang Zhengyou's camera calibration method.

3. The multi-view global map stitching method based on the calibration plate according to claim 2 is characterized in that: In step S1, the camera distortion expression is as follows: Among them, (x, y) represents the original image coordinates obtained by the camera without distortion correction, (x d ,y d ) represents the coordinates of the image after eliminating radial distortion and tangential distortion, r represents (x d ,y d ) to the origin of the coordinate system, k1, k1, k3 represent radial distortion parameters, and p1, p2 represent tangential distortion parameters.

4. The multi-view global map stitching method based on the calibration plate according to claim 1 is characterized in that: The S2 includes the following steps: S21: Convert the original image from RGB mode to HSV mode, obtain the HSV value in the mouse monitoring function, expand the HSV value to a certain extent, and perform binarization on the image; S22: performing a closing operation on the obtained binary image, first dilating and then corroding, eliminating the holes in the binary image, and obtaining a calibration mask; S23: Use the Canny operator to perform edge detection on the calibration plate, use the contour extraction algorithm to obtain the calibration plate contour, and then fit the contour into a quadrilateral, with the vertices of the quadrilateral being the corner points of the calibration plate.

5. The multi-view global map stitching method based on the calibration plate according to claim 4 is characterized in that: In step S23, the canny operator uses a Gaussian function to smooth the image. The two-dimensional Gaussian function is as follows: Among them, σ is the standard deviation of the Gaussian filter function, x is the horizontal coordinate variable, and y is the vertical coordinate variable.

6. The multi-view global map stitching method based on the calibration plate according to claim 1 is characterized in that: The S3 includes the following steps: S31: Calculate the first-order difference of the coordinates of each corner point based on the sum of the horizontal and vertical coordinates of the corner points. The point with the smallest sum is the upper left corner point, and the point with the largest sum is the lower right corner point. The point with the smallest difference is the upper right corner point, and the point with the largest difference is the lower left corner point. S32: Based on the extracted corner points and side lengths of the calibration plate, the target point coordinates are calculated to obtain a set of four pairs of matching coordinates, and the image can be adaptively perspective transformed into an orthographic projection view.

7. The multi-view global map stitching method based on the calibration plate according to claim 6 is characterized in that: In step S32, the image adaptive perspective transformation formula is as follows: According to the transformation of formula (5), we can get: Among them, H o2d Is the homography matrix, perform perspective transformation, (h 11 , h 22 )、(h 12 , h 21 )、(h 13 , h 31 )、(h 23 , h 32 ) are the coordinates of the upper left corner, upper right corner, lower right corner and lower left corner of the calibration plate under adaptive perspective, and the coordinates of the midpoint O of the original image to be perspective transformed are (x o ,y o ), the coordinates of point D corresponding to point O after perspective transformation are (x d ,y d ).

Images