Camera calibration device and calibration method for single-shot image captured by wide-angle fisheye lens

Abstract

The present invention relates to a camera calibration device and a calibration method for a single-shot image of a wide-angle fisheye lens, wherein the camera calibration device for a single-shot image of a wide-angle fisheye lens is a fisheye box, which is an open rectangular parallelepiped with five faces, and each inner surface of the fisheye box is painted with black and white square chessboards with regular n×m rows and columns, and the outer edges of the chessboards are spliced ​​around a circle of black and white chessboard small grids, and a closed black frame is arranged outside the chessboard small grids, and the black frame does not intersect with the chessboard small grids therein; a gap is arranged between two adjacent black frames of the fisheye box, and each gap is respectively located at the edge or edge of two adjacent inner surfaces of the fisheye box; two groups of mutually perpendicular line segments are drawn at the center position of the center surface of the fisheye box to form a crosshair; and circular marks are drawn at the middle position of a row of chessboard grids near the center surface of the chessboards on the left and right inner surfaces of the fisheye box. The fisheye box lens calibration provided by the present invention saves time and effort, has high calculation accuracy, and good robustness.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a camera calibration device and a calibration method for a single-shot image shot by a wide-angle fisheye lens. Background Art

[0002] In the field of computer vision and image processing, fisheye lens calibration is a key technology used to correct image distortion caused by wide-angle or fisheye lenses to ensure image accuracy and reliability. Traditional fisheye lens calibration methods usually require taking multiple images and using calibration targets such as checkerboards for calibration. The main problems of these methods include:

[0003] 1. Time-consuming and labor-intensive: Traditional methods require taking images at different angles multiple times and then analyzing these images, which is a tedious and time-consuming task for users, which limits its practicality in some applications.

[0004] 2. High data set requirements: The calibration process requires a large amount of data, including multiple images from different shooting angles and corresponding calibration targets, which increases the difficulty of data collection and processing.

[0005] 3. Low calculation accuracy: Traditional calibration methods often have certain limitations in calculation accuracy. This may result in the accuracy of the calibration results being insufficient to meet the requirements of high-precision applications. Especially for wide-angle fisheye lenses, due to the large image distortion, the estimation of the calibration parameters may not be accurate enough, thus affecting the accuracy of subsequent image processing and applications.

[0006] 4. Poor robustness: Traditional calibration methods are usually sensitive to image quality and environmental factors. Noise in the image, changes in illumination, or lens quality issues may cause unstable calibration results, which means that the same calibration method may produce inconsistent results in different situations. This reduces the robustness of the calibration method and makes it difficult to obtain reliable results in practical applications.

[0007] In recent years, researchers have proposed a series of improved fisheye lens calibration methods to overcome the limitations of traditional methods. These methods include accurate fisheye lens calibration based on differential geometry and arc curvature, radial distortion removal based on homography matrix, deep learning methods, etc. These methods have made some progress in improving calibration accuracy and efficiency. However, they still require multiple images or complex calculations, and in the case of wide-angle fisheye lenses, there are still certain limitations. Summary of the invention

[0008] One object of the present invention is to provide a camera calibration device for single-shot images taken with a large wide-angle fisheye lens. The camera calibration device for single-shot images taken with a large wide-angle fisheye lens is used to solve the problems of traditional fisheye lens calibration methods being time-consuming and labor-intensive, requiring high data sets, having low calculation accuracy, and having poor robustness. Another object of the present invention is to provide a calibration method for the camera calibration device for single-shot images taken with a large wide-angle fisheye lens.

[0009] The technical solution adopted by the present invention to solve the technical problem is as follows: the camera calibration device for the single-shot image shooting of the wide-angle fisheye lens is a fisheye box, which is an open rectangular parallelepiped with five faces, and each inner surface of the fisheye box is painted with black and white square chessboards with regular n×m rows and columns, and the outer edges of the black and white square chessboards are spliced ​​and surrounded by a circle of black and white chessboard small grids, and a closed black frame is arranged outside the chessboard small grids, and the black frame does not intersect with the chessboard small grids therein; a gap is arranged between two adjacent black frames of the fisheye box, and each gap is respectively located at the edge or edge of two adjacent inner surfaces of the fisheye box; two groups of mutually perpendicular line segments are drawn at the center position of the center surface of the fisheye box to form a crosshair; and circular marks are respectively drawn at the middle position of a row of chessboard grids of the black and white square chessboards on the left and right inner surfaces of the fisheye box close to the center surface.

[0010] The method for performing camera calibration by the camera calibration device for the above-mentioned wide-angle fisheye lens single-shot image is as follows:

[0011] Step 1: Segment the single-shot fisheye box image once to obtain multiple regional segmentation images, each of which contains only a single black and white square chessboard image;

[0012] Step 2: Perform secondary segmentation on multiple regional segmentation images to obtain multiple detailed chessboard images;

[0013] Step 3: Perform corner point detection and sub-pixel precision processing on multiple detailed checkerboard images to generate an accurate set of two-dimensional corner point coordinates;

[0014] Step 4: Sort the corner points according to the precise two-dimensional corner point coordinate set, map them to the world coordinate system, calculate the corner point coverage, and obtain a three-dimensional world coordinate set that corresponds one-to-one to the two-dimensional corner point coordinates;

[0015] Step 5: Based on the precise two-dimensional corner point coordinate set and the precise two-dimensional corner point coordinate set, the fisheye lens calibration is realized by using the six-parameter division model to obtain the high-precision fisheye lens internal parameters.

[0016] The primary segmentation method described in step 1 of the above scheme is an image segmentation method based on convex polygon approximation:

[0017] (1) Preprocess the single-shot fisheye box image, convert it into a grayscale image, and then perform histogram equalization and bilateral filtering;

[0018] (2) Using the Canny operator to perform edge detection on the preprocessed image, the outer contours of the edges are connected to eliminate discontinuous outer edges and ensure that the periphery of the chessboard is completely detected;

[0019] (3) In the connected edge map, find a new contour and judge it based on its area to determine whether it is a valid contour. A valid contour must contain a complete closed chessboard.

[0020] (4) For the effective contour, a convex polygon approximation method is used to create a convex polygon mask;

[0021] (5) A single chessboard area image is cut out from the original image using a convex polygon mask. The segmented chessboard area image contains two parts, namely, the chessboards on the four inner sides of the fisheye box and a small number of mis-segmented center chessboards.

[0022] The secondary segmentation in step 2 of the above scheme adopts an adaptive threshold segmentation method based on image inversion;

[0023] (1) Convert the cut single chessboard area image into a grayscale image and perform adaptive thresholding to generate a binary image;

[0024] (2) Perform contour detection on the binary image;

[0025] (3) Calculate the areas of all detected contours, find the contour with the largest area, and mark it as a valid contour;

[0026] (4) Create a mask based on the effective contour to segment the binary image and obtain a segmented image. At the same time, subtract the original binary image from the segmented image to generate a pixel difference image;

[0027] (5) Setting an image pixel threshold to screen the segmented image and pixel difference image obtained in step (4), marking the images that meet the conditions as valid secondary segmentation images, and discarding the images that do not meet the conditions;

[0028] (6) Perform an inversion operation on the pixels of the secondary segmented effective image, and set some pixels in the original image that do not belong to the maximum contour area to white.

[0029] The corner detection method in step 3 of the above scheme adopts a sub-pixel corner detection method based on feature screening:

[0030] Define two different corner point models: the corner point model when the chessboard is in the ideal position and the corner point model when the chessboard is rotated 45°;

[0031] Use the above two corner point models to calculate the corner point similarity of each pixel, and set f x i is the convolution response of the convolution kernel X and model i for a specific pixel, then the corner similarity of the pixel is defined by taking the maximum value of the combination of the two models:

[0032]

[0033] in, Represents the corner point model in two different situations, that is, the left diagonal is black, the right diagonal is white, and the left diagonal is white, and the right diagonal is black;

[0034] If any of the four convolution kernels has a weak convolution response, the corner similarity of the pixel at that location is low;

[0035] Through the above calculation, a series of pixel points that meet the corner point characteristics are obtained. These pixel points are called corner point candidates, and sub-pixel precision processing is performed, including precision of position and edge direction, to determine the position and direction information of the corner points in the image;

[0036] Assume that C is the ideal corner point, P and Q are two pixels in the local neighborhood of C, and g P , g Q are the image gradient vectors of point P and point Q respectively. Since point P is located in the flat pixel area, the gradient of the flat pixel area g P =0, so Point Q is on the boundary, and the gradient g Q The direction of the vector is vertically downward, and the direction of (QC) is horizontally to the right. The directions of the two vectors are perpendicular to each other, so So for corner point C, g N (N is the pixel point in the neighborhood near the corner point C) and (NC) are always orthogonal. That is:

[0037]

[0038] Among them, C′ is the candidate corner point that meets C≥K after setting the judgment threshold K. is the neighborhood of the corner candidate point C′, and the size of the neighborhood is 11×11;

[0039] Adjacent pixels can be automatically weighted by the gradient magnitude, as shown below:

[0040]

[0041] The edge direction vectors e1, e2 are taken by minimizing the error between their normal and the image gradient:

[0042]

[0043] in, is a set of adjacent pixels that satisfies Its gradient m with corner model i i Alignment, m i =[cos(a i )sin(a i )] T , by filtering pixels, we can obtain an accurate set of two-dimensional corner point coordinates.

[0044] The corner point sorting method in step 4 of the above scheme is a corner point sorting method based on bidirectional expansion of feature points, and generates corresponding world coordinates;

[0045] Step 4.1: Find the sorting feature points to achieve bidirectional expansion sorting. The sorting feature points refer to the corner points that serve as the starting points for sorting, and they need to meet the following characteristics:

[0046] (1) The sorting feature points are located at the edge of the corner point matrix, which is curved rather than straight.

[0047] (2) The sorting feature point is located in the middle of the above edge;

[0048] (3) For the fisheye box, the sorting feature points of mutually symmetric planes are in an axisymmetric relationship;

[0049] Step 4.2, calculate the center point coordinates of the detected corner point matrix. For the right side, the sorting feature points that meet the conditions are located on the left side of the center point. The sorting feature points of the right side are located by the following steps:

[0050] (1) Calculate the coordinates of the center point C1(x1,y1);

[0051] (2) From all detected corner points, select the n corner points closest to the vertical coordinate of C1, where n is at least the number of corner points in a column;

[0052] (3) From these n corner points, select the corner point with the smallest horizontal coordinate as the sorting feature point of the right side face;

[0053] Step 4.3, after locating the sorting feature point, use this point as the sorting starting point to sort the corner points and generate world coordinates that correspond to the corner points one by one. The corner point sorting process is as follows:

[0054] (1) The sorting feature point O on the right side is the sorting starting point, and its world coordinates are set to (0,0,0);

[0055] (2) From all detected corner points, find the three corner points with the smallest Euclidean distance from point O, and then select the corner point with the largest horizontal coordinate as point B (1, 0, 0); next, from the remaining two corner points, select the corner point with the smallest vertical coordinate that is smaller than point O as point C (0, -1, 0), and select the corner point with the largest vertical coordinate that is larger than point O as point A (0, 1, 0);

[0056] (3) Take point C as the new starting point, find the three corner points with the smallest Euclidean distance from point C, then select the corner point with the smallest ordinate that is smaller than point C as point D (0, -2, 0), repeat this process until no corner point that meets the conditions is found, and return to point A;

[0057] (4) Start from point A and sort downwards to find the three corner points with the smallest Euclidean distance from point A. Then, select the corner point with the largest ordinate that is larger than point A as point E(0,2,0). Repeat this process until no corner point that meets the conditions is found. Return to point B, complete the sorting of the first column of the corner point matrix, and generate the corresponding world coordinates.

[0058] (5) Excluding the already sorted corner points, take point B as the new starting point, and follow the same method to find the three corner points with the smallest Euclidean distance from point B. Select the corner point with the largest horizontal coordinate as point G(2,0,0). Then, from the remaining two corner points, select the corner point with the smallest vertical coordinate that is smaller than point B as point H(0,-2,0), and select the corner point with the largest vertical coordinate that is larger than point B as point F(0,2,0).

[0059] (6) Similarly, a new starting point is set to perform bidirectional expansion sorting. After each column of corner points is sorted, the next column is sorted until all corner points are sorted and the corresponding world coordinates are generated.

[0060] Assume point K (h, 0, 0) is the last corner point of the row with the least number of corner points. When executing the corner point sorting algorithm based on bidirectional expansion of feature points, after sorting the column where point K is located, it is necessary to sort the remaining corner points. First, arrange the remaining corner points from small to large according to the ordinate. Then, select the corner point L with the smallest ordinate as the starting point, and find the corner point M (h, l, 0) closest to corner point L in the column where point K is located. Finally, the ordinate of point L (i, j, 0) is inherited from point M, and its world coordinate is calculated by the following formula:

[0061]

[0062] Similarly, sort all the remaining corner points on the right side, and use the same method to sort the corner points on other sides and generate the corresponding world coordinates;

[0063] Since the center plane is a corner point matrix of standard rows and columns, the traditional corner point sorting method is used to sort the center plane and generate the corresponding world coordinates.

[0064] The fisheye lens calibration model used in step 5 of the above scheme is a six-parameter division model. Let (x d ,y d ) is the coordinate of the image distortion point, (x u ,y u ) is the coordinate of the undistorted point of the image, and the six-parameter DM model expression is:

[0065]

[0066] Among them, r d and r u Distortion points (x d ,y d ) and the undistorted point (x u ,y u ) to the Euclidean distance of the distortion center P, λ i is the radial distortion parameter, and the single-parameter DM model is adopted:

[0067]

[0068] Assume that the coordinates of the distortion center P are (x0, y0), then the coordinates of the undistorted point in the image are:

[0069]

[0070] Among them, r d The image distortion point and the distortion center P(x0, y0) satisfy:

[0071]

[0072] Suppose the equation of the line is y=kx+b, where k is the slope and b is the intercept, that is, the equation of the undistorted line is y u =kx u +b, substituting into the above formula we get:

[0073]

[0074] It can be seen from the above formula that under the single-parameter DM model, the ideal straight line is distorted into a circular arc curve. After sorting it out, the following equation is obtained:

[0075]

[0076] Among them, A, B, and C meet the following conditions:

[0077]

[0078] It obtains the parameters A, B, C and the distortion center by obtaining three points from the image, and obtains the following relationship based on A, B, and C:

[0079]

[0080] By solving A, B, C and the distortion center, the distortion parameter λ is obtained and the six-parameter DM model is solved.

[0081] Beneficial effects:

[0082] 1. Aiming at the shortcoming of the traditional fisheye calibration method that requires multiple image captures, the present invention designs a new calibration device, the fisheye box, which contains rich spatial information and can achieve the same effect as the traditional method of multiple captures with a single capture, greatly shortening the calibration time of the fisheye lens. The present invention only needs to process a few images after segmentation, while the traditional method needs to process a large number of images, so it has an advantage in the running time of the overall algorithm. In addition, the present invention also has an advantage in image capture time, because only one image needs to be captured, which greatly shortens the time cost of calibration compared to the traditional method of capturing 11 images.

[0083] 2. The present invention proposes a novel fisheye lens calibration method based on the fisheye box, which includes a sub-pixel corner point detection method based on feature screening. The method can accurately locate the corner point position and improve the accuracy of the corner point coordinates through sub-pixel precision processing, thereby improving the accuracy of subsequent fisheye calibration.

[0084] 3. The present invention proposes a new corner point sorting method called corner point sorting method based on bidirectional expansion of feature points. This method can automatically locate and sort feature points, and sort corner points based on this, accurately generating a world coordinate set that corresponds one-to-one to the corner point set, thereby ensuring the accuracy of fisheye calibration.

[0085] 4. The present invention uses the division model to calibrate the fisheye lens, and on this basis, the division model is calculated to the six-parameter level, which greatly improves the calibration accuracy of the algorithm. Compared with the traditional method, the present invention has achieved a significant improvement in calibration accuracy, and its RMS is reduced by 8.56% compared with the traditional method.

[0086] 5. The present invention proposes a calibration device fisheye box and a corresponding calibration method, aiming to improve the calculation accuracy and robustness. By using the fisheye box, high-precision fisheye lens calibration can be achieved in a single shot image, reducing the possibility of error accumulation. The present invention also adopts a series of improvements in image segmentation, corner point detection, corner point sorting and calibration algorithms to improve the calculation accuracy, algorithm robustness and calibration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 The design diagram of a camera calibration device for single-shot image capture with a wide-angle fisheye lens.

[0088] Figure 2 The present invention is a schematic diagram of an embodiment of a camera calibration device for single-shot image capture by a wide-angle fisheye lens.

[0089] Figure 3 The present invention is a flow chart of a camera calibration method for a single shot image of a wide-angle fisheye lens.

[0090] Figure 4 This is a flow chart of the segmentation method.

[0091] Figure 5 It is a flow chart of the secondary segmentation method.

[0092] Figure 6 There are two types of corner point model diagrams.

[0093] Figure 7 It is a schematic diagram of the corner point position.

[0094] Figure 8 2 is a schematic diagram of an implementation example of sorting candidate feature points on the right side.

[0095] Fig. 9 It is a schematic diagram of the right side implementation example of the corner point sorting method based on bidirectional expansion of feature points.

[0096] Fig.10 It is a schematic diagram of an embodiment of edge corner point missing.

[0097] Fig.11 It is a comparison chart of experimental results of the present invention and the traditional method embodiment.

[0098] Fig.12 It is a comparison diagram of discrete points of reprojection errors between the present invention and the traditional method.

[0099] In the picture: 1 chessboard, 2 small chessboard grids, 3 black border, 4 gap, 5 crosshair, 6 circular mark. DETAILED DESCRIPTION

[0100] The present invention will be further described below in conjunction with the accompanying drawings:

[0101] Embodiment 1:

[0102] Combination Figure 1 , Figure 2 As shown, the camera calibration device for the single-shot image of the wide-angle fisheye lens is a fisheye box, which is used to obtain an image with rich spatial information in a single shot. The specific structure of the fisheye box is as follows:

[0103] (1) The fisheye box is a five-sided rectangular parallelepiped. Compared with the traditional calibration plate, this structure helps the fisheye lens capture a larger calibration area.

[0104] (2) A square black and white checkerboard pattern with regular n×m rows and columns is drawn on each surface of the fisheye box. To facilitate image segmentation, a closed black border 3 is set outside each surface checkerboard pattern and does not intersect with the black and white checkerboard. A gap 4 is set between two adjacent black borders of the fisheye box. In order to enhance the detection of more corner points during the calibration process, a circle of regularly arranged small checkerboard grids 2 surrounds the outer edge of the black and white checkerboard 1 on each surface.

[0105] (3) The black and white checkerboard pattern and border of each face do not intersect with other faces, ensuring that there is enough white area between faces to facilitate subsequent image segmentation.

[0106] (4) Two sets of perpendicular line segments are drawn at the center position of the center plane of the fisheye box to form a crosshair 5. These line segments are used to accurately locate the position of the camera when taking images to prevent the influence of factors such as offset or misalignment.

[0107] (5) Two circular marks 6 are drawn in the middle of the inner edges of the left and right surfaces of the fisheye box to verify the accuracy of the feature points found by the subsequent corner point sorting algorithm.

[0108] The size of the center surface of the calibration box is 500mm×500mm, and it contains a checkerboard pattern with 14×14 rows and columns; the size of the surrounding surfaces of the calibration box is 500mm×300mm, and each surface contains a checkerboard pattern with 14×8 rows and columns; the black frame width of each surface of the calibration box is 6mm, the checkerboard size is 33mm×33mm, and the small grid size of the checkerboard edge is 15mm. The calibration box of this specification is moderate in size, and even if it is shot with a 200° fisheye lens, it can still retain relatively complete feature information, and the edge information of each surface will not be confused due to the close distance.

[0109] Figure 3 The flowchart of a camera calibration method for a single shot image of a wide-angle fisheye lens is shown, comprising the following steps:

[0110] Step 1: Segment the single-shot fisheye box image once to obtain multiple regional segmented images.

[0111] In order to calibrate a single-shot fisheye box image, the image needs to be decomposed into multiple images containing only a single checkerboard. This is beneficial to utilize the diversity of corner information in the fisheye box image to obtain reliable calibration results. Given that the images taken by the fisheye lens have large distortion, resulting in the peripheral sides of the image being curved, the one-shot segmentation method uses an image segmentation method based on convex polygon approximation. The flow chart of the one-shot segmentation method is shown in the figure below. Figure 4 As shown, it includes the following processes:

[0112] (1) Preprocess the single-shot fisheye box image and convert it into a grayscale image for subsequent processing. After that, histogram equalization and bilateral filtering are performed to enhance the image contrast and make the image smooth.

[0113] (2) The Canny operator is used to perform edge detection on the preprocessed image, and the outer contours of the edges are connected to eliminate discontinuous outer edges to ensure that the periphery of the chessboard is completely detected.

[0114] (3) In the connected edge map, find new contours and judge them based on their areas to determine whether they are valid contours. These valid contours must contain complete closed checkerboards.

[0115] (4) For the effective contour, a convex polygon approximation method is used to create a convex polygon mask.

[0116] (5) Use this mask to cut out a single checkerboard area image from the original image.

[0117] Step 2: Perform secondary segmentation on multiple regionalized segmented images to obtain multiple detailed chessboard images.

[0118] The checkerboard area image obtained by one segmentation contains two parts, namely the checkerboards on the surrounding sides and a small number of mis-segmented center checkerboards. The interference factor of the checkerboard pattern in the image will have a great impact on the accuracy of corner point detection. However, one segmentation will approach the detected contour to a convex polygon, resulting in the original non-convex polygon checkerboard pattern cannot be segmented independently. Therefore, step two uses an adaptive threshold segmentation method based on image inversion for secondary segmentation. The flowchart of the secondary segmentation method is as follows: Figure 5 As shown, the method includes the following process:

[0119] (1) For the image after the first segmentation, it is converted into a grayscale image and an adaptive thresholding process is performed to generate a binary image.

[0120] (2) Perform contour detection on the binary image. After one segmentation, the contours of the side checkerboard and the interference checkerboard in the surrounding side images are relatively independent. Therefore, in an ideal case, the largest contour is the desired checkerboard.

[0121] (3) Calculate the areas of all detected contours, find the contour with the largest area, and mark it as a valid contour.

[0122] (4) Create a mask based on the valid contours to segment the binary image. At the same time, perform pixel subtraction between the original binary image and the segmented image to generate a pixel difference image. The reason for performing the pixel difference operation is that sometimes the valid checkerboard pattern does not form a complete contour, so the detected contour may contain factors from the center surface. After the difference operation, the valid contour can be better judged.

[0123] (5) Setting an image pixel threshold to screen the above two images, marking the images that meet the conditions as valid images for secondary segmentation, and eliminating the images that do not meet the conditions.

[0124] (6) Perform an inversion operation on the pixels of the secondary segmented effective image, and set some pixels in the original image that do not belong to the maximum contour area to white.

[0125] Step 3: Perform corner detection and sub-pixel precision processing on multiple detailed checkerboard images to generate an accurate two-dimensional corner point coordinate set.

[0126] Traditional corner detection methods need to specify the number of rows and columns of corner points in advance, and require that these rows and columns must be regular and complete. However, when actually shooting fisheye box images, in order to improve the accuracy of calibration, it is necessary to shorten the distance between the lens and the calibration box to increase the coverage of the corner points in the image, which causes the checkerboard pattern on the surrounding sides to become incomplete. Therefore, the corner detection method described in step three adopts a sub-pixel corner detection method based on feature screening. It mainly relies on the grayscale changes and geometric changes of the checkerboard. Corner points at different positions have similar features. This feature is called corner similarity (Corner Likelihood). The present invention defines two different corner point models, namely the corner point model when the checkerboard is in the ideal position and the corner point model when the checkerboard is rotated 45°. As Figure 6 Two corner point models are shown, namely the corner point model when the chessboard is in the ideal position and the corner point model when the chessboard is rotated 45°. For fisheye box images, these two corner point models can describe the features of all required corner points.

[0127] The above two corner point models are used to calculate the corner point similarity of each pixel. Ideally, the convolution response of the corner point {A, B} should be greater than the average response of {A, B, C, D}, and the response of {C, D} should be smaller. For corner points with rotation angles, the opposite is true. Let f x i is the convolution response of the convolution kernel X and model i for a specific pixel, then the corner similarity C of the pixel can be defined by taking the maximum value of the combination of the two models:

[0128]

[0129] in, Represents the corner point model in two different situations, namely, the left diagonal is black squares, the right diagonal is white squares and the left diagonal is white squares, and the right diagonal is black squares.

[0130] It can be seen from the above formula that if any of the four convolution kernels has a weak convolution response, the corner similarity of the pixel at that location is low. Therefore, it is very important to ensure that the checkerboard is checked and calibrated before taking the image to eliminate potential influencing factors as much as possible.

[0131] Through the above calculations, a series of pixels that meet the corner point characteristics can be obtained. These pixels are called corner point candidates. However, the positions of these candidate points are usually not accurate enough, so sub-pixel level precision processing is required, including the precision of position and edge direction. This step aims to improve the accuracy of corner point detection so as to more accurately determine the position and direction information of corner points in the image.

[0132] like Figure 7 As shown in the figure, a schematic diagram of the corner point position. Assume that C is the ideal corner point position, P and Q are two pixels in the local neighborhood of C, and g P , g Q are the image gradient vectors of point P and point Q respectively. Since point P is located in the pixel flat area, the pixel flat area gradient g P =0, so Point Q is on the boundary, and the gradient g Q The direction of the vector is vertically downward, and the direction of (QC) is horizontally to the right. The directions of the two vectors are perpendicular to each other, so So for corner point C, g N (N is the pixel point in the neighborhood near the corner point C) and (NC) are always approximately orthogonal. That is:

[0133]

[0134] Among them, C′ is the candidate corner point that meets C≥K after setting the judgment threshold K. It is the neighborhood of the corner candidate point C′, and the size of the neighborhood is generally 11×11.

[0135] Adjacent pixels can be automatically weighted by the gradient magnitude, as shown below:

[0136]

[0137] In order to refine the calculation of the edge direction vectors e1, e2, the method of minimizing the error between their normal and the image gradient is adopted:

[0138]

[0139] in, is a set of adjacent pixels that satisfies Its gradient m with corner model i i Alignment, m i =[cos(a i )sin(a i )] T By filtering pixels, an accurate set of two-dimensional corner point coordinates is obtained.

[0140] Step 4: Sort the corner points according to the precise two-dimensional corner point coordinate set, map them to the world coordinate system, calculate the corner point coverage, and obtain a three-dimensional world coordinate set that corresponds one-to-one to the two-dimensional corner point coordinates.

[0141] The corner point sorting method is a corner point sorting method based on bidirectional expansion of feature points. For fisheye boxes, usually, in order to capture images with a high corner point coverage rate, it may be necessary to sacrifice the corner points on the edge to a certain extent. However, for those irregularly arranged corner point matrices, conventional corner point sorting methods may no longer be applicable. Therefore, the present invention proposes a corner point sorting method based on bidirectional expansion of feature points to cope with this situation and generate corresponding world coordinates.

[0142] This method needs to find sorting feature points to achieve bidirectional expansion sorting. Sorting feature points actually refer to the corner points that serve as the starting points for sorting, which need to meet the following characteristics:

[0143] (1) The sorting feature points are located at the edge of the corner point matrix, which is curved rather than straight.

[0144] (2) The sorting feature point is located in the middle of the edge. It does not need to be located strictly in the center of the edge, and 1-2 corner point position deviations can be tolerated.

[0145] (3) For the fisheye box, the sorting feature points of the mutually symmetric planes should be in an axisymmetric relationship.

[0146] For the convenience of subsequent description, the following corner point sorting principle will take the right side as an example. Figure 8 The figure shows a schematic diagram of the right side embodiment of the sorting feature point candidate point, C1 is the center point of the corner point, M1 and N1 are the sorting feature point candidate points that meet the conditions. In order to accurately locate the sorting feature points, it is necessary to calculate the center point coordinates of the detected corner point matrix. For the right side, the sorting feature point that meets the conditions is located on the left side of the center point. Therefore, the sorting feature points on the right side can be located by the following steps:

[0147] (1) Calculate the coordinates of the center point C1(x1,y1).

[0148] (2) From all detected corner points, select the n corner points closest to the vertical coordinate of C1. n is at least the number of corner points in a column.

[0149] (3) From these n corner points, select the corner point with the smallest horizontal coordinate, which can be used as the sorting feature point of the right side.

[0150] like Fig. 9 The figure shows a schematic diagram of the right side embodiment of the corner point sorting method based on bidirectional expansion of feature points. After locating the sorting feature point, the point is used as the sorting starting point to sort the corner points, and the world coordinates corresponding to the corner points are generated. The corner point sorting process is as follows:

[0151] (1) The sorting feature point O on the right side is the sorting starting point, and its world coordinates are set to (0,0,0).

[0152] (2) From all detected corner points, find the three corner points with the smallest Euclidean distance from point O. Then, select the corner point with the largest horizontal coordinate as point B (1, 0, 0). Next, from the remaining two corner points, select the corner point with the smallest vertical coordinate that is smaller than point O as point C (0, -1, 0), and select the corner point with the largest vertical coordinate that is larger than point O as point A (0, 1, 0).

[0153] (3) Take point C as the new starting point and find the three corner points with the smallest Euclidean distance from point C. Then, select the corner point with the smallest ordinate that is smaller than point C as point D (0, -2, 0). Continue this process until no corner point that meets the conditions is found, and return to point A.

[0154] (4) Start from point A and sort downwards to find the three corner points with the smallest Euclidean distance from point A. Then, select the corner point with the largest ordinate that is greater than point A as point E(0,2,0). Continue this process until no corner point that meets the conditions is found, and return to point B. This completes the sorting of the first column of the corner point matrix and generates the corresponding world coordinates.

[0155] (5) Excluding the already sorted corner points, take point B as the new starting point, and follow the same principle to find the three corner points with the smallest Euclidean distance from point B. Select the corner point with the largest horizontal coordinate as point G(2,0,0), and then from the remaining two corner points, select the corner point with the smallest vertical coordinate that is smaller than point B as point H(0,-2,0), and select the corner point with the largest vertical coordinate that is larger than point B as point F(0,2,0).

[0156] (6) Continue to sort in both directions with the new sorting starting point. After sorting each column of corner points, sort the next column. And so on, until all corner points are sorted and the corresponding world coordinates are generated.

[0157] Fig.10The figure shows a schematic diagram of an embodiment of missing edge corner points. In order to pursue a higher corner point coverage rate, the images captured on the left and right sides of the fisheye box often lack some edge corner points, and sometimes even the number of corner points in the row where the sorted feature points are located is less than that in other rows. In this case, point K (h, 0, 0) is set as the last corner point of the row with the least number of corner points. When executing the corner point sorting algorithm based on bidirectional expansion of feature points, after the sorting of the column where point K is located is completed, the remaining corner points need to be sorted. First, the remaining corner points are arranged from small to large according to the ordinate. Then, the corner point L with the smallest ordinate is selected as the starting point, and the corner point M (h, l, 0) closest to the corner point L in the column where point K is located is found. Finally, the ordinate of point L (i, j, 0) is inherited from point M, so its world coordinate can be calculated by the following formula:

[0158]

[0159] And so on, all the remaining corner points are sorted. The same principle of feature point positioning and corner point sorting also applies to other sides.

[0160] Since the center plane is a corner point matrix of standard rows and columns, it can be sorted using the traditional corner point sorting method to generate the corresponding world coordinates.

[0161] Step 5: Based on the precise two-dimensional corner point coordinate set and the precise two-dimensional corner point coordinate set, the fisheye lens calibration is realized by using the six-parameter division model to obtain the high-precision fisheye lens internal parameters.

[0162] The fisheye calibration method used in step 5 is a fisheye lens calibration method based on a six-parameter division model. d ,y d ) is the coordinate of the image distortion point, (x u ,y u ) is the coordinate of the undistorted point of the image, and the six-parameter division model expression can be expressed as follows:

[0163]

[0164] Among them, r d and r u Distortion points (x d ,y d ) and the undistorted point (x u ,y u ) to the Euclidean distance of the distortion center P, λ i is the radial distortion parameter. In order to simplify the subsequent solution process, the following detailed explanation will use a single-parameter DM model:

[0165]

[0166] Assume that the coordinates of the distortion center P are (x0, y0), then the coordinates of the undistorted point in the image are:

[0167]

[0168] Among them, r d The image distortion point and the distortion center P(x0, y0) satisfy:

[0169]

[0170] Suppose the equation of the line is y=kx+b, where k is the slope and b is the intercept, that is, the equation of the undistorted line is y u =kx u +b, substituting into the above formula we get:

[0171]

[0172] It can be seen from the above formula that under the single-parameter DM model, the ideal straight line is distorted into an arc curve. After sorting it out, the following equation is obtained:

[0173]

[0174] Among them, A, B, and C meet the following conditions:

[0175]

[0176] It can obtain three points from the image to solve the parameters A, B, C and the distortion center. Based on the relationship between A, B, and C, it can be obtained:

[0177]

[0178] By substituting the obtained A, B, C and distortion center into the above formula and performing a series of operations, the distortion parameter λ can be obtained. Based on the above principle, the six-parameter DM model can be solved.

[0179] Based on the large local distortion of the fisheye box, the above method is used to solve the DM model into six distortion parameters, so as to more accurately describe the degree of distortion of the fisheye box image. The above method can be used to calibrate the wide-angle fisheye lens with high precision.

[0180] Experimental verification of the present invention:

[0181] In order to verify the effectiveness of the camera calibration device and method for single-shot images of a wide-angle fisheye lens proposed in the present invention, five experiments were conducted under the condition of 32 light source intensity using a No. 1 fisheye lens with a field of view of 200°. The experimental results are shown in Table 1. The experimental data analysis found that when calibrating 5 images taken continuously at a fixed position, the jump of the effective focal length (EFL) was less than 0.15, the jump of the center of distortion (COD) was also significantly less than 0.15, and the jump of the distortion parameters did not exceed 0.01. The experimental results show that this method has excellent calibration stability.

[0182] Table 1 Experimental data of lens No. 1 (light source intensity 32)

[0183]

[0184] In order to verify the accuracy of the camera calibration device and method for a single shot image of a wide-angle fisheye lens proposed in the present invention, three fisheye lenses with a field of view of 200° were used to conduct five experiments under the condition of 32 light source intensity. The experimental results are shown in Table 1, Table 2, and Table 3. The experimental data analysis found that the reprojection errors (RMS) of the three lenses were all less than 0.2, and their RMS jumps were significantly less than 0.005. These experimental results show that this method has excellent calibration accuracy.

[0185] Table 2 Experimental data of lens No. 2 (light source intensity 32)

[0186]

[0187] Table 3 Experimental data of lens No. 3 (light source intensity 32)

[0188]

[0189] In order to verify the robustness of the camera calibration device and method for a single-shot image of a wide-angle fisheye lens proposed in the present invention, a fisheye lens No. 1 with a field of view of 200° was used to conduct five experiments under the conditions of 32 and 42 light source intensities, and the experimental results are shown in Tables 1 and 4. The experimental data analysis found that the reprojection errors (RMS) of the three lenses were all less than 0.2, and their RMS jumps were significantly less than 0.005. These experimental results show that the method has excellent calibration accuracy. Analysis of the experimental data shows that for the same lens, the calibration method of the present invention is performed under different light intensities, and the differences in EFL and COD are less than 0.1, and the difference in RMS is less than 0.005. This shows that its calibration accuracy is almost unaffected. In addition, by observing the jumps of the calibration results under different light intensities, it is found that the jump of EFL is still less than 0.15, the jump of COD is still less than 0.15, and the jump of distortion parameters is less than 0.005, indicating that its calibration stability is almost unaffected by changes in light intensity.

[0190] Table 4 Experimental data of lens No. 1 (light source intensity 42)

[0191]

[0192] In order to verify the superiority of the camera calibration device and method for single-shot image shooting with a wide-angle fisheye lens proposed in the present invention, five experiments were conducted using a No. 1 fisheye lens with a 200° field of view under the condition of 32 light intensity. Under the same conditions, 11 images were taken using the No. 1 lens as the experimental objects of the traditional fisheye calibration method. The experimental results are shown in Figure 2. Fig.11 , 12 As shown. Fig.11 , 12 It can be seen that the present invention has achieved significant improvement in calibration accuracy compared with the traditional method. Compared with the traditional method, its RMS is reduced by 8.56%. In addition, the present invention also has obvious advantages in operation time. The main reasons for these advantages include the following points:

[0193] (1) The present invention adopts a high-precision corner point detection method, which can optimize the corner point coordinates at the sub-pixel level, ensuring the accuracy of the corner point coordinates.

[0194] (2) A fisheye calibration method based on a six-parameter division model was adopted to improve the accuracy of distortion correction.

[0195] (3) The present invention only needs to process a few images after segmentation, while the traditional method needs to process a large number of images, so it has an advantage in the running time of the overall algorithm.

[0196] In addition, the present invention also has an advantage in image shooting time, because only one image needs to be shot, which greatly shortens the time cost of calibration compared to the traditional method of shooting 11 images. Therefore, considering the accuracy, calculation time and shooting time, the calibration method proposed in the present invention has obvious advantages in many aspects.

[0197] The present invention proposes a camera calibration device and method suitable for wide-angle fisheye lenses. The method includes two segmentation algorithms, a sub-pixel corner point detection method based on feature screening, and a fisheye calibration method based on a six-parameter division model. Compared with traditional calibration methods, the main advantages of this method are as follows:

[0198] (1) Only one image needs to be taken, which greatly shortens the calibration time and improves efficiency.

[0199] (2) Under the same lens, the calibration results show good algorithm stability.

[0200] (3) For images taken with different lenses, the RMS of the calibration results is less than 0.2, demonstrating good algorithm accuracy.

[0201] (4) In images taken under different light intensity conditions, the calibration results have a small fluctuation and a small RMS value, which shows good algorithm robustness.

[0202] (5) Compared with the traditional fisheye calibration algorithm, the calibration error of this method is reduced by 8.56% and has a faster execution rate.

[0203] Therefore, the camera calibration device and method suitable for a wide-angle fisheye lens proposed in the present invention show good performance in terms of calibration accuracy and efficiency, and can meet the calibration requirements of the fisheye lens.

Claims

1. A camera calibration method for a camera calibration device for a single-shot image shot by a wide-angle fisheye lens, characterized in that: The camera calibration device for the single-shot image shooting of the wide-angle fisheye lens is a fisheye box. The fisheye box is an open rectangular parallelepiped with five faces. Each inner surface of the fisheye box is painted with black and white square checkerboards in regular n×m rows and columns. The outer edges of the black and white square checkerboards are spliced ​​with a circle of black and white checkerboard small grids, and a closed black frame is arranged outside the small checkerboard grids, and the black frame does not intersect with the small checkerboard grids therein; a gap is arranged between two adjacent black frames of the fisheye box, and each gap is respectively located at the edge or edge of two adjacent inner surfaces of the fisheye box; two groups of mutually perpendicular line segments are drawn at the center position of the center surface of the fisheye box to form a crosshair; circular marks are drawn at the middle position of a row of checkerboards close to the center surface of the black and white square checkerboards on the left and right inner surfaces of the fisheye box; The camera calibration method of the camera calibration device for single-shot image shooting of a wide-angle fisheye lens comprises the following steps: Step 1: Segment the single-shot fisheye box image once to obtain multiple regional segmentation images, each of which contains only a single black and white square chessboard image; Step 2: Perform secondary segmentation on multiple regional segmentation images to obtain multiple detailed chessboard images; Step 3: Perform corner point detection and sub-pixel precision processing on multiple detailed checkerboard images to generate an accurate two-dimensional corner point coordinate set; Step 4: Sort the corner points according to the precise two-dimensional corner point coordinate set, map them to the world coordinate system, calculate the corner point coverage, and obtain a three-dimensional world coordinate set that corresponds one-to-one to the two-dimensional corner point coordinates; Step 5: Based on the precise two-dimensional corner point coordinate set, the six-parameter division model is used to calibrate the fisheye lens and obtain high-precision fisheye lens internal parameters.

2. The camera calibration method according to claim 1, wherein: The primary segmentation method described in step 1 is an image segmentation method based on convex polygon approximation: (1) Preprocess the single-shot fisheye box image, convert it into a grayscale image, and then perform histogram equalization and bilateral filtering; (2) Using the Canny operator to perform edge detection on the preprocessed image, the outer contours of the edges are connected to eliminate discontinuous outer edges and ensure that the periphery of the chessboard is completely detected; (3) In the connected edge map, find a new contour and judge it based on its area to determine whether it is a valid contour. A valid contour must contain a complete closed chessboard. (4) For the effective contour, a convex polygon approximation method is used to create a convex polygon mask; (5) A single chessboard area image is cut out from the original image using a convex polygon mask. The segmented chessboard area image contains two parts, namely, the chessboards on the four inner sides of the fisheye box and a small number of mis-segmented center chessboards.

3. The camera calibration method according to claim 2, wherein: The secondary segmentation in step 2 adopts an adaptive threshold segmentation method based on image inversion; (1) Convert the cut single chessboard area image into a grayscale image and perform adaptive thresholding to generate a binary image; (2) Perform contour detection on the binary image; (3) Calculate the areas of all detected contours, find the contour with the largest area, and mark it as a valid contour; (4) Creating a mask based on the effective contour to segment the binary image and obtain a segmented image. At the same time, subtracting the original binary image from the segmented image to generate a pixel difference image. (5) Setting an image pixel threshold to screen the segmented image and pixel difference image obtained in step (4), marking the images that meet the conditions as valid secondary segmentation images, and discarding the images that do not meet the conditions; (6) Perform an inversion operation on the pixels of the secondary segmented effective image, and set some pixels in the original image that do not belong to the maximum contour area to white.

4. The camera calibration method according to claim 3, wherein: The corner detection method in step 3 adopts a sub-pixel corner detection method based on feature screening: Define two different corner point models: the corner point model when the chessboard is in the ideal position and the corner point model when the chessboard is rotated 45°; The above two corner point models are used to calculate the corner point similarity of each pixel. is the convolution response of the convolution kernel X and model i for a specific pixel, then the corner similarity of the pixel is defined by taking the maximum value of the combination of the two models: in, Indicates the corner point model in two different cases, n = 1, 2; that is, the left diagonal is black, the right diagonal is white, and the left diagonal is white, and the right diagonal is black; If any of the four convolution kernels has a weak convolution response, the corner similarity of the pixel at that location is low; Through the above calculation, a series of pixel points that meet the corner point characteristics are obtained. These pixel points are called corner point candidates, and sub-pixel precision processing is performed, including precision of position and edge direction, to determine the position and direction information of the corner points in the image; Assume that C is the ideal corner point, P and Q are two pixels in the local neighborhood of C, and g P , g Q are the image gradient vectors of point P and point Q respectively. Since point P is located in the pixel flat area, the pixel flat area gradient gP=0, so Point Q is on the boundary, and the gradient g Q The direction of the vector is vertically downward, and the direction of (QC) is horizontally to the right. The directions of the two vectors are perpendicular to each other, so So for corner point C, g N is always orthogonal to (NC), N is the pixel point in the neighborhood near the corner point C, that is: Among them, C′ is the candidate corner point that meets C≥K after setting the judgment threshold K. is the neighborhood of the corner candidate point C′, and the size of the neighborhood is 11×11; Adjacent pixels can be automatically weighted by the gradient magnitude, as shown below: The edge direction vectors e1, e2 are taken by minimizing the error between their normal and the image gradient: in, is a set of adjacent pixels that satisfies Its gradient m with corner model i i Alignment, m i =[cos(a i )sin(a i )] T , by filtering pixels, we can obtain an accurate set of two-dimensional corner point coordinates.

5. The camera calibration method according to claim 4, wherein: The corner point sorting method in step 4 is a corner point sorting method based on bidirectional expansion of feature points, and generates corresponding world coordinates; Step 4.1: Find the sorting feature points to achieve bidirectional expansion sorting. The sorting feature points refer to the corner points that serve as the starting points for sorting, and they need to meet the following characteristics: (1) The sorting feature points are located at the edge of the corner point matrix, which is curved rather than straight. (2) The sorting feature point is located in the middle of the above edge; (3) For the fisheye box, the sorting feature points of mutually symmetric planes are in an axisymmetric relationship; Step 4.2, calculate the center point coordinates of the detected corner point matrix. For the right side, the sorting feature points that meet the conditions are located on the left side of the center point. The sorting feature points of the right side are located by the following steps: (1) Calculate the coordinates of the center point C1(x1,y1); (2) From all detected corner points, select the n corner points closest to the vertical coordinate of C1, where n is at least the number of corner points in a column; (3) From these n corner points, select the corner point with the smallest horizontal coordinate as the sorting feature point of the right side face; Step 4.3, after locating the sorting feature point, use this point as the sorting starting point to sort the corner points and generate world coordinates that correspond to the corner points one by one. The corner point sorting process is as follows: (1) The sorting feature point O on the right side is the sorting starting point, and its world coordinate is set to (0,0,0); (2) From all detected corner points, find the three corner points with the smallest Euclidean distance from point O, and then select the corner point with the largest horizontal coordinate as point B (1, 0, 0); next, from the remaining two corner points, select the corner point with the smallest vertical coordinate that is smaller than point O as point C (0, -1, 0), and select the corner point with the largest vertical coordinate that is larger than point O as point A (0, 1, 0); (3) Take point C as the new starting point, find the three corner points with the smallest Euclidean distance from point C, then select the corner point with the smallest ordinate that is smaller than point C as point D (0, -2, 0), repeat this process until no corner point that meets the conditions is found, and return to point A; (4) Start from point A and sort downwards to find the three corner points with the smallest Euclidean distance from point A. Then, select the corner point with the largest ordinate that is larger than point A as point E(0,2,0). Repeat this process until no corner point that meets the conditions is found. Return to point B, complete the sorting of the first column of the corner point matrix, and generate the corresponding world coordinates. (5) Excluding the already sorted corner points, take point B as the new starting point, and follow the same method to find the three corner points with the smallest Euclidean distance from point B. Select the corner point with the largest horizontal coordinate as point G(2,0,0). Then, from the remaining two corner points, select the corner point with the smallest vertical coordinate that is smaller than point B as point H(0,-2,0), and select the corner point with the largest vertical coordinate that is larger than point B as point F(0,2,0). (6) Similarly, a new starting point is set to perform bidirectional expansion sorting. After each column of corner points is sorted, the next column is sorted until all corner points are sorted and the corresponding world coordinates are generated. Assume point K (h, 0, 0) is the last corner point of the row with the least number of corner points. When executing the corner point sorting algorithm based on bidirectional expansion of feature points, after sorting the column where point K is located, it is necessary to sort the remaining corner points. First, arrange the remaining corner points from small to large according to the ordinate. Then, select the corner point L with the smallest ordinate as the starting point, and find the corner point M (h, l, 0) closest to corner point L in the column where point K is located. Finally, the ordinate of point L (i, j, 0) is inherited from point M, and its world coordinate is calculated by the following formula: Similarly, sort all the remaining corner points on the right side, and use the same method to sort the corner points on other sides and generate the corresponding world coordinates; Since the center plane is a corner point matrix of standard rows and columns, the traditional corner point sorting method is used to sort the center plane and generate the corresponding world coordinates.

6. The camera calibration method according to claim 5, characterized in that: The fisheye lens calibration model used in step 5 is a six-parameter division model. Let (x d ,y d ) is the coordinate of the image distortion point, (x u ,y u ) is the coordinate of the undistorted point of the image, and the six-parameter DM model expression is: Among them, r d and r u Distortion points (x d ,y d ) and the undistorted point (x u ,y u ) to the Euclidean distance of the distortion center P, λ i is the radial distortion parameter, and the single-parameter DM model is adopted: Assume that the coordinates of the distortion center P are (x0, y0), then the coordinates of the undistorted point in the image are: Among them, r d The image distortion point and the distortion center P(x0, y0) satisfy: Suppose the equation of the line is y=kx+b, where k is the slope and b is the intercept, that is, the equation of the undistorted line is y u =kx u +b, substituting into the above formula we get: It can be seen from the above formula that under the single-parameter DM model, the ideal straight line is distorted into a circular arc curve. After sorting it out, the following equation is obtained: Among them, A, B, and C meet the following conditions: It obtains the parameters A, B, C and the distortion center by obtaining three points from the image, and obtains the following relationship based on A, B, and C: By solving A, B, C and the distortion center, the distortion parameter λ is obtained and the six-parameter DM model is solved.

Images