Calibration of a Depth Camera Based on a Hollowed-out Chessboard

Abstract

This invention discloses a calibration method for a depth camera based on a hollowed-out chessboard. The method involves obtaining a depth image using an RGB camera or a depth camera, locating the chessboard grid within the depth image, binarizing the image, merging the results, performing a closing operation, extracting edges, performing a Hough transform, relocalizing lines using SVM, optimizing line positions using parallel and equidistant constraints, calculating chessboard corner points, and completing the calibration. The method uses CAPE to find the calibration board plane, locating the four vertices of a square, projecting the line at infinity onto the calibration board plane in the image coordinate system, constructing a projection matrix that restores the parallel and equidistant properties of the original chessboard lines after the projective transformation, mapping the intersection points of the optimized chessboard lines back to the original plane, and calibrating the depth camera based on the optimized chessboard corner points. This method solves the problem of finding the corner points of the depth camera on the calibration plane.

Description

Technical Field

[0001] This invention belongs to the field of UAV vision technology, specifically relating to a camera calibration method based on a calibration object. Background Technology

[0002] Calibration based on a calibration object is the mainstream method for camera calibration. For RGB cameras, the chessboard, as the calibration object, is easy to acquire and manipulate. However, the depth image values ​​represent the distance between points and the camera, so corner points cannot be found in the depth image of a black and white chessboard. To address this issue, some researchers have proposed self-calibration methods that do not use a calibration object. Existing self-calibration methods are based on active vision and can be implemented through solving Kruppa equations, stepwise layer calibration, and quadratic surface methods. However, the above self-calibration methods all suffer from drawbacks such as high cost, complex manufacturing processes, poor practicality, and poor robustness. Some methods utilize a specific calibration object observed simultaneously by an RGB camera and a depth sensor.

[0003] The main problem with depth camera calibration is the difficulty in finding corner points on the calibration plane. Currently, researchers have improved upon the corner-finding method used in color images and applied it to depth images. They removed the black checkerboard pattern from the calibration plate, so that the depth information obtained by the depth camera at these locations is simply the background behind the plate. The corner points are estimated based on the depth difference between the calibration plate and the background, and then calibration is performed similarly to that of a color camera. Summary of the Invention

[0004] The purpose of this invention is to address the problem that it is difficult to find the corner points of a depth camera on the calibration plane, and to provide a calibration method for a depth camera based on a hollowed-out checkerboard.

[0005] A calibration method for a depth camera based on a perforated chessboard, comprising:

[0006] 1. Obtain a depth image using an RGB camera or a depth camera, and locate the checkerboard pattern within the depth image;

[0007] 2. Perform a binarization operation on the image;

[0008] 3. Merge the results and perform a closing operation;

[0009] 4. Extract edges and perform Hough transform;

[0010] 5. Perform SVM line repositioning;

[0011] 6. Optimize line positions using parallel equidistant constraints; calculate chessboard corner points and complete calibration;

[0012] Step 6, which involves optimizing the line position using parallel equidistant constraints, includes: finding the calibration plate plane using the CAPE method; locating the four vertices of a square on this plane; finding the line at infinity on the calibration plate plane based on the projection of the four vertices onto the image coordinate system; and constructing a projection matrix based on the line at infinity. This process restores the parallel and equidistant properties of the original chessboard lines after projective transformation. The projected chessboard lines are then optimized based on the constraints of parallel and equidistant lines. Finally, the intersection points of the optimized chessboard lines are mapped back to the original plane. The depth camera is then calibrated based on the corner points of the optimized chessboard.

[0013] The specific method for optimizing the chessboard lines is as follows:

[0014] The first step is to use the CAPE method to find the calibration plate plane:

[0015] π0=[a0 b0 c0 d0] T .......................(1)

[0016] The second step is to determine the four planes:

[0017]

[0018] These four planes satisfy the following relationship:

[0019]

[0020] The third step is to calculate the four vertices l of the square determined by π0, π1, π3; π0, π3, π2; π0, π2, π1; and π0, π1, π4 (three faces determine one point). SQ1 l SQ2 l SQ3 l SQ4 The fourth step is to calculate the x-coordinates of the four vertices on the pixel plane. SI1 x SI2 x SI3 x SI4 The fifth step is to find the four sides l of the square. SI1 l SI2 l SI3 l SI4 The method is as follows:

[0021]

[0022] Step 6: Determine the two points w at infinity based on the two sets of parallel lines. SI1 w SI2 :

[0023]

[0024] Step 7: Find the line w1 at infinity based on the two points at infinity:

[0025] wl = w SI1 ×w SI2 ..........................(6)

[0026] Where wl is:

[0027] wl = [wl1 wl2 wl3] T ..........................(7)

[0028] Step 8: Calculate the projective matrix H based on wl. IP To restore its parallel and equidistant property, H IP for:

[0029]

[0030] Step 9: Calculate the projective projections of the horizontal and vertical lines of the chessboard obtained using the SVM classifier (transformation matrix is...). The functions `row(row1, row2, row3)` and `col(col1, col2, col3, col4)` are used, where the horizontal lines are segments intercepted by `col1` and `col4`, and the vertical lines are segments intercepted by `row1` and `row3`. Step 10: Calculate the projective projections of the chessboard boundary points (`bounds`). Step 11: Calculate the midpoints of `row` and `col`. Step 12: Calculate the coordinates of `row`, `col`, and the pixel coordinate system `v`. p The average angles along the axis (vertically downward) are used to calculate the new angles between row and col. Step 13: Find the new midpoint that minimizes the movement distance of row and col using the least squares method, and update row and col using the new midpoint. Step 14: Using the midpoint as the axis and taking the rotation angles of row and col as variables, find the rotation angle θ that minimizes the sum of the squares of the distances from bounds to row and col, respectively. row With θ col And update row and col accordingly. Step 15 fixes row, col, and v. p Given the included angle of the axes, and taking the displacement as a variable, find the displacement d that minimizes the sum of the squares of the distances from bounds to row and col, respectively. row With d col The sixteenth step is to find the intersection points (points) between row and col, which are the corner points of the chessboard. The seventeenth step is to use the projective matrix H... IP The eighteenth step involves projecting points onto the original projective space to obtain points1, which allows for the calibration of the depth camera based on points1.

[0031] This invention addresses the difficulty of locating corner points on the calibration plane using depth cameras. It applies the constraint of parallel and equidistant chessboard lines to optimize the corner point positions, resulting in more accurate corner point locations and better calibration results. First, the chessboard grid positions are located in the depth image, and the grid image is binarized to fuse the background and occluded areas. Multiple depth images of the calibration chessboard are captured at the same location, and the binarized images are merged. A closing operation is used to remove noise from the calibration chessboard grid. Hough transform is performed to detect the lines on the chessboard, and then SVM is used for corner relocation. Finally, the positions of the chessboard lines are further optimized by adding constraints of parallel and equidistant lines. First, the CAPE method is used to find the calibration plate plane. The four vertices of a square are located on this plane. Based on the projection of the four vertices onto the image coordinate system, the line at infinity on the calibration plate plane is found. A projection matrix is ​​constructed based on the line at infinity, so that the original chessboard lines recover their parallel and equidistant properties after the projection transformation. The projected chessboard lines are optimized based on the parallel and equidistant constraints. Then, the intersection points of the optimized chessboard lines are mapped back to the original plane. Finally, the depth camera is calibrated based on the corner points of the optimized chessboard.

[0032] By applying the constraint of parallel and equidistant chessboard lines to optimize the position of chessboard corner points, this method can obtain more accurate chessboard corner point positions, thus achieving better calibration results. Through the above optimization method, the first thirteen steps restore the parallel and equidistant constraints of the chessboard lines. The subsequent steps further optimize the chessboard lines by reusing the previously obtained chessboard boundary points. By fine-tuning the angle between the chessboard lines and the vp axis and the position of the midpoint of the chessboard lines using the least squares method, more accurate chessboard corner points are calculated, ultimately improving the calibration accuracy of the depth camera. Attached Figure Description

[0033] Figure 1 Depth camera calibration process;

[0034] Figure 2 To calibrate the chessboard depth map;

[0035] Figure 3 To calibrate the depth distribution map of the chessboard board;

[0036] Figure 4 To calibrate the binary image of the chessboard board;

[0037] Figure 5 The result of the closed operation is shown in the figure;

[0038] Figure 6 Detecting chessboard lines using Hough transform;

[0039] Figure 7 Here is a flowchart of the CAPE method;

[0040] Figure 8 Eight binarized and merged calibration images were collected from different locations. These images were then processed by a closing operation to produce the final image. Detailed Implementation

[0041] Example 1: A calibration method for a depth camera based on a hollowed-out chessboard.

[0042] To address the difficulty in locating corner points of depth cameras on the calibration plane, the constraint of parallel and equidistant chessboard lines is applied to optimize the positions of chessboard corner points. This method yields more accurate chessboard corner point positions, resulting in better calibration results. Figure 1 The process for calibrating depth cameras.

[0043] A calibration method for a depth camera based on a perforated chessboard, comprising:

[0044] 1. Obtain depth images using a depth camera.

[0045] One drawback of depth cameras that obtain depth information using structured light is accuracy issues, including noise from the camera itself and occlusion caused by positional differences between the infrared receiver and transmitter. To accurately locate corner points, checkerboard positions on the depth image are first selected using spatial proximity between the RGB camera and the depth camera.

[0046] 2. Perform binarization on the image.

[0047] The checkerboard image from the depth camera is binarized. The depth values ​​of the entire checkerboard image are divided into three types: foreground with low depth values, background with high depth values, and occluded areas with a depth value of 0mm. The calibration checkerboard depth map is shown below. Figure 2 As shown.

[0048] The purpose of binarization is to blend the background with the occluded areas, thereby eliminating the interference of occlusion. We binarize the image by treating the pixel values ​​in the checkerboard region as a function of the depth value, such as... Figure 3 As shown, the curve has three distinct peaks, corresponding to the foreground, background, and occluded areas mentioned above. By setting the pixel values ​​near the second peak to 255 and setting the pixel values ​​in other areas to 0, a binarized image of the checkerboard region is obtained. Figure 4 As shown.

[0049] 3. Merge the results and perform a closing operation.

[0050] Because depth images have poor accuracy at object edges, the binarized image here will contain some erroneous points at the boundaries of the background and occluded areas. Two methods are used to address this problem. The first method involves taking multiple depth images of the calibration chessboard at the same location and merging the resulting binarized images to reduce the number of erroneous points. Specifically, this involves counting the number of times each pixel in all binarized images is identified as foreground; only pixels with a frequency greater than 78% are identified as foreground, otherwise they are identified as background. The second method uses morphological closing operations to solve this problem. The structured elements we use in the closing operation are shown below:

[0051]

[0052] This structural element design has two advantages: First, the distance between each pixel at the edge and the center is closer than in a regular rectangle, better preserving the original shape. Second, the chessboard is roughly horizontal, so the sharp points in both the vertical and horizontal directions retain the detail of the chessboard's edges. Figure 5 As shown, most of the noise on the calibration chessboard can be effectively removed using the second method.

[0053] 4. Extract edges and perform Hough transform.

[0054] After obtaining the binarized depth map, a Hough transform is performed to detect lines on the chessboard, and then corner points are located based on the constraints between the lines, such as... Figure 6 As shown. Due to noise, we will detect checkerboard lines and other lines. Prior knowledge can be used to eliminate these interfering lines, such as parallelism, checkerboard layout, the number of checkerboard lines, and determining line lengths based on the size of the checkerboard grid area. Due to noise, the corner positions are inaccurate, so we will use an SVM method for corner relocation next.

[0055] Using the checkerboard lines obtained from the Hough transform as initial positions, for each checkerboard line, we can find all points adjacent to that line in the edge image. Before performing the closing operation, we first perform erosion and dilation operations on the image. The structured element for the erosion operation is K1, and the structured element for the dilation operation is K2. The result of this operation can be seen as a dilation operation, so the background region of the binary image shrinks inward. On the one hand, it separates the background that has been merged due to noise. On the other hand, it distributes the edge points of the checkerboard grid on both sides of the checkerboard lines, which is beneficial for corner relocation using SVM.

[0056]

[0057] 5. Perform SVM line repositioning

[0058] We extract the edge points near the checkerboard lines from the edge image of the black background region and then cluster these edge points based on their spatial continuity. This groups the edge points of each black background region together. We smooth the edges of the black background regions by moving each point to its nearest centroid. The next step is to find an optimal boundary between these two classes of points. As the black region shrinks inward, these points should be on either side of the line's true location. We consider the optimal boundary between these two classes to be the actual location of the line, and an SVM classifier is used to find this optimal boundary.

[0059] 6. Optimize line position using parallel equidistant constraints.

[0060] The chessboard lines obtained using the SVM classifier did not meet the constraint of parallelism and equidistant spacing between the lines, resulting in inaccurate line positions. Therefore, the following optimizes the chessboard line positions by adding constraints on parallelism and equidistant spacing between the lines. This requires using a method for extracting point cloud plane equations. The Cylinder and Plane Extraction Method (CAPE) is a method for extracting planes from point cloud data through region growing. It can process 640×480px depth maps at a frequency of 300Hz on a single-core CPU, which is 4-10 times faster than the PEAC method. Figure 7 The flowchart shown is for the CAPE method. CAPE is a method for extracting planes and cylinders from point clouds, enabling rapid processing of depth maps. First, the image is divided into small grids, and planes are fitted to these grids. Then, larger planes are found by performing region growing on these planar units. During this process, adjacent planes with the same planar parameters are merged. Finally, the boundaries of the planes are refined pixel by pixel within the cells selected through morphological operations. Using the CAPE method, we can find the position of the calibration board plane in the camera coordinate system, providing conditions for optimizing the position of the checkerboard lines.

[0061] The method for optimizing the chessboard lines is as follows. First, use the CAPE method to find the calibration board plane. Locate the four vertices of a square on this plane. Based on the projections of these four vertices into the image coordinate system, find the line at infinity on the calibration board plane. Construct a projection matrix H based on this line at infinity. IP This process restores the parallel and equidistant properties of the original chessboard lines after projective transformation. The projected chessboard lines are then optimized based on the constraints of parallelism and equidism. The intersections of the optimized chessboard lines are then mapped back to the original plane. Finally, the depth camera is calibrated based on the corner points of the optimized chessboard.

[0062] The specific method for optimizing the chessboard lines is as follows:

[0063] The first step is to use the CAPE method to find the calibration plate plane:

[0064] π0=[a0 b0 c0 d0]T......................(1)

[0065] The second step is to determine the four planes:

[0066]

[0067] These four planes satisfy the following relationship:

[0068]

[0069] The third step is to calculate the four vertices l of the square determined by π0, π1, π3; π0, π3, π2; π0, π2, π1; and π0, π1, π4 (three faces determine one point). SQ1 l SQ2 l SQ3 l SQ4 The fourth step is to calculate the x-coordinates of the four vertices on the pixel plane. SI1 x SI2 x SI3 x SI4 The fifth step is to find the four sides l of the square. SI1 l SI2 l SI3 l SI4 The method is as follows:

[0070]

[0071] Step 6: Determine the two points w at infinity based on the two sets of parallel lines. SI1 w SI2 :

[0072]

[0073] Step 7: Find the line wl at infinity based on two points at infinity.

[0074] wl = W SI1 ×w SI2 ..........................(6)

[0075] Where wl is:

[0076] wl = [wl1 wl2 wl3] T ..........................(7)

[0077] Step 8: Calculate the projective matrix H based on wl. IP To restore its parallel and equidistant property, HIP for:

[0078]

[0079] Step 9: Calculate the projective projections of the horizontal and vertical lines of the chessboard obtained using the SVM classifier (transformation matrix is...). The functions `row(row1,row2,row3)` and `col(col1,col2,col3,col4)` are used, where the horizontal lines are segments intercepted by `col1` and `col4`, and the vertical lines are segments intercepted by `row1` and `row3`. Step 10: Calculate the projective projections of the chessboard boundary points (`bounds`). Step 11: Calculate the midpoints of `row` and `col`. Step 12: Calculate the coordinates of `row`, `col`, and the pixel coordinate system `v`. p The average angles along the axis (vertically downward) are used to calculate the new angles between row and col. Step 13: Find the new midpoint that minimizes the movement distance of row and col using the least squares method, and update row and col using the new midpoint. Step 14: Using the midpoint as the axis and taking the rotation angles of row and col as variables, find the rotation angle θ that minimizes the sum of the squares of the distances from bounds to row and col, respectively. row With θ col And update row and col accordingly. Step 15 fixes row, col, and v. p Given the included angle of the axes, and taking the displacement as a variable, find the displacement d that minimizes the sum of the squares of the distances from bounds to row and col, respectively. row With d col The sixteenth step is to find the intersection points (points) between row and col, which are the corner points of the chessboard. The seventeenth step is to use the projective matrix H... IP The eighteenth step involves projecting points onto the original projective space to obtain points1, which allows for the calibration of the depth camera based on points1.

[0080] Using the optimization method described above, the first thirteen steps restore the chessboard lines to meet the parallel and equidistant constraints. Subsequent steps further optimize the chessboard lines by reusing the previously obtained boundary points, fine-tuning the chessboard lines and v using the least squares method. p By calculating the angle between the axes and the position of the midpoint of the chessboard lines, more accurate chessboard corner points can be calculated, ultimately improving the calibration accuracy of the depth camera.

[0081] Experimental Example 1

[0082] Eight binarized and merged calibration images were collected from different locations. The resulting images after undergoing a closing operation are shown below. Figure 8 As shown.

[0083] In the method for optimizing the position of chessboard lines, the variance of the included angle and the variance of the spacing of the chessboard lines in the twelfth step are shown in Table 1. This proves that the original chessboard line fitting method does not meet the constraints of parallelism and equidistance between chessboard lines. It is necessary to further optimize it to make the chessboard lines parallel and equidistant, so as to calculate more accurate chessboard corner points and improve the calibration accuracy of the depth camera.

[0084] Table 1. Variance of included angle and spacing

[0085] Table 1. Variance of included angle and spacing

[0086]

[0087] The depth camera calibration reprojection error before using the optimized chessboard line method was 0.783. The calibrated depth camera intrinsic parameter was K. depth0 :

[0088]

[0089] The corner coordinates of one of the calibration images and the reprojection coordinates based on world coordinates and calibration results are shown in Table 2.

[0090] Table 2 Corner coordinates and reprojected corner coordinates

[0091] Table 2 Corner coordinates and reprojected corner coordinates

[0092]

[0093] The depth camera calibration reprojection error after using the optimized checkerboard line method is 0.1957. The calibrated depth camera intrinsic parameter is K. depth1 :

[0094]

[0095] The coordinates of the inner corner points of one of the calibration images and the reprojection coordinates based on the world coordinates and calibration results are shown in Table 3.

[0096] Table 3. Corner coordinates after optimization and reprojection of corner positions.

[0097] Table 3. Corner coordinates after optimization and reprojection of corner positions.

[0098]

[0099] Comparing the reprojection errors before and after optimization with the corner reprojection coordinates and corner coordinates shown in Tables 2 and 3, although the reprojection error of a few points increased, the overall reprojection error of the corner points decreased after optimization. This indicates that the camera's intrinsic parameter calibration is more accurate after optimization.

Claims

1. A calibration method for a depth camera based on a perforated chessboard, comprising: 1) Obtain a depth image using an RGB camera or a depth camera, and locate the checkerboard pattern within the depth image; 2) Perform binarization on the image; 3) Merge the results and perform a closing operation; 4) Extract the edges and perform Hough transform; 5) Perform SVM line repositioning; 6) Optimize line positions using parallel equidistant constraints; Calculate and mark the corner points of the chessboard; Step 6, which involves optimizing the line position using parallel equidistant constraints, includes: finding the calibration plate plane using the CAPE method; locating the four vertices of a square on this plane; finding the line at infinity on the calibration plate plane based on the projection of the four vertices onto the image coordinate system; and constructing a projection matrix based on the line at infinity. This process restores the parallel and equidistant properties of the original chessboard lines after projective transformation. The projected chessboard lines are then optimized based on the constraints of parallel and equidistant transformation. Finally, the intersection points of the optimized chessboard lines are mapped back to the original plane. The depth camera is then calibrated based on the corner points of the optimized chessboard. Step 6, which involves optimizing the line position using parallel equidistant constraints, includes: The first step is to use the CAPE method to find the calibration plate plane: ； The second step is to determine the four planes: ； These four planes satisfy the following relationship: ； The third step is to calculate π0, π1, π3; π0, π3, π2; π0, π2, π1; and π0, π1, π4 respectively. For every three faces, a point is determined, thus defining the four vertices l of the square. SQ1 l SQ2 l SQ3 l SQ4 ； The fourth step is to calculate the x-coordinates of the four vertices on the pixel plane. SI1 x SI2 x SI3 x SI4 ; Step 5: Find the four sides l of the square. SI1 l SI2 l SI3 l SI4 The method is as follows: ； Step 6: Determine the two points w at infinity based on the two sets of parallel lines. SI1 w SI2 : ； Step 7: Find the line wl at infinity based on two points at infinity. ； Where wl is: ； Step 8: Calculate the projective matrix H based on wl. IP To restore its parallel and equidistant property, H IP for: ； The ninth step is to find the projective projections row and col of the horizontal and vertical lines of the chessboard obtained by the SVM classifier. row includes row1, row2, and row3, and col includes col1, col2, col3, and col4. Here, the horizontal lines are all line segments intercepted by col1 and col4, and the vertical lines are all line segments intercepted by row1 and row3. Step 10: Calculate the projective bounds of the chessboard boundary points; Step 11: Find the midpoints of row and col; Step 12: Calculate row, col, and pixel coordinate system v respectively. p The average angle between the axes is used to update the angles between row and col, where v is the average angle. p The axis is in the vertically downward direction; Step 13: Find the new midpoint that minimizes the movement distance of row and col using the least squares method, and update row and col using the new midpoint; Step fourteen: Using the midpoint as the axis, and taking the rotation angles of row and col as variables, find the rotation angle θ that minimizes the sum of the squares of the distances from bounds to row and col, respectively. row With θ col And update row and col accordingly; Step 15: Fix row, col, and v p Given the included angle of the axes, and taking the displacement as a variable, find the displacement d that minimizes the sum of the squares of the distances from bounds to row and col, respectively. row With d col And update row and col accordingly; The sixteenth step is to find the intersection points (points) between row and col, which are the corner points of the chessboard; Step 17: Based on the projective matrix H IP The eighteenth step involves projecting points onto the original projective space to obtain points1, which allows for the calibration of the depth camera based on points1.

Images