A monocular camera intrinsic and extrinsic parameter calibration method and system

Abstract

The application discloses a monocular camera internal and external parameter calibration method and system, and relates to the field of unmanned vehicles.The method comprises the following steps: acquiring calibration board data when the calibration board is located at the first, second and third positions in front of the vehicle respectively; the three sets of calibration board data each comprise point cloud data and camera image data; the calibration board comprises a square main board and rectangular board surfaces connected to the main board, and each rectangular board surface is at a set angle to the square main board; obtaining the three-dimensional coordinates of the center of a circle in a laser radar coordinate system corresponding to each frame of point cloud data; using a symmetrical circle point calibration pattern on the calibration board as a monocular camera internal parameter calibration pattern to calibrate the internal parameters of the monocular camera; obtaining the three-dimensional coordinates of the center of a circle hole in a monocular camera coordinate system corresponding to each frame of camera data; constructing a least square problem using the three-dimensional coordinates of the center of a circle in the laser radar coordinate system and the center of a circle in the monocular camera coordinate system corresponding to each frame of point cloud data, and obtaining an external parameter calibration result by solving the least square problem.The application improves the accuracy of calibration.

Description

Technical Field

[0001] This invention relates to the field of autonomous vehicle technology, and in particular to a method and system for calibrating the intrinsic and extrinsic parameters of a monocular camera. Background Technology

[0002] Autonomous vehicles represent a significant technological advancement, embodying a revolutionary technology for future transportation and possessing substantial strategic importance and broad application prospects. The three main technological modules of autonomous vehicles are perception, decision-making, and control. Among these, the perception module is a crucial component. Utilizing sensors such as LiDAR and monocular cameras, the perception module collects environmental information including roads, obstacles, and traffic participants. This information is then processed using technologies like computer vision, point cloud processing, and deep learning for environmental perception and target recognition. The accuracy and stability of the perception module are paramount to the safety and reliability of autonomous vehicles. It enables real-time acquisition of the surrounding environment, helping the vehicle make correct decisions and plans, thus ensuring safe driving and adaptation to diverse traffic conditions. Different sensors are susceptible to environmental influences, leading to decreased accuracy in the perception module. To ensure the normal operation of autonomous vehicles in dynamic environments, multi-sensor collaborative technologies have emerged. Among numerous sensor data fusion methods, the combination of LiDAR and a monocular camera is one of the most commonly used sensor pairs for driving environment perception. LiDAR primarily acquires distance and position information of objects in the environment by emitting laser beams and measuring the reflected light. Monocular cameras acquire visual information about the environment by capturing images and analyzing features within them. LiDAR excels in accuracy and stability, while monocular cameras are superior in recognizing and analyzing object features. Fusion of these two technologies can compensate for their respective limitations, improving the robustness and accuracy of perception. The main challenge in fusing these two heterogeneous sensors is finding the precise intrinsic parameters of the monocular camera and the rigid body transformation between the sensor coordinate system through external calibration.

[0003] Option 1: This method relies on markers and a robotic arm for extrinsic parameter calibration of the LiDAR and monocular camera, requiring the placement of specific calibration targets. The robotic arm, carrying both the monocular camera and LiDAR, is moved sequentially to each monocular camera sampling point. When the LiDAR acquires the 3D coordinate information of the calibration board's position, a 2D coordinate image is generated. If a discontinuity is detected in the 2D coordinate image among the fitted straight line segments from various scattered points, the LiDAR data representing the calibration board is extracted from the 2D coordinate image and used as the image to be processed. The attitude calibration parameters of the LiDAR relative to the monocular camera are then calculated.

[0004] Option 2: A calibration system and method for monocular cameras and lidar in autonomous driving environmental perception based on markers. A sensor system based on a customized calibration board processes the data collected by the monocular camera and lidar to obtain the three-dimensional coordinates of the center of the hollow circle on the calibration board in the coordinates of each sensor. Then, the two sets of points are registered based on the improved Iterative Closest Point (ICP) algorithm to solve the relative pose relationship between the monocular camera and lidar and obtain the calibration result.

[0005] Existing marker-based LiDAR and monocular camera extrinsic parameter calibration methods require first calibrating the monocular camera's intrinsic parameters, then acquiring 3D point cloud data of the LiDAR and image data of the monocular camera from a calibration board located in the shared field of view of both the LiDAR and the monocular camera. After acquiring multiple frames of data, the extrinsic parameters of the LiDAR and monocular camera are calculated. Marker-based LiDAR and monocular camera extrinsic parameter calibration methods can be divided into two types: region-based feature matching and point-based feature matching. Region-based feature matching requires finding regions with specific semantic meanings in both the image and point cloud data. This type of method extracts point cloud data of specific regions and designs a mask containing only those specific regions of the image. Then, it designs an object function to evaluate whether enough point cloud data of the specific region is projected onto the image mask. Finally, iterates through optimization methods to maximize or minimize the object function, thereby calculating the extrinsic parameters of the LiDAR and monocular camera. The core idea of ​​point feature matching is similar to that of region feature matching. However, this type of method extracts point features (such as edge points) from LiDAR and images and attempts to solve the extrinsic parameters by constructing PnP.

[0006] The existing extrinsic parameter calibration methods for LiDAR and monocular cameras relying on markers described above all require an additional calibration board for monocular camera intrinsic parameter calibration. Traditional monocular camera intrinsic parameter calibration data acquisition requires dozens of images of the calibration board in different poses, resulting in high calibration costs and cumbersome processes. Most current extrinsic parameter calibration methods still rely on manually selecting point cloud data and feature points. Solving for the extrinsic parameter calibration results requires dozens of sets of data to ensure accuracy, leading to a significant manual workload and low automation, making the overall calibration process extremely time-consuming.

[0007] Furthermore, as mentioned above, existing extrinsic parameter calibration methods for LiDAR and monocular cameras relying on markers do not achieve high accuracy, making it difficult to meet the requirements for precise fusion of LiDAR and monocular cameras. Existing extrinsic parameter calibration methods based on region feature matching for LiDAR and monocular cameras involve manually selecting calibration board point cloud data and aligning it with an image mask. Since manual selection of point clouds is difficult and prone to errors, the point cloud input data required for calculating the calibration extrinsic parameters is inaccurate, leading to significant errors in the calculated extrinsic parameters. Existing point feature matching LiDAR and monocular camera extrinsic parameter calibration methods are not robust enough to feature point extraction due to factors such as illumination changes, occlusion, or noise. Feature point matching also suffers from mismatches or missing matches. Moreover, some existing point feature matching calibration algorithms still rely on manual point selection, which is inaccurate under manual operation. The accuracy of the corresponding point pair data has a significant impact on the calibration results, and accumulated errors can lead to substantial calibration errors. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for calibrating the intrinsic and extrinsic parameters of a monocular camera, thereby improving the accuracy of calibration.

[0009] To achieve the above objectives, the present invention provides the following solution:

[0010] A method for calibrating the intrinsic and extrinsic parameters of a monocular camera, comprising:

[0011] Data from the first, second, and third calibration boards are collected when the calibration board is located at a first, second, and third position in front of the vehicle. All three calibration boards include point cloud data collected by the vehicle's lidar and camera image data collected by the vehicle's monocular camera. The calibration board comprises a square main board and rectangular surfaces connected to the four sides of the square main board, each rectangular surface forming a predetermined angle with the square main board. The square main board has M circular holes, and symmetrical dot calibration patterns with different orientations are provided on both the square main board and each rectangular surface; M is a positive integer greater than 1.

[0012] N frames of point cloud data and camera image data are selected from the first calibration board data, the second calibration board data, and the third calibration board data respectively to obtain the first input data, the second input data, and the third input data; N is a positive integer greater than 1;

[0013] Based on the point cloud data in the first input data, the second input data, and the third input data, the three-dimensional coordinates of the center of the circular hole in the lidar coordinate system corresponding to each frame of point cloud data are obtained by using the point cloud pass-through filtering algorithm and the RANSAC algorithm.

[0014] The symmetrical dot calibration pattern is used as the intrinsic parameter calibration pattern for the monocular camera. The intrinsic parameters of the monocular camera are calibrated based on the camera image data in the first input data, the second input data, and the third input data.

[0015] Based on the camera image data in the first input data, the second input data, and the third input data, as well as the three-dimensional center coordinates of the centers of the M circular holes in the calibration plate coordinate system, the three-dimensional coordinates of the center of the circular holes in the monocular camera coordinate system corresponding to each frame of camera data are obtained.

[0016] By constructing a least-squares problem, the three-dimensional coordinates of the center of the aperture in the lidar coordinate system corresponding to each frame of point cloud data and the three-dimensional coordinates of the center of the aperture in the monocular camera coordinate system corresponding to each frame of camera data are jointly calibrated to obtain the rotation matrix and translation matrix of the lidar relative to the monocular camera.

[0017] Optionally, the first position is a first predetermined distance directly in front of the vehicle, the second position is a second predetermined distance to the right of the first position, and the third position is a second predetermined distance to the left of the first position. The connection between the second position and the third position is perpendicular to the vehicle's centerline.

[0018] Optionally, based on the point cloud data in the first input data, the second input data, and the third input data, a point cloud pass-through filtering algorithm and a RANSAC algorithm are used to obtain the three-dimensional coordinates of the center of the circular aperture in the lidar coordinate system corresponding to each frame of point cloud data, specifically including:

[0019] For each frame of point cloud data in the first input data, the second input data, and the third input data:

[0020] A point cloud pass-through filtering algorithm is used to filter out points whose values ​​are not within the given range, resulting in filtered point cloud data. The given range is defined as the range within 2m to 5m in front of the lidar in the x, y, and z directions, within 3m to the left and right of the lidar, and above 0.1m above the ground.

[0021] The RANSAC algorithm is used to extract the point cloud data of the calibration board surface from the filtered point cloud data;

[0022] Edge extraction is performed on the point cloud data of the calibration board surface to obtain calibration board edge point cloud data, which includes the outer contour of the calibration board and the contours of M circular holes;

[0023] The RANSAC algorithm is used to fit the contours of M circular holes in the edge point cloud data of the calibration board, and the three-dimensional coordinates of the center of the circular hole in the lidar coordinate system corresponding to each frame of point cloud data are obtained based on the fitted M circular hole contours.

[0024] Optionally, the symmetrical dot marking pattern is a square, and the square main board and each of the rectangular boards are provided with symmetrical dot marking patterns in two postures. The first posture is that the four sides of the symmetrical dot marking pattern are parallel to the four sides of the board surface, and the second posture is the posture after rotating the first posture by 45 degrees.

[0025] The symmetrical dot marking pattern is a 7×7 arrangement of symmetrical dots, with each dot having a radius of 12.5 mm and a center-to-center distance of 25 mm between each dot.

[0026] Optionally, the symmetrical dot calibration pattern is used as the monocular camera intrinsic parameter calibration pattern. Based on the camera image data from the first input data, the second input data, and the third input data, the monocular camera's intrinsic parameters are calibrated, specifically including:

[0027] Extract the corner points of each symmetrical dot calibration pattern from each frame of camera image data in the first input data, the second input data, and the third input data;

[0028] Based on the corner points of each symmetrical dot calibration pattern, the intrinsic parameters of the monocular camera are calibrated using Zhang Zhengyou's intrinsic parameter calibration method, resulting in the intrinsic parameter matrix and distortion matrix of the monocular camera.

[0029] Optionally, based on the camera image data in the first input data, the second input data, and the third input data, and the three-dimensional center coordinates of the centers of the M circular holes in the calibration board coordinate system, the three-dimensional coordinates of the center of the circular holes in the monocular camera coordinate system corresponding to each frame of camera data are obtained, specifically including:

[0030] For each frame point camera image data in the first input data, the second input data, and the third input data:

[0031] Based on the parameter matrix and distortion matrix within the monocular camera, the ChArUco mark in the calibration board is identified from the camera image data in the first input data, the second input data, and the third input data, and the rotation vector and translation vector of the calibration board relative to the monocular camera are obtained.

[0032] The rotation vector is transformed into a rotation matrix of the calibration plate relative to the monocular camera using the Rodriguez rotation formula;

[0033] The rotation matrix and the translation vector are combined to obtain the homogeneous transformation matrix;

[0034] Based on the homogeneous transformation matrix, the three-dimensional center coordinates of the M circular holes in the calibration plate coordinate system are transformed to the monocular camera coordinate system, thereby obtaining the three-dimensional center coordinates of the M circular holes in the monocular camera coordinate system corresponding to each frame of camera image data.

[0035] Optionally, the three-dimensional coordinates of the center of the aperture in the lidar coordinate system corresponding to each frame of point cloud data and the three-dimensional coordinates of the center of the aperture in the monocular camera coordinate system corresponding to each frame of camera data are jointly calibrated by constructing a least-squares problem to obtain the rotation matrix and translation matrix of the lidar relative to the monocular camera, specifically including:

[0036] The three-dimensional coordinates of the center of the circular hole in the lidar coordinate system corresponding to each frame of point cloud data are taken as an element to form the first set of center data.

[0037] The three-dimensional center coordinates of the center of the circular aperture in the monocular camera coordinate system corresponding to each frame of camera image data are used as an element to form the second set of center data.

[0038] Remove elements from the first set of center data that have a number of centers other than M to obtain the third set of center data;

[0039] Remove elements from the second set of center data that have a number of centers other than M to obtain the fourth set of center data;

[0040] The third and fourth center data sets are used to construct a least squares problem. The goal of the least squares problem is to solve the rotation and translation matrices of the lidar relative to the monocular camera.

[0041] The least squares problem is solved by matrix singular value decomposition to obtain the rotation and translation matrices of the lidar relative to the monocular camera.

[0042] Optionally, M can be 6 and N can be 15.

[0043] Optionally, the front of the calibration plate is the side printed with the ChArUco logo, and the positions of each circular hole on the back of the calibration plate are chamfered.

[0044] This invention discloses a monocular camera intrinsic and extrinsic parameter calibration system, comprising:

[0045] The calibration data acquisition module is used to acquire first, second, and third calibration board data when the calibration board is located at a first, second, and third position in front of the vehicle, respectively. The first, second, and third calibration board data all include point cloud data acquired by the vehicle's lidar and camera image data acquired by the vehicle's monocular camera. The calibration board includes a square main board and rectangular plates connected to the four sides of the square main board, with each rectangular plate forming a predetermined angle with the square main board. The square main board has M circular holes, and symmetrical dot calibration patterns with different orientations are provided on both the square main board and each of the rectangular plates; M is a positive integer greater than 1.

[0046] The data selection module is used to select N frames of point cloud data and camera image data from the first calibration board data, the second calibration board data and the third calibration board data respectively to obtain the first input data, the second input data and the third input data; N is a positive integer greater than 1;

[0047] The module for determining the center coordinates of the circular aperture in the lidar coordinate system is used to obtain the three-dimensional coordinates of the center of the circular aperture in the lidar coordinate system corresponding to each frame of point cloud data by using the point cloud pass-through filtering algorithm and the RANSAC algorithm based on the point cloud data in the first input data, the second input data and the third input data.

[0048] The monocular camera intrinsic parameter calibration module is used to use the symmetrical dot calibration pattern as the monocular camera intrinsic parameter calibration pattern, and to perform intrinsic parameter calibration on the monocular camera based on the camera image data in the first input data, the second input data, and the third input data.

[0049] The module for determining the center coordinates of the circular aperture in the monocular camera coordinate system is used to obtain the three-dimensional coordinates of the center of the circular aperture in the monocular camera coordinate system corresponding to each frame of camera data based on the camera image data in the first input data, the second input data, and the third input data, as well as the three-dimensional center coordinates of the centers of the M circular apertures in the calibration plate coordinate system.

[0050] The joint calibration module is used to perform joint calibration by constructing a least squares problem to obtain the rotation matrix and translation matrix of the LiDAR relative to the monocular camera by combining the three-dimensional coordinates of the center of the circular aperture in the LiDAR coordinate system corresponding to each frame of point cloud data and the three-dimensional coordinates of the center of the circular aperture in the monocular camera coordinate system corresponding to each frame of camera data.

[0051] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0052] This invention requires only one calibration plate. By collecting point cloud data and camera image data from the calibration plate located at the left, center, and right positions of the vehicle, accurate monocular camera intrinsic parameter data can be obtained. By setting M through holes, feature points can be extracted from the point cloud data and camera image data more accurately without the need for manual selection, thus improving the accuracy of calibration. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic flowchart of a method for calibrating the intrinsic and extrinsic parameters of a monocular camera, provided in an embodiment of the present invention.

[0055] Figure 2 This is a detailed flowchart illustrating a method for calibrating the intrinsic and extrinsic parameters of a monocular camera, as provided in an embodiment of the present invention.

[0056] Figure 3 This is a front view of the calibration plate provided in an embodiment of the present invention;

[0057] Figure 4 This is a front view of the back of the calibration plate provided in an embodiment of the present invention;

[0058] Figure 5 This is a front side view of the calibration plate provided in an embodiment of the present invention;

[0059] Figure 6 This is a rear side view of the calibration plate provided in an embodiment of the present invention;

[0060] Figure 7 This is a schematic diagram of the scene arrangement for extrinsic parameter calibration of a lidar and a monocular camera provided in an embodiment of the present invention;

[0061] Figure 8 This is a schematic diagram of the calibration data acquisition process for lidar and monocular camera provided in an embodiment of the present invention;

[0062] Figure 9 This is a schematic diagram of a monocular camera intrinsic and extrinsic parameter calibration system provided in an embodiment of the present invention.

[0063] Symbol explanation:

[0064] 1-Square motherboard, 2-Rectangular board surface, 3-Round hole, 4-Symmetrical dot marking pattern, 5-ChArUco logo. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] The purpose of this invention is to provide a method and system for calibrating the intrinsic and extrinsic parameters of a monocular camera, thereby improving the accuracy of calibration.

[0067] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Example 1

[0069] like Figure 1 and Figure 2 As shown in the figure, the method for calibrating the intrinsic and extrinsic parameters of a monocular camera provided in this embodiment specifically includes the following steps:

[0070] Step 101: Collect the first, second, and third calibration board data when the calibration board is located at the first, second, and third positions in front of the vehicle, respectively; the first, second, and third calibration board data all include point cloud data collected by the lidar on the vehicle and camera image data collected by the monocular camera on the vehicle; the calibration board includes a square main board and rectangular plates connected to the four sides of the square main board, and each of the rectangular plates is at a set angle to the square main board; the square main board is provided with M circular holes, and the square main board and each of the rectangular plates are provided with symmetrical circular dot calibration patterns in different postures; M is a positive integer greater than 1.

[0071] The value of M is 6.

[0072] The front of the calibration plate is printed with the ChArUco logo, and the circular holes on the back of the calibration plate are chamfered. The front of the calibration plate is shown below. Figure 3 and Figure 5 The back of the calibration plate is as follows Figure 4 and Figure 6 .

[0073] The angle is set to 130 degrees, meaning that each of the rectangular panels forms a 130-degree angle with the square main board.

[0074] The symmetrical dot marking pattern is a square. The square main board and each of the rectangular boards are provided with two postures of the symmetrical dot marking pattern. The first posture is that the four sides of the symmetrical dot marking pattern are parallel to the four sides of the board surface. The second posture is the posture after rotating the first posture by 45 degrees.

[0075] The symmetrical dot marking pattern is a 7×7 arrangement of symmetrical dots, with each dot having a radius of 12.5 mm and a center-to-center distance of 25 mm between each dot.

[0076] The first position is a predetermined distance directly in front of the vehicle; the second position is a predetermined distance to the right of the first position; and the third position is a predetermined distance to the left of the first position. The connection between the second and third positions is perpendicular to the vehicle's centerline. The first predetermined distance is 3 meters. The second predetermined distance is 2 meters.

[0077] Step 101 specifically includes:

[0078] First, the scene setup for extrinsic parameter calibration of the LiDAR and monocular camera is performed. For example... Figure 7 As shown, drive the vehicle to an open area with ample light. There should be no vertical panels or walls within 5 meters in front of the vehicle to facilitate the LiDAR point cloud data processing program's identification and processing of the calibration board's point cloud data. Then, manually place the calibration board 3 meters directly in front of the vehicle. The calibration board should be as perpendicular to the ground as possible and without any bending.

[0079] After the scene setup is completed, data collection begins. The specific collection process is as follows: Figure 8 As shown.

[0080] First, move the calibration board to a position 3m directly in front of the vehicle (first position), leave it stationary for 30 seconds, and collect the corresponding lidar point cloud data and monocular camera image data.

[0081] Then, the calibration board is moved to a position 3m in front of the vehicle and 2m to the right, offset from the vehicle's centerline (second position), held still for 30 seconds, and the corresponding LiDAR point cloud data and monocular camera image data are collected. Finally, the calibration board is moved to a position 3m in front of the vehicle and 2m to the left, offset from the vehicle's centerline (third position), held still for 30 seconds, and the corresponding LiDAR point cloud data and monocular camera image data are collected. The point cloud data and camera image data are saved after collection.

[0082] This involves synchronizing point cloud data with camera image data in time.

[0083] Step 102: Select N frames of point cloud data and camera image data from the first calibration board data, the second calibration board data and the third calibration board data respectively to obtain the first input data, the second input data and the third input data; N is a positive integer greater than 1.

[0084] N takes the value 15.

[0085] Step 102 specifically includes:

[0086] The point cloud data and camera image data are timestamped and synchronized.

[0087] Point cloud data and camera image data pairs are extracted from the vehicle's front right, center, and left directions. Five frames of point cloud data and camera image data are selected from each of the three directions in front of the vehicle and extracted as input data for the external parameter calibration calculation of the LiDAR and monocular camera, namely the first input data, the second input data, and the third input data.

[0088] Step 103: Based on the point cloud data in the first input data, the second input data, and the third input data, the three-dimensional coordinates of the center of the circular aperture in the lidar coordinate system corresponding to each frame of point cloud data are obtained by using the point cloud pass-through filtering algorithm and the RANSAC algorithm.

[0089] Step 103 mainly processes and calculates the input fifteen frames of point cloud data, ultimately obtaining the 3D (three-dimensional) coordinates of the centers of the six circular holes in each frame of point cloud data, and placing them into a set P. L ={{p l11 ,p l12 ,p l13 ,p l14 ,p l15 ,p l16},…,{p l151 ,p l152 ,p l153 ,p l154 ,p l155 ,p l156}}middle.

[0090] For each frame of point cloud data in the first input data, the second input data, and the third input data, the specific process includes:

[0091] Point cloud pass-through filtering: The point cloud data processing section only requires point cloud data from the calibration board. LiDAR has a wide sensing range, generating a large amount of point cloud data that is not computationally necessary. The point cloud pass-through filtering algorithm from the Point Cloud Library (PCL) is used to filter out this data. Specified dimensional direction Points whose values ​​are outside the given range are selected to obtain filtered point cloud data. The given value range is within 2m to 5m in front of the lidar in the x, y, and z directions, within 3m to the left and right of the lidar, and above 0.1m above the ground.

[0092] Extracting point cloud data from the calibration board surface: The Random Sample Consensus (RANSAC) algorithm is used to extract the point cloud data from the filtered point cloud data, that is, the RANSAC algorithm is used to fit the plane in the clipped point cloud (filtered point cloud data).

[0093] The specific process of fitting a plane in a spatial point cloud using the RANSAC algorithm is as follows: First, three points are randomly selected from the point cloud. Then, these three points form a plane, and the distances to all other points from this plane are calculated. If the distance is less than a threshold t (adjusted according to the density of the point cloud), the points are considered to be on the same plane. If there are more than n points on the same plane, this plane is saved, and all points on this plane are marked as matched. The RANSAC algorithm terminates when, after L1 iterations, the plane found has fewer than n points, or when no three unmarked points can be found. Since the point cloud has undergone pass-through filtering beforehand, there are no other planes in the environment besides the calibration board surface; the resulting plane is the calibration board surface. Finally, the clipped point cloud data is used to... The calibration plate surface data is obtained from the calibration plate. Extract point cloud data from the calibration board surface This is to eliminate interference from other objects in the environment on the calibration, so that the edge extraction results of the subsequent calibration board point cloud data are more accurate.

[0094] Extracting Calibration Board Edge Point Cloud Data: The subsequent center detection section only requires the calibration board edge point cloud data. To make the center recognition program more efficient and accurate, edge extraction processing of the calibration board surface data is necessary first. Edge extraction is performed on the point cloud data of the calibration board surface to obtain calibration board edge point cloud data. This calibration board edge point cloud data includes the outer contour of the calibration board and the contours of M circular holes. Specifically, this includes calling the pcl::BoundarEstimation function in PCL based on the normal estimation method to process the calibration board surface data. Edge extraction was performed to obtain point cloud data of the calibration board edge, including the outer contour of the calibration board and the six circular holes. Extracting point cloud data from the calibration board edge This is to facilitate the subsequent extraction of the center of the six holes. The RANSAC center recognition algorithm will traverse all point clouds. If only the point cloud data of the calibration board edge is processed, the running efficiency of the calibration program can be greatly improved.

[0095] The RANSAC algorithm is used to fit the contours of M circular holes in the edge point cloud data of the calibration board, and the three-dimensional coordinates of the center of the circular hole in the lidar coordinate system corresponding to each frame of point cloud data are obtained based on the fitted M circular hole contours.

[0096] The specific process of fitting circles (M circular hole contours) in a spatial point cloud using the RANSAC algorithm is as follows: First, three coordinate points are randomly selected from all points, and the equation of the circle obtained from these three points is calculated to obtain the center and radius. Based on a distance threshold t (determined according to the density of the point cloud), all points are iterated. Points whose difference between the distance and radius from the obtained center is less than t are considered interior points; otherwise, they are considered outliers. The above steps are repeated continuously, recording the number of interior points and the number of interior points obtained each time, and the iteration is repeated L2 times to end the iteration. Finally, the iteration result with the largest number of interior points is selected, and the final circle fitting is performed based on the interior points of this result, and the 3D circular point coordinates of the circle are output. Finally, the 3D coordinates of the six circle centers of this frame are placed into a set P. L middle.

[0097] Steps 104 and 105 mainly process and calculate the fifteen frames of camera image data obtained in step 102, ultimately obtaining the 3D coordinates of the centers of the six circular holes in each frame of the camera image data, and storing these six 3D coordinates of the centers of the six circular holes in a set.

[0098] P C ={{p c11 ,p c12 ,p c13 ,p c14 ,p c15 ,p c16},…,{p c151 ,p c152 ,p c153 ,p c154 ,p c155 ,p c156}}middle.

[0099] Step 104: Use the symmetrical dot calibration pattern as the monocular camera intrinsic parameter calibration pattern, and perform intrinsic parameter calibration on the monocular camera based on the camera image data in the first input data, the second input data, and the third input data.

[0100] Step 104 specifically includes:

[0101] Extract the corner points of each symmetrical dot calibration pattern from each frame of camera image data in the first input data, the second input data, and the third input data.

[0102] Based on the corner points of each symmetrical dot calibration pattern, the intrinsic parameters of the monocular camera are calibrated using the Zhang Zhengyou intrinsic parameter calibration method, yielding the intrinsic parameter matrix and distortion matrix of the monocular camera. The Zhang Zhengyou camera intrinsic parameter calibration method requires calibration diagrams of the calibration pattern rotated around the x, y, and z axes in each direction. The four tilted small calibration plates around the main board of the specially designed calibration board used in this invention perfectly meet the data requirements of the Zhang Zhengyou intrinsic parameter calibration method. Specifically, the corner points of several symmetrical dot calibration patterns with different poses on the input fifteen frames of image data are identified and extracted by calling the cv::findCircleGrid function in the OpenCV image processing library. Then, the monocular camera intrinsic parameters are calibrated using the cv::calibrateCamera function in the OpenCV image processing library (the core algorithm of the cv::calibrateCamera function in the OpenCV image processing library is based on the Zhang Zhengyou intrinsic parameter calibration method). The function output finally yields the monocular camera intrinsic parameter matrix. With the distortion matrix [k1k2 k3 p1 p2].

[0103] The camera intrinsic matrix is ​​a 3x3 matrix used to represent the camera's intrinsic parameters. It is used to convert 3D world coordinates to 2D image coordinates. The camera intrinsic matrix contains information about the camera's focal length and the image center. Where f... x This represents the camera's focal length on the x-axis, which determines the degree of scaling of the image in the horizontal direction. y This represents the camera's focal length on the y-axis, which determines the degree of image scaling in the vertical direction. These two parameters are typically expressed in pixels and are used to map points in the camera coordinate system onto the image plane. A larger focal length results in greater image scaling and a smaller view of the scene, while a smaller focal length results in a larger view of the scene. x c represents the x-coordinate of the principal point of the image on the image plane. y This represents the y-coordinate of the principal point of the image on the image plane. The principal point is a point on the image plane that is the intersection of the camera's optical axis and the image plane, and is also the location of the image center. In the camera intrinsic parameter matrix, c x and c yThe camera distortion matrix defines the position of the origin of the image coordinate system in pixels, typically in pixels. It is a set of parameters used to describe camera distortion and is commonly used to correct distortion in images. k1, k2, and k3 represent radial distortion coefficients. They are modeled based on the distance of light rays from the optical axis. k1, k2, and k3 represent the first, second, and third order radial distortion coefficients, respectively. Positive values ​​indicate shrinkage (counter-clockwise bending) near the image center, while negative values ​​indicate stretching (clockwise bending) at the image edges. p1 and p2 represent tangential distortion coefficients. These are relative to pixel coordinates within the image plane. p1 and p2 represent the effects of tangential distortion in the x and y directions, respectively. Positive values ​​indicate that pixels in the upper part of the image will shift to the right, and pixels in the lower part of the image will shift downwards. When correcting an image, the camera distortion matrix adjusts the pixel coordinates in the image based on these parameters to eliminate radial and tangential distortion, resulting in a more accurate and straighter image.

[0104] Step 105: Based on the camera image data in the first input data, the second input data, and the third input data, and the three-dimensional center coordinates of the centers of the M circular holes in the calibration plate coordinate system, obtain the three-dimensional coordinates of the center of the circular hole in the monocular camera coordinate system corresponding to each frame of camera data.

[0105] Step 105 specifically includes:

[0106] For each frame point camera image data in the first input data, the second input data, and the third input data:

[0107] 3D coordinate data of the centers of the six circular holes in the preset calibration plate coordinate system by For example, the coordinates of this point are

[0108] Based on the intrinsic parameters and distortion matrix of the monocular camera, the ChArUco mark in the calibration board is identified from the camera image data in the first input data, the second input data, and the third input data. The rotation and translation vectors of the calibration board relative to the monocular camera are then obtained. Specifically, this includes: based on the intrinsic parameters of the monocular camera... Using the distortion matrix [k1 k2 k3 p1 p2] and the ChArUco flag, the cv::aruco::estimatePoseCharucoBoard function of the charuco class in the OpenCV image processing library is used to estimate the pose of the ChArUco flag, and the output is the rotation vector of the calibration board relative to the camera. With translation vector Where v x vy v z t represents the angles of rotation of the calibration plate coordinate system around the x-axis, y-axis, and z-axis of the camera coordinate system, respectively. x t y t z These represent the distances by which the calibration plate coordinate system is translated along the x-axis, y-axis, and z-axis of the camera coordinate system, respectively, and T represents transpose.

[0109] The rotation vector is obtained by using the Rodriguez rotation formula. Transformed into the rotation matrix of the calibration plate relative to the monocular camera

[0110] The standard form of the Rodriguez rotation formula is: R = I + (1 - cosθ)U 2 +sinθU.

[0111] Where I is the identity matrix, U represents the antisymmetric matrix of the rotation vector transformation, R represents the rotation matrix, which describes the rotation transformation of an object in three-dimensional space, and θ represents the norm of the rotation vector, that is, the length of the rotation vector.

[0112] Based on the above Rodriguez rotation formula and get Specific format:

[0113]

[0114] The rotation matrix With the translation vector Combining the results, we obtain the homogeneous transformation matrix.

[0115] Homogeneous transformation matrix The specific form is as follows:

[0116]

[0117] Based on the homogeneous transformation matrix, the three-dimensional center coordinates of the M circular holes in the calibration plate coordinate system are transformed to the monocular camera coordinate system, thus obtaining the three-dimensional center coordinates of the M circular holes in the monocular camera coordinate system corresponding to each frame of camera image data. For P C One element.

[0118] The coordinates of the center of the first circular aperture in the monocular camera coordinate system For example, through homogeneous transformation matrix The coordinates of the center of the circle in the calibration plate coordinate system The formula for transforming the 3D center of a circle to the camera coordinate system is:

[0119]

[0120] Step 106: Combine the three-dimensional coordinates of the center of the aperture in the lidar coordinate system corresponding to each frame of point cloud data and the three-dimensional coordinates of the center of the aperture in the monocular camera coordinate system corresponding to each frame of camera data by constructing a least squares problem to obtain the rotation matrix and translation matrix of the lidar relative to the monocular camera.

[0121] Step 106 specifically includes:

[0122] The three-dimensional coordinates of the center of the circular aperture in the lidar coordinate system corresponding to each frame of point cloud data are taken as an element to form the first set of center data, i.e., set P. L .

[0123] The three-dimensional center coordinates of the aperture center in the monocular camera coordinate system corresponding to each frame of camera image data are used as one element to form the second set of aperture center data, i.e., set P. c .

[0124] Remove elements from the first set of center data that have a number of centers other than M to obtain the third set of center data. By removing elements from the second set of center data that have a number of centers other than M, we obtain the fourth set of center data. If the number of center coordinates in each element of either the first or second center data set is not six, then that element and its corresponding point cloud data or camera image data are discarded. A new point cloud center data set is then obtained. With the new image center data set n is a positive integer less than 15.

[0125] A least-squares problem is constructed using the third and fourth center data sets. The objective of this least-squares problem is to find the rotation and translation matrices of the lidar relative to the monocular camera. Singular value decomposition is used to solve the least-squares problem, yielding the rotation and translation matrices of the lidar relative to the monocular camera.

[0126] The specific solution process for joint calibration is as follows:

[0127] The formula for transforming from the lidar coordinate system to the monocular camera coordinate system is:

[0128] in, To determine the coordinates of the center of a circle on the calibration board in the camera coordinate system, These are the coordinates of the same center point in the lidar coordinate system. By multiplying by the rotation matrix of the lidar relative to the monocular camera In addition, the translation matrix of the lidar relative to the monocular camera get

[0129] Find by constructing a least squares problem and The optimal matching least squares problem is represented as:

[0130]

[0131] in, The optimal solution for the rotation matrix of the lidar relative to the monocular camera. The optimal solution represents the translation matrix of the lidar relative to the monocular camera. SO3 represents the group of all rotations in three-dimensional space about a fixed point, and the rotation matrix of the lidar relative to the monocular camera is... Since it belongs to this group, R∈SO3. The translation matrix of the lidar relative to the monocular camera in a three-dimensional vector space representing real numbers. It is a three-dimensional vector, therefore we have t generally refers to all three-dimensional vectors belonging to the three-dimensional vector space, and specifically refers to the three-dimensional vector translation matrix in this context. w i This represents the weighting coefficient, which varies depending on the scenario. Here, it is set to 1 to ensure general applicability. The main idea of ​​the least squares method is to minimize the sum of squared errors (also known as residuals) between the true and predicted values ​​by determining the unknown parameters. This indicates that the objective function When taking the minimum value and The variable value, that is, the optimal solution required for calibration. and The solution process is based on the Singular Value Decomposition (SVD) method.

[0132] Rotate the matrix of the joint calibration results of lidar and monocular camera. With translation matrix Export to a YAML file and save; calibration complete.

[0133] This invention achieves rapid calibration of monocular camera intrinsic parameters and automatic, accurate calibration of LiDAR and monocular camera extrinsic parameters by setting six hollowed-out circles (holes), a ChArUco mark, and several symmetrical dot calibration patterns on a calibration plate. Using several symmetrical dot calibration patterns in different orientations to calibrate the monocular camera intrinsic parameters eliminates the need for an additional monocular camera intrinsic parameter calibration plate and the need to adjust the calibration plate's tilt and tilt during data acquisition to obtain accurate monocular camera intrinsic parameter data. The six hollowed-out circles enable more accurate extraction of feature points from the point cloud and image; the center extraction of these feature points is automated, eliminating the need for manual selection. The ChArUco mark provides depth information to the two-dimensional data of the monocular camera image. Finally, an iterative nearest-point algorithm based on SVD is used to pair and solve the three-dimensional point cloud data and three-dimensional image data, resulting in more accurate calibration results.

[0134] Example 2

[0135] like Figure 9 As shown, this embodiment provides a monocular camera intrinsic and extrinsic parameter calibration system, including:

[0136] The calibration data acquisition module 201 is used to acquire first calibration board data, second calibration board data, and third calibration board data when the calibration board is located at a first, second, and third position in front of the vehicle, respectively. The first calibration board data, the second calibration board data, and the third calibration board data all include point cloud data acquired by the lidar on the vehicle and camera image data acquired by the monocular camera on the vehicle. The calibration board includes a square main board and rectangular plates connected to the four sides of the square main board, and each of the rectangular plates is at a set angle to the square main board. The square main board is provided with M circular holes, and the square main board and each of the rectangular plates are provided with symmetrical circular dot calibration patterns in different postures; M is a positive integer greater than 1.

[0137] The data selection module 202 is used to select N frames of point cloud data and camera image data from the first calibration board data, the second calibration board data and the third calibration board data respectively to obtain the first input data, the second input data and the third input data; N is a positive integer greater than 1.

[0138] The LiDAR coordinate system circular aperture center coordinate determination module 203 is used to obtain the three-dimensional coordinates of the circular aperture center in the LiDAR coordinate system corresponding to each frame of point cloud data by using the point cloud pass-through filtering algorithm and the RANSAC algorithm based on the point cloud data in the first input data, the second input data and the third input data.

[0139] The monocular camera intrinsic parameter calibration module 204 is used to use the symmetrical dot calibration pattern as the monocular camera intrinsic parameter calibration pattern, and to perform intrinsic parameter calibration on the monocular camera based on the camera image data in the first input data, the second input data, and the third input data.

[0140] The module 205 for determining the center coordinates of the circular holes in the monocular camera coordinate system is used to obtain the three-dimensional coordinates of the center of the circular holes in the monocular camera coordinate system corresponding to each frame of camera data based on the camera image data in the first input data, the second input data, and the third input data, as well as the three-dimensional center coordinates of the centers of the M circular holes in the calibration plate coordinate system.

[0141] The joint calibration module 206 is used to jointly calibrate the three-dimensional coordinates of the center of the circular aperture in the lidar coordinate system corresponding to each frame of point cloud data and the three-dimensional coordinates of the center of the circular aperture in the monocular camera coordinate system corresponding to each frame of camera data by constructing a least squares problem, so as to obtain the rotation matrix and translation matrix of the lidar relative to the monocular camera.

[0142] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0143] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for calibrating the intrinsic and extrinsic parameters of a monocular camera, characterized in that, include: Data from the first, second, and third calibration boards are collected when the calibration board is located at the first, second, and third positions in front of the vehicle. All three calibration boards include point cloud data collected by the vehicle's lidar and camera image data collected by the vehicle's monocular camera. The calibration board comprises a square main board and rectangular surfaces connected to the four sides of the square main board. Each rectangular surface forms a set angle with the square main board. The square main board has M circular holes, and symmetrical dot calibration patterns with different orientations are provided on both the square main board and each rectangular surface. M is a positive integer greater than 1. The symmetrical dot calibration pattern is square, and the square main board and each rectangular surface have two orientations of symmetrical dot calibration patterns: the first orientation is where the four sides of the symmetrical dot calibration pattern are parallel to the four sides of the surface it occupies, and the second orientation is the first orientation rotated 45 degrees. N frames of point cloud data and camera image data are selected from the first calibration board data, the second calibration board data, and the third calibration board data respectively to obtain the first input data, the second input data, and the third input data; N is a positive integer greater than 1; Based on the point cloud data in the first input data, the second input data, and the third input data, the three-dimensional coordinates of the center of the circular hole in the lidar coordinate system corresponding to each frame of point cloud data are obtained by using the point cloud pass-through filtering algorithm and the RANSAC algorithm. The symmetrical dot calibration pattern is used as the intrinsic parameter calibration pattern for the monocular camera. The intrinsic parameters of the monocular camera are calibrated based on the camera image data in the first input data, the second input data, and the third input data. Based on the camera image data in the first input data, the second input data, and the third input data, and the three-dimensional center coordinates of the centers of the M holes in the calibration board coordinate system, the three-dimensional coordinates of the center of the holes in the monocular camera coordinate system corresponding to each frame of camera data are obtained. The three-dimensional coordinates of the center of the circular hole in the lidar coordinate system corresponding to each frame of point cloud data are taken as an element to form the first set of center data. The three-dimensional center coordinates of the center of the circular aperture in the monocular camera coordinate system corresponding to each frame of camera image data are used as an element to form the second set of center data. Remove elements from the first set of center data that have a number of centers other than M to obtain the third set of center data; Remove elements from the second set of center data that have a number of centers other than M to obtain the fourth set of center data; The third and fourth center data sets are used to construct a least squares problem. The goal of the least squares problem is to solve the rotation and translation matrices of the lidar relative to the monocular camera. The least squares problem is solved by matrix singular value decomposition to obtain the rotation and translation matrices of the lidar relative to the monocular camera.

2. The method for calibrating the intrinsic and extrinsic parameters of a monocular camera according to claim 1, characterized in that, The first position is a first predetermined distance directly in front of the vehicle, the second position is a second predetermined distance to the right of the first position, and the third position is a second predetermined distance to the left of the first position. The line connecting the second position and the third position is perpendicular to the centerline of the vehicle.

3. The method for calibrating the intrinsic and extrinsic parameters of a monocular camera according to claim 1, characterized in that, Based on the point cloud data in the first input data, the second input data, and the third input data, the point cloud pass-through filtering algorithm and the RANSAC algorithm are used to obtain the three-dimensional coordinates of the center of the circular aperture in the lidar coordinate system corresponding to each frame of point cloud data, specifically including: For each frame of point cloud data in the first input data, the second input data, and the third input data: A point cloud pass-through filtering algorithm is used to filter out points whose values ​​are outside a given range, resulting in filtered point cloud data; the given range is defined as... The range is within 2m to 5m in front of the lidar in three directions, within 3m to the left and right of the lidar, and above 0.1m above the ground. The RANSAC algorithm is used to extract the point cloud data of the calibration board surface from the filtered point cloud data; Edge extraction is performed on the point cloud data of the calibration board surface to obtain calibration board edge point cloud data, which includes the outer contour of the calibration board and the contours of M circular holes; The RANSAC algorithm is used to fit the contours of M circular holes in the edge point cloud data of the calibration board, and the three-dimensional coordinates of the center of the circular hole in the lidar coordinate system corresponding to each frame of point cloud data are obtained based on the fitted M circular hole contours.

4. The method for calibrating the intrinsic and extrinsic parameters of a monocular camera according to claim 1, characterized in that, The symmetrical dot marking pattern is as follows: The pattern consists of symmetrical dots arranged in a circle, each dot having a radius of 12.5 mm and a center-to-center distance of 25 mm between each dot.

5. The method for calibrating the intrinsic and extrinsic parameters of a monocular camera according to claim 4, characterized in that, Using the symmetrical dot calibration pattern as the intrinsic parameter calibration pattern for the monocular camera, the intrinsic parameter calibration of the monocular camera is performed based on the camera image data in the first input data, the second input data, and the third input data. Specifically, this includes: Extract the corner points of each symmetrical dot calibration pattern from each frame of camera image data in the first input data, the second input data, and the third input data; Based on the corner points of each symmetrical dot calibration pattern, the intrinsic parameters of the monocular camera are calibrated using Zhang Zhengyou's intrinsic parameter calibration method, resulting in the intrinsic parameter matrix and distortion matrix of the monocular camera.

6. The method for calibrating the intrinsic and extrinsic parameters of a monocular camera according to claim 5, characterized in that, Based on the camera image data in the first input data, the second input data, and the third input data, and the three-dimensional center coordinates of the centers of the M circular holes in the calibration board coordinate system, the three-dimensional coordinates of the center of the circular holes in the monocular camera coordinate system corresponding to each frame of camera data are obtained, specifically including: For each frame point camera image data in the first input data, the second input data, and the third input data: Based on the parameter matrix and distortion matrix within the monocular camera, the ChArUco mark in the calibration board is identified from the camera image data in the first input data, the second input data, and the third input data, and the rotation vector and translation vector of the calibration board relative to the monocular camera are obtained. The rotation vector is transformed into a rotation matrix of the calibration plate relative to the monocular camera using the Rodriguez rotation formula; The rotation matrix and the translation vector are combined to obtain the homogeneous transformation matrix; Based on the homogeneous transformation matrix, the three-dimensional center coordinates of the M circular holes in the calibration plate coordinate system are transformed to the monocular camera coordinate system, thereby obtaining the three-dimensional center coordinates of the M circular holes in the monocular camera coordinate system corresponding to each frame of camera image data.

7. The method for calibrating the intrinsic and extrinsic parameters of a monocular camera according to claim 1, characterized in that, M takes the value 6, and N takes the value 15.

8. The method for calibrating the intrinsic and extrinsic parameters of a monocular camera according to claim 1, characterized in that, The front of the calibration plate is printed with the ChArUco logo, and the positions of the round holes on the back of the calibration plate are chamfered.

9. A monocular camera intrinsic and extrinsic parameter calibration system, characterized in that, The monocular camera intrinsic and extrinsic parameter calibration system uses the monocular camera intrinsic and extrinsic parameter calibration method of claim 1, and the monocular camera intrinsic and extrinsic parameter calibration system includes: The calibration data acquisition module is used to acquire first, second, and third calibration board data when the calibration board is located at a first, second, and third position in front of the vehicle, respectively. The first, second, and third calibration board data all include point cloud data acquired by the vehicle's lidar and camera image data acquired by the vehicle's monocular camera. The calibration board includes a square main board and rectangular plates connected to the four sides of the square main board, with each rectangular plate forming a predetermined angle with the square main board. The square main board has M circular holes, and symmetrical dot calibration patterns with different orientations are provided on both the square main board and each of the rectangular plates; M is a positive integer greater than 1. The data selection module is used to select N frames of point cloud data and camera image data from the first calibration board data, the second calibration board data and the third calibration board data respectively to obtain the first input data, the second input data and the third input data; N is a positive integer greater than 1; The module for determining the center coordinates of the circular aperture in the lidar coordinate system is used to obtain the three-dimensional coordinates of the center of the circular aperture in the lidar coordinate system corresponding to each frame of point cloud data by using the point cloud pass-through filtering algorithm and the RANSAC algorithm based on the point cloud data in the first input data, the second input data and the third input data. The monocular camera intrinsic parameter calibration module is used to use the symmetrical dot calibration pattern as the monocular camera intrinsic parameter calibration pattern, and to perform intrinsic parameter calibration on the monocular camera based on the camera image data in the first input data, the second input data, and the third input data. The module for determining the center coordinates of the circular aperture in the monocular camera coordinate system is used to obtain the three-dimensional coordinates of the center of the circular aperture in the monocular camera coordinate system corresponding to each frame of camera data based on the camera image data in the first input data, the second input data, and the third input data, as well as the three-dimensional center coordinates of the centers of the M circular apertures in the calibration plate coordinate system. The joint calibration module is used to perform joint calibration by constructing a least squares problem to obtain the rotation matrix and translation matrix of the LiDAR relative to the monocular camera by combining the three-dimensional coordinates of the center of the circular aperture in the LiDAR coordinate system corresponding to each frame of point cloud data and the three-dimensional coordinates of the center of the circular aperture in the monocular camera coordinate system corresponding to each frame of camera data.

Images