A calibration method and a calibration system for joint calibration of a laser radar and a monocular camera

Abstract

This invention relates to the field of sensor calibration technology, specifically providing a calibration method and system for joint calibration of a lidar and a monocular camera. The method includes: placing a calibration board at multiple different positions within the field of view of both the lidar and the monocular camera; simultaneously acquiring point cloud data and image data; measuring the distance data of multiple ranging points on the surface of the calibration board using an ultrasonic sensor array; fitting the ranging points to a three-dimensional reference plane; calculating the ray equation by combining pixel coordinates and camera intrinsic parameters; obtaining the intersection point of the ray and the reference plane as the camera's three-dimensional coordinates; constructing a search tree to match the three-dimensional coordinates of the camera and the lidar to form corresponding point groups; defining the target optimization function using the Euclidean distance of the corresponding point groups; and solving for the optimal rotation matrix and translation vector through nonlinear optimization iteration. This invention introduces an ultrasonic sensor, achieving high-precision joint calibration of lidar and a monocular camera at low cost.

Description

Technical Field

[0001] This invention relates to the field of sensor calibration technology, specifically providing a calibration method and calibration system for joint calibration of lidar and monocular camera. Background Technology

[0002] In industrial operating environments, environmental perception systems are a key technology for achieving automation and intelligence in tracked vehicles (such as excavators, bulldozers, and mining trucks). To obtain comprehensive environmental information, it is usually necessary to fuse multiple sensors, with lidar and cameras being the most common combination. To achieve effective fusion of these two sensors, the extrinsic parameters of the sensors must first be accurately calibrated, that is, the spatial transformation relationship between the lidar coordinate system and the camera coordinate system must be determined.

[0003] However, due to the harsh working environment, monocular industrial cameras, which lack depth information, are often used in camera selection. While monocular industrial cameras have advantages such as simple structure, strong impact resistance, low cost, and simple deployment, they also have several problems during calibration: First, monocular cameras lack depth information, making it impossible to directly establish the correspondence between image pixels and three-dimensional space; second, in the LiDAR-camera calibration process, due to the lack of depth constraints, the calibration parameter solution becomes an underconstrained problem, requiring a complex nonlinear optimization process. This makes traditional monocular-LiDAR calibration methods require a large amount of calibration data and complex iterative optimization, resulting in an unstable calibration process that is prone to getting trapped in local optima; third, the strong vibrations when tracked vehicles are in motion often cause the calibrated extrinsic parameters to be distorted by sensor vibrations, leading to sensor fusion failure. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a calibration method and system for joint calibration of a lidar and a monocular camera. It utilizes ultrasonic waves to perform depth compensation on the monocular camera, achieving high-precision and low-cost joint calibration.

[0005] The calibration method for joint calibration of lidar and monocular camera provided by this invention includes: S1: The calibration board containing feature points is set one by one at multiple different positions within the field of view of the lidar and monocular camera. The lidar and monocular camera are used to collect the original point cloud data and original image data corresponding to each position. The original point cloud data and original image data are preprocessed to obtain the original point cloud data and image data. S2: Using image data, obtain the pixel coordinates of feature points in the monocular camera coordinate system; using point cloud data, obtain the 3D coordinates of feature points in the lidar coordinate system. S3: Utilize an ultrasonic sensor array to measure ultrasonic data corresponding to multiple ranging points on the surface of the calibration plate; based on the ultrasonic data, fit the three-dimensional coordinates of the multiple ranging points into a three-dimensional reference plane. The ray equation of the feature point in the monocular camera coordinate system is calculated based on the pixel coordinates and the intrinsic parameters of the monocular camera; the coordinates of the intersection point of the ray equation and the three-dimensional reference plane are obtained as the three-dimensional coordinates of the camera. S4: Construct a kd-tree to search for camera 3D coordinates and LiDAR 3D coordinates, forming corresponding point groups; calculate the Euclidean distance between the corresponding point groups, define the objective optimization function based on the Euclidean distance, construct a nonlinear optimization problem, and iteratively obtain the optimal rotation matrix. Translation vector Objective optimization function The expression is: ; in, Represents the regularization parameter. Indicates the number of feature points. express Any one of the feature points, This indicates the effective ranging value of the ultrasonic sensor. Represents any coordinate in the camera's 3D coordinate set. This represents any coordinate in the set of three-dimensional coordinates of the lidar.

[0006] Preferably, when vibration exists between the lidar and the monocular camera, the calibration method further includes: S5: Use IMU to obtain vibration data of lidar and monocular camera in real time, establish vibration displacement model based on vibration data, the vibration displacement model is used to reflect the displacement of lidar and monocular camera caused by vibration; obtain the translation vector of lidar relative to monocular camera based on vibration displacement model. S6: Calculate the change in rotation angle based on vibration data: S7: Based on the translation vector and rotation angle changes of the lidar relative to the monocular camera, the matrix and vector of the corrected extrinsic parameters are obtained using a Kalman filter: ; ; in, This indicates the change in rotation angle. This represents the translation vector of the lidar relative to the monocular camera. This represents the rotation matrix of the corrected extrinsic parameters. This represents the translation vector of the corrected extrinsic parameters.

[0007] Preferably, the expression for the three-dimensional reference plane is: ; in, , , and All represent the coefficients of a linear equation. Represents three-dimensional coordinates; The expression for the ray equation is: , in, Represents ray parameters, This represents the direction vector of the ray.

[0008] The preferred expression for minimizing the Euclidean distance is: .

[0009] Preferably, the vibration data includes triaxial acceleration and triaxial angular velocity.

[0010] Preferably, the expression for the vibration displacement model is: ; in, Indicates amplitude. Indicates the phase angle. Indicates the vibration frequency. It represents the vibration displacement in all directions.

[0011] Preferably, the expression for the lidar relative to the monocular camera is: : , in, , , These represent the vibration displacements of the lidar in each axial direction. , , These represent the vibration displacements of the monocular camera along each axis. express Change in direction of translation express Change in direction of translation express Change in directional translation.

[0012] Preferably, the change in rotation angle The expression is: , in, Indicates circling Change in axis rotation Indicates circling Change in axis rotation Indicates circling Change in axis rotation.

[0013] A calibration system for joint calibration of lidar and monocular camera is provided to realize the calibration method for joint calibration of lidar and monocular camera. The calibration system includes: a data acquisition module, a feature point extraction module, an ultrasonic fitting module, and a joint calibration module. The data acquisition module is used to acquire image data, point cloud data, and ultrasonic data. The feature point extraction module is used to extract feature points from image data and point cloud data, and obtain pixel coordinates or LiDAR 3D coordinates in the corresponding coordinate system; The ultrasonic fitting module is used to generate a three-dimensional reference plane and calculate the camera's three-dimensional coordinate system; The joint calibration module is used to obtain the relative conversion relationship between the monocular camera and the lidar.

[0014] Preferably, when the application environment of the lidar and monocular camera is subject to vibration, the calibration system also includes a detection and calibration module; The detection and calibration module is used to detect the displacement of the lidar and monocular camera caused by vibration, calculate the rotation matrix and translation vector of the corrected extrinsic parameters, and perform dynamic calibration compensation on the extrinsic parameters.

[0015] Compared with the prior art, the present invention can achieve the following beneficial effects: This invention employs an ultrasonic sensor for depth compensation, which is significantly less expensive than binocular cameras or depth cameras. Ultrasonic sensors are inexpensive and highly reliable, making them suitable for large-scale industrial applications. Furthermore, by introducing an ultrasonic sensor array to provide depth compensation, the two-dimensional pixel coordinates of a monocular camera are converted into three-dimensional spatial coordinates, effectively solving the problem of missing depth information in monocular cameras. This provides sufficient constraints for LiDAR-camera calibration, significantly improving calibration accuracy and stability. In addition, this invention integrates multi-sensor data and proposes a dynamic calibration method based on a vibration model and Kalman filtering to address vibration issues in the application environment. This method can correct calibration parameters in real time, ensuring the accuracy and stability of calibration parameters in dynamic environments. Attached Figure Description

[0016] Figure 1 This is a flowchart of a calibration method for joint calibration of a lidar and a monocular camera provided according to an embodiment of the present invention; Figure 2 This is a schematic diagram of coordinate system transformation provided according to an embodiment of the present invention; Figure 3 This is an example installation diagram of a lidar, monocular camera, and ultrasonic sensor provided according to an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and do not constitute a limitation thereof. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined to form various implementations. Furthermore, the order of the steps or actions in the method description can be changed or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various orders in the specification and drawings are merely for the clear description of a particular embodiment and do not imply a mandatory order, unless otherwise stated that a particular order must be followed.

[0019] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] Some types of special-purpose vehicles are equipped with multiple sensors, including lidar and monocular cameras. The lidar and monocular cameras require joint calibration before use. For example... Figure 1 As shown, this embodiment of the invention takes a tracked vehicle as an example and provides a calibration method for joint calibration of a lidar and a monocular camera. First, static calibration is performed. Static calibration refers to calibration performed when both the lidar and the monocular camera are in a state free from external vibrations or other interference, including: S1: The calibration board containing feature points is set one by one at multiple different positions within the field of view of the lidar and monocular camera. The lidar and monocular camera are used to collect the original point cloud data and original image data corresponding to each position. The original point cloud data and original image data are preprocessed to obtain the original point cloud data and image data.

[0023] Before joint calibration, relevant data is acquired using the lidar and monocular camera to be calibrated (hereinafter referred to as lidar and monocular camera). Specifically, a calibration board is prepared, which must contain feature points that can be quickly, accurately, and robustly identified; ideally, these points should have high contrast and be precisely located. Multiple feature points are selected on the calibration board, and then the lidar and monocular camera are used to acquire the raw point cloud data and raw image data of the calibration board, respectively. When acquiring the raw point cloud data and raw image data, the calibration board is placed sequentially at multiple different locations within the field of view of the lidar and monocular camera, acquiring a set of raw point cloud data and raw image data at each location from different angles and orientations. During acquisition, the acquisition command is triggered by the PPS external trigger edge to ensure that the timestamps of the acquired raw point cloud data and raw image data are aligned. In this embodiment of the invention, raw point cloud data is collected, and the calibration board is set at close, medium and far distances from the lidar, respectively. When collecting raw image data, the calibration board is set at the left, middle and right positions of the monocular camera's viewpoint to collect multiple sets of images.

[0024] The original point cloud data and original image data are preprocessed to obtain point cloud data and image data, respectively. Preprocessing methods include, but are not limited to, image denoising and point cloud filtering. This embodiment of the invention also removes outliers from the point cloud data to reduce subsequent computation; and blurs the image data to reduce useless pixels.

[0025] In particular, the camera intrinsics of a monocular camera are known data.

[0026] S2: Using image data, obtain the pixel coordinates of feature points in the monocular camera coordinate system; using point cloud data, obtain the 3D coordinates of feature points in the lidar coordinate system.

[0027] The acquired image data includes feature points, and the pixel coordinates corresponding to these feature points in the monocular camera coordinate system are obtained. There are various methods for obtaining pixel coordinates from an image. This embodiment of the invention utilizes the built-in algorithms of the open-source library OpenCV or deep learning networks to extract feature points from the image data, performs sub-pixel precision optimization on the detected feature points, and refines the pixels to ultimately obtain... The set of pixel coordinates of feature points is represented as ,in, This represents the x-coordinate of each feature point in the pixel coordinate system. This represents the ordinate of each feature point in the pixel coordinate system. express Any one of the feature points.

[0028] like Figure 2 As shown, planar or edge features of the calibration board are extracted from point cloud data to obtain the three-dimensional coordinates of the feature points in the lidar coordinate system. In this embodiment, these are referred to as lidar three-dimensional coordinates. The set of lidar three-dimensional coordinates is represented as... In this embodiment of the invention, the RANSAC algorithm is used to detect planar features in point cloud data, identify the plane where the calibration board is located, and the number of iterations is set to 1000, with the interior point distance set to 0.02 meters. On the detected plane, edge feature points of the calibration board or the center point of the plane are extracted, and the point cloud data is clustered to group the points within the plane. For each cluster, the centroid is calculated as a feature point or the corner points of the plane boundary are extracted as feature points, ultimately obtaining... The set of three-dimensional coordinates of a feature point in the lidar coordinate system is represented as follows: . Figure 2 middle This represents the transformation matrix from lidar to the monocular camera, used to transform the lidar coordinate system. Transform the points below to the monocular camera coordinate system . This indicates the X, Y, and Z axes corresponding to the lidar coordinate system. These represent the X-axis, Y-axis, and Z-axis corresponding to the monocular camera coordinate system, respectively.

[0029] S3: Measure ultrasonic data from multiple ranging points on the surface of the calibration plate using an ultrasonic sensor array; fit the multiple ranging points into a three-dimensional reference plane based on the ultrasonic data; calculate the ray equation of the feature points in the monocular camera coordinate system based on the pixel coordinates and the intrinsic parameters of the monocular camera; obtain the coordinates of the intersection of the ray equation and the three-dimensional reference plane as the camera's three-dimensional coordinates.

[0030] Since image data acquired by a monocular camera lacks depth information, this embodiment of the invention employs ultrasonic waves for depth compensation. An ultrasonic sensor array is used to measure ultrasonic data corresponding to multiple ranging points on the calibration plate surface to characterize depth information. The obtained ultrasonic data requires preprocessing; this embodiment performs de-processing on the ultrasonic data. This embodiment calculates depth anchor points by randomly selecting 50 ultrasonic data points. These ultrasonic data include the distance to the ranging points measured by the ultrasonic waves, the installation position of the ultrasonic sensor array in the monocular camera coordinate system, and the unit vector of the measurement direction. The three-dimensional coordinates of the ranging points in the monocular camera coordinate system are calculated, i.e., the depth anchor points. In this embodiment, after measuring the distance to the calibration plate surface, the depth information is processed by median filtering with a window size of 5 to remove outliers; the effective ranging value of each ultrasonic sensor is recorded. and its installation position in the monocular camera coordinate system and measurement direction unit vector Calculate the three-dimensional coordinates of the ranging point on the calibration plate in the monocular camera coordinate system. : , In an embodiment of the present invention, , The value is set to 50, which means we get 50 depth anchor points. The coordinates of these 50 depth anchor points are all based on the camera coordinate system.

[0031] The ranging points acquired by the ultrasonic sensor array can form a three-dimensional reference plane. In this embodiment of the invention, the RANSAC (Random Sample Consensus) algorithm is used to fit the three-dimensional coordinates of the ranging points in the monocular camera coordinate system to a three-dimensional reference plane. The fitting process is as follows: First, establish a linear equation, expressed as: , in, , , and All represent the coefficients of a linear equation. Represents three-dimensional coordinates.

[0032] During the RANSAC iteration process, after randomly sampling three ranging points, a 3D reference plane is fitted using Singular Value Decomposition (SVD), and interior points are counted. Based on the optimal set of interior points, SVD is used again to accurately fit the final 3D reference plane. The specific SVD solution process is as follows: The above linear equation can be written in matrix form as follows: , Given three random distance measurement points, the equation problem can be transformed into a solution problem. The least squares solution.

[0033] After obtaining the 3D reference plane, the ray equation of the feature point in the monocular camera coordinate system is calculated based on the pixel coordinates of the feature point in the monocular camera coordinate system and the intrinsic parameters of the monocular camera: , in, Represents ray parameters, This represents the direction vector of the ray. The coordinates of the intersection point between the ray equation and the three-dimensional reference plane are obtained as the camera's three-dimensional coordinates. In this embodiment of the invention, the set of camera three-dimensional coordinates is represented as follows: .

[0034] S4: Construct a kd-tree to search for camera 3D coordinates and LiDAR 3D coordinates, forming corresponding point groups; calculate the Euclidean distance between the corresponding point groups, define the objective optimization function based on the Euclidean distance, construct a nonlinear optimization problem, and iteratively obtain the optimal rotation matrix. Translation vector .

[0035] Obtaining the 3D coordinates of the camera and the LiDAR allows the calibration problem to be transformed into an ICP (3D-3D) problem for extrinsic parameter calibration. The number of obtained camera and LiDAR 3D coordinates may not be the same, and they may not overlap one-to-one. Therefore, it is necessary to first establish the correspondence between the camera and LiDAR 3D coordinates. This embodiment of the invention constructs a kd-tree to perform accelerated nearest neighbor search, extracting the nearest points from both the camera and LiDAR 3D coordinate sets to form corresponding point groups. The Euclidean distance between the camera and LiDAR 3D coordinates in each corresponding point group is calculated, and the Euclidean distance is minimized. The corresponding expression is as follows: , Singular value decomposition (SVD) is used to estimate the initial extrinsic parameter transformation. Specifically, the centroid is first calculated and then decentroided. Based on this, the covariance matrix is ​​constructed, and then singular value decomposition is performed to obtain the initial extrinsic parameter transformation. and Among these, centroid calculation and centroid removal are common operations for reducing computational load, and will not be elaborated further here.

[0036] Because the depth information provided by ultrasound has errors, a depth constraint is introduced when constructing the objective optimization function. The expression of the constructed objective optimization function is as follows: , in, Denotes the objective optimization function. Represents the regularization parameter. Represents any coordinate in the camera's 3D coordinate set. This represents any coordinate in the set of three-dimensional coordinates of the lidar.

[0037] In the objective function, the first term is the geometric error, which is the Euclidean distance of the corresponding point group. The second term is the depth constraint, which ensures that the distance from the feature point in the monocular camera coordinate system to the monocular camera is consistent with the ultrasonic ranging value. The depth constraint corresponds to the objective optimization function. .

[0038] Construct a nonlinear optimization problem, perform nonlinear optimization using Levenberg-Marquardt method, and transform the initial extrinsic parameters obtained by SVD. As a guiding term, the optimal rotation matrix can then be obtained iteratively. Translation vector Rotation matrix Translation vector Express the correspondence between the extrinsic parameters of the monocular camera and the lidar, and obtain the rotation matrix. Translation vector Joint calibration completed.

[0039] When the application environment of the monocular camera and lidar is subject to vibration, causing them to vibrate—for example, in this embodiment of the invention, the lidar and monocular camera are used on a tracked vehicle—they will vibrate in sync with the vibration of the tracked vehicle. In this case, it is necessary to calibrate and compensate the results of the joint calibration. The specific method is as follows: S5: Use IMU to obtain vibration data of LiDAR and monocular camera in real time, establish vibration displacement model based on vibration data, the vibration displacement model is used to reflect the displacement of LiDAR and monocular camera caused by vibration; obtain the translation vector of LiDAR relative to monocular camera based on vibration displacement model.

[0040] This embodiment of the invention uses two IMUs (Inertial Measurement Units), which are rigidly connected to a lidar and a monocular camera, respectively. First, the IMUs acquire vibration data in real time, including triaxial acceleration and triaxial angular velocity. The acquired vibration data is then denoised, including adjustments to the vibration frequency and inverse low-pass filtering. This embodiment uses two identical six-axis IMUs for vibration data acquisition. A Fast Fourier Transform is performed on the triaxial acceleration data to extract the amplitude and phase in each direction, establishing a vibration displacement model, the expression of which is as follows: , in, Indicates amplitude. Indicates the phase angle. Indicates the vibration frequency. It represents the vibration displacement in all directions.

[0041] Based on the vibration displacement model, the translation vector of the lidar relative to the monocular camera can be obtained as follows: , in, This represents the translation vector of the lidar relative to the monocular camera. , , These represent the vibration displacements of the lidar in each axial direction. , , These represent the vibration displacements of the monocular camera along each axis. express Change in direction of translation express Change in direction of translation express Change in directional translation.

[0042] S6: Calculate the change in rotation angle based on vibration data.

[0043] Integrating the triaxial angular velocities of the two IMUs yields the rotation angle, which is then interpolated to obtain the change in rotation angle around the corresponding axis. : , in, Indicates circling Change in axis rotation Indicates circling Change in axis rotation Indicates circling Change in axis rotation.

[0044] S7: Based on the translation vector and rotation angle changes of the lidar relative to the monocular camera, the matrix and vector of the corrected extrinsic parameters are obtained using a Kalman filter. Establish state vector : , The six elements in the state vector represent the changes in the extrinsic parameter transformation relationship between the lidar and the monocular camera.

[0045] A state transition equation is established to characterize the changes in the extrinsic parameters of the lidar and monocular camera predicted by the IMU. The expression of the state transition equation is as follows: , in, express The change in external parameters at time t, express time, express The moment before the moment, These represent correction coefficients used to correct the extrinsic parameter matrix. Represents the IMU input matrix. Indicates IMU measurement value, This represents process noise, which follows a normal distribution, i.e. The mean is 0 and the variance is 0. .

[0046] An observation equation is established to characterize the changes in the extrinsic parameters of the lidar and monocular camera as observed visually. The expression for the observation equation is: , in, This represents the reprojection error of static feature points. Represents the observation matrix. This represents observation noise, which follows a normal distribution, i.e. The mean is 0 and the variance is 0. .

[0047] Establish a Kalman filter based on the state vector from the Kalman filter. Extracting external parameter changes The vibration data from the IMU is treated as "prediction," and the reprojection error from static calibration is treated as "observation." After fusing the two, the corrected rotation matrix and translation vector of the extrinsic parameters are output in real time. , , in, This represents the rotation matrix of the corrected extrinsic parameters. This represents the translation vector of the corrected extrinsic parameters.

[0048] Based on the above-described calibration method for joint calibration of lidar and monocular camera, this embodiment of the invention also provides a calibration system for joint calibration of lidar and monocular camera to realize the above-described calibration method for joint calibration of lidar and monocular camera. The calibration system includes: a data acquisition module, a feature point extraction module, an ultrasonic fitting module, a joint calibration module, and a detection and calibration module.

[0049] The data acquisition module includes a monocular camera module, a lidar module, and an ultrasonic sensor array module (with at least six ultrasonic sensors, such as...). Figure 3 The ultrasonic sensors 1, 2, 3, 4, 5, and 6 are shown in the image. Figure 3 The diagram shows the installation of a monocular camera, LiDAR, and ultrasonic sensor. The data acquisition module is used to acquire image data, point cloud data, and ultrasonic data.

[0050] The feature point extraction module includes an image extraction submodule and a point cloud extraction submodule, which are used to extract feature points from image data and point cloud data, respectively, and obtain pixel coordinates or LiDAR 3D coordinates in the corresponding coordinate system. The feature point extraction module is compatible with deep learning for feature point extraction.

[0051] Based on the ultrasonic data transmitted by the ultrasonic sensor array, the ultrasonic fitting module is used to generate a three-dimensional reference plane and calculate the three-dimensional coordinate system of the camera, thereby compensating for the depth information of the image data measured by the monocular camera.

[0052] The joint calibration module solves a 3D-3D optimization problem based on the 3D reference plane fitted by the ultrasonic wave, the 3D coordinates of the camera, and the 3D coordinates of the lidar, in order to obtain the relative transformation relationship between the monocular camera and the lidar and perform static calibration.

[0053] When vibration occurs in the application environment of the lidar and monocular camera, the calibration system for joint calibration of lidar and monocular camera also includes a detection and calibration module. This module detects the displacement of the lidar and monocular camera caused by vibration, calculates the rotation matrix and translation vector of the corrected extrinsic parameters, and dynamically calibrates and compensates for the extrinsic parameters in real time when vibration exists between the monocular camera and lidar.

[0054] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0055] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A calibration method for joint calibration of lidar and monocular camera, characterized in that, include: S1: The calibration board containing feature points is set one by one at multiple different positions within the field of view of the lidar and the monocular camera. The lidar and the monocular camera are used to collect the original point cloud data and original image data corresponding to each position. The original point cloud data and original image data are preprocessed to obtain point cloud data and image data. S2: Using the image data, obtain the pixel coordinates of the feature points in the monocular camera coordinate system; using the point cloud data, obtain the three-dimensional coordinates of the feature points in the lidar coordinate system. S3: Measure the ultrasonic data corresponding to multiple ranging points on the surface of the calibration plate using an ultrasonic sensor array; fit the three-dimensional coordinates of the multiple ranging points into a three-dimensional reference plane based on the ultrasonic data. The ray equation of the feature point in the monocular camera coordinate system is calculated based on the pixel coordinates and the intrinsic parameters of the monocular camera; the coordinates of the intersection point of the ray equation and the three-dimensional reference plane are obtained as the three-dimensional coordinates of the camera. S4: Construct a kd-tree to search for the 3D coordinates of the camera and the 3D coordinates of the LiDAR, forming a corresponding point group; calculate the Euclidean distance of the corresponding point group, define the objective optimization function based on the Euclidean distance, construct a nonlinear optimization problem, and iteratively obtain the optimal rotation matrix. Translation vector The objective optimization function The expression is: ； in, Represents the regularization parameter. Indicates the number of feature points. express Any one of the feature points, This indicates the effective ranging value of the ultrasonic sensor. Represents any coordinate in the camera's 3D coordinate set. This represents any coordinate in the set of three-dimensional coordinates of the lidar.

2. The calibration method for joint calibration of lidar and monocular camera according to claim 1, characterized in that, When there is vibration between the lidar and the monocular camera, the calibration method also includes: S5: Use an IMU to obtain vibration data of the lidar and monocular camera in real time, establish a vibration displacement model based on the vibration data, and use the vibration displacement model to reflect the displacement of the lidar and monocular camera caused by vibration; obtain the translation vector of the lidar relative to the monocular camera based on the vibration displacement model. S6: Calculate the change in rotation angle based on vibration data: S7: Based on the translation vector of the lidar relative to the monocular camera and the change in the rotation angle, the matrix and vector of the corrected extrinsic parameters are obtained using a Kalman filter: ； ； in, This indicates the change in rotation angle. This represents the translation vector of the lidar relative to the monocular camera. This represents the rotation matrix of the corrected extrinsic parameters. This represents the translation vector of the corrected extrinsic parameters.

3. The calibration method for joint calibration of lidar and monocular camera according to claim 1, characterized in that, The expression for the three-dimensional reference plane is: ； in, , , and All represent the coefficients of a linear equation. Represents three-dimensional coordinates; The expression for the ray equation is: ， in, Represents ray parameters, This represents the direction vector of the ray.

4. The calibration method for joint calibration of lidar and monocular camera according to claim 1, characterized in that, The expression for minimizing the Euclidean distance is: 。 5. The calibration method for joint calibration of lidar and monocular camera according to claim 2, characterized in that, The vibration data includes triaxial acceleration and triaxial angular velocity.

6. The calibration method for joint calibration of lidar and monocular camera according to claim 2, characterized in that, The expression for the vibration displacement model is: ； in, Indicates amplitude. Indicates the phase angle. Indicates the vibration frequency. It represents the vibration displacement in all directions.

7. The calibration method for joint calibration of lidar and monocular camera according to claim 2, characterized in that, The expression for lidar relative to a monocular camera is: : ， in, , , These represent the vibration displacements of the lidar in each axial direction. , , These represent the vibration displacements of the monocular camera along each axis. express Change in direction of translation express Change in direction of translation express Change in directional translation.

8. The calibration method for joint calibration of lidar and monocular camera according to claim 2, characterized in that, The change in rotation angle The expression is: ， in, Indicates circling Change in axis rotation Indicates circling Change in axis rotation Indicates circling Change in axis rotation.

9. A calibration system for joint calibration of lidar and monocular camera, used to implement the calibration method for joint calibration of lidar and monocular camera as described in any one of claims 1-8, characterized in that, The calibration system includes: a data acquisition module, a feature point extraction module, an ultrasonic fitting module, and a joint calibration module; The data acquisition module is used to acquire image data, point cloud data, and ultrasonic data. The feature point extraction module is used to extract feature points from image data and point cloud data, and obtain pixel coordinates or LiDAR three-dimensional coordinates in the corresponding coordinate system. The ultrasonic fitting module is used to generate a three-dimensional reference plane and calculate the camera's three-dimensional coordinate system. The joint calibration module is used to obtain the relative conversion relationship between the monocular camera and the lidar.

10. The calibration system for joint calibration of lidar and monocular camera according to claim 9, characterized in that, When the application environment of lidar and monocular camera is subject to vibration, the calibration system also includes a detection and calibration module; The detection and calibration module is used to detect the displacement of the lidar and monocular camera caused by vibration, calculate the rotation matrix and translation vector of the corrected extrinsic parameters, and perform dynamic calibration compensation on the extrinsic parameters.

Images