A method and apparatus for joint calibration of camera intrinsic parameters and camera extrinsic parameters relative to lidar.

Abstract

This invention relates to the field of autonomous driving technology, specifically to a joint calibration method and apparatus for camera intrinsic parameters and camera extrinsic parameters relative to LiDAR. By using a combined galvanometer and a three-degree-of-freedom rotary table structure, the calibration depth is extended, the relative pose of the perception system with respect to the calibration board is precisely adjusted, the number of feature points is increased, and the calibration accuracy is improved. Using the aforementioned auxiliary calibration apparatus, the calibration process can be fully automated, saving labor costs and improving calibration efficiency. The auxiliary calibration apparatus can expand the calibration space and improve calibration accuracy in a limited space, meeting the needs of large-scale practical applications (such as autonomous driving scenarios) while reducing the requirements of the calibration environment and improving calibration accuracy.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and apparatus for joint calibration of camera intrinsic parameters and camera extrinsic parameters relative to lidar. Background Technology

[0002] In the fields of autonomous driving, robotics, and augmented / virtual reality (AR / VR), LiDAR and cameras are commonly used sensors. Each has its unique advantages, but also its drawbacks and limitations. To achieve multi-sensor fusion, an increasing number of solutions employ multi-camera and multi-LiDAR configurations. However, multi-sensor calibration is a prerequisite for the proper functioning of these systems. Currently, multimodal sensor calibration still faces several challenges, requiring significant manual intervention. The most common issue is the calibration between cameras and LiDAR. Therefore, extrinsic parameter calibration of cameras and LiDAR becomes crucial. By solving the coordinate transformation relationship between these two sensors, the fusion of camera imaging data and LiDAR point cloud data can be achieved.

[0003] Currently, existing camera and LiDAR calibration processes involve calibrating extrinsic parameters before the environmental perception system operates, and these parameters are not adjusted during system operation. Many mature camera and LiDAR calibration tools exist, such as Autoware for autonomous driving, the lidar_camera_calibration toolkit, and Baidu's Apollo system. Existing LiDAR systems can be divided into two categories: traditional mechanical LiDAR (such as Hesai) and emerging solid-state LiDAR (such as Livox). Traditional mechanical LiDAR is expensive (a 32-line LiDAR can cost hundreds of thousands of yuan) and requires a mechanical rotating structure to rotate the laser emitter. In contrast, solid-state LiDAR is relatively inexpensive (costing only a few thousand yuan). Furthermore, existing open-source calibration tools are primarily designed for traditional mechanical LiDAR and do not provide direct support for solid-state LiDAR, thus making them unsuitable for direct calibration. Current calibration methods involve attaching calibration information in a specific order indoors, taking a single photograph, analyzing the feature information, and performing feature matching to achieve feature calibration. For outdoor calibration, a single calibration board is typically moved manually or automatically to change its relative position to the calibration equipment. Feature matching from a large number of images, combined with calibration board size information, is then processed by an optimization function to obtain the camera's intrinsic parameters relative to the solid-state LiDAR. However, the former indoor calibration method requires deploying a large amount of calibration information in the scene, resulting in high material costs, significant manpower and resource consumption, and a lack of portability. The latter outdoor method, using a fully automatic or manually moving calibration board model, is time-consuming and labor-intensive for large-scale calibration; while the fully automatic calibration scheme achieves multi-angle shooting by moving the calibration board, each calibration requires multiple adjustments to the calibration board's angle and displacement, with large and frequent adjustments that severely impact equipment lifespan and increase costs.

[0004] Existing methods focus on checkerboard calibration or feature point matching calibration, and the typical application scenarios for LiDAR and cameras, such as autonomous driving, involve distances of hundreds of meters. However, traditional calibration methods often suffer from drawbacks in these scenarios, such as the calibration conditions not being suitable for the deployment environment and the sparse feature points of solid-state LiDAR at long distances. Furthermore, using multiple checkerboard calibration strategies requires large sites, while single checkerboard calibration is labor-intensive and unsuitable for production line calibration. Feature point matching calibration, on the other hand, has high requirements for the number of feature points in the scene and ambient lighting, and in scenes lacking texture and with high repetition, it leads to difficulties in feature extraction and matching, severely impacting calibration results. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a method and apparatus for joint calibration of camera intrinsic parameters and camera extrinsic parameters relative to lidar.

[0006] In a first aspect, embodiments of the present invention provide a joint calibration method for camera intrinsic parameters and camera extrinsic parameters relative to a lidar. A sensing system is pre-constructed, comprising a three-degree-of-freedom rotary table, a camera and a solid-state lidar mounted on the rotary table, and a combined galvanometer. The camera and the solid-state lidar are relatively fixed in position. The combined galvanometer illuminates a calibration plate with the beam emitted by the solid-state lidar and then returns to be received by the solid-state lidar. The method includes:

[0007] The corresponding calibration scheme is determined according to the type of calibration camera and solid-state lidar. The calibration scheme includes the pre-selection of the calibration board, the number of calibration acquisitions, and the setting of the calibration pose sequence of the sensing system relative to the calibration board.

[0008] The pose adjustment plan for the sensing system relative to the calibration board is based on the calibration scheme.

[0009] The lower-level computer is used to control the auxiliary calibration device and the sensing system to complete the pose adjustment plan, and to control the calibration camera and the solid-state lidar to collect calibration data;

[0010] The calibration data is subjected to feature matching and minimization iteration to obtain the intrinsic parameters of the camera relative to the extrinsic lidar.

[0011] As an optional approach, the calibration scheme is determined based on the type of calibration camera and solid-state LiDAR. The calibration scheme includes pre-selection of the calibration board, setting the number of calibration acquisitions, and the calibration pose sequence of the sensing system relative to the calibration board, including:

[0012] A corresponding calibration scheme is set according to the hardware characteristics of the calibration camera and the solid-state lidar. The hardware characteristics include the resolution of the camera, the field of view and pixel size of the camera, the detection range of the solid-state lidar, the field of view of the solid-state lidar, and the point cloud density and angular resolution of the solid-state lidar.

[0013] The selection of the calibration board, the number of calibration acquisitions, and the setting of the calibration pose sequence of the sensing system relative to the calibration board are determined.

[0014] As an optional solution, in the sensing system, the combined galvanometer is set to be parallel to each other at a vertical distance of 50m and at a 45° angle relative to the ground, and an auxiliary calibration device is placed there so that the beam of the solid-state lidar is projected onto the calibration plate and reflected back to be received by the solid-state lidar.

[0015] As an optional solution, the three-degree-of-freedom rotary table has multiple stepper motors, the combined galvanometer includes multiple sub-galvanometers, and the pose adjustment planning of the sensing system relative to the calibration plate according to the calibration scheme includes:

[0016] Based on the pattern features of the calibration plate, the pose adjustment plan of the sensing system relative to the calibration plate is planned, and the inverse kinematics solution of the pose adjustment plan is obtained by using the motion transfer equation to obtain the rotation parameters of each stepper motor and the placement position of the sub-mirror. The rotation parameters and the placement position are converted into the number of rotation steps of the stepper motor controlling the sub-mirror and the three-degree-of-freedom rotary table.

[0017] As an optional solution, the step of using a lower-level computer to control the auxiliary calibration device and the sensing system to complete the pose adjustment planning, and controlling the calibration camera and the solid-state lidar to collect calibration data, includes:

[0018] Based on the rotation steps obtained from the inverse kinematics, the lower-level machine sends pulse signals to drive the stepper motor to rotate. When the sensing system reaches the specified pose, the lower-level machine runs the driver program of the sensing system, uses the camera to capture two-dimensional images, and records the three-dimensional point cloud data of the solid-state lidar. The data is collected cyclically according to the calibration number of acquisitions until the calibration number of acquisitions is reached.

[0019] As an optional approach, the calibration data is subjected to feature matching and minimization iteration processing to obtain the camera's intrinsic parameters and extrinsic parameters relative to the solid-state lidar. This includes solving for the camera's intrinsic parameters and the extrinsic parameter matrix of the camera relative to the solid-state lidar. The specific steps are as follows:

[0020] The process of solving the camera intrinsic parameters includes:

[0021] Image processing is performed on the two-dimensional image in the calibration data, including grayscale conversion and threshold segmentation, to obtain the pixel coordinates corresponding to the feature points of the calibration board in the two-dimensional image. The correspondence between the pixels in the two-dimensional image and the real points is determined through the image attributes of the calibration board.

[0022] Based on the feature points and the correspondence between pixels and real points, and substituting the positional relationships between pixels into the initial camera intrinsic parameter matrix, we obtain the predicted positional relationship L_1 between real points. The distance error is obtained by subtracting the predicted positional relationship L_1 from the actual point-to-point positional relationship L.

[0023] ΔL1=L-L_1

[0024] Extending the distance error to the distance relationships between all n types of features yields the error equation constructed by deriving the camera intrinsic parameter matrix:

[0025]

[0026] Where e1 represents the average of the squares of the errors between the distances of n feature points and the distances of the real points. The camera intrinsic parameter matrix can be obtained by using the Ceres library to minimize the distance error through iterative processing.

[0027] The process of solving the extrinsic parameter matrix of the camera relative to the solid-state lidar:

[0028] Given the intrinsic parameter matrices of the camera and the solid-state lidar, the camera acquires feature points in the two-dimensional image. Multiplying the intrinsic parameter matrix of the world coordinate system relative to the camera coordinate system by the matrix maps the points in the world coordinate system. Fix feature points in the three-dimensional point cloud data of the solid-state lidar The intrinsic parameter matrix of the world coordinate system relative to the solid-state lidar coordinate system is mapped to points in the world coordinate system.

[0029] The extrinsic parameter matrix of the camera coordinate system relative to the solid-state LiDAR coordinate system is initialized, and the error equation is constructed based on the distance error between the matching feature point pairs as follows:

[0030]

[0031] In the formula, k represents the number of feature points that are matched. This represents a point in the camera's world coordinate system. This indicates that the points in the solid-state lidar coordinate system The external parameter matrix of the solid-state lidar coordinate system relative to the camera Convert the coordinates of the points into the camera coordinate system, accumulate the squared distances of the above-mentioned two matched points, and then average them to obtain the distance error e2;

[0032] The minimization operation is performed, and the specific formula for the minimization operation is as follows:

[0033]

[0034] The minimization operation formula is transformed into a least squares problem, and the optimal extrinsic parameter coefficients are obtained through iterative optimization using the Ceres library.

[0035] Secondly, this invention provides a joint calibration device for camera-in-camera relative to lidar extrinsic parameters, applied to the aforementioned joint calibration method for camera-in-camera relative to lidar extrinsic parameters. The device includes a sensing system and a calibration plate. The system comprises a three-degree-of-freedom rotary table, a camera and a solid-state lidar mounted on the rotary table, and a combined galvanometer. The camera and the solid-state lidar are relatively fixed in position. The combined galvanometer illuminates the calibration plate with a beam emitted by the solid-state lidar, which then returns to be received by the solid-state lidar.

[0036] As an optional solution, the three-degree-of-freedom rotary table includes a first stepper motor, a second stepper motor, a third stepper motor, and a fixed support. The first stepper motor is used for the fixed support to rotate around the X-axis, the second stepper motor is used for the fixed support to rotate around the Y-axis, and the third stepper motor is used for the fixed support to rotate around the Z-axis. The camera and the solid-state lidar are mounted on the fixed support.

[0037] As an optional solution, the combined galvanometer includes a first sub-galvanometer, a second sub-galvanometer, a fourth stepper motor, a fifth stepper motor, and a support frame. The first sub-galvanometer is mounted on the support frame via the fourth stepper motor, and the second sub-galvanometer is mounted on the support frame via the fifth stepper motor.

[0038] As an alternative, the calibration board can be calibrated using a checkerboard pattern or Gray code.

[0039] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0040] This invention provides a method and apparatus for joint calibration of camera intrinsic parameters and camera extrinsic parameters relative to a LiDAR. First, a suitable calibration method is formulated based on the type of calibration camera and solid-state LiDAR. Second, according to the designed calibration scheme, a pose adjustment plan for the sensing system relative to the calibration board is made. The pose adjustment plan is then solved using inverse kinematics to obtain the rotation parameters of each stepper motor and the position of the sub-mirror. This is then converted into the number of rotation steps for the stepper motors controlling the sub-mirror and the three-degree-of-freedom rotary table. While adjusting the stepper motor rotation, the lower-level computer runs the driver program, the camera captures two-dimensional images, and the three-dimensional point cloud data of the solid-state LiDAR is recorded. The 3D image and 3D point cloud data are used as calibration data. The above acquisition process is repeated to achieve automated acquisition of calibration data. Finally, the acquired calibration data is processed, including feature extraction and matching, and iterative processing to minimize reprojection error. Ultimately, the intrinsic parameters of the camera relative to the extrinsic parameters of the solid-state LiDAR are obtained. In this embodiment, by using a combined galvanometer and a three-degree-of-freedom rotary table structure, the calibration depth is extended, the relative pose of the sensing system with respect to the calibration board is precisely adjusted, the number of feature points is increased, and the calibration accuracy is improved. Using the above-mentioned auxiliary calibration device, the calibration process can be fully automated, saving labor costs and improving calibration efficiency. The auxiliary calibration device can expand the calibration space and improve calibration accuracy in a limited space, meeting the needs of large-scale practical applications (such as autonomous driving scenarios) while reducing the requirements of the calibration environment and improving calibration accuracy. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating a method for joint calibration of camera intrinsic parameters and camera relative lidar extrinsic parameters according to an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the structure of a joint calibration device for camera internal parameters and camera relative lidar external parameters provided in an embodiment of the present invention. Detailed Implementation

[0043] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0045] Combination Figure 1As shown, this embodiment of the invention provides a joint calibration method for camera intrinsic parameters and camera extrinsic parameters relative to a lidar. A sensing system is pre-constructed, comprising a three-degree-of-freedom rotary table, a camera and a solid-state lidar mounted on the rotary table, and a combined galvanometer. The camera and the solid-state lidar are relatively fixed in position. The combined galvanometer illuminates a calibration plate with the beam emitted by the solid-state lidar and then returns to be received by the solid-state lidar. The method includes:

[0046] S101. Determine the corresponding calibration scheme according to the type of calibration camera and solid-state lidar. The calibration scheme includes the pre-selection of the calibration board, the number of calibration acquisitions, and the setting of the calibration pose sequence of the sensing system relative to the calibration board.

[0047] Based on the hardware characteristics of the calibration camera and the solid-state lidar, a corresponding calibration scheme is set. The hardware characteristics include the resolution of the camera, the influence of the camera's field of view and pixel size, the detection range of the solid-state lidar, the field of view of the solid-state lidar, the point cloud density and angular resolution of the solid-state lidar, and the selection of the calibration board, the number of calibration acquisitions, and the setting of the calibration pose sequence of the sensing system relative to the calibration board.

[0048] In the sensing system, the combined galvanometer is set to be parallel to each other at a vertical distance of 50m and at a 45° angle relative to the ground. An auxiliary calibration device is placed so that the beam of the solid-state lidar is projected onto the calibration plate and reflected back to be received by the solid-state lidar.

[0049] S102. Plan the pose adjustment of the sensing system relative to the calibration board according to the calibration scheme.

[0050] Based on the pattern features of the calibration plate, the pose adjustment plan of the sensing system relative to the calibration plate is planned, and the inverse kinematics solution of the pose adjustment plan is obtained by using the motion transfer equation to obtain the rotation parameters of each stepper motor and the placement position of the sub-mirror. The rotation parameters and the placement position are converted into the number of rotation steps of the stepper motor controlling the sub-mirror and the three-degree-of-freedom rotary table.

[0051] S103. The lower-level computer is used to control the auxiliary calibration device and the sensing system to complete the pose adjustment plan, and the calibration camera and the solid-state lidar are controlled to collect calibration data.

[0052] Based on the rotation steps obtained from the inverse kinematics, the lower-level machine sends pulse signals to drive the stepper motor to rotate. When the sensing system reaches the specified pose, the lower-level machine runs the driver program of the sensing system, uses the camera to capture two-dimensional images, and records the three-dimensional point cloud data of the solid-state lidar. The data is collected cyclically according to the calibration number of acquisitions until the calibration number of acquisitions is reached.

[0053] S104. Perform feature matching and minimization iteration processing on the calibration data to obtain the extrinsic parameters of the camera intrinsic parameters relative to the solid-state lidar.

[0054] This includes solving for the camera's intrinsic parameters and the extrinsic parameter matrix of the camera relative to the solid-state LiDAR. The specific steps are as follows:

[0055] The process of solving the camera intrinsic parameters includes:

[0056] Image processing is performed on the two-dimensional image in the calibration data, including grayscale conversion and threshold segmentation, to obtain the pixel coordinates corresponding to the feature points of the calibration board in the two-dimensional image. The correspondence between the pixels in the two-dimensional image and the real points is determined through the image attributes of the calibration board.

[0057] Based on the feature points and the correspondence between pixels and real points, and substituting the positional relationships between pixels into the initial camera intrinsic parameter matrix, we obtain the predicted positional relationship L_1 between real points. The distance error is obtained by subtracting the predicted positional relationship L_1 from the actual point-to-point positional relationship L.

[0058] ΔL1=L-L_1

[0059] Extending the distance error to the distance relationships between all n types of features yields the error equation constructed by deriving the camera intrinsic parameter matrix:

[0060]

[0061] Where e1 represents the average of the squares of the errors between the distances of n feature points and the distances of the real points. The camera intrinsic parameter matrix can be obtained by using the Ceres library to minimize the distance error through iterative processing.

[0062] The process of solving the extrinsic parameter matrix of the camera relative to the solid-state lidar:

[0063] Given the intrinsic parameter matrices of the camera and the solid-state lidar, the camera acquires feature points in the two-dimensional image. Multiplying the intrinsic parameter matrix of the world coordinate system relative to the camera coordinate system by the matrix maps the points in the world coordinate system. Fix feature points in the three-dimensional point cloud data of the solid-state lidar The intrinsic parameter matrix of the world coordinate system relative to the solid-state lidar coordinate system is mapped to points in the world coordinate system.

[0064] The extrinsic parameter matrix of the camera coordinate system relative to the solid-state LiDAR coordinate system is initialized, and the error equation is constructed based on the distance error between the matching feature point pairs as follows:

[0065]

[0066] In the formula, k represents the number of feature points that are matched. This represents a point in the camera's world coordinate system. This indicates that the points in the solid-state lidar coordinate system The external parameter matrix of the solid-state lidar coordinate system relative to the camera Convert the coordinates of the points into the camera coordinate system, accumulate the squared distances of the above-mentioned two matched points, and then average them to obtain the distance error e2;

[0067] The minimization operation is performed, and the specific formula for the minimization operation is as follows:

[0068]

[0069] The minimization operation formula is transformed into a least squares problem, and the optimal extrinsic parameter coefficients are obtained through iterative optimization using the Ceres library. This system can instantly calibrate the extrinsic parameters of a sensing system consisting of a camera and a solid-state LiDAR, as well as the camera's intrinsic parameters. This includes the automated acquisition of calibration data, which consists of 2D image data captured by the camera and 3D point cloud data received by the solid-state LiDAR. The entire calibration process is fully automated and unrestricted by location.

[0070] This invention provides a joint calibration method for camera intrinsic parameters and camera extrinsic parameters relative to LiDAR. First, a suitable calibration method is formulated based on the type of calibration camera and solid-state LiDAR. Second, according to the designed calibration scheme, a pose adjustment plan for the sensing system relative to the calibration board is made. The pose adjustment plan is then solved using inverse kinematics to obtain the rotation parameters of each stepper motor and the position of the sub-mirror. This is then converted into the number of rotation steps for the stepper motors controlling the sub-mirror and the three-degree-of-freedom rotary table. While adjusting the stepper motor rotation, the lower-level computer runs the driver program, the camera captures two-dimensional images, and the three-dimensional point cloud data of the solid-state LiDAR is recorded. The two-dimensional images are then processed... Image and 3D point cloud data are used as calibration data. The above acquisition process is repeated to achieve automated acquisition of calibration data. Finally, the acquired calibration data is processed, feature extraction and matching are performed, and iterative processing to minimize reprojection error is considered to finally obtain the intrinsic parameters of the camera relative to the extrinsic LiDAR. In this embodiment of the invention, by using a combined galvanometer and a three-degree-of-freedom rotary table structure, the calibration depth is extended, the relative pose of the perception system with respect to the calibration board is precisely adjusted, the number of feature points is increased, and the calibration accuracy is improved. Using the above-mentioned auxiliary calibration device, the calibration process can be fully automated, saving labor costs and improving calibration efficiency. The auxiliary calibration device can expand the calibration space and improve the calibration accuracy in a limited space. While meeting the needs of large-scale practical applications (such as autonomous driving scenarios), it reduces the requirements of the calibration environment and improves the calibration accuracy.

[0071] Combination Figure 2 As shown, this embodiment of the invention provides a joint calibration device for camera-in-camera relative to lidar extrinsic parameters, applied to the aforementioned joint calibration method for camera-in-camera relative to lidar extrinsic parameters. It includes a sensing system and a calibration plate 21. The system includes a three-degree-of-freedom rotary table 20, a camera 22 and a solid-state lidar 23 mounted on the three-degree-of-freedom rotary table 20, and a combined galvanometer 30. The positions of the camera 22 and the solid-state lidar 23 are relatively fixed. The combined galvanometer 30 illuminates the calibration plate 21 with the beam emitted by the solid-state lidar 23, and the beam returns to be received by the solid-state lidar 23.

[0072] In some embodiments, the three-degree-of-freedom rotary table 20 includes a first stepper motor 24, a second stepper motor 25, a third stepper motor 26, and a fixed support 27. The first stepper motor 24 is used for the fixed support 27 to rotate around the X-axis, the second stepper motor 24 is used for the fixed support to rotate around the Y-axis, and the third stepper motor is used for the fixed support 27 to rotate around the Z-axis. The camera 22 and the solid-state lidar 23 are mounted on the fixed support 27. Specifically, the first stepper motor 24, the second stepper motor 25, and the third stepper motor 26 are all high-precision stepper motors. Combined with differential microstepping drive, the static accuracy of each step can reach step angle* (±5%). The pulse signal output by the controller (not shown in the figure) sequentially triggers the driver (not shown in the figure). Under the action of the driver, the three stepper motors are controlled to rotate precisely by a preset angle. Three stepper motors move simultaneously, enabling the rotating fixed bracket to rotate around the X-axis, Y-axis, and Z-axis. The camera and solid-state LiDAR 23 are connected by fasteners (not shown in the figure), and finally connected to the fixed bracket 27 by screws and nuts (not shown in the figure).

[0073] In some embodiments, the combined galvanometer 30 includes a first sub-galvanometer 31, a second sub-galvanometer 32, a fourth stepper motor (not shown in the figure), a fifth stepper motor (not shown in the figure), and a support frame 33. The first sub-galvanometer 31 is mounted on the support frame 33 via the fourth stepper motor, and the second sub-galvanometer 32 is mounted on the support frame 33 via the fifth stepper motor.

[0074] In some embodiments, the calibration plate 21 is calibrated using a checkerboard or Gray code. The calibration plate 21 is placed and fixed parallel to the front end face of the auxiliary calibration device and perpendicular to the side end face. By precisely adjusting the laser beam pose of the solid-state lidar 23, three-dimensional point cloud data and two-dimensional images are collected and processed to obtain calibration feature points.

[0075] This invention provides a joint calibration device for camera-internal parameters and camera-relative lidar extrinsic parameters. By using a combined galvanometer and a three-degree-of-freedom rotary table structure, the calibration depth is extended, the relative pose of the sensing system with respect to the calibration board is precisely adjusted, the number of feature points is increased, and the calibration accuracy is improved. Using the above-mentioned auxiliary calibration device, the calibration process can be fully automated, saving labor costs and improving calibration efficiency. The auxiliary calibration device can expand the calibration space and improve calibration accuracy in a limited space, meeting the needs of large-scale practical applications (such as autonomous driving scenarios) while reducing the requirements of the calibration environment and improving calibration accuracy.

[0076] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0077] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for joint calibration of camera extrinsic parameters relative to a laser radar, characterized in that, A pre-constructed sensing system includes a three-degree-of-freedom rotary table, a camera mounted on the rotary table, a solid-state lidar, and a combined galvanometer. The camera and the solid-state lidar are relatively fixed in position. The combined galvanometer illuminates a calibration plate with the beam emitted by the solid-state lidar, and the beam returns to be received by the solid-state lidar. The system includes: The corresponding calibration scheme is determined according to the type of calibration camera and solid-state lidar. The calibration scheme includes the pre-selection of the calibration board, the number of calibration acquisitions, and the setting of the calibration pose sequence of the sensing system relative to the calibration board. The pose adjustment plan for the sensing system relative to the calibration board is based on the calibration scheme. The lower-level computer is used to control the auxiliary calibration device and the sensing system to complete the pose adjustment plan, and to control the calibration camera and the solid-state lidar to collect calibration data; The calibration data is subjected to feature matching and minimization iteration to obtain the intrinsic parameters of the camera relative to the extrinsic lidar.

2. The method of claim 1, wherein, The calibration scheme is determined based on the type of calibration camera and solid-state lidar. The calibration scheme includes pre-selection of the calibration board, setting the number of calibration acquisitions, and the calibration pose sequence of the sensing system relative to the calibration board. A corresponding calibration scheme is set according to the hardware characteristics of the calibration camera and the solid-state lidar. The hardware characteristics include the resolution of the camera, the field of view and pixel size of the camera, the detection range of the solid-state lidar, the field of view of the solid-state lidar, and the point cloud density and angular resolution of the solid-state lidar. The selection of the calibration board, the number of calibration acquisitions, and the setting of the calibration pose sequence of the sensing system relative to the calibration board are determined.

3. The method of claim 2, wherein, In the sensing system, the combined galvanometer is set to be parallel to each other at a vertical distance of 50m and at a 45° angle relative to the ground. An auxiliary calibration device is placed so that the beam of the solid-state lidar is projected onto the calibration plate and reflected back to be received by the solid-state lidar.

4. The method of claim 2, wherein, The three-degree-of-freedom rotary table has multiple stepper motors, the combined galvanometer includes multiple sub-galvanometers, and the pose adjustment planning of the sensing system relative to the calibration plate according to the calibration scheme includes: Based on the pattern features of the calibration plate, the pose adjustment plan of the sensing system relative to the calibration plate is planned, and the inverse kinematics solution of the pose adjustment plan is obtained by using the motion transfer equation to obtain the rotation parameters of each stepper motor and the placement position of the sub-mirror. The rotation parameters and the placement position are converted into the number of rotation steps of the stepper motor controlling the sub-mirror and the three-degree-of-freedom rotary table.

5. The method for joint calibration of camera intrinsic parameters and camera relative lidar extrinsic parameters according to claim 4, characterized in that, The process of using a lower-level computer to control the auxiliary calibration device and the sensing system to complete the pose adjustment planning, and controlling the calibration camera and the solid-state lidar to collect calibration data, includes: Based on the rotation steps obtained from the inverse kinematics, the lower-level machine sends pulse signals to drive the stepper motor to rotate. When the sensing system reaches the specified pose, the lower-level machine runs the driver program of the sensing system, uses the camera to capture two-dimensional images, and records the three-dimensional point cloud data of the solid-state lidar. The data is collected cyclically according to the calibration number of acquisitions until the calibration number of acquisitions is reached.

6. The method for joint calibration of camera intrinsic parameters and camera relative lidar extrinsic parameters according to claim 5, characterized in that, The calibration data is subjected to feature matching and minimization iteration processing to obtain the camera's intrinsic parameters and extrinsic parameters relative to the solid-state lidar. This includes solving for the camera's intrinsic parameters and solving for the camera's extrinsic parameter matrix relative to the solid-state lidar. The specific steps are as follows: The process of solving the camera intrinsic parameters includes: Image processing is performed on the two-dimensional image in the calibration data, including grayscale conversion and threshold segmentation, to obtain the pixel coordinates corresponding to the feature points of the calibration board in the two-dimensional image. The correspondence between the pixels in the two-dimensional image and the real points is determined through the image attributes of the calibration board. Based on the feature points and the correspondence between pixels and real points, and substituting the positional relationships between pixels into the initial camera intrinsic parameter matrix, we obtain the predicted positional relationship L_1 between real points. The distance error is obtained by subtracting the predicted positional relationship L_1 from the actual point-to-point positional relationship L. Extending the distance error to the distance relationships between all n types of features yields the error equation constructed by deriving the camera intrinsic parameter matrix: in It is expressed as the average of the squares of the errors between the distances of n feature points and the distances of the real points. The camera intrinsic parameter matrix can be obtained by using the Ceres library to perform a minimum iteration process on the distance error. The process of solving the extrinsic parameter matrix of the camera relative to the solid-state lidar: Given the intrinsic parameter matrices of the camera and the solid-state lidar, the camera acquires feature points in the two-dimensional image. Multiplying the intrinsic parameter matrix of the world coordinate system relative to the camera coordinate system by the matrix maps the points in the world coordinate system. Fixed feature points in the three-dimensional point cloud data of the solid-state lidar The intrinsic parameter matrix of the world coordinate system relative to the solid-state lidar coordinate system is mapped to points in the world coordinate system. ; The extrinsic parameter matrix of the camera coordinate system relative to the solid-state LiDAR coordinate system is initialized, and the error equation is constructed based on the distance error between the matching feature point pairs as follows: Among them, in the formula This indicates the number of feature points that match. This represents a point in the camera's world coordinate system. This indicates that the points in the solid-state lidar coordinate system The external parameter matrix of the solid-state lidar coordinate system relative to the camera Convert the coordinates of the points to those in the camera coordinate system, accumulate the squared distances between the matched feature points, and then average them to obtain the distance error. ; The minimization operation is performed, and the specific formula for the minimization operation is as follows: The minimization operation formula is transformed into a least squares problem, and the optimal extrinsic parameter coefficients are obtained through iterative optimization using the Ceres library. .

7. A joint calibration apparatus for camera-in-camera and camera-relative lidar extrinsic parameters, applied to the joint calibration method for camera-in-camera and camera-relative lidar extrinsic parameters as described in any one of claims 1 to 6, characterized in that, The system includes a sensing system and a calibration board. The sensing system includes a three-degree-of-freedom rotary table, a camera and a solid-state lidar mounted on the three-degree-of-freedom rotary table, and a combined galvanometer. The camera and the solid-state lidar are in relatively fixed positions. The combined galvanometer illuminates the calibration board with the beam emitted by the solid-state lidar and then returns to be received by the solid-state lidar.

8. The joint calibration device for camera intrinsic parameters and camera relative lidar extrinsic parameters according to claim 7, characterized in that, The three-degree-of-freedom rotary table includes a first stepper motor, a second stepper motor, a third stepper motor, and a fixed support. The first stepper motor is used for the fixed support to rotate around the X-axis, the second stepper motor is used for the fixed support to rotate around the Y-axis, and the third stepper motor is used for the fixed support to rotate around the Z-axis. The camera and the solid-state lidar are mounted on the fixed support.

9. The joint calibration device for camera intrinsic parameters and camera relative lidar extrinsic parameters according to claim 7, characterized in that, The combined galvanometer includes a first sub-galvanometer, a second sub-galvanometer, a fourth stepper motor, a fifth stepper motor, and a support frame. The first sub-galvanometer is mounted on the support frame via the fourth stepper motor, and the second sub-galvanometer is mounted on the support frame via the fifth stepper motor.

10. The joint calibration device for camera intrinsic parameters and camera relative lidar extrinsic parameters according to claim 7, characterized in that, The calibration board uses checkerboard calibration or Gray code calibration.

Images