Sensor autonomous external parameter calibration method and system based on convex optimization and reflection intensity assistance

Abstract

The invention discloses a sensor autonomous external parameter calibration method and system based on convex optimization and reflection intensity assistance, and mainly solves the problem of external parameter calibration of a vehicle-mounted look-around fisheye camera and a laser radar. According to the method, firstly, calibration scene arrangement and data acquisition are carried out; secondly, detecting calibration board angular points in the image, describing an optimization problem by a convex function, obtaining camera internal parameters and estimating calibration board external parameters; analyzing the topological relation between the calibration board and the camera, proposing a corresponding calibration board three-dimensional object combination and camera grouping mechanism, and obtaining the initial external participation calibration board index mark number of the camera; then, accurate three-dimensional angular points of the calibration plate are extracted according to the point cloud data; and finally, integrating the index number of the calibration board, the initial external parameter of the camera and the accurate three-dimensional angular point of the calibration board, carrying out multi-sensor internal and external parameter joint optimization calibration, and obtaining accurate external parameters of the laser radar and the look-around fisheye camera. In the field of vehicle-mounted sensor calibration, the calibration precision and the autonomy degree are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle-mounted sensor calibration, and in particular relates to a sensor autonomous extrinsic parameter calibration method and system based on convex optimization and reflection intensity assistance. Background Art

[0002] With the rapid development of autonomous driving technology, on-board surround-view fisheye cameras and lidars are key perception sensors, and their precise calibration is crucial to the accuracy of vehicle environmental perception. Traditional calibration methods, such as those based on feature matching, usually rely on complex calibration equipment and manual operations, which not only increases the calibration cost, but also makes it difficult to maintain calibration stability and accuracy in a dynamically changing environment. In addition, calibration methods based on deep learning require a large amount of training data and have high requirements for computing resources, which may not be flexible and efficient in practical applications. In order to improve calibration accuracy and reduce costs, the present invention introduces the Charuco calibration plate as a reference object, and uses the stable reference point it provides to calibrate the sensor, thereby improving the accuracy and reliability of calibration.

[0003] Surround-view fisheye cameras are widely used in autonomous vehicle surround-view systems due to their wide-angle capabilities. They provide a broad field of view, helping vehicles navigate complex environments. LiDAR, with its high-precision distance measurement capabilities, plays a key role in obstacle detection and path planning. Summary of the Invention

[0004] The purpose of the present invention is to address the problem of insufficient calibration accuracy of vehicle-mounted surround-view fisheye cameras and lidars in the prior art, and to propose a sensor autonomous extrinsic parameter calibration method and system based on convex optimization and reflection intensity assistance.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance, the specific steps are as follows:

[0007] Step 1: Use the vehicle-mounted surround-view fisheye camera and lidar to calibrate the scene layout and perform data collection to obtain point cloud data with a calibration plate and vehicle-mounted surround-view fisheye image data;

[0008] Step 2: Detect the corner points of the Charuco calibration plate based on the fisheye camera model, and use a convex optimization function to optimize the minimum reprojection error in the chessboard detection and calibration method of computer vision technology. Then, use the cost function to perform a convex optimization of the minimum reprojection error on the corner point observation position of each frame based on the image data obtained by each camera, and preliminarily obtain the camera intrinsic parameters and estimate the calibration plate extrinsic parameters.

[0009] Step 3: Based on the obtained internal and external parameters, the initial external parameters of the vehicle-mounted surround-view fisheye camera and the external parameters of each calibration plate in the reference camera coordinate system are obtained by combining the calibration plate 3D objects with the camera grouping mechanism and the camera group joint convex optimization function;

[0010] Step 4: Segment and denoise the point cloud data with the calibration plate, extract the precise range of the calibration plate, and detect the precise 3D extrinsic parameters of each calibration plate based on the reflection intensity of the point cloud within the calibration plate range;

[0011] Step 5: Combine the calibration plate index in the lidar coordinate system, the precise 3D pose of the calibration plate, and the initial extrinsic parameters of the fisheye camera in the reference camera coordinate system to perform a joint optimization calibration of the internal and external parameters of multiple sensors. This will obtain the precise extrinsic parameters of the vehicle-mounted lidar and surround-view fisheye camera, thereby improving the accuracy of the vehicle-mounted sensor calibration.

[0012] Furthermore, step one includes four surround-view fisheye cameras and a vehicle-mounted lidar located on the roof; eight Charuco calibration plates are used as reference objects. The calibration plate has a geometric shape of a square with a side length of 5000 mm, which contains a 5×5 chessboard pattern with a side length of 880 mm and ten Aruco markers with a side length of 660 mm.

[0013] Furthermore, the internal parameters of step 2 include: fisheye camera focal length, camera optical center coordinates, and fisheye model distortion coefficient; the external parameters of the calibration plate include: estimated three-dimensional pose of the calibration plate corner points in the camera coordinate system and the index of the camera coordinate system where the calibration plate is located.

[0014] Furthermore, in step 2, the convex optimization function is used to optimize the minimum reprojection error in the chessboard detection and calibration method of computer vision technology:

[0015]

[0016] Among them, T represents the total number of frames of the calibration dataset, i represents the fisheye camera index, t represents the frame number index of a frame in the dataset, and M b Indicates the total number of calibration plates used, j indicates the index of each calibration plate, S indicates the number of visible corner points on the calibration plate, s indicates the index of the corner point of the calibration plate shown, It represents the two-dimensional pixel coordinates of the corner point with index s on the j-th calibration plate observed by the i-th fisheye camera in the t-th frame in the image coordinate system. Represents the three-dimensional coordinates of the corner point with index s on the j-th calibration plate in the camera coordinate system, Represents the external parameter matrix from the calibration plate with index j to the fisheye camera with index i in the t-th frame, Represents the projection mapping from 3D camera coordinates to 2D pixel coordinates, Represents the rotation matrix from the calibration plate with index j to the fisheye camera with index i, Represents the translation vector from the calibration plate with index j to the fisheye camera with index i, Represents the intrinsic parameter matrix of the fisheye camera with index i, Represents the distortion coefficient of the fisheye camera with index i;

[0017] Given initial estimates of these extrinsic and intrinsic parameters, the following cost function is optimized separately for all images acquired by each camera:

[0018]

[0019] in, For any value of i.

[0020] Furthermore, the camera extrinsic parameters in step three include: the transfer matrix of each fisheye camera relative to the origin in the reference camera coordinate system and the index number of each fisheye camera; the calibration plate extrinsic parameters include: the estimated transfer matrix and index number of the calibration plate relative to the origin.

[0021] Furthermore, the step 3 is first performed according to the matrix multiplication Estimate the relative poses between three or more chessboards simultaneously visible in a single fisheye image and combine them into a 3D calibration object;

[0022] The calibration plate extrinsics in each 3D calibration object are refined in a nonlinear manner by minimizing the reprojection error:

[0023]

[0024] All fisheye camera relative pose extrinsics are individually refined using Levenberg-Marquardt by minimizing the following cost function:

[0025]

[0026] Among them, N g is the total number of fisheye cameras, M o is the total number of 3D calibration objects, S o is the total number of corner points in the 3D calibration object.

[0027] After obtaining the initial external parameters of the surround fisheye camera system, the relative positions, camera poses, and camera intrinsic parameters of all Charuco cameras in the system are refined by minimizing the reprojection error in all frames:

[0028]

[0029] Where Nc represents the number of fisheye cameras, i represents the fisheye camera index, Represents the external parameter matrix from the reference calibration plate of frame t to the calibration plate with index j, represents the rotation matrix from the calibration plate with index j to the reference calibration plate, Represents the translation vector from the calibration plate with index j to the reference calibration plate, Represents the rotation matrix from the fisheye camera with index i to the reference fisheye camera, Indicates the translation vector from the fisheye camera with index i to the reference fisheye camera.

[0030] Furthermore, the precise external parameters of step four include: the transfer matrix of each calibration plate corner point in the laser radar coordinate system and the corresponding index of the calibration plate obtained in step three.

[0031] Furthermore, in step 4, the precise three-dimensional extrinsic parameters of each calibration plate are detected based on the reflection intensity of the point cloud within the calibration plate range. Specifically, the point cloud data is first filtered out using features such as planarity, boundary, and segmented point distribution as conditions for automatic filtering and segmentation; a planarity verification analysis is performed based on the calibration plate point cloud data, and a checkerboard point cloud model is further constructed. A cost function is formulated based on the constraint condition of the correspondence between intensity and color to detect the precise three-dimensional extrinsic parameters of the corner points. The cost function is formulated based on the constraint condition of the correspondence between intensity and color and is defined in the following equation:

[0032]

[0033] Among them, r i is the reflection intensity value; f Y (·) Determine whether the reflection intensity is valid. If it is in the invalid range, it is 0, and if it is outside the invalid range, it is 1. G represents the four corners of the calibration plate. for point Is it located in the plane with G as the vertex? If yes, the value is 1, otherwise 0; To convert the calibration plate point cloud from the LiDAR coordinate system to the plane coordinate system, the 3D points are reduced to 2D after being rotated by the matrix of three PCA vectors, that is, all P M z=0, and the xy axis coordinates are based on the center of the calibration plate as the origin; the plane transformation parameter is the rotation angle vector With translation vector

[0034] According to the camera extrinsics, calibration plate extrinsics and corner points detected in the fisheye image in step 3, a common counting order is defined to match the corner point indexes detected in the image and point cloud, and calculate the precise extrinsic parameter transfer matrix between the calibration plates.

[0035] Furthermore, the multi-sensor internal and external parameter joint optimization calibration in step 5 is expressed as:

[0036]

[0037] in, Represents the three-dimensional coordinates of the corner point with index s on the j-th calibration plate detected by the point cloud data in the lidar coordinate system, Represents the external parameter matrix from the fisheye camera to the lidar with index i at frame t, represents the external parameter matrix from the lidar to the reference calibration plate, Represents the extrinsic parameter matrix from the reference calibration plate to the calibration plate with index j.

[0038] A computer device / equipment / system comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a sensor autonomous extrinsic parameter calibration method based on convex optimization and assisted by reflection intensity.

[0039] The beneficial effects of the present invention are:

[0040] The present invention uses a calibration method that combines a laser radar with a fisheye camera to effectively reduce the impact of the fisheye camera's distortion on the calibration accuracy, thereby significantly improving the accuracy of the calibration. The calibration method of the present invention is not limited by the fixed calibration workshop environment. It only requires a sufficient number of Charuco calibration plates and suitable lighting conditions to complete accurate calibration according to the steps described in the present invention, greatly expanding the scope of application of the calibration method. The present invention achieves full automation of the calibration process, reduces the need for manual intervention, and thus significantly reduces labor costs and improves calibration efficiency. By improving the accuracy of sensor calibration, the present invention enhances the environmental perception capability of the autonomous driving system, thereby improving the safety and reliability of autonomous driving, and providing important technical support for the development of autonomous driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the Charuco calibration plate of the present invention;

[0042] Figure 2 This is a schematic diagram of the placement of the Charuco calibration plate of the present invention;

[0043] Figure 3 This is a schematic diagram of corner point detection on the Charuco calibration plate of the present invention;

[0044] Figure 4 An initial estimation diagram of the extrinsic parameters of the camera group and calibration plate combination of the present invention;

[0045] Figure 5This is a schematic diagram of corner point detection on the Charuco calibration plate of the laser radar of the present invention;

[0046] Figure 6 This is the Charuco calibration plate corner reprojection error map of the present invention;

[0047] Figure 7 Flowchart of the method of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below with reference to the accompanying drawings.

[0049] The present invention provides a sensor autonomous external parameter calibration method and system based on convex optimization and reflection intensity assistance, according to Figure 7 As shown, the specific steps of the method are as follows:

[0050] Step 1: Calibrate the scene layout and data collection to obtain point cloud data with calibration plate and vehicle-mounted surround fisheye image data.

[0051] This vehicle-mounted sensor calibration system includes surround-view fisheye cameras deployed in the front, back, left, and right directions of the vehicle, as well as a lidar located on the roof.

[0052] The present invention uses eight Charuco calibration plates as reference objects as shown in the attached Figure 1 As shown in the figure, the calibration plate is a 5000 mm square, containing a 5×5 checkerboard pattern with 880 mm sides and ten Aruco markers with 660 mm sides. This design makes the calibration plate simple and easy to deploy in various environments, providing a stable reference point for surround-view fisheye cameras and lidar, thereby improving calibration accuracy and reliability.

[0053] Place eight calibration plates around the vehicle to be calibrated as shown in the attached Figure 2 As shown, there is no requirement for the specific placement of the calibration plate, and it is only necessary to ensure that the adjacent fisheye cameras have overlapping calibration plate field of view. Further, data collection begins, and the present invention collects 80 frames.

[0054] Step 2: Detect the corner points of the Charuco calibration plate based on the fisheye camera model, use the convex optimization function to optimize the minimum reprojection error in the chessboard detection and calibration method of computer vision technology, and then use the cost function to perform convex optimization of the minimum reprojection error on the corner point observation position of each frame according to the image data obtained by each camera, preliminarily obtain the camera intrinsic parameters and estimate the calibration plate extrinsic parameters; the intrinsic parameters include: the focal length of the fisheye camera, the coordinates of the camera optical center, and the distortion coefficient of the fisheye model; the calibration plate extrinsic parameters include: the estimated three-dimensional pose of the calibration plate corner points in the camera coordinate system and the index of the camera coordinate system where the calibration plate is located.

[0055] The initial stage of the calibration process is to detect the chessboard in the image and accurately locate its 2D corners as shown in the following figure. Figure 3 Specifically, the Charuco standard planar checkerboard pattern is mixed with the ArUco fiducial marker board.

[0056] At this stage, all images from all cameras are processed to store their 2D keypoint locations and their corresponding 3D points in the chessboard reference. This detection process is very important, and the calibration of the entire system strongly relies on the robustness and accuracy of these 2D keypoints.

[0057] To improve calibration accuracy, the present invention employs an efficient corner refinement process. Furthermore, to avoid degenerate configurations, the present invention applies an additional collinearity check. Furthermore, to enhance overall robustness, boards with fewer than a certain percentage of visible corners are discarded from further consideration; this threshold is typically set at 40%.

[0058] In this paper, we propose a chessboard detection and calibration method based on computer vision technology, which involves multiple key technologies, including the PnP algorithm, the RANSAC algorithm and the application of the OpenCV library.

[0059] For each camera c i , first collect all 3D to 2D corresponding point pairs in the image containing the chessboard. These corresponding point pairs are used to initialize the intrinsic parameters K i and distortion coefficient k i Specifically, the OpenCV library provides an implementation for calibrating fisheye cameras using the Kannala distortion model. This model more accurately describes the imaging process of fisheye cameras, especially when dealing with large fields of view. The Kannala model uses polynomial approximations to describe lens distortion, enabling more accurate distortion correction.

[0060] The initialization process can be relatively slow when processing a large number of images. To improve efficiency, we subsample the images by randomly selecting a subset of 80 checkerboard observations for each camera. If fewer than 80 checkerboard observations are available, all image data is used. Note that the intrinsic parameters will be refined using all images in the next stage to improve calibration accuracy and robustness.

[0061] Based on the calculated initial intrinsic parameters, we further estimate the relative poses of all cameras for each observed chessboard. These pose estimates are implemented using the PnP algorithm and embedded in a RANSAC robust estimation process. This RANSAC stage aims to remove only very large outliers (e.g., reprojection errors exceeding 10 pixels) to improve the overall robustness of our process. The inliers are then used to refine the pose estimates of the chessboard relative to the fisheye camera using the Levenberg-Marquardt nonlinear optimization method.

[0062] The PnP (Perspective-n-Point) algorithm is used in this paper to estimate the pose of the chessboard relative to the camera. Specifically, this algorithm is a widely used algorithm in computer vision that can estimate the pose of the camera from a set of 2D image points and their corresponding 3D world points.

[0063] The RANSAC (RANdom SAmple Consensus) algorithm is used in this paper to remove large error points generated during the chessboard pose estimation process, thereby improving the overall robustness. Specifically, the algorithm is an iterative method specifically designed for estimating the parameters of a mathematical model from a dataset containing outliers.

[0064] Furthermore, the extrinsic parameters of the fisheye camera relative to the chessboard are refined by minimizing the reprojection error:

[0065]

[0066] Among them, T represents the total number of frames of the calibration dataset, i represents the fisheye camera index, t represents the frame number index of a frame in the dataset, and M b Indicates the total number of calibration plates used, j indicates the index of each calibration plate, S indicates the number of visible corner points on the calibration plate, s indicates the index of the corner point of the calibration plate shown, It represents the two-dimensional pixel coordinates of the corner point with index s on the j-th calibration plate observed by the i-th fisheye camera in the t-th frame in the image coordinate system, Represents the three-dimensional coordinates of the corner point with index s on the j-th calibration plate in the camera coordinate system, Represents the external parameter matrix from the calibration plate with index j to the fisheye camera with index i in the t-th frame, Represents the projection mapping from 3D camera coordinates to 2D pixel coordinates, Represents the rotation matrix from the calibration plate with index j to the fisheye camera with index i, Represents the translation vector from the calibration plate with index j to the fisheye camera with index i, Represents the intrinsic parameter matrix of the fisheye camera with index i, Indicates the distortion coefficient of the fisheye camera with index i.

[0067] Given initial estimates of these extrinsic and intrinsic parameters, the following cost function is optimized separately for all images acquired by each camera:

[0068]

[0069] Among them, T represents the total number of frames of the calibration data set, t represents the frame index of a frame in the data set, and M b Indicates the total number of calibration plates used, j indicates the index of each calibration plate, S indicates the number of visible corner points on the calibration plate, s indicates the index of the corner point of the calibration plate shown, It represents the two-dimensional pixel coordinates of the corner point with index s on the j-th calibration plate observed by the i-th fisheye camera in the t-th frame in the image coordinate system, Represents the three-dimensional coordinates of the corner point with index s on the j-th calibration plate in the camera coordinate system, Represents the external parameter matrix from the calibration plate with index j to the fisheye camera with index i in the t-th frame, Represents the projection mapping from 3D camera coordinates to 2D pixel coordinates, Represents the rotation matrix from the calibration plate with index j to the fisheye camera with index i, Represents the translation vector from the calibration plate with index j to the fisheye camera with index i, Represents the intrinsic parameter matrix of the fisheye camera with index i, Represents the distortion coefficient of the fisheye camera with index i, For any value of i.

[0070] Step 3: Based on the obtained internal and external parameters, and through the calibration plate three-dimensional object combination and camera grouping mechanism and camera group joint convex optimization function, the initial external parameters of the vehicle-mounted surround fisheye camera and the external parameters of each calibration plate in the reference camera coordinate system are obtained; the camera external parameters include: the transfer matrix of each fisheye camera relative to the origin in the reference camera coordinate system and the index number of each fisheye camera; the calibration plate external parameters include: the estimated transfer matrix and index number of the calibration plate relative to the origin.

[0071] In step 2, the present invention has estimated the relative pose of each individually observed chessboard with respect to all cameras. Furthermore, the relative poses between chessboards are estimated to combine them into a 3D calibration object.

[0072] If three or more chessboards are visible simultaneously in a single fisheye image, then matrix multiplication can be used to Estimate the relative pose between them.

[0073] To enhance robustness, we collect measurements of pairs of chessboards visible in the same image across all images and compute the average rotation and translation between them. For example, if chessboards b0 and b1 are observed together in 10 images by one or more cameras, the relative relationship between the chessboards will be the robust average of these 10 measurements.

[0074] This strategy can significantly improve robustness and allow the system to perform well in the presence of potential outliers, by providing a reliable prior estimate of the pose across the chessboards through this averaging strategy.

[0075] Furthermore, 3D calibration objects are constructed through the extrinsic relationship between the chessboards, and the calibration plates are stored in a directed weighted graph. For each 3D calibration object, the calibration plate with the lowest index number is selected as the reference calibration plate.

[0076] Finally, the Dijkstra shortest path algorithm is used to determine the optimal posture transformation combination, and then the eight calibration plates are stored in a directed weighted graph to form a 3D calibration object.

[0077] Furthermore, the calibration plate extrinsics in each 3D calibration object are refined in a nonlinear manner by minimizing the reprojection error:

[0078]

[0079] After merging the eight Charuco calibration plates into a 3D calibration object, the pose extrinsics of the fisheye camera relative to the 3D calibration object in all frames are estimated using a PnP algorithm similar to step 2.

[0080] Furthermore, the fisheye camera with the lowest index is used as the reference camera, and the Dijkstra algorithm is used to determine the initial relative poses of other fisheye cameras.

[0081] Finally, all fisheye camera relative pose extrinsics are individually refined using Levenberg-Marquardt by minimizing the following cost function:

[0082]

[0083] Among them, N g is the total number of fisheye cameras, M o is the total number of 3D calibration objects, S o is the total number of corner points in the 3D calibration object.

[0084] Furthermore, after obtaining the initial external parameters of the surround fisheye camera system, the relative positions, camera poses, and camera intrinsic parameters of all Charuco objects in the system are refined by minimizing the reprojection error in all frames:

[0085]

[0086] Among them, Nc represents the number of fisheye cameras, i represents the fisheye camera index, T represents the total number of frames of the calibration data set, t represents the frame index of a frame in the data set, and M b Indicates the total number of calibration plates used, j indicates the index of each calibration plate, S indicates the number of visible corner points on the calibration plate, s indicates the index of the corner point of the calibration plate shown, It represents the two-dimensional pixel coordinates of the corner point with index s on the j-th calibration plate observed by the i-th fisheye camera in the t-th frame in the image coordinate system. Represents the three-dimensional coordinates of the corner point with index s on the j-th calibration plate in the camera coordinate system, Represents the external parameter matrix from the fisheye camera with index i to the reference fisheye camera, Represents the external parameter matrix from the reference calibration plate of frame t to the fisheye camera with index i, Represents the external parameter matrix from the reference calibration plate of frame t to the calibration plate with index j, Represents the projection mapping from 3D camera coordinates to 2D pixel coordinates, represents the rotation matrix from the calibration plate with index j to the reference calibration plate, Represents the translation vector from the calibration plate with index j to the reference calibration plate, Represents the rotation matrix from the fisheye camera with index i to the reference fisheye camera, Represents the translation vector from the fisheye camera with index i to the reference fisheye camera, Represents the intrinsic parameter matrix of the fisheye camera with index i, Indicates the distortion coefficient of the fisheye camera with index i.

[0087] The present invention adopts a layered calibration strategy to provide a near-convergent initialization, thereby obtaining very stable and accurate results as shown in the attached figure. Figure 4 shown.

[0088] Step 4: Segment and denoise the point cloud data with the calibration plate, extract the precise range of the calibration plate, and detect the precise three-dimensional extrinsic parameters of each calibration plate based on the reflection intensity of the point cloud within the calibration plate range; the precise extrinsic parameters include: the transfer matrix of each calibration plate corner point in the lidar coordinate system and the corresponding index of the calibration plate obtained in step 3.

[0089] For point cloud data with a calibration plate, scanline segmentation is first performed. Specifically, each frame of point cloud data is segmented into scanlines. Each scanline is clustered to determine whether consecutive points belong to the same object. Clustering is performed based on distance and angle thresholds.

[0090] Furthermore, the clustered scan lines are aggregated into the final 3D segments, and the Euclidean distance and the similarity of the PCA results are used as the aggregation criteria.

[0091] The newly acquired point cloud data is segmented, and the newly segmented segments are merged with the segmentation results of the previous frame to obtain the segmented point cloud data.

[0092] The checkerboard calibration plate is then retrieved from the objects in the segmented point cloud.

[0093] Features such as planarity, boundaries, and point distribution of segments are used as conditions for automatic segment filtering.

[0094] By using the theoretical number of points, the point cloud on the non-calibrated board is filtered out. Theoretically, the number of point clouds on a board should be in the interval [η theo *n theo ,n theo ], where η theo The coefficient is assumed to be 0.6, n theo The calculation formula for the theoretical number of points on the calibration plate is as follows:

[0095]

[0096] Among them, d W d H denote the chessboard width and height respectively, Δh and Δv denote the horizontal and vertical resolutions of the lidar respectively, r is the Euclidean distance from the centroid of the segmented object to the lidar sensor, Indicates rounding up a real number.

[0097] Furthermore, the planarity is verified. The matrix M composed of all points in the segmented point cloud nx3 Decompose V along 3 basis vectors b =(μ1,μ2,μ3) T The component ratios on each basis vector are μ1, μ2, and μ3. If the minimum ratio μ3 is less than 0.01, the segment is considered to be a planar object. The segmented planar object is estimated as a plane using RANSAC, and the fitted plane points are M nx3,fr .

[0098] Put M nx3,fr Perform rotation and translation so that the origin is the center of mass of the plane points, μ1, μ2 are aligned with the xy direction, and then the segmentation plane after rotation and translation is obtained.

[0099]

[0100] M n×3,frt =M n×3,fr -mean(M n×3,fr )

[0101] Furthermore, the uniformity of the plane point cloud is tested. The uniformity of the point cloud distribution is determined by the difference in point distribution in four equally divided areas. The boundary of the segmentation plane is [0.8d W ,1.6d W ] and [0.8d H ,1.6d H ] are considered potential chessboards.

[0102] The uniformity is calculated as follows:

[0103]

[0104] Among them, n all is the total number of points in the segment, assuming the maximum number of points in the region is n max , the minimum value is n min The larger ∈ norm The value indicates that the point cloud is distributed normally. The threshold value of uniformity ∈ norm Set to 0.85.

[0105] If there are multiple segmentation planes that meet the above conditions, the point cloud segmentation with higher uniformity is selected. The point set in the detected chessboard segmentation is represented as P M .

[0106] Furthermore, based on the chessboard point cloud plane, the point cloud reflection intensity is used to estimate the corner points.

[0107] In order to process the intensity data adaptively, a reflection intensity invalid range is defined, which is expressed as [I l ,I h ]. Intensity is less than I l Points with an intensity greater than I are considered to be reflected from the black pattern. h The points are considered to be reflected from the white pattern. l and I h The value of the histogram is created and the peaks on both sides of the average intensity (R L ,R H Then, the reflection intensity five-effect range [I l ,I h ] is defined as follows:

[0108]

[0109] Among them, ∈ g ≥2 is a constant. In the present invention, ∈ g is set to 2 so that all points are used when there are enough points for corner estimation (the invalid area point cloud will be zero); and when there is enough confidence in the pattern color from the reflection intensity for error estimation, ∈ g Set to 4.

[0110] The cost function is based on the constraint of the correspondence between intensity and color and is defined in the following equation:

[0111]

[0112] Among them, r i is the reflection intensity value; f Y (·) Determine whether the reflection intensity is valid. If it is in the invalid range, it is 0, and if it is outside the invalid range, it is 1. G represents the four corners of the calibration plate. for point Is it located in the plane with G as the vertex? If yes, the value is 1, otherwise 0; To convert the calibration plate point cloud from the LiDAR coordinate system to the plane coordinate system, the 3D points are reduced to 2D after being rotated by the matrix of three PCA vectors, that is, all P M z=0, and the xy axis coordinates are based on the center of the calibration plate as the origin; the plane transformation parameter is the rotation angle vector With translation vector

[0113] Furthermore, the Powell optimization method is used to solve the p when the cost function J is the minimum. i , which is the precise 3D coordinates of the corner points of the calibration plate, as shown in the attached Figure 5 shown.

[0114] At the same time, by defining a common counting order starting from the lower left corner of the calibration plate and the index order of the eight calibration plates, the corner points detected in the image and point cloud are matched, and the precise extrinsic parameter transfer matrix between the calibration plates is calculated.

[0115] Step 5: Combine the calibration plate index in the lidar coordinate system, the precise 3D pose of the calibration plate, and the initial extrinsic parameters of the fisheye camera in the reference camera coordinate system to perform a joint optimization calibration of the internal and external parameters of multiple sensors to obtain the precise extrinsic parameters of the lidar and surround-view fisheye camera.

[0116] Furthermore, the minimum reprojection error is used as a convex function, and the internal and external parameters of the sensor are used as variables for the final joint optimization:

[0117]

[0118] Among them, N c Indicates the number of fisheye cameras, i indicates the fisheye camera index, T indicates the total number of frames of the calibration dataset, t indicates the frame index of a frame in the dataset, j indicates the index of each calibration plate, S indicates the number of visible corner points on the calibration plate, and s indicates the index of the corner point of the calibration plate shown. It represents the two-dimensional pixel coordinates of the corner point with index s on the j-th calibration plate observed by the i-th fisheye camera in the t-th frame in the image coordinate system. Represents the three-dimensional coordinates of the corner point with index s on the j-th calibration plate detected by the point cloud data in the lidar coordinate system, Represents the external parameter matrix from the fisheye camera to the lidar with index i at frame t, represents the external parameter matrix from the lidar to the reference calibration plate, represents the extrinsic parameter matrix from the reference calibration plate to the calibration plate with index j, Represents the projection mapping from 3D camera coordinates to 2D pixel coordinates, represents the rotation matrix from the calibration plate with index j to the reference calibration plate, Represents the translation vector from the calibration plate with index j to the reference calibration plate, Represents the rotation matrix from the fisheye camera with index i to the reference fisheye camera, Represents the translation vector from the fisheye camera with index i to the reference fisheye camera, Represents the intrinsic parameter matrix of the fisheye camera with index i, Indicates the distortion coefficient of the fisheye camera with index i.

[0119] The calibration reprojection error effect is shown in the attached Figure 6 As shown, this figure reveals the implementation effect of the average reprojection error per frame. Specifically, the horizontal axis represents the frame serial number, ranging from 0 to 80; the vertical axis represents the average reprojection error, ranging from 0.00 to 0.25. It can be seen from the figure that the average reprojection error fluctuates between different frame serial numbers. Near the frame serial numbers 30 and 80, the error values ​​reach local peaks, which are approximately 0.25 and 0.22, respectively. At other frame serial numbers, the error values ​​are relatively stable, mostly concentrated between 0.15 and 0.20. In summary, the average reprojection error peak is ≤0.25 pixels, and the average reprojection error at 80 frames is 0.187 pixels, which shows the high precision effect of the method of the present invention.

[0120] In summary, this invention achieves precise calibration of surround-view fisheye cameras and lidar by combining the PnP (Perspective-n-Point) algorithm, the RANSAC (Random Sample Consensus) algorithm, a fisheye camera model, and convex optimization techniques. Furthermore, the precise three-dimensional corner detection of the lidar effectively reduces the effects of fisheye camera distortion, thereby improving calibration accuracy. Although this invention is primarily applicable to static environments, its advantage in calibration accuracy is still significant, providing a reliable foundation for environmental perception for autonomous vehicles.

[0121] During the calibration process, this invention uses point cloud segmentation and denoising technology to extract the precise three-dimensional corner points of the calibration plate from the LiDAR data. These corner points are then used to calibrate the fisheye camera. Convex optimization techniques are used to minimize reprojection errors, further improving calibration accuracy. This invention's high-precision calibration capability in static environments provides a reliable foundation for environmental perception for autonomous vehicles, particularly in applications requiring high-precision maps and environmental modeling.

[0122] In addition, the function of the sensor autonomous external parameter calibration system based on convex optimization and reflection intensity assistance of the present invention can be described by the aforementioned sensor autonomous external parameter calibration method based on convex optimization and reflection intensity assistance. The system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. Various programs and data required for the operation of the rescue response system are also stored in RAM 303. CPU 301, ROM 302 and RAM 303 are connected to each other via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0123] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, push button switches, etc.; an output section 307 including a liquid crystal display (LCD), an audio output device, indicator lights, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 310 as needed, so that computer programs read therefrom can be installed into the storage section 308 as needed.

[0124] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a computer-readable medium, the computer program including a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 309 and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the various functions defined in the present invention are performed.

[0125] It should be noted that specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0126] Specifically, the sensor autonomous extrinsic parameter calibration system based on convex optimization and assisted by reflection intensity in this embodiment includes a processor and a memory, and a computer program is stored in the memory. When the computer program is executed by the processor, the sensor autonomous extrinsic parameter calibration method based on convex optimization and assisted by reflection intensity provided in the above embodiment is implemented.

[0127] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the sensor-autonomous extrinsic parameter calibration system based on convex optimization and reflection intensity assistance described in the above embodiments; or it may exist independently and not be assembled into the sensor-autonomous extrinsic parameter calibration system based on convex optimization and reflection intensity assistance. The above storage medium carries one or more computer programs, and when the one or more computer programs are executed by a processor of the sensor-autonomous extrinsic parameter calibration system based on convex optimization and reflection intensity assistance, the sensor-autonomous extrinsic parameter calibration system based on convex optimization and reflection intensity assistance implements the sensor-autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance provided in the above embodiments.

[0128] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance, characterized by: The specific steps are as follows: Step 1: Use the vehicle-mounted surround-view fisheye camera and lidar to calibrate the scene layout and perform data collection to obtain point cloud data with a calibration plate and vehicle-mounted surround-view fisheye image data; Step 2: Detect the corner points of the Charuco calibration plate based on the fisheye camera model, and use a convex optimization function to optimize the minimum reprojection error in the chessboard detection and calibration method of computer vision technology. Then, use the cost function to perform a convex optimization of the minimum reprojection error on the corner point observation position of each frame based on the image data obtained by each camera, and preliminarily obtain the camera intrinsic parameters and estimate the calibration plate extrinsic parameters; Step 3: Based on the obtained internal and external parameters, the initial external parameters of the vehicle-mounted surround-view fisheye camera and the external parameters of each calibration plate in the reference camera coordinate system are obtained by combining the calibration plate 3D objects with the camera grouping mechanism and the camera group joint convex optimization function; Step 4: Segment and denoise the point cloud data with the calibration plate, extract the precise range of the calibration plate, and detect the precise three-dimensional external parameters of each calibration plate based on the reflection intensity of the point cloud within the calibration plate range; Step 5: Combine the calibration plate index in the lidar coordinate system, the precise 3D pose of the calibration plate, and the initial extrinsic parameters of the fisheye camera in the reference camera coordinate system to perform a joint optimization calibration of the internal and external parameters of multiple sensors. This will obtain the precise extrinsic parameters of the vehicle-mounted lidar and surround-view fisheye camera, thereby improving the accuracy of the vehicle-mounted sensor calibration.

2. The sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance according to claim 1 is characterized by: Step 1 includes four surround-view fisheye cameras and a roof-mounted lidar; eight Charuco calibration plates are used as reference objects. The calibration plates have a geometric shape of a square with a side length of 5000 mm, containing a 5×5 checkerboard pattern with a side length of 880 mm, and ten Aruco markers with a side length of 660 mm.

3. The sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance according to claim 1 is characterized in that: The internal parameters of step 2 include: the focal length of the fisheye camera, the coordinates of the camera optical center, and the distortion coefficient of the fisheye model; the external parameters of the calibration plate include: the estimated three-dimensional pose of the corner points of the calibration plate in the camera coordinate system and the index of the camera coordinate system where the calibration plate is located.

4. The sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance according to claim 3 is characterized by: The step 2 uses a convex optimization function to optimize the minimum reprojection error in the chessboard detection and calibration method of computer vision technology: Among them, T represents the total number of frames of the calibration dataset, i represents the fisheye camera index, t represents the frame number index of a frame in the dataset, and M b Indicates the total number of calibration plates used, j indicates the index of each calibration plate, S indicates the number of visible corner points on the calibration plate, s indicates the index of the corner point of the calibration plate shown, It represents the two-dimensional pixel coordinates of the corner point with index s on the j-th calibration plate observed by the i-th fisheye camera in the t-th frame in the image coordinate system, Represents the three-dimensional coordinates of the corner point with index s on the j-th calibration plate in the camera coordinate system, Represents the external parameter matrix from the calibration plate with index j to the fisheye camera with index i in the t-th frame, Represents the projection mapping from 3D camera coordinates to 2D pixel coordinates, Represents the rotation matrix from the calibration plate with index j to the fisheye camera with index i, Represents the translation vector from the calibration plate with index j to the fisheye camera with index i, Represents the intrinsic parameter matrix of the fisheye camera with index i, Represents the distortion coefficient of the fisheye camera with index i; Given initial estimates of these extrinsic and intrinsic parameters, the following cost function is optimized separately for all images acquired by each camera: in, For any value of i.

5. The sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance according to claim 1 is characterized in that: The camera extrinsic parameters in step 3 include: the transfer matrix of each fisheye camera relative to the origin in the reference camera coordinate system and the index number of each fisheye camera; the calibration plate extrinsic parameters include: the estimated transfer matrix and index number of the calibration plate relative to the origin.

6. The sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance according to claim 5 is characterized in that: The step 3 firstly performs matrix multiplication Estimate the relative poses between three or more chessboards simultaneously visible in a single fisheye image and combine them into a 3D calibration object; The calibration plate extrinsics in each 3D calibration object are refined in a nonlinear manner by minimizing the reprojection error: All fisheye camera relative pose extrinsics are individually refined using Levenberg-Marquardt by minimizing the following cost function: Among them, N g is the total number of fisheye cameras, M o is the total number of 3D calibration objects, S o is the total number of corner points in the 3D calibration object. After obtaining the initial external parameters of the surround fisheye camera system, the relative positions, camera poses, and camera intrinsic parameters of all Charuco cameras in the system are refined by minimizing the reprojection error in all frames: Where Nc represents the number of fisheye cameras, i represents the fisheye camera index, Represents the external parameter matrix from the reference calibration plate of frame t to the calibration plate with index j, represents the rotation matrix from the calibration plate with index j to the reference calibration plate, Represents the translation vector from the calibration plate with index j to the reference calibration plate, Represents the rotation matrix from the fisheye camera with index i to the reference fisheye camera, Indicates the translation vector from the fisheye camera with index i to the reference fisheye camera.

7. The sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance according to claim 1 is characterized in that: The precise external parameters of step 4 include: the transfer matrix of each calibration plate corner point in the laser radar coordinate system and the corresponding index of the calibration plate obtained in step 3.

8. The sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance according to claim 7 is characterized in that: In step 4, the precise three-dimensional external parameters of each calibration plate are detected based on the reflection intensity of the point cloud within the calibration plate range. Specifically, the point cloud data is first filtered out using features such as planarity, boundary, and segmented point distribution as conditions for automatic filtering and segmentation; a planarity verification analysis is performed based on the calibration plate point cloud data, and a checkerboard point cloud model is further constructed. A cost function is formulated based on the constraint condition of the correspondence between intensity and color to detect the precise three-dimensional external parameters of the corner points. The cost function is formulated based on the constraint condition of the correspondence between intensity and color and is defined in the following equation: Among them, r i is the reflection intensity value; f Y (·) Determine whether the reflection intensity is valid. If it is in the invalid range, it is 0, and if it is outside the invalid range, it is 1. G represents the four corners of the calibration plate. for point Is it located in the plane with G as the vertex? If yes, the value is 1, otherwise 0; To convert the calibration plate point cloud from the LiDAR coordinate system to the plane coordinate system, the 3D points are reduced to 2D after being rotated by the matrix of three PCA vectors, that is, all P M z=0, and the xy axis coordinates are based on the center of the calibration plate as the origin; the plane transformation parameter is the rotation angle vector With translation vector According to the camera extrinsics, calibration plate extrinsics and corner points detected in the fisheye image in step 3, a common counting order is defined to match the corner point indexes detected in the image and point cloud, and calculate the precise extrinsic parameter transfer matrix between the calibration plates.

9. The sensor autonomous extrinsic parameter calibration method based on convex optimization and reflection intensity assistance according to claim 1 is characterized in that: The multi-sensor internal and external parameter joint optimization calibration in step 5 is expressed as: in, Represents the three-dimensional coordinates of the corner point with index s on the j-th calibration plate detected by the point cloud data in the lidar coordinate system, Represents the external parameter matrix from the fisheye camera to the lidar with index i at frame t, represents the external parameter matrix from the lidar to the reference calibration plate, Represents the extrinsic parameter matrix from the reference calibration plate to the calibration plate with index j.

10. A computer device / apparatus / system comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images