Camera calibration methods, devices, electronic equipment and storage media

Abstract

This invention provides a camera calibration method, apparatus, electronic device, and storage medium in the field of data processing technology. The method includes: acquiring LiDAR point cloud data, inertial measurement unit data, and reference camera image data of a vehicle; acquiring initial calibration parameters for the camera to be calibrated; constructing a point cloud map containing color texture and the vehicle's pose trajectory based on the LiDAR point cloud data, inertial measurement unit data, and reference camera image data; converting the point cloud map into a surface mesh model; rendering the surface mesh model to the viewpoint of the camera to be calibrated according to the pose trajectory and initial calibration parameters; generating a rendered image and a rendered depth map corresponding to the camera to be calibrated; establishing a correspondence between two-dimensional image features and three-dimensional spatial points in the real image data based on the real image data, the rendered image, and the rendered depth map; constructing a reprojection error objective function based on the correspondence; and optimizing the extrinsic and intrinsic parameters of the camera to be calibrated by minimizing the objective function.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a camera calibration method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of autonomous driving, to achieve comprehensive perception of the surrounding environment, it is typically necessary to deploy multiple sensors on the vehicle, including LiDAR, inertial measurement units, and multiple cameras arranged around the vehicle body. The prerequisite for these sensors to work together is to obtain accurate calibration parameters, including the extrinsic parameters between the LiDAR and each camera, as well as the intrinsic parameters of the cameras themselves.

[0003] Traditional calibration methods primarily rely on static calibration using calibration boards such as checkerboards in laboratories or dedicated facilities. While this can yield relatively accurate initial calibration results, it requires specialized facilities and manual operation, resulting in high maintenance costs. More importantly, during actual vehicle operation, factors such as vibration and temperature changes can cause slight shifts in the sensor's installation position, leading to the gradual invalidation of calibration parameters. Existing static calibration methods cannot detect and correct this parameter drift online.

[0004] Therefore, how to more effectively calibrate vehicle cameras has become an urgent problem to be solved in the industry. Summary of the Invention

[0005] This invention provides a camera calibration method, apparatus, electronic device, and storage medium to solve the problem of how to more effectively calibrate vehicle cameras in the prior art.

[0006] This invention provides a camera calibration method, comprising:

[0007] Acquire the vehicle's lidar point cloud data, inertial measurement unit data, reference camera image data, and real image data of the camera to be calibrated, and obtain the initial calibration parameters of the camera to be calibrated;

[0008] Based on the lidar point cloud data, the inertial measurement unit data, and the reference camera image data, a point cloud map containing color texture and the vehicle's pose trajectory are constructed, and the point cloud map is converted into a surface mesh model.

[0009] Based on the pose trajectory and the initial calibration parameters, the surface mesh model is rendered to the viewpoint of the camera to be calibrated, generating a rendered image and a rendered depth map corresponding to the camera to be calibrated;

[0010] Based on the real image data, the rendered image, and the rendered depth map, establish the correspondence between two-dimensional image features and three-dimensional spatial points in the real image data;

[0011] Based on the correspondence, a reprojection error objective function is constructed, and the extrinsic and intrinsic parameters of the camera to be calibrated are optimized by minimizing the objective function.

[0012] According to a camera calibration method provided by the present invention, establishing the correspondence between two-dimensional image features and three-dimensional spatial points in the real image data includes:

[0013] Feature extraction and matching are performed on the real image data and the rendered image to obtain feature matching pairs, wherein the feature matching pairs contain mutually matching real image features and rendered image features;

[0014] Based on the rendered depth map, the pose trajectory, and the initial calibration parameters, the rendered image features are back-projected into three-dimensional space to obtain the three-dimensional space point coordinates corresponding to the rendered image features;

[0015] The two-dimensional pixel coordinates of the real image features are associated with the three-dimensional spatial point coordinates corresponding to the rendered image features, which is used as the correspondence.

[0016] According to a camera calibration method provided by the present invention, the feature matching pairs include: point feature matching pairs and line segment feature matching pairs; feature extraction and matching of the real image data and the rendered image include:

[0017] Point feature extraction and line segment feature extraction are performed on the real image data to obtain a first point feature set and a first line segment feature set;

[0018] Point feature extraction and line segment feature extraction are performed on the rendered image to obtain a second point feature set and a second line segment feature set;

[0019] The first point feature set and the second point feature set are matched to obtain the point feature matching pair;

[0020] The first line segment feature set and the second line segment feature set are matched to obtain the line segment feature matching pair.

[0021] According to a camera calibration method provided by the present invention, the rendered image features are back-projected into a three-dimensional space, comprising:

[0022] For the point feature matching pair, the depth value of the corresponding point feature in the second point feature set is obtained from the rendered depth map, and back projection is performed based on the depth value, the pose trajectory and the initial calibration parameters to obtain the three-dimensional point coordinates of the corresponding point feature;

[0023] For the line segment feature matching pair, the depth values ​​of multiple sampling points on the corresponding line segment feature in the second line segment feature set are obtained from the rendered depth map. Based on the depth values, the pose trajectory and the initial calibration parameters, back projection is performed to obtain the coordinates of multiple three-dimensional sampling points of the corresponding line segment feature.

[0024] The three-dimensional point coordinates and the coordinates of the plurality of three-dimensional sampling points together constitute the three-dimensional spatial point coordinates.

[0025] According to a camera calibration method provided by the present invention, a reprojection error objective function is constructed based on the correspondence, including:

[0026] For the point feature matching pair, the three-dimensional point coordinates are projected onto the image plane according to the current extrinsic and intrinsic parameters to obtain the predicted point coordinates. The distance between the predicted point coordinates and the pixel coordinates of the corresponding point feature in the first point feature set is calculated as the point reprojection error.

[0027] For the line segment feature matching pair, the coordinates of the multiple three-dimensional sampling points are projected onto the image plane according to the current extrinsic and intrinsic parameters to obtain multiple predicted projection points. The distance from each predicted projection point to the line containing the corresponding line segment feature in the first line segment feature set is calculated as the line segment reprojection error.

[0028] The objective function for the reprojection error is constructed based on the weighted sum of the point reprojection error and the line segment reprojection error of the vehicle at multiple acquisition times.

[0029] According to a camera calibration method provided by the present invention, the cameras to be calibrated are multiple; after optimizing the extrinsic and intrinsic parameters of each camera to be calibrated, the method further includes:

[0030] Identify adjacent pairs of cameras to be calibrated where their fields of view overlap.

[0031] Feature extraction and matching are performed on the real image data of the two cameras to be calibrated in the adjacent camera pair to obtain feature matching pairs between cameras;

[0032] Based on the feature matching pairs between cameras, relative pose constraints between cameras are constructed, and joint optimization is performed to correct the extrinsic parameters of the adjacent pairs of cameras to be calibrated.

[0033] According to a camera calibration method provided by the present invention, the construction of a point cloud map including color texture and the pose trajectory of the vehicle includes:

[0034] The inertial measurement unit data, the lidar point cloud data, and the reference camera image data are fused using a tightly coupled method.

[0035] Based on the fused data, synchronous positioning and mapping are performed to generate the point cloud map and the pose trajectory.

[0036] The present invention also provides a camera calibration device, comprising:

[0037] The acquisition module is used to acquire the vehicle's lidar point cloud data, inertial measurement unit data, reference camera image data, and real image data of the camera to be calibrated, and to acquire the initial calibration parameters of the camera to be calibrated.

[0038] The construction module is used to construct a point cloud map containing color texture and the vehicle's pose trajectory based on the lidar point cloud data, the inertial measurement unit data and the reference camera image data, and to convert the point cloud map into a surface mesh model.

[0039] The rendering module is used to render the surface mesh model to the viewpoint of the camera to be calibrated based on the pose trajectory and the initial calibration parameters, and generate a rendered image and a rendered depth map corresponding to the camera to be calibrated.

[0040] A module is established to establish a correspondence between two-dimensional image features and three-dimensional spatial points in the real image data based on the real image data, the rendered image, and the rendered depth map.

[0041] The calibration module is used to construct a reprojection error objective function based on the correspondence, and to optimize the extrinsic and intrinsic parameters of the camera to be calibrated by minimizing the objective function.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the camera calibration method as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the camera calibration method as described above.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the camera calibration method as described above.

[0045] This invention provides a camera calibration method, apparatus, electronic device, and storage medium. By utilizing a point cloud map with color texture constructed through the fusion of a LiDAR, an inertial measurement unit, and a reference camera, and converting it into a surface mesh model, it is rendered to the viewpoint of the camera to be calibrated. This allows the camera to establish a geometric association with the LiDAR coordinate system without direct overlap with the LiDAR's field of view, breaking through the dependence of traditional calibration methods on dedicated calibration scenes and calibration boards. Through feature matching between the rendered image and the real image, combined with the rendered depth map, accurate mapping from 2D image features to 3D spatial points is achieved, providing reliable geometric constraints for calibration optimization. Optimization based on the reprojection error objective function can simultaneously correct the extrinsic and intrinsic parameters of the camera to be calibrated, compensating for calibration parameter drift caused by factors such as vehicle vibration and temperature changes. This enables automatic and precise calibration of sensor parameters during normal vehicle operation, achieving automatic calibration without a calibration scene, and improving the calibration accuracy and long-term stability of the multi-sensor fusion sensing system. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the camera calibration method provided by the present invention;

[0048] Figure 2 The overall flowchart provided for this invention;

[0049] Figure 3 This is a schematic diagram of the overall steps of the present invention;

[0050] Figure 4 This is a schematic diagram of the camera calibration device provided by the present invention;

[0051] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] The camera calibration method provided by this invention can be implemented by various execution entities, including but not limited to: vehicle controller, vehicle computing platform, edge computing device, cloud server, processor or processor cluster with data processing capabilities, etc.

[0054] In practical applications, the execution entity of this method can be flexibly selected based on computing resource requirements and real-time requirements. For example, when online real-time calibration is required, an on-board high-performance computing platform can be used as the execution entity; when offline batch calibration is used, the collected data can be uploaded to a cloud server for processing.

[0055] Figure 1 This is a flowchart illustrating the camera calibration method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:

[0056] Step 110: Obtain the vehicle's lidar point cloud data, inertial measurement unit data, reference camera image data, and real image data of the camera to be calibrated, and obtain the initial calibration parameters of the camera to be calibrated;

[0057] In this application, lidar point cloud data refers to the set of three-dimensional spatial points output by the lidar equipment installed on the vehicle during operation.

[0058] LiDAR measures the three-dimensional coordinates of points on the surface of objects in the surrounding environment by emitting laser pulses and receiving reflected signals.

[0059] In one embodiment, the lidar can be a rotating lidar capable of scanning a 360-degree area around the vehicle. Each frame of point cloud data contains a large number of three-dimensional points, each of which typically includes spatial coordinates (X, Y, Z) and reflection intensity information (I). The lidar point cloud data can be single-frame data or a sequence of multiple frames {L_i} continuously collected over a period of time, where i = 1, 2, ..., N represents the frame number.

[0060] Inertial measurement unit data refers to motion measurement data output by an inertial measurement unit (IMU).

[0061] An inertial measurement unit typically includes a three-axis accelerometer and a three-axis gyroscope, which can measure the linear acceleration and angular velocity of a vehicle in three-dimensional space.

[0062] Since the sampling frequency of the inertial measurement unit is usually much higher than that of the lidar and camera, the number of frames of inertial measurement unit data in the same time period can be tens or even hundreds of times that of lidar frames.

[0063] For example, when a lidar operates at a frequency of 10Hz and collects N frames of data, an inertial measurement unit operating at 100Hz can generate approximately 100×N frames of data.

[0064] Reference camera image data refers to image data acquired by a reference camera. A reference camera is a camera on a vehicle used in conjunction with lidar and inertial measurement units to construct an environmental map.

[0065] In one embodiment, the reference camera may be a forward-facing camera of the vehicle, which has good field-of-view overlap with the lidar, facilitating multi-sensor fusion mapping.

[0066] Reference camera image data can contain color information, such as RGB three-channel images, to add color texture to the point cloud map that is subsequently built.

[0067] The actual image data of the camera to be calibrated refers to the actual image data collected by the camera that needs to be calibrated. The camera to be calibrated can be other cameras on the vehicle besides the reference camera, such as the vehicle's rear-view camera, side-view camera, surround-view camera, etc.

[0068] In one embodiment, 11 cameras can be installed on the vehicle, one of which serves as a reference camera, such as a forward-looking camera, and the remaining 10 serve as cameras to be calibrated, distributed at different locations on the vehicle to cover an all-around field of view around the vehicle. The image data collected by each camera to be calibrated within the same time period constitutes the camera's real image data sequence {I_k_i}, where k represents the camera number and i represents the frame number.

[0069] Initial calibration parameters refer to the calibration parameters that the camera to be calibrated already possesses before the current calibration process begins. Initial calibration parameters are usually obtained through static calibration methods, such as calibration using a calibration board (such as a checkerboard or dotted board) before the vehicle leaves the factory.

[0070] The initial calibration parameters consist of two parts: extrinsic parameters and intrinsic parameters. Extrinsic parameters describe the spatial geometric relationship between the camera to be calibrated and the lidar, including rotation and translation parameters.

[0071] Rotation parameters can be represented by rotation matrices R (3×3 matrices), rotation vectors, quaternions, or Euler angles, describing the rotational relationship between two coordinate systems. Translation parameters are usually represented by translation vectors t (3×1 vectors), describing the displacement relationship between the origins of two coordinate systems.

[0072] Intrinsic parameters describe the imaging characteristics of the camera itself, including but not limited to: focal length parameters (f_x, f_y) representing the equivalent focal length of the camera in the horizontal and vertical directions; principal point coordinates (c_x, c_y) representing the pixel coordinates of the intersection of the camera's optical axis and the imaging plane; and distortion coefficients (d) describing the image distortion caused by the camera lens, which may include radial distortion coefficients and tangential distortion coefficients, etc.

[0073] It should be noted that during actual driving, due to factors such as vehicle vibration, temperature changes, and mechanical structural deformation, the sensor's installation position may shift slightly, causing the initial calibration parameters to become inaccurate.

[0074] This invention automatically optimizes and corrects the extrinsic and intrinsic parameters of the camera to be calibrated by using data collected during normal vehicle operation, based on the initial calibration parameters.

[0075] Step 120: Based on the lidar point cloud data, the inertial measurement unit data, and the reference camera image data, construct a point cloud map containing color texture and the vehicle's pose trajectory, and convert the point cloud map into a surface mesh model.

[0076] Constructing a point cloud map containing color texture and the vehicle's pose trajectory refers to fusing LiDAR point cloud data, inertial measurement unit data, and reference camera image data, and outputting two results simultaneously.

[0077] First, the vehicle's pose information (position and attitude) at various moments during the data collection process constitutes the pose trajectory;

[0078] Second, the point cloud data of each frame is registered into a unified world coordinate system according to the corresponding pose to form a complete environmental point cloud map, and color information is added to each point in the point cloud using reference camera images.

[0079] A point cloud map is a three-dimensional geometric representation of the environment surrounding a vehicle, consisting of a large number of spatial points. When a point cloud map includes color textures, each point has color information (R, G, B) in addition to its three-dimensional coordinates (X, Y, Z) and reflectance intensity (I), forming a colored point cloud.

[0080] Color information originates from images captured by a reference camera. During map construction, LiDAR points are projected onto the image plane of the reference camera based on calibration parameters between the LiDAR and the reference camera, thereby obtaining the pixel color values ​​at the corresponding locations. This point cloud map, containing color textures, provides the foundation for subsequent multi-view rendering, enabling the rendered image to be similar in color and texture to images captured by the actual camera.

[0081] The pose trajectory is the pose sequence {T_WL_i} of the vehicle at various moments during the data acquisition process. Each pose T_WL_i describes the transformation relationship between the vehicle (or lidar) coordinate system and the world coordinate system at the i-th frame, including rotation and translation.

[0082] The pose trajectory provides crucial information for subsequent multi-view rendering. By combining the pose trajectory with the extrinsic parameters between the camera and the LiDAR, the pose of each camera in the world coordinate system at each moment can be calculated, thereby determining the rendering viewpoint.

[0083] The technology for constructing point cloud maps and pose trajectories can be achieved using Simultaneous Localization and Mapping (SLAM) technology.

[0084] SLAM (Simultaneous Localization and Mapping) technology is a class of algorithms capable of simultaneously estimating sensor poses and constructing environmental maps in unknown environments. In this invention, a multi-sensor fusion SLAM method integrating LiDAR, inertial measurement unit (IMU), and camera can be employed. By comprehensively utilizing the precise ranging capability of LiDAR, the high-frequency motion measurement capability of IMU, and the rich texture information of the camera, high-precision pose estimation and high-quality environmental maps can be obtained.

[0085] Converting a point cloud map into a surface mesh model refers to performing surface reconstruction processing on the constructed point cloud map to generate a mesh model composed of polygonal patches (usually triangular patches).

[0086] A surface mesh model is a three-dimensional representation of an object's surface that approximates the surface using polygonal meshes.

[0087] Compared to discrete point clouds, surface mesh models have continuous surfaces, which better represent the geometry and spatial continuity of objects in the environment. Each facet of a surface mesh model can be associated with color and texture information to generate realistic images during rendering.

[0088] Methods for converting point clouds into surface mesh models include, but are not limited to, triangulation algorithms, Poisson surface reconstruction algorithms, and moving cube algorithms; these algorithms generate triangular meshes that approximate the surface of real objects based on the positional distribution of points in the point cloud.

[0089] Step 130: Based on the pose trajectory and the initial calibration parameters, render the surface mesh model to the viewpoint of the camera to be calibrated, and generate a rendered image and a rendered depth map corresponding to the camera to be calibrated.

[0090] In this application, rendering the surface mesh model to the viewpoint of the camera to be calibrated means using the pose of the camera to be calibrated as a virtual viewing viewpoint to project and render the surface mesh model to generate a two-dimensional image from that viewpoint.

[0091] Specifically, for each camera k to be calibrated and each time frame i, the pose of the LiDAR in the world coordinate system at that moment is first obtained based on the pose trajectory T_WL_i; then, combined with the initial extrinsic parameters {R_LC_k, t_LC_k} of the camera to be calibrated, the pose T_WC_k_i of the camera to be calibrated in the world coordinate system at that moment is calculated; finally, using the pose and the initial intrinsic parameters {K_k, d_k} of the camera to be calibrated, the surface mesh model is projected and rendered onto the imaging plane of the camera.

[0092] The rendered image is an RGB image obtained through the above rendering process, and its pixel values ​​come from the color texture of each facet in the surface mesh model.

[0093] Since the surface mesh model is converted from a color point cloud map and contains color information of the environment, the rendered image should visually resemble a real image taken by an actual camera at the same viewpoint.

[0094] The difference between the rendered image and the real image mainly stems from errors in the initial calibration parameters. When there are deviations in the initial extrinsic or intrinsic parameters, the camera pose and projection model used during rendering are not entirely consistent with the real situation, resulting in differences in the position, size, and shape of objects in the rendered image compared to the real image. It is precisely by utilizing this difference that subsequent steps can correct the calibration parameters through optimization algorithms.

[0095] A rendered depth map is a depth information image generated simultaneously during the rendering process, where each pixel value represents the distance from the corresponding 3D point on the surface mesh model to the camera's imaging plane. The rendered depth map provides crucial depth information for subsequent steps of backprojecting 2D image features into 3D space.

[0096] Specifically, for any pixel in the rendered image, by combining its depth value and camera intrinsic parameters, the two-dimensional pixel can be back-projected into three-dimensional space to obtain its three-dimensional coordinates in the camera coordinate system.

[0097] For each frame of data from each camera to be calibrated, the onboard computing platform generates a corresponding rendered image RImage_k_i and a rendered depth map Depth_k_i. These rendered images and depth maps provide a benchmark for subsequent feature extraction and matching.

[0098] Step 140: Based on the real image data, the rendered image, and the rendered depth map, establish the correspondence between the two-dimensional image features and the three-dimensional spatial points in the real image data;

[0099] In this application, two-dimensional image features refer to representative image features extracted from real image data. Image features are local patterns or structures in an image that can be stably detected and described; commonly used image feature types include point features and line features.

[0100] A three-dimensional point is a point that has definite three-dimensional coordinates in a three-dimensional spatial coordinate system (such as the lidar coordinate system or the world coordinate system).

[0101] Establishing the correspondence between two-dimensional image features and three-dimensional spatial points means determining which position in three-dimensional space corresponds to a two-dimensional feature point detected in a real image.

[0102] This 2D-3D correspondence is the basis for establishing constraint equations in camera calibration: by minimizing the error between the predicted position and the actual observed position of a 3D point after it has been projected onto the image plane according to the calibration parameters, the calibration parameters can be optimized.

[0103] In this invention, the method for establishing a 2D-3D correspondence is to use a rendered image as an intermediate bridge. The specific process includes:

[0104] First, feature extraction is performed on the real image data and the rendered image to obtain the features in the two images respectively. Then, the two sets of features are matched to find the corresponding feature pairs. Next, the depth information of the matching features in the rendered image is obtained using the rendered depth map and back-projected onto the three-dimensional space. Finally, since the real image features match the rendered image features, the three-dimensional space position corresponding to the real image features is the three-dimensional coordinate obtained by back-projecting the matched rendered image features.

[0105] The correspondence between two-dimensional image features and three-dimensional spatial points can be represented as a dataset, where each element contains the two-dimensional pixel coordinates of the feature in the real image and the corresponding three-dimensional spatial coordinates of that feature.

[0106] Step 150: Construct a reprojection error objective function based on the correspondence, and optimize the extrinsic and intrinsic parameters of the camera to be calibrated by minimizing the objective function.

[0107] In this application, reprojection error refers to the deviation between the predicted pixel coordinates obtained by projecting a three-dimensional spatial point onto the image plane according to the currently estimated calibration parameters and the actual pixel coordinates of the real image features corresponding to that three-dimensional point.

[0108] Reprojection error is the most commonly used error metric in camera calibration. Its geometric meaning is intuitive: if the calibration parameters are accurate, the position of a 3D point after projection should be consistent with its actual observed position in the image, and the reprojection error is zero; conversely, the less accurate the calibration parameters are, the greater the reprojection error.

[0109] The reprojection error objective function is a mathematical function constructed based on the reprojection errors of all 2D-3D correspondences. It is a function of the parameters to be optimized (extrinsic and intrinsic parameters). The objective function is usually in the form of an aggregation of all reprojection errors, and the goal of optimization is to minimize this objective function.

[0110] The optimization of the extrinsic and intrinsic parameters of the camera to be calibrated refers to finding the extrinsic and intrinsic parameters that minimize the objective function through mathematical optimization methods.

[0111] Commonly used optimization methods include nonlinear least squares methods, which can be solved using algorithms such as the Levenberg-Marquardt algorithm and the Gauss-Newton algorithm.

[0112] In practice, a mature optimization solver can be used to solve this nonlinear optimization problem.

[0113] The optimization process uses the initial calibration parameters as the initial values ​​for iteration and searches for parameter values ​​that minimize the objective function within their neighborhood.

[0114] Since there are only minor deviations caused by factors such as vibration and temperature changes, the initial calibration parameters are usually quite close to the true values. The optimization problem has a good starting point, which is conducive to the algorithm quickly converging to the global optimum or a local optimum close to the global optimum.

[0115] The extrinsic parameters to be optimized can include rotation and translation parameters from the LiDAR coordinate system to the camera coordinate system. The rotation parameters can be represented by three Euler angles to correct the initial values, and the translation parameters can be represented by three translation amounts to correct the initial values.

[0116] The intrinsic parameters to be optimized may include correction values ​​for focal length, principal point coordinates, and distortion coefficients. During the optimization process, you can choose to optimize all or some of the intrinsic parameters, depending on the actual needs.

[0117] It should be noted that since the reference camera, such as the forward-looking camera, has already built a map together with SLAM and LiDAR, its initial extrinsic parameters are relatively accurate. In the optimization process, the reference camera can be fixed as a reference, and the parameters of the other cameras to be calibrated should be optimized.

[0118] After optimization, the optimized extrinsic and intrinsic parameters of each camera to be calibrated were obtained, thus completing the core optimization process of camera calibration.

[0119] The final output consists of optimized calibration parameters for the cameras to be calibrated, including the external parameters (rotation matrix and translation vector) of each camera relative to the LiDAR coordinate system, as well as the internal parameters (focal length, principal point, distortion coefficients, etc.). These updated calibration parameters can be used in subsequent multi-sensor fusion sensing algorithms, significantly improving the accuracy of point cloud and image fusion and reducing perception bias caused by calibration errors.

[0120] In this application, by acquiring multi-sensor data and initial calibration parameters, a point cloud map containing color texture and pose trajectory is constructed. The point cloud map is then converted into a surface mesh model and rendered to the viewpoint of the camera to be calibrated, generating a rendered image and a rendered depth map. A correspondence between 2D image features and 3D spatial points is established based on the real image and the rendered image. Finally, the extrinsic and intrinsic parameters of the camera to be calibrated are optimized by minimizing the reprojection error objective function. This method eliminates the need for an additional calibration scene, relying solely on sensor data collected during normal vehicle operation to automatically correct camera calibration parameters. This solves the problem of traditional static calibration methods requiring dedicated sites and manual operation. Furthermore, it compensates for calibration parameter drift caused by factors such as vehicle vibration and temperature changes, improving the accuracy and reliability of the multi-sensor fusion perception system.

[0121] Optionally, establishing the correspondence between two-dimensional image features and three-dimensional spatial points in the real image data includes:

[0122] Feature extraction and matching are performed on the real image data and the rendered image to obtain feature matching pairs, wherein the feature matching pairs contain mutually matching real image features and rendered image features;

[0123] Based on the rendered depth map, the pose trajectory, and the initial calibration parameters, the rendered image features are back-projected into three-dimensional space to obtain the three-dimensional space point coordinates corresponding to the rendered image features;

[0124] The two-dimensional pixel coordinates of the real image features are associated with the three-dimensional spatial point coordinates corresponding to the rendered image features, which is used as the correspondence.

[0125] In this application, feature extraction refers to the process of detecting and describing representative local features from an image.

[0126] Feature extraction typically involves two stages: feature detection, which determines the location of features in an image; and feature description, which generates a description vector for each feature for subsequent matching.

[0127] Feature matching refers to the process of establishing a correspondence between features extracted from two images. Feature matching is usually based on the similarity of feature description vectors: for a certain feature in one image, the feature with the most similar description vector in the feature set of another image is found as its matching point.

[0128] To improve the accuracy of matching, strategies such as ratio testing, cross-validation, and geometric consistency testing can be used to filter out incorrect matches.

[0129] Feature matching pairs are the result of feature matching. Each feature matching pair contains two corresponding features: one from the real image (real image feature) and one from the rendered image (rendered image feature).

[0130] Feature matching pairs indicate that the two features describe the same physical location or structure in their respective images.

[0131] There are various methods for feature extraction and matching. Traditional feature detection and description algorithms can be used, as well as deep learning-based feature detection and matching methods. Deep learning-based methods generally have better robustness and can obtain more stable matching results under conditions such as changes in lighting and viewing angle.

[0132] The vehicle-mounted computing platform uses a rendering depth map to upscale two-dimensional features in a rendered image to three-dimensional space.

[0133] Backprojecting rendered image features into three-dimensional space means calculating the coordinates of the feature in three-dimensional space by using the inverse process of the camera projection model, based on the pixel coordinates and corresponding depth values ​​of the rendered image feature.

[0134] The specific back projection process is as follows: Let the pixel coordinates of the rendered image feature be (u, v), and obtain the depth value Z at this location from the rendered depth map. The camera intrinsic parameters include the focal length (f_x, f_y) and the principal point (c_x, c_y). Then, the three-dimensional coordinates of this feature in the camera coordinate system can be calculated by the back projection formula.

[0135] After obtaining the three-dimensional coordinates in the camera coordinate system, they can be further transformed to the lidar coordinate system or the world coordinate system based on the pose trajectory and initial calibration parameters.

[0136] Since the initial extrinsic parameters have been considered during rendering, the 3D points obtained by backprojection should fall into the corresponding positions in the lidar coordinate system under the initial calibration. This coordinate system transformation allows 3D points from different frames and different cameras to be compared and processed in a unified coordinate system.

[0137] The onboard computing platform establishes the final 2D-3D correspondence data. Since the real image features and rendered image features in the feature matching pairs correspond to each other and describe the same physical location, the two-dimensional pixel coordinates of the real image features on the image plane should correspond to the three-dimensional spatial coordinates obtained by back-projecting the rendered image features. By associating the two, the 2D-3D correspondence of the real image features is obtained.

[0138] In this application, feature matching pairs are obtained by extracting and matching features from real images and rendered images. The rendered image features are then back-projected into 3D space using a rendered depth map to obtain 3D coordinates. Finally, the 2D coordinates of the real image features are associated with their corresponding 3D coordinates, thus establishing a correspondence between 2D image features and 3D spatial points. By using the rendered image as an intermediary bridge, an indirect but effective connection is established between the real image and the 3D map, avoiding the difficult cross-modal matching between heterogeneous data and improving the accuracy and robustness of the correspondence establishment.

[0139] Optionally, the feature matching pairs include: point feature matching pairs and line segment feature matching pairs; feature extraction and matching of the real image data and the rendered image includes:

[0140] Point feature extraction and line segment feature extraction are performed on the real image data to obtain a first point feature set and a first line segment feature set;

[0141] Point feature extraction and line segment feature extraction are performed on the rendered image to obtain a second point feature set and a second line segment feature set;

[0142] The first point feature set and the second point feature set are matched to obtain the point feature matching pair;

[0143] The first line segment feature set and the second line segment feature set are matched to obtain the line segment feature matching pair.

[0144] In this application, feature matching pairs include point feature matching pairs and line segment feature matching pairs.

[0145] Point feature matching pairs are based on point features, such as corners and spots, and establish matching relationships. Each point feature matching pair contains a point feature in the real image and a matching point feature in the rendered image.

[0146] Line segment feature matching pairs are based on line segment features, such as the edges of straight lines in an image, to establish matching relationships. Each line segment feature matching pair contains a line segment in the real image and a matching line segment in the rendered image.

[0147] In this application, point feature extraction refers to detecting local point features such as corners and spots in an image and calculating their descriptors.

[0148] Possible point feature detection algorithms include, but are not limited to: Harris corner detection, FAST corner detection, SuperPoint algorithm, etc.

[0149] For example, the SuperPoint algorithm can be used to extract local feature descriptors such as corner points. The first set of extracted point features contains all detected point features and their descriptors in the real image.

[0150] Line segment feature extraction refers to the detection of straight line segments from an image. Road scenes contain abundant straight line structures, such as lane lines, road edges, building edges, and lampposts. These straight line features can provide important geometric constraints for calibration.

[0151] Possible line segment detection algorithms include, but are not limited to: Hough transform-based line detection, LSD (Line Segment Detector) line segment detector, and CNN-based detector. The extracted first line segment feature set contains all detected line segments in the real image. Each line segment can be represented by the pixel coordinates of its two endpoints or by a line equation.

[0152] The same point feature and line segment feature extraction process as the real image is performed on the rendered image to obtain the second point feature set and the second line segment feature set.

[0153] Point feature matching establishes a correspondence between a first set of point features and a second set of point features. Matching is usually based on the similarity of feature descriptors, such as calculating the Euclidean distance or cosine similarity between descriptors, and selecting the feature with the closest distance as a matching candidate.

[0154] For example, the SuperGlue network can be used to match feature point pairs between two images. To improve matching quality, strategies such as ratio testing, cross-validation, and geometric consistency checks can be used to filter out incorrect matches.

[0155] Line segment feature matching establishes a correspondence between a first set of line segment features and a second set of line segment features. Line segment matching is more challenging than point matching because the endpoints of line segments may not be perfectly consistent due to occlusion or detection accuracy issues.

[0156] Line segment matching can be used to match line segments in real and rendered images based on geometric relationships, such as epipolar constraints and parallel / perpendicular relationships.

[0157] In this application, by simultaneously extracting and matching point features and line segment features, two types of matching information are obtained: point feature matching pairs and line segment feature matching pairs. This point-line feature fusion method is more robust than using only a single type of feature: when there are many corner points in the environment, point features provide the main constraints; when there are few obvious corner points in the environment but the straight line structure is obvious, line features can still provide effective constraints. The fusion of the two types of features enables the calibration method to adapt to a wider range of scene conditions, improving its adaptability to different scenarios.

[0158] Optionally, back-projecting the rendered image features into three-dimensional space includes:

[0159] For the point feature matching pair, the depth value of the corresponding point feature in the second point feature set is obtained from the rendered depth map, and back projection is performed based on the depth value, the pose trajectory and the initial calibration parameters to obtain the three-dimensional point coordinates of the corresponding point feature;

[0160] For the line segment feature matching pair, the depth values ​​of multiple sampling points on the corresponding line segment feature in the second line segment feature set are obtained from the rendered depth map. Based on the depth values, the pose trajectory and the initial calibration parameters, back projection is performed to obtain the coordinates of multiple three-dimensional sampling points of the corresponding line segment feature.

[0161] The three-dimensional point coordinates and the coordinates of the plurality of three-dimensional sampling points together constitute the three-dimensional spatial point coordinates.

[0162] In this application, for the point feature matching pair, the depth value of the corresponding point feature in the second point feature set is obtained from the rendered depth map, and back projection is performed based on the depth value, the pose trajectory and the initial calibration parameters to obtain the three-dimensional point coordinates of the corresponding point feature.

[0163] Specifically, for each matching point, the depth value of that pixel is obtained from the rendered depth map. Combined with the camera intrinsic parameters, it can be back-projected onto the camera coordinate system to obtain the corresponding 3D point. Since the initial extrinsic parameters have been considered during rendering, this 3D point should fall on the corresponding position in the LiDAR coordinate system under the initial calibration.

[0164] For the line segment feature matching pair, the depth values ​​of multiple sampling points on the corresponding line segment feature in the second line segment feature set are obtained from the rendered depth map. Based on the depth values, the pose trajectory and the initial calibration parameters, back projection is performed to obtain the coordinates of multiple three-dimensional sampling points of the corresponding line segment feature.

[0165] Since line segments are continuous geometric elements rather than single points, multiple sampling points need to be selected on the line segment for back projection.

[0166] Specifically, for each pair of matching line segments, the depth of a series of pixels on the line segment can be obtained from the rendering depth map. These pixels are then back-projected in batches into several three-dimensional points in the rendering camera coordinate system, which are regarded as the set of sampling points of the physical line in three-dimensional space.

[0167] The methods for selecting sampling points include, but are not limited to, selecting the two endpoints of the line segment; uniformly sampling several intermediate points on the line segment; or using an adaptive sampling strategy to determine the sampling density based on the length of the line segment.

[0168] The coordinates of three-dimensional points and the coordinates of the multiple three-dimensional sampling points together constitute the coordinates of the three-dimensional spatial points. That is, the coordinates of the three-dimensional points obtained by back-projection of all point features, plus the coordinates of the three-dimensional sampling points obtained by back-projection of the sampling points on all line segment features, together form a complete set of three-dimensional spatial points used to establish the 2D-3D correspondence, covering both discrete point features and continuous line features, providing constraints for subsequent solutions.

[0169] In this application, point features are obtained by directly querying depth and backprojecting to obtain a single 3D point; line segment features are obtained by sampling multiple points on the line segment, querying depth for each point, and backprojecting to obtain multiple 3D sampling points. This differentiated processing for different feature types enables the backprojection process to accurately elevate various 2D features to 3D space, providing accurate 3D reference points for subsequent reprojection error calculation.

[0170] Optionally, a reprojection error objective function is constructed based on the correspondence, including:

[0171] For the point feature matching pair, the three-dimensional point coordinates are projected onto the image plane according to the current extrinsic and intrinsic parameters to obtain the predicted point coordinates. The distance between the predicted point coordinates and the pixel coordinates of the corresponding point feature in the first point feature set is calculated as the point reprojection error.

[0172] For the line segment feature matching pair, the coordinates of the multiple three-dimensional sampling points are projected onto the image plane according to the current extrinsic and intrinsic parameters to obtain multiple predicted projection points. The distance from each predicted projection point to the line containing the corresponding line segment feature in the first line segment feature set is calculated as the line segment reprojection error.

[0173] The objective function for the reprojection error is constructed based on the weighted sum of the point reprojection error and the line segment reprojection error of the vehicle at multiple acquisition times.

[0174] In this application, for the point feature matching pair, the three-dimensional point coordinates are projected onto the image plane according to the current extrinsic and intrinsic parameters to obtain the predicted point coordinates. The distance between the predicted point coordinates and the pixel coordinates of the corresponding point feature in the first point feature set is calculated as the point reprojection error.

[0175] Specifically, in a certain point feature matching pair, the pixel coordinates of the corresponding point feature in the first point feature set, that is, the point feature in the real image, are (u_obs, v_obs), and the corresponding three-dimensional point coordinates are P_L=(x_L, y_L, z_L)^T (in the lidar coordinate system).

[0176] The projection process is as follows:

[0177] (1) Coordinate system transformation: Based on the currently estimated extrinsic parameters (rotation matrix R_LC and translation vector t_LC), the 3D points are transformed from the lidar coordinate system to the camera coordinate system:

[0178] P_C=R_LC×P_L+t_LC, record P_C=(X_C,Y_C,Z_C)^T.

[0179] Where R_LC is a 3×3 rotation matrix, representing the rotation transformation from the lidar coordinate system to the camera coordinate system; t_LC is a 3×1 translation vector, representing the translation transformation from the lidar coordinate system to the camera coordinate system.

[0180] (2) Normalized projection: Projecting the 3D points in the camera coordinate system onto the normalized plane, the coordinates of the normalized plane are:

[0181] x_n = X_C / Z_C;

[0182] y_n = Y_C / Z_C;

[0183] (3) Distortion model: Consider the distortion function Distort(x_n,y_n,d). Based on the currently estimated distortion coefficient d, apply distortion to the normalized coordinates to obtain the distorted coordinates (x_d,y_d):

[0184] (x_d,y_d)=Distort(x_n,y_n,d);

[0185] (4) Pixel coordinate calculation: Based on the currently estimated intrinsic parameters (focal length f_x, f_y and principal point c_x, c_y), the predicted coordinates in the pixel plane are:

[0186] u_pred=f_x × x_d+c_x;

[0187] v_pred = f_y × y_d + c_y;

[0188] (5) Error Calculation: Let the coordinates of the corresponding point in the real image be (u_obs, v_obs), then the reprojection error of a single point is:

[0189] e_u = u_pred - u_obs;

[0190] e_v = v_pred - v_obs;

[0191] The overall point reprojection error can be expressed as the sum of squared errors of all matching points:

[0192] E_point=Σ(e_u 2 + e_v 2 );

[0193] For the line segment feature matching pair, the coordinates of the multiple three-dimensional sampling points are projected onto the image plane according to the current extrinsic and intrinsic parameters to obtain multiple predicted projection points. The distance from each predicted projection point to the line containing the corresponding line segment feature in the first line segment feature set is calculated as the line segment reprojection error.

[0194] Unlike point features, the reprojection error of line segment features is not a simple point-to-point distance, but rather the distance from a point to a line.

[0195] This is because a line segment observed in a real image represents a straight line, or a segment of a straight line. Ideally, all points on the corresponding three-dimensional line segment should fall on this straight line after being projected into the image. Therefore, the line segment reprojection error is defined as the distance from the projection point to the observed straight line.

[0196] The specific calculation process is as follows:

[0197] (1) Determine the equation of the observed line: Let the endpoints of the matching line segment in the real image be (u1_obs, v1_obs) and (u2_obs, v2_obs). The corresponding observed line can be written as the equation of the line on the image plane:

[0198] a×u+b×v+c=0;

[0199] a = v2_obs - v1_obs;

[0200] b = u1_obs - u2_obs;

[0201] c=u2_obs×v1_obs-u1_obs×v2_obs;

[0202] Where (a, b) is the unit normal vector.

[0203] (2) Projecting three-dimensional sampling points: For multiple three-dimensional sampling points corresponding to the line segment, they are projected onto the image plane according to the above point projection process (coordinate system transformation → normalized projection → distortion model → pixel coordinate calculation) to obtain multiple predicted projection points.

[0204] (3) Calculate the distance from the point to the line: The distance from any predicted projection point (u_pred, v_pred) to the observed line is:

[0205] dist=(a×u_pred+b×v_pred+c) / sqrt(a 2 +b 2 );

[0206] (4) Accumulate the line segment error: The line segment reprojection distance error is defined as the sum of the squares of the distances between all sampling points (e.g., the two ends and the midpoint of the line segment, or the midpoints of several discretely sampled line segments):

[0207] E_line=Σ(dist 2 );

[0208] Based on the weighted sum of the point reprojection error and the line segment reprojection error of the vehicle at multiple acquisition times, a reprojection error objective function is constructed. The joint optimization objective function is:

[0209] E_total=w_point×E_point+w_line×E_line;

[0210] Here, w_point and w_line are weighting coefficients used to balance the relative contributions of point features and line features in the optimization.

[0211] It should be noted that both E_point and E_line mentioned above include the cumulative error of multiple acquisition times (multiple frames of data).

[0212] Joint optimization using multi-frame data is employed, which means simultaneously using N frames of data to solve for a unified set of calibration parameters. This is equivalent to adding geometric consistency constraints of multiple frames to the objective function, making the optimization more stable and less prone to local extrema caused by random noise in a single frame.

[0213] By using nonlinear optimization solvers such as Ceres, the objective function is minimized to obtain the optimal solution for each optimization parameter, thereby updating the extrinsic parameters of the lidar and the camera, as well as the intrinsic parameters of the camera.

[0214] In this application, by defining separate calculation methods for point reprojection error and line segment reprojection error, and constructing an objective function by weighted summation of point and line feature errors, joint optimization integrating point and line features is achieved. The point reprojection error is calculated as the sum of squares of the pixel coordinate differences between the predicted and observed points; the line segment reprojection error is calculated as the sum of squares of the distances from the predicted projection point to the observed line, which better reflects the geometric characteristics of line segment features. Simultaneously, the optimization process integrates errors from both point and line features, improving adaptability to different scenarios. Multi-frame joint optimization enhances the observability of the optimization problem, reduces the impact of single-frame noise, and makes the calibration results more stable and reliable.

[0215] Optionally, there are multiple cameras to be calibrated; after optimizing the extrinsic and intrinsic parameters of each camera, the method further includes:

[0216] Identify adjacent pairs of cameras to be calibrated where their fields of view overlap.

[0217] Feature extraction and matching are performed on the real image data of the two cameras to be calibrated in the adjacent camera pair to obtain feature matching pairs between cameras;

[0218] Based on the feature matching pairs between cameras, relative pose constraints between cameras are constructed, and joint optimization is performed to correct the extrinsic parameters of the adjacent pairs of cameras to be calibrated.

[0219] In this application, adjacent camera pairs to be calibrated refer to a combination of two cameras installed on a vehicle that are adjacent in position and have overlapping or adjacent fields of view. For example, on a vehicle equipped with surround view cameras, there may be overlapping fields of view between the left front camera and the front view camera, and similarly, there may be overlapping fields of view between the front view camera and the right front camera, or between the left front camera and the left rear camera.

[0220] The methods for determining adjacent pairs of cameras to be calibrated can be as follows: First, determine in advance which pairs of cameras may have overlapping fields of view based on the camera's installation position and field of view. Second, analyze the images acquired by each camera to detect whether there are any commonly visible scene areas or features.

[0221] Camera-to-camera feature matching pairs establish feature correspondences between real images captured by the two cameras in an adjacent pair of cameras to be calibrated. Unlike the matching between real and rendered images in the previous steps, this involves direct matching between two real images (in this case, matching between real images is used, and rendered images are not required).

[0222] Within the overlapping fields of view of adjacent cameras, the same physical scene structure will be imaged in both camera images. By performing feature extraction and feature matching on the two images, feature pairs representing the same physical location can be found, thereby obtaining the correspondence between the coordinates of the two camera images.

[0223] Feature matching between cameras can be performed between two time-synchronized images, i.e., images captured by two adjacent cameras at the same time. The steps are similar to the feature extraction and matching described above: feature points and line segments are extracted and matched from images captured by adjacent cameras.

[0224] Relative pose constraints between cameras are geometric constraints established based on feature matching relationships between adjacent cameras. When two cameras observe the same 3D point, there are definite geometric relationships between the image position of that 3D point in the images of the two cameras, the intrinsic parameters of the two cameras, and the relative pose, relative rotation, and relative translation between the two cameras. This relationship can be described using epipolar geometry, with the core constraints being essential matrix constraints or fundamental matrix constraints.

[0225] Joint optimization refers to jointly optimizing the relative attitude parameters between the two cameras while keeping the basic framework of the extrinsic parameters of the LiDAR and the camera unchanged, or equivalently optimizing the difference constraints between the two relative to the extrinsic parameters of the LiDAR, in order to reduce the inconsistency of feature projection in the overlapping area between the cameras.

[0226] Incorporating this joint optimization between pairs of cameras into the overall optimization process can improve the global consistency of the extrinsic parameters of all cameras in the system and avoid the minor inconsistencies that may arise from optimizing only one camera at a time using LiDAR.

[0227] Multi-camera joint calibration can be performed based on epipolar geometric constraints between images or the mixed error between triangulated reconstruction points and lidar points, so that camera-to-camera and camera-to-liquid data can be accurately aligned after unified calibration.

[0228] By performing the above processing on all adjacent pairs of cameras to be calibrated, the relative attitude relationship of all cameras in the system will be more coordinated, and the calibration results of each camera to be calibrated will achieve consistent high accuracy relative to the reference camera / LiDAR benchmark.

[0229] In this application, by identifying adjacent camera pairs with overlapping fields of view, feature matching is performed on their real images to construct relative pose constraints between cameras and conduct joint optimization, thus achieving global consistency correction for a multi-camera system. This method overcomes the relative pose inconsistency problem that may occur when calibrating each camera independently, improves the overall accuracy of the multi-camera and LiDAR fusion system, and ensures reliable alignment of the perception system across the omnidirectional coverage of each camera. This provides a reliable calibration foundation for subsequent tasks such as surround-view perception, panoramic stitching, and bird's-eye view fusion.

[0230] Optionally, constructing a point cloud map including color texture and the vehicle's pose trajectory includes:

[0231] The inertial measurement unit data, the lidar point cloud data, and the reference camera image data are fused using a tightly coupled method.

[0232] Based on the fused data, synchronous positioning and mapping are performed to generate the point cloud map and the pose trajectory.

[0233] In this application, tight coupling is a fusion strategy in multi-sensor fusion. In tight coupling, the raw measurement data of each sensor directly participate in the joint optimization or filtering process, and the constraint relationship between sensors is uniformly modeled.

[0234] The advantage of tight coupling is that it can make fuller use of the complementary information of each sensor. When the data quality of a certain sensor deteriorates (such as when the camera is in low light conditions or when the lidar is blocked at close range), the data of other sensors can provide effective compensation, thereby obtaining more robust and accurate fusion results.

[0235] Fusion of inertial measurement unit data, point cloud data, and reference camera image data involves combining measurement information from these three sensors.

[0236] This SLAM system tightly couples and fuses the inertial measurement unit (IMU), LiDAR scans, and camera image frames. The IMU provides high-frequency motion state prediction; the LiDAR point cloud provides accurate 3D geometric measurements; and the reference camera images provide rich texture and visual feature information. In one embodiment, existing fast laser-inertial-visual odometry can be used for fusion.

[0237] Simultaneous Localization and Mapping (SLAM) is a technology that can estimate sensor poses and build environmental maps in real time in unknown environments.

[0238] SLAM systems typically consist of two parts: a front-end and a back-end. The front-end is responsible for processing sensor data, extracting features, performing data association, and estimating motion. The back-end is responsible for globally optimizing historical data, eliminating accumulated errors, and ensuring global consistency of maps and trajectories.

[0239] Based on a tightly coupled fusion-based multi-sensor SLAM system, the vehicle's pose trajectory (odometer information) is output and a dense point cloud map of the environment is constructed.

[0240] Because it incorporates visual information from the camera, the point cloud map generated by SLAM includes color texture, forming a colored point cloud where each point contains XYZI information and RGB color. This provides a foundation for subsequent comparisons in the image domain.

[0241] In this application, by employing a tightly coupled approach to fuse inertial measurement unit (IMU) data, lidar point cloud data, and reference camera image data, and performing synchronous localization and mapping based on the fused data, high-precision pose and trajectory estimation and high-quality color point cloud map construction are achieved. This tightly coupled fusion method fully leverages the complementary advantages of each sensor, improving mapping accuracy and robustness, and providing a reliable data foundation for subsequent multi-view rendering and calibration optimization.

[0242] In one alternative embodiment, Figure 2 The overall flowchart provided for this invention is as follows: Figure 2 As shown, the camera calibration system mainly consists of a data acquisition terminal, a calibration processing unit, and an output terminal.

[0243] The data acquisition unit is responsible for acquiring multi-source sensor data during vehicle operation. Specifically, it includes a lidar for scanning three-dimensional environmental information, an inertial measurement unit for measuring vehicle motion attitude data, and eleven cameras arranged around the vehicle, including one reference camera and ten cameras to be calibrated.

[0244] The raw data collected by these sensors is transmitted to the calibration processing unit, which serves as the core computing module. This unit can be implemented by an onboard computing platform or a remote server and is responsible for executing subsequent data fusion, mapping, and parameter optimization algorithms.

[0245] After calculation by the calibration processing unit, the system finally outputs the calibration result, which specifically includes the extrinsic parameters of each camera relative to the lidar and the intrinsic parameters of each camera itself, thereby realizing the automatic correction of sensor parameters.

[0246] In one alternative embodiment, Figure 3 This is a schematic diagram of the overall steps of the present invention, as shown below. Figure 3 As shown, firstly, the system receives N frames of LiDAR point cloud data, N frames of camera image data, inertial measurement unit data, and the initial calibration parameters of the camera to be calibrated.

[0247] After acquiring the data, SLAM mapping is performed using data from LiDAR, inertial measurement unit, and forward-looking camera to construct a point cloud map containing color texture.

[0248] Subsequently, the generated point cloud map is converted into a mesh model of a continuous surface, and the mesh model is rendered to the viewpoint of other cameras to be calibrated, excluding the forward-looking camera, according to the initial calibration parameters, thereby generating a rendered image.

[0249] Next, feature matching is performed between real image data and rendered images to establish the correspondence between two-dimensional image features and three-dimensional spatial points, and line segment features in the image are further extracted to supplement geometric constraints.

[0250] Based on the extracted point and line segment features, a reprojection error objective function is constructed for each camera, and its extrinsic and intrinsic parameters are optimized.

[0251] Finally, for adjacent cameras with overlapping fields of view, joint optimization processing is performed to correct global consistency, and the refined extrinsic and intrinsic parameters are output as calibration results.

[0252] Figure 4 This is a schematic diagram of the camera calibration device provided by the present invention, as shown below. Figure 4 As shown, it includes:

[0253] The acquisition module 410 is used to acquire the lidar point cloud data, inertial measurement unit data, reference camera image data and real image data of the camera to be calibrated of the vehicle, and to acquire the initial calibration parameters of the camera to be calibrated.

[0254] The construction module 420 is used to construct a point cloud map containing color texture and the vehicle's pose trajectory based on the lidar point cloud data, the inertial measurement unit data and the reference camera image data, and convert the point cloud map into a surface mesh model.

[0255] The rendering module 430 is used to render the surface mesh model to the viewpoint of the camera to be calibrated according to the pose trajectory and the initial calibration parameters, and generate a rendered image and a rendered depth map corresponding to the camera to be calibrated.

[0256] The module 440 is used to establish a correspondence between two-dimensional image features and three-dimensional spatial points in the real image data based on the real image data, the rendered image, and the rendered depth map.

[0257] The calibration module 450 is used to construct a reprojection error objective function based on the correspondence, and optimize the extrinsic and intrinsic parameters of the camera to be calibrated by minimizing the objective function.

[0258] In this application, a point cloud map with color texture is constructed by fusing LiDAR, inertial measurement unit, and reference camera, and then converted into a surface mesh model and rendered to the view of the camera to be calibrated. This allows the camera to establish a geometric relationship with the LiDAR coordinate system without direct overlap of the view with the LiDAR, breaking through the dependence of traditional calibration methods on dedicated calibration scenes and calibration boards. By matching the features of the rendered image with the real image and combining it with the rendered depth map, accurate mapping of two-dimensional image features to three-dimensional spatial points is achieved, providing reliable geometric constraints for calibration optimization. Optimization based on the reprojection error objective function can simultaneously correct the extrinsic and intrinsic parameters of the camera to be calibrated, compensating for calibration parameter drift caused by factors such as vehicle vibration and temperature changes. This enables automatic and fine calibration of sensor calibration parameters during normal vehicle operation, improving the calibration accuracy and long-term stability of the multi-sensor fusion perception system.

[0259] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a camera calibration method, which includes: acquiring LiDAR point cloud data of the vehicle, inertial measurement unit data, reference camera image data, and real image data of the camera to be calibrated, and acquiring the initial calibration parameters of the camera to be calibrated;

[0260] Based on the lidar point cloud data, the inertial measurement unit data, and the reference camera image data, a point cloud map containing color texture and the vehicle's pose trajectory are constructed, and the point cloud map is converted into a surface mesh model.

[0261] Based on the pose trajectory and the initial calibration parameters, the surface mesh model is rendered to the viewpoint of the camera to be calibrated, generating a rendered image and a rendered depth map corresponding to the camera to be calibrated;

[0262] Based on the real image data, the rendered image, and the rendered depth map, establish the correspondence between two-dimensional image features and three-dimensional spatial points in the real image data;

[0263] Based on the correspondence, a reprojection error objective function is constructed, and the extrinsic and intrinsic parameters of the camera to be calibrated are optimized by minimizing the objective function.

[0264] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0265] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the camera calibration method provided by the above methods, the method including: acquiring the vehicle's lidar point cloud data, inertial measurement unit data, reference camera image data and real image data of the camera to be calibrated, and acquiring the initial calibration parameters of the camera to be calibrated;

[0266] Based on the lidar point cloud data, the inertial measurement unit data, and the reference camera image data, a point cloud map containing color texture and the vehicle's pose trajectory are constructed, and the point cloud map is converted into a surface mesh model.

[0267] Based on the pose trajectory and the initial calibration parameters, the surface mesh model is rendered to the viewpoint of the camera to be calibrated, generating a rendered image and a rendered depth map corresponding to the camera to be calibrated;

[0268] Based on the real image data, the rendered image, and the rendered depth map, establish the correspondence between two-dimensional image features and three-dimensional spatial points in the real image data;

[0269] Based on the correspondence, a reprojection error objective function is constructed, and the extrinsic and intrinsic parameters of the camera to be calibrated are optimized by minimizing the objective function.

[0270] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the camera calibration method provided by the above methods, the method comprising: acquiring LiDAR point cloud data of a vehicle, inertial measurement unit data, reference camera image data and real image data of the camera to be calibrated, and acquiring initial calibration parameters of the camera to be calibrated;

[0271] Based on the lidar point cloud data, the inertial measurement unit data, and the reference camera image data, a point cloud map containing color texture and the vehicle's pose trajectory are constructed, and the point cloud map is converted into a surface mesh model.

[0272] Based on the pose trajectory and the initial calibration parameters, the surface mesh model is rendered to the viewpoint of the camera to be calibrated, generating a rendered image and a rendered depth map corresponding to the camera to be calibrated;

[0273] Based on the real image data, the rendered image, and the rendered depth map, establish the correspondence between two-dimensional image features and three-dimensional spatial points in the real image data;

[0274] Based on the correspondence, a reprojection error objective function is constructed, and the extrinsic and intrinsic parameters of the camera to be calibrated are optimized by minimizing the objective function.

[0275] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0276] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0277] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A camera calibration method, characterized by, include: Acquire the vehicle's lidar point cloud data, inertial measurement unit data, reference camera image data, and real image data of the camera to be calibrated, and obtain the initial calibration parameters of the camera to be calibrated; Based on the lidar point cloud data, the inertial measurement unit data, and the reference camera image data, a point cloud map containing color texture and the vehicle's pose trajectory are constructed, and the point cloud map is converted into a surface mesh model. Based on the pose trajectory and the initial calibration parameters, the surface mesh model is rendered to the viewpoint of the camera to be calibrated, generating a rendered image and a rendered depth map corresponding to the camera to be calibrated; Feature extraction and matching are performed on the real image data and the rendered image to obtain feature matching pairs, wherein the feature matching pairs contain mutually matching real image features and rendered image features; Based on the rendered depth map, the pose trajectory, and the initial calibration parameters, the rendered image features are back-projected into three-dimensional space to obtain the three-dimensional space point coordinates corresponding to the rendered image features; The two-dimensional pixel coordinates of the real image features are associated with the three-dimensional spatial point coordinates corresponding to the rendered image features, which is used as the correspondence. Based on the correspondence, a reprojection error objective function is constructed, and the extrinsic and intrinsic parameters of the camera to be calibrated are optimized by minimizing the objective function.

2. The camera calibration method according to claim 1, characterized in that, The feature matching pairs include: point feature matching pairs and line segment feature matching pairs; feature extraction and matching of the real image data and the rendered image include: Point feature extraction and line segment feature extraction are performed on the real image data to obtain a first point feature set and a first line segment feature set; Point feature extraction and line segment feature extraction are performed on the rendered image to obtain a second point feature set and a second line segment feature set; The first point feature set and the second point feature set are matched to obtain the point feature matching pair; The first line segment feature set and the second line segment feature set are matched to obtain the line segment feature matching pair.

3. The camera calibration method according to claim 2, characterized in that, Back-projecting the rendered image features into three-dimensional space includes: For the point feature matching pair, the depth value of the corresponding point feature in the second point feature set is obtained from the rendered depth map, and back projection is performed based on the depth value, the pose trajectory and the initial calibration parameters to obtain the three-dimensional point coordinates of the corresponding point feature; For the line segment feature matching pair, the depth values ​​of multiple sampling points on the corresponding line segment feature in the second line segment feature set are obtained from the rendered depth map. Based on the depth values, the pose trajectory and the initial calibration parameters, back projection is performed to obtain the coordinates of multiple three-dimensional sampling points of the corresponding line segment feature. The three-dimensional point coordinates and the coordinates of the plurality of three-dimensional sampling points together constitute the three-dimensional spatial point coordinates.

4. The camera calibration method according to claim 3, characterized in that, Based on the aforementioned correspondence, a reprojection error objective function is constructed, including: For the point feature matching pair, the three-dimensional point coordinates are projected onto the image plane according to the current extrinsic and intrinsic parameters to obtain the predicted point coordinates. The distance between the predicted point coordinates and the pixel coordinates of the corresponding point feature in the first point feature set is calculated as the point reprojection error. For the line segment feature matching pair, the coordinates of the multiple three-dimensional sampling points are projected onto the image plane according to the current extrinsic and intrinsic parameters to obtain multiple predicted projection points. The distance from each predicted projection point to the line containing the corresponding line segment feature in the first line segment feature set is calculated as the line segment reprojection error. The objective function for the reprojection error is constructed based on the weighted sum of the point reprojection error and the line segment reprojection error of the vehicle at multiple acquisition times.

5. The camera calibration method according to claim 1, characterized in that, The cameras to be calibrated are multiple; after optimizing the extrinsic and intrinsic parameters of each camera, the following are also included: Identify adjacent pairs of cameras to be calibrated where their fields of view overlap. Feature extraction and matching are performed on the real image data of the two cameras to be calibrated in the adjacent camera pair to obtain feature matching pairs between cameras; Based on the feature matching pairs between cameras, relative pose constraints between cameras are constructed, and joint optimization is performed to correct the extrinsic parameters of the adjacent pairs of cameras to be calibrated.

6. The camera calibration method according to claim 1, characterized in that, The construction of the point cloud map containing color texture and the vehicle's pose trajectory includes: The inertial measurement unit data, the lidar point cloud data, and the reference camera image data are fused using a tightly coupled method. Based on the fused data, synchronous positioning and mapping are performed to generate the point cloud map and the pose trajectory.

7. A camera calibration device, characterized in that, include: The acquisition module is used to acquire the vehicle's lidar point cloud data, inertial measurement unit data, reference camera image data, and real image data of the camera to be calibrated, and to acquire the initial calibration parameters of the camera to be calibrated. The construction module is used to construct a point cloud map containing color texture and the vehicle's pose trajectory based on the lidar point cloud data, the inertial measurement unit data and the reference camera image data, and to convert the point cloud map into a surface mesh model. The rendering module is used to render the surface mesh model to the viewpoint of the camera to be calibrated based on the pose trajectory and the initial calibration parameters, and generate a rendered image and a rendered depth map corresponding to the camera to be calibrated. A module is established to perform feature extraction and matching on the real image data and the rendered image to obtain feature matching pairs, wherein the feature matching pairs contain mutually matching real image features and rendered image features; Based on the rendered depth map, the pose trajectory, and the initial calibration parameters, the rendered image features are back-projected into three-dimensional space to obtain the three-dimensional space point coordinates corresponding to the rendered image features; The two-dimensional pixel coordinates of the real image features are associated with the three-dimensional spatial point coordinates corresponding to the rendered image features, which is used as the correspondence. The calibration module is used to construct a reprojection error objective function based on the correspondence, and to optimize the extrinsic and intrinsic parameters of the camera to be calibrated by minimizing the objective function.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the camera calibration method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the camera calibration method as described in any one of claims 1 to 6.

Images