Sensor automatic calibration method and device based on automatic driving, and storage medium

Abstract

This application discloses an automatic sensor calibration method, device, and storage medium based on autonomous driving, comprising: acquiring relative position calibration data of a surround-view calibration pattern using surround-view cameras; acquiring radar calibration data corresponding to at least three integrated calibration objects at each parking position; calculating the intrinsic parameters of the forward-looking camera, a first extrinsic parameter between the forward-looking camera and the millimeter-wave radar, a second extrinsic parameter between the forward-looking camera and the lidar, and the intrinsic parameters of each surround-view camera and a third extrinsic parameter between the surround-view cameras based on the relative position calibration data and the multiple radar calibration data; and fusing and optimizing the multiple sensor intrinsic and extrinsic parameters to generate an autonomous driving perception calibration parameter set for the vehicle to be calibrated. This application achieves efficient synchronous calibration of multiple sensors through integrated calibration objects and multi-position acquisition, and ensures global consistency and accuracy of parameters through data fusion optimization.

Description

Technical Field

[0001] This application relates to the field of autonomous driving development, and in particular to an automatic sensor calibration method, device and storage medium based on autonomous driving. Background Technology

[0002] In the development and mass production of autonomous driving systems, the high-precision collaborative operation of multiple types of sensors (such as forward-looking cameras, surround-view cameras, millimeter-wave radar, and lidar) is fundamental to the reliability of environmental perception. To achieve this collaboration, precise internal parameter calibration of each sensor is necessary, along with determining the spatial position and attitude transformation relationships between different sensors—that is, extrinsic parameter calibration. Typically, specific calibration materials and independent calibration procedures need to be designed for each type of sensor or each sensor pair.

[0003] Due to the significant differences in the working principles and data modes of various sensors, traditional calibration methods often require preparing different calibration fields for cameras, millimeter-wave radars, and lidars, and arranging multiple, sequential calibration operations. This discrete, serial calibration mode not only results in a lengthy and complex overall process but also introduces unavoidable cumulative errors during multiple vehicle movements and calibration scene switching. Furthermore, the lack of a unified spatial reference and geometric constraints between parameters calibrated in different batches and under different environments makes it difficult to guarantee the consistency and overall optimality of the entire sensor system's parameters within a unified coordinate system.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide an automatic sensor calibration method, device and storage medium based on autonomous driving, which aims to solve the technical problems of cumbersome and inefficient multi-sensor calibration process in existing autonomous driving systems, and the difficulty in ensuring the consistency between parameters.

[0006] To achieve the above objectives, this application proposes an automatic sensor calibration method based on autonomous driving, the method comprising: The vehicle to be calibrated is controlled to park at an initial position relative to at least three integrated calibration objects. The relative position calibration data of the integrated calibration objects are collected by a surround-view camera. A surround-view calibration pattern containing color difference feature points and generated by projection is set on the ground of the calibration space. The vehicle to be calibrated is controlled to stop sequentially at multiple calibration positions. At each stopping position, radar calibration data corresponding to the at least three integrated calibration objects are collected by a forward-looking camera, millimeter-wave radar, and lidar, respectively. At least three preset integrated calibration objects are arranged in the calibration space. The preset integrated calibration objects include a radar corner reflector and a checkerboard calibration plate fixedly connected to the same support structure, and the at least three preset integrated calibration objects are geometrically distributed in space. Based on the relative position calibration data and multiple radar calibration data, calculate the intrinsic parameters of the forward-looking camera, the first extrinsic parameter between the forward-looking camera and the millimeter-wave radar, the second extrinsic parameter between the forward-looking camera and the lidar, and the intrinsic parameters of each surround-view camera and the third extrinsic parameter between the surround-view cameras. Based on statistical fusion of multi-location data and cross-validation of geometric constraints among sensors, the intrinsic and extrinsic parameters of multiple sensors are fused and optimized to generate an autonomous driving perception calibration parameter set for the vehicle to be calibrated. The sensors include surround-view cameras, forward-view cameras, millimeter-wave radar, and lidar.

[0007] In one embodiment, the step of controlling the vehicle to be calibrated to park at an initial position relative to at least three integrated calibration objects, and acquiring relative position calibration data of the integrated calibration objects through a surround-view camera, includes: Control the surround-view camera to acquire an image containing the surround-view calibration pattern; Identify each color difference feature point in the image and obtain the coordinates of each feature point in the corresponding panoramic camera image coordinate system; The coordinates are used as the relative position calibration data.

[0008] In one embodiment, the step of controlling the vehicle to be calibrated to sequentially stop at multiple calibration positions, and at each stopping position, collecting radar calibration data corresponding to the at least three integrated calibration objects through a forward-looking camera, millimeter-wave radar, and lidar respectively, includes: The vehicle to be calibrated is controlled to sequentially stop at the initial position, the first forward position, and the second forward position relative to the three preset integrated calibration objects; At each stop, calibration data corresponding to the vehicle to be calibrated and the integrated calibration object are collected by the forward-looking camera, millimeter-wave radar and lidar respectively. The radar calibration data is generated based on the calibration data.

[0009] In one embodiment, the step of collecting calibration data corresponding to the vehicle to be calibrated and the integrated calibration object at each parking position using the forward-looking camera, millimeter-wave radar, and lidar respectively includes: The forward-looking camera acquires an image containing the integrated calibration object, and extracts the two-dimensional pixel coordinates of multiple corner points on the checkerboard calibration board and the two-dimensional pixel coordinates of the radar corner reflector from the image as the first calibration data. The point cloud data of the integrated calibration object is collected by the lidar, and the three-dimensional spatial coordinates corresponding to the multiple corner points of the chessboard calibration board are extracted from the point cloud data as the second calibration data. The millimeter-wave radar collects radar echo data of the integrated calibration object, and extracts two-dimensional spatial coordinates corresponding to the radar corner reflector from the radar echo data as third calibration data. The first calibration data, the second calibration data, and the third calibration data are associated and stored as radar calibration data at the current docking position.

[0010] In one embodiment, the step of calculating the intrinsic parameters of the forward-looking camera based on the relative position calibration data and multiple radar calibration data includes: Obtain the two-dimensional pixel coordinates of multiple corner points on the chessboard calibration board extracted from the relative position calibration data at multiple docking positions; Establish multiple sets of point correspondences based on the two-dimensional pixel coordinates and the known three-dimensional coordinates of the corner points in the world coordinate system; Based on the aforementioned multiple sets of point correspondences, the intrinsic parameter matrix and distortion coefficients of the forward-looking camera are solved using a nonlinear optimization algorithm to obtain the intrinsic parameters of the forward-looking camera.

[0011] In one embodiment, the step of calculating the first extrinsic parameter between the forward-looking camera and the millimeter-wave radar based on the relative position calibration data and multiple radar calibration data includes: Extract the two-dimensional pixel coordinates of each radar corner reflector from the image captured by the forward-looking camera; The two-dimensional spatial coordinates of each radar corner reflector detected by the millimeter-wave radar are obtained from the radar calibration data. Match the two-dimensional pixel coordinates of each radar corner reflector with its corresponding two-dimensional spatial coordinates to construct multiple two-dimensional point pairs; Based on multiple two-dimensional point pairs, a nonlinear optimization algorithm based on a perspective projection model and a robust RANSAC estimation strategy are used to solve for the rotation matrix and translation vector between the forward-looking camera and the millimeter-wave radar, thereby obtaining the first extrinsic parameters.

[0012] In one embodiment, the step of calculating a second extrinsic parameter between the forward-looking camera and the lidar based on the relative position calibration data and multiple radar calibration data includes: Extract the two-dimensional pixel coordinates of multiple corner points on the checkerboard calibration board from the image captured by the forward-looking camera; Extract the three-dimensional point cloud coordinates corresponding to the same multiple corner points in the image from the radar calibration data collected by the lidar; Match the two-dimensional pixel coordinates of each corner point with the corresponding three-dimensional point cloud coordinates to construct multiple sets of point correspondences; Based on the multiple sets of point correspondences, a point cloud and image registration algorithm is used to iteratively optimize and solve the rotation matrix and translation vector between the forward-looking camera and the lidar, thereby obtaining the second extrinsic parameter.

[0013] In one embodiment, the step of calculating the intrinsic parameters of each surround-view camera and the third extrinsic parameters between surround-view cameras based on the relative position calibration data and multiple radar calibration data includes: Based on the image coordinates of each color difference feature point in the relative position calibration data and their known coordinates in the world coordinate system, the intrinsic parameters of each surround-view camera are solved respectively. Establish multiple sets of image coordinate correspondences by using observation data of the same feature point within the overlapping field of view of adjacent panoramic cameras; Based on the correspondence of multiple sets of image coordinates and the intrinsic parameters of each surround view camera, the relative rotation and translation between adjacent surround view cameras are calculated using a multi-view geometric algorithm as initial extrinsic parameters; A beam adjustment algorithm is used to jointly optimize the intrinsic parameters of all surround-view cameras and the initial extrinsic parameters to obtain the optimized third extrinsic parameters and the updated surround-view camera intrinsic parameters.

[0014] In addition, to achieve the above objectives, this application also proposes an automatic sensor calibration device based on autonomous driving, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the automatic sensor calibration method based on autonomous driving as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the automatic sensor calibration method based on autonomous driving as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: The technical solution of this application involves controlling the vehicle to be calibrated to stop at an initial position relative to at least three integrated calibration objects, and collecting relative position calibration data of the integrated calibration objects through a surround-view camera. A surround-view calibration pattern containing color difference feature points and generated by projection is set on the ground of the calibration space. The vehicle to be calibrated is controlled to stop sequentially at multiple calibration positions. At each stopping position, radar calibration data corresponding to the at least three integrated calibration objects is collected by a forward-looking camera, millimeter-wave radar, and lidar, respectively. At least three preset integrated calibration objects are arranged in the calibration space, and each preset integrated calibration object includes a radar angle reflector fixedly connected to the same support structure. The system includes a calibration board with a checkerboard pattern, and at least three of the preset integrated calibration objects are geometrically distributed in space. Based on the relative position calibration data and multiple radar calibration data, the system calculates the intrinsic parameters of the forward-looking camera, the first extrinsic parameter between the forward-looking camera and the millimeter-wave radar, the second extrinsic parameter between the forward-looking camera and the lidar, and the intrinsic parameters and third extrinsic parameters between each surround-view camera. Based on the statistical fusion of multi-position data and the cross-validation of geometric constraints between sensors, the system fuses and optimizes the intrinsic and extrinsic parameters of multiple sensors to generate the autonomous driving perception calibration parameter set for the vehicle to be calibrated. The sensors include surround-view cameras, forward-looking cameras, millimeter-wave radar, and lidar.

[0017] This application achieves efficient synchronous calibration of multiple sensors through integrated calibration material and multi-location data acquisition, and ensures global consistency and accuracy of parameters through data fusion optimization. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the automatic sensor calibration method based on autonomous driving in this application; Figure 2 This is a detailed process diagram based on step S10 in the first embodiment; Figure 3 This is a detailed process diagram based on step S20 in the first embodiment; Figure 4 This is a detailed schematic diagram of step S30 in the first embodiment; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the automatic sensor calibration method based on autonomous driving in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] In related technologies, multi-sensor calibration for autonomous driving typically adopts a discrete, serial calibration method, which involves designing dedicated calibration objects and independent processes for different types of sensors. This results in a lengthy and complex calibration process with high repetitiveness. Furthermore, it is prone to introducing cumulative errors when moving the vehicle multiple times and switching calibration scenarios. There is also a lack of a unified spatial reference and joint geometric constraints between different sensor parameters, making it difficult to achieve consistency, overall optimality, and efficient calibration of the entire sensor system under a unified coordinate system.

[0024] Based on the aforementioned deficiencies in related technologies, this application proposes an automatic sensor calibration method for autonomous driving. In this method, at least three integrated calibration objects are arranged in the calibration space. Each calibration object includes a fixedly connected radar corner reflector and a checkerboard calibration board, and a surround-view calibration pattern with color-differential feature points is projected onto the ground. This allows for simultaneous calibration of LiDAR, millimeter-wave radar, forward-looking camera, and surround-view camera in the same scenario. By controlling the vehicle to sequentially stop at multiple calibration positions and systematically collecting calibration data from each sensor, combined with statistical fusion of multi-position data and cross-validation of geometric constraints between sensors, the intrinsic and extrinsic parameters of each sensor are jointly calculated and optimized. This achieves a high degree of integration, automation, and high precision in the multi-sensor calibration process, significantly improving calibration efficiency, parameter consistency, and overall system perception reliability.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] Based on this, embodiments of this application provide an automatic sensor calibration method for autonomous driving, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the automatic sensor calibration method based on autonomous driving according to this application. In this embodiment, the automatic sensor calibration method based on autonomous driving includes steps S10 to S40: Step S10: Control the vehicle to be calibrated to stop at an initial position relative to at least three integrated calibration objects, and collect the relative position calibration data of the integrated calibration objects through a surround-view camera. In this step, a surround-view calibration pattern containing color difference feature points and generated by projection is set on the ground of the calibration space. In this embodiment, a surround-view camera system is used to collect relative position calibration data of a specific projected pattern on the ground, providing basic data for subsequent calculations of the intrinsic and extrinsic parameters of the surround-view camera. The key to this process is to use a high-contrast, color-differentiated projected pattern as a calibration reference to achieve automated and high-precision extraction of the coordinates of the calibration point image.

[0027] First, a surround-view calibration pattern is set on the ground in the calibration space. This pattern is formed by projecting a pre-designed digital image onto the ground using a high-brightness digital projector. The projected pattern contains multiple feature points with distinct color differences, such as high-contrast color combinations of red, green, blue, and white, and these feature points are evenly distributed within the projection area according to a specific geometric pattern. The projection area needs to cover a sufficiently large area, typically no less than 8 meters by 6 meters, to ensure complete coverage of the combined field of view of the vehicle surround-view camera system. Before projection, the projector needs to be installed and tested, including adjusting the installation height to 3 to 5 meters, performing keystone correction to eliminate image distortion, and adjusting the focus and brightness to ensure that the projected pattern on the ground is clear, distortion-free, and has sufficient contrast.

[0028] Once the vehicle to be calibrated is parked in the pre-marked initial position within the calibration space as guided, the surround-view calibration data acquisition process can be initiated. The system first controls multiple surround-view cameras on the vehicle to simultaneously or synchronously acquire images containing ground projection patterns. The acquired raw images undergo a series of preprocessing steps, such as noise reduction and enhancement, to improve image quality.

[0029] Subsequently, the system performs automated calibration point identification and coordinate extraction. The processing logic typically includes converting the image from the RGB color space to the more suitable HSV or Lab color space for color segmentation, and then segmenting and extracting calibration points of different colors based on preset color thresholds. Morphological operations such as opening and closing operations are used to optimize the segmented binary image, eliminating noise and small regions. Next, connected component analysis is performed on the processed image to identify each independent calibration point region.

[0030] For each identified calibration point, its precise position in the image coordinate system is calculated. Typically, pixel coordinates are obtained by calculating the centroid of the connected component, and sub-pixel precision algorithms can be used to further improve the accuracy of coordinate positioning. The system records the image pixel coordinates of all identified calibration points and numbers or categorizes them according to preset identification rules.

[0031] Finally, the system encapsulates and stores the coordinate data of the calibration points acquired and processed by the front, rear, left, and right surround-view cameras in a unified format, forming a relative position calibration dataset for the initial location. Throughout the acquisition process, the software continuously evaluates the completeness of pattern recognition and the quality of coordinate extraction, such as checking whether the number of detected calibration points matches expectations and whether the coordinate distribution is reasonable, thereby ensuring the validity of the data and providing reliable input for subsequent calculations.

[0032] Step S20: Control the vehicle to be calibrated to park sequentially at multiple calibration positions. At each parking position, radar calibration data corresponding to the at least three integrated calibration objects are collected by a forward-looking camera, millimeter-wave radar, and lidar, respectively. At least three preset integrated calibration objects are arranged in the calibration space. The preset integrated calibration objects include a radar corner reflector and a checkerboard calibration plate fixedly connected to the same support structure, and the at least three preset integrated calibration objects are geometrically distributed in space. In this embodiment, by controlling the vehicle to stop at different spatial locations, observation data from a forward-looking camera, millimeter-wave radar, and lidar are collected from a fixed integrated calibration object. The key to this process is to increase the diversity of geometric constraints by utilizing multi-location observations and to rely on the integrated calibration object to achieve a natural correlation between multi-sensor data in physical space.

[0033] Before data acquisition begins, at least three pre-designed integrated calibration objects need to be placed in the calibration space. This calibration object is a composite structure that rigidly connects and fixes a corner reflector specifically for millimeter-wave radar and a checkerboard calibration plate specifically for vision sensors via a supporting structure such as a metal rod, ensuring that their relative spatial positions remain constant. These integrated calibration objects are geometrically distributed in space, for example, arranged at three points—left, center, and right—in front of the vehicle's expected parking area, forming a spatial geometry conducive to calibration solutions.

[0034] Starting from the initial position, the vehicle is controlled to move forward sequentially and stop at multiple, for example, three different calibration positions. The distance moved each time is pre-set, for example, 1 to 2 meters, and the vehicle is required to maintain its lateral position and heading angle as much as possible during movement to ensure that the calibration object remains within the effective field of view of each sensor. At each stopping position, after the vehicle has come to a stable stop, the forward-looking camera, millimeter-wave radar, and lidar are triggered synchronously or sequentially to collect data.

[0035] For LiDAR, the data processing logic is as follows: First, raw point clouds are acquired. Then, spatial filtering is used to retain point clouds in areas that may contain calibration objects, such as filtering based on distance and installation height. Next, the plane containing the checkerboard calibration board is identified in the filtered point cloud, and the corner points of the checkerboard are further detected on this plane. Using the reflection intensity information and geometric constraints of the point cloud, the system can extract the three-dimensional spatial coordinates of at least three checkerboard corner points.

[0036] For the forward-looking camera, the data processing logic is as follows: Target detection is performed on the acquired image to locate the checkerboard calibration area within the image. Then, corner detection algorithms, such as Harris corner detection combined with sub-pixel thinning technology, are applied within this area to accurately extract the pixel coordinates of the inner corners of the checkerboard. The system records the image corner coordinates corresponding to the physical corners extracted by the LiDAR.

[0037] For millimeter-wave radar, the data processing logic is as follows: analyze the radar echo signal, filter by radar cross-section value and range range, and identify strong reflective static targets from corner reflectors from numerous detection points. Separate the echoes from different corner reflectors through signal processing technology, and accurately measure the relative distance and azimuth information between the vehicle radar and each corner reflector.

[0038] At each docking location, the three sensors independently collect and extract their calibration data, packaging the data into a radar calibration data package for that location. Three such data packages are generated across the three location moves. These data packages together constitute a multi-location, multi-modal radar calibration dataset. An implicit spatial correspondence is established within the dataset through an integrated calibration object and a docking location index, providing rich observational data for subsequent joint calibration calculations.

[0039] Step S30: Based on the relative position calibration data and multiple radar calibration data, calculate the intrinsic parameters of the forward-looking camera, the first extrinsic parameters between the forward-looking camera and the millimeter-wave radar, the second extrinsic parameters between the forward-looking camera and the lidar, and the intrinsic parameters of each surround-view camera and the third extrinsic parameters between the surround-view cameras. This embodiment focuses on the core step of calibration calculation, calculating the intrinsic and extrinsic parameters for different sensor combinations based on two main types of data acquired previously. The core of its data processing logic is establishing a mathematical model between the observed data and the parameters to be determined, and then using an optimization algorithm to solve for the optimal parameters.

[0040] First, the intrinsic parameters of the forward-looking camera are calculated. This process utilizes the checkerboard corner data acquired from multiple locations in the above embodiment, relying solely on the images acquired by the forward-looking camera itself and not involving any radar measurement data. The system establishes a one-to-one correspondence between the two-dimensional coordinates of all checkerboard corner points extracted from different locations and calibration plates and their corresponding three-dimensional world coordinates of the checkerboard of known size. Based on the pinhole model and distortion model of camera imaging, such as the Brown model including radial and tangential distortion, a reprojection error function is constructed. Through a nonlinear optimization algorithm, such as the Levenberg-Marquardt algorithm, the reprojection error of all corner points is minimized, ultimately solving for the camera's intrinsic parameter matrix and distortion coefficient vector. During the solution process, the parameters are reasonably initialized and constrained to avoid getting trapped in local optima or obtaining physically unreasonable solutions.

[0041] Secondly, the first extrinsic parameter between the forward-looking camera and the millimeter-wave radar is calculated. The key here is establishing the correspondence between image pixels and radar 3D points. Using the data acquired in step S20, the system matches the centroid pixel coordinates of the corner reflectors detected in the camera image with the 2D spatial coordinates (e.g., distance and angle in the radar polar coordinate system, or 2D coordinates transformed to the radar Cartesian coordinate system) of the same corner reflector detected by the millimeter-wave radar, forming multiple 2D-to-2D point pairs. Since the millimeter-wave radar only provides 2D measurement information (lacking height information), while the camera provides 2D image coordinates, solving for the extrinsic parameter between them requires constructing an optimization problem based on perspective projection. Nonlinear optimization algorithms, such as the Levenberg-Marquardt algorithm, are typically used to solve for the rotation matrix and translation vector between the camera and the millimeter-wave radar by minimizing the reprojection error of the radar detection points on the image plane. To enhance robustness, the RANSAC algorithm is used to remove erroneous matching point pairs, followed by fine optimization based on the interior point set, and the calibration accuracy is evaluated by calculating the reprojection error of the radar points on the image.

[0042] Next, the second extrinsic parameter between the forward-looking camera and the LiDAR is calculated. The principle is similar to camera-radar calibration, but the data source is different. Here, the checkerboard corner point information extracted from the same calibration object is used. The system matches the pixel coordinates of the checkerboard corner points in the camera image with the 3D coordinates of the corresponding corner points extracted from the LiDAR point cloud. Since the 3D coordinates provided by the point cloud are highly accurate, the registration optimization from the point cloud to the image plane can be performed using an iterative nearest-point algorithm or its variants, or an optimization problem based on reprojection error can be directly constructed to solve the rigid body transformation relationship between the two, i.e., rotation and translation.

[0043] Finally, the intrinsic parameters of each surround-view camera and the third extrinsic parameter between the surround-view cameras are calculated. The calculation method for the intrinsic parameters of the surround-view cameras is the same as the calibration principle of the intrinsic parameters of the forward-looking cameras, but the ground projection calibration pattern data collected in step S10 is used. By using the coordinates of the known feature points of the projection pattern on the ground plane in the world coordinate system and the corresponding pixel coordinates extracted from the images of each surround-view camera, the intrinsic parameters and extrinsic parameters relative to the ground of each surround-view camera can be calibrated respectively. The extrinsic parameters between the surround-view cameras can be calculated through their common field-of-view overlap area. For example, by using calibration points that appear simultaneously in the images of two adjacent cameras, the relative pose between them can be calculated, and the intrinsic and extrinsic parameters of all surround-view cameras can be optimized through global binding adjustment to ensure the consistency of the parameters of the entire surround-view system.

[0044] Specifically, the calculation of the first, second, and third extrinsic parameters based on the collected relative position calibration parameters and radar calibration data includes the following steps: The step of calculating the first extrinsic parameter between the forward-looking camera and the millimeter-wave radar based on the relative position calibration data and multiple sets of radar calibration data includes: Extract the two-dimensional pixel coordinates of each radar corner reflector from the image captured by the forward-looking camera; The two-dimensional spatial coordinates of each radar corner reflector detected by the millimeter-wave radar are obtained from the radar calibration data. Match the two-dimensional pixel coordinates of each radar corner reflector with its corresponding two-dimensional spatial coordinates to construct multiple two-dimensional point pairs; Based on multiple two-dimensional point pairs, a nonlinear optimization algorithm based on a perspective projection model and a robust RANSAC estimation strategy are used to solve for the rotation matrix and translation vector between the forward-looking camera and the millimeter-wave radar, thereby obtaining the first extrinsic parameters.

[0045] This embodiment details the process of calculating the spatial transformation relationship between the forward-looking camera and the millimeter-wave radar, i.e., the first extrinsic parameter. Its core data processing logic involves using the radar corner reflector in the integrated calibration device as a common observation target, and correlating and matching the two-dimensional image coordinates acquired by the forward-looking camera with the two-dimensional spatial coordinates measured by the millimeter-wave radar. The rotation and translation relationship between the two sensor coordinate systems is determined by solving an optimization problem based on a perspective projection model. The key to this process is establishing accurate point-to-point correspondences and employing a robust estimation algorithm.

[0046] First, the system needs to extract observational data for extrinsic parameter calculations from data collected at multiple docking locations. On one hand, from the image data acquired and processed by the forward-looking camera in the previous steps, the precise two-dimensional pixel coordinates of each radar corner reflector in the corresponding image are extracted. These coordinates are obtained by previously identifying corner reflector features and calculating their centroid positions, and have typically undergone sub-pixel accuracy optimization and preliminary lens distortion correction. On the other hand, the two-dimensional spatial coordinates of each corner reflector detected by the millimeter-wave radar are obtained from its calibration data. Millimeter-wave radars typically output the target's range and azimuth in a polar coordinate system (some radars may output elevation angles, but in two-dimensional calibration scenarios, only range and azimuth are usually used to construct the two-dimensional coordinates). The system needs to convert this data into coordinates in a two-dimensional Cartesian coordinate system with the radar as the origin, or directly use polar coordinate parameterization during the optimization process. The radar's internal calibration parameters are applied during the conversion process to ensure coordinate accuracy.

[0047] Next, the system performs a crucial data association step, constructing a two-dimensional point pair for each identifiable radar corner reflector. Since each integrated calibration object is fixedly connected to a corner reflector and a checkerboard pattern, and the calibration objects are discretely arranged in space, the system can uniquely match a pixel of a corner reflector in the image with a two-dimensional spatial point detected by the radar from the same physical calibration object using the calibration object number and acquisition location information. The system iterates through all valid locations and all calibration objects, constructing multiple such point pairs. During the construction process, consistency checks are performed, such as verifying the geometric relationship of the same corner reflector under different vehicle position observations, to filter out obviously erroneous matches.

[0048] After obtaining a reliable set of 2D-to-2D point pairs, the system inputs them into the extrinsic parameter solving algorithm. A nonlinear optimization algorithm based on a perspective projection model is used here. The process involves transforming 2D spatial points (in the radar coordinate system) to the camera coordinate system through a rigid body transformation (rotation and translation), and then projecting them onto the 2D image plane using camera intrinsic parameters. Since millimeter-wave radar points only provide 2D information (lacking height dimension measurement), this optimization problem needs to incorporate appropriate geometric constraints, such as assuming corner reflectors in the calibration scene are on the same horizontal plane or using known calibration object height information as prior. The goal is to find the optimal rotation matrix and translation vector that minimizes the error between the projected image positions of all 3D points and their actual observed pixel coordinates.

[0049] To improve the robustness of the solution and prevent a few erroneous matching point pairs (exterior points) from interfering with the final result, the system integrates a RANSAC random sampling consensus strategy. This strategy randomly selects a subset of points (e.g., 3 pairs, the minimum number of points required for a 2D-to-2D point pair) from the point pair set, calculates an initial extrinsic parameter hypothesis model, and then uses this model to test all point pairs, counting the number of points (interior points) that meet the model's error threshold. This process is repeated multiple times, and finally, the model with the largest number of interior points is selected as the basis for the optimal solution. After RANSAC selects the interior point set, the system uses all interior points to refine the solution of the rotation matrix and translation vector through a nonlinear optimization algorithm, minimizing the reprojection error of all interior points. Finally, the rotation matrix and translation vector obtained after optimization convergence are the first extrinsic parameters between the forward-looking camera and the millimeter-wave radar, defining how the radar point cloud data is accurately transformed into the spatial relationship in the forward-looking camera image coordinate system.

[0050] The subsequent step of calculating the second extrinsic parameter between the forward-looking camera and the lidar based on the relative position calibration data and multiple radar calibration data includes: Extract the two-dimensional pixel coordinates of multiple corner points on the checkerboard calibration board from the image captured by the forward-looking camera; Extract the three-dimensional point cloud coordinates corresponding to the same multiple corner points in the image from the radar calibration data collected by the lidar; Match the two-dimensional pixel coordinates of each corner point with the corresponding three-dimensional point cloud coordinates to construct multiple sets of point correspondences; Based on the multiple sets of point correspondences, a point cloud and image registration algorithm is used to iteratively optimize and solve the rotation matrix and translation vector between the forward-looking camera and the lidar, thereby obtaining the second extrinsic parameter.

[0051] This embodiment details the implementation process for calculating the spatial transformation relationship between the forward-looking camera and the LiDAR, i.e., the second extrinsic parameter. Its core data processing logic is to establish a dense correspondence between the two-dimensional corner point pixel coordinates in the forward-looking camera image and the three-dimensional corner point spatial coordinates in the LiDAR point cloud, using the checkerboard corner points on the integrated calibration object as high-precision feature references. Then, through point cloud-to-image registration optimization techniques, the six-degree-of-freedom pose transformation parameters between the two sensors are solved. The advantage of this method lies in utilizing the fact that checkerboard corner points are easy to extract accurately from both modal data and are abundant in number.

[0052] The implementation process begins with the extraction and alignment of multi-source feature data. For each vehicle parking location, the system needs to acquire corresponding observation data from the forward-looking camera and LiDAR, targeting the same checkerboard calibration board. First, from the camera image, the system extracts the two-dimensional pixel coordinates of several predefined interior corner points on the checkerboard. These coordinates have already been obtained in previous processing using a sub-pixel-level corner detection algorithm and associated with their index numbers on the standard checkerboard template. Simultaneously, the system processes the LiDAR point cloud data acquired at the same location. By performing planar segmentation, clustering, and geometric shape analysis on the point cloud, the system can identify point cloud clusters belonging to the checkerboard plane and further utilize the reflection intensity and spatial distribution of the point cloud to extract the three-dimensional spatial coordinates corresponding to corner points with the same index numbers as those in the camera image. The LiDAR provides three-dimensional points with millimeter-level precision in its own coordinate system.

[0053] After establishing multiple correspondences between 2D and 3D corner point coordinates, the system needs to solve a rigid body transformation so that the 3D corner point set in the lidar coordinate system, after this transformation, best matches the observation in the forward-looking camera coordinate system. This is essentially a 3D-to-2D registration problem. A direct solution approach is to construct it as a nonlinear optimization problem, where the objective function is the sum of reprojection errors between the positions of the lidar's 3D corner points projected onto the image plane and the actual detected image corner point positions. The optimization variables are the rotation matrix and translation vector to be determined.

[0054] The system typically employs an iterative optimization approach to solve the problem. The algorithm requires an initial estimate of the extrinsic parameters, which can be provided by the coarse installation location information of the sensor or preliminarily estimated using other linear methods. In each iteration, the algorithm projects the 3D corner points of the LiDAR onto the image plane based on the current extrinsic parameter values, calculates the error between the projected points and the actual corner points, and then adjusts the extrinsic parameter estimates based on the error. Algorithms such as the Gauss-Newton method or Levenberg-Marquardt are typically used to guide the direction of parameter updates. Iteration continues until the sum of the reprojection errors converges to a minimum and no longer decreases significantly.

[0055] Ultimately, the rotation matrix and translation vector output by the optimization process become the high-precision second extrinsic parameter between the forward-looking camera and the LiDAR. To verify the calibration results, the system typically performs a verification step: transforming the entire LiDAR point cloud, not just the corner points, into the camera coordinate system using the just-solved extrinsic parameter, and projecting it onto the image. By observing whether the projected point cloud is precisely aligned with the object contours in the image, the calibration quality can be intuitively evaluated. This second extrinsic parameter enables pixel-level fusion of the rich semantic information from the forward-looking camera with the precise 3D geometric information from the LiDAR, and is a key parameter for the perception system to achieve 3D environment reconstruction.

[0056] Finally, the step of calculating the intrinsic parameters of each surround-view camera and the third extrinsic parameters between surround-view cameras based on the relative position calibration data and multiple radar calibration data includes: Based on the image coordinates of each color difference feature point in the relative position calibration data and their known coordinates in the world coordinate system, the intrinsic parameters of each surround-view camera are solved respectively. Establish multiple sets of image coordinate correspondences by using observation data of the same feature point within the overlapping field of view of adjacent panoramic cameras; Based on the correspondence of multiple sets of image coordinates and the intrinsic parameters of each surround view camera, the relative rotation and translation between adjacent surround view cameras are calculated using a multi-view geometric algorithm as initial extrinsic parameters; A beam adjustment algorithm is used to jointly optimize the intrinsic parameters of all surround-view cameras and the initial extrinsic parameters to obtain the optimized third extrinsic parameters and the updated surround-view camera intrinsic parameters.

[0057] This embodiment details the calibration process for a vehicle surround-view camera system. This process not only calculates the individual intrinsic parameters of each surround-view camera, but more importantly, determines their relative positions and attitude relationships—the third extrinsic parameter—to support seamless 360-degree surround-view image stitching. Its data processing logic is divided into three levels: first, the intrinsic parameters of each camera are independently calibrated using a known pattern from the ground projection; then, preliminary relative relationships are calculated using common feature points in the overlapping fields of view of adjacent cameras; finally, all parameters are uniformly optimized through global optimization to ensure overall consistency.

[0058] First, the intrinsic parameters of each surround-view camera are calibrated. This step relies on relative position calibration data acquired at the initial location. This data includes ground projection pattern images captured by each surround-view camera, and the image coordinates of a large number of color feature points automatically identified from these images. Simultaneously, since the projection pattern is pre-designed digitally and precisely projected onto a known level ground, the three-dimensional coordinates of each feature point on the ground plane in a unified world coordinate system are known. The system establishes a separate calibration task for each surround-view camera. A correspondence is established between the two-dimensional image coordinates of all feature points observed by that camera and their corresponding three-dimensional ground coordinates. Based on these corresponding point pairs, the same nonlinear optimization principle used for calibrating the intrinsic parameters of the forward-looking camera—namely, minimizing the reprojection error—is employed to solve for the intrinsic parameter matrix and distortion coefficients of each surround-view camera. This completes the independent calibration of the imaging model for each camera.

[0059] Secondly, the initial values ​​of the extrinsic parameters between the surround-view cameras, i.e., the initial values ​​of the third extrinsic parameter, are calculated. A characteristic of surround-view systems is that adjacent cameras, such as those looking forward and left, or left and back, have overlapping fields of view. In the acquired ground projection pattern images, feature points located within the overlapping areas are simultaneously captured by two adjacent cameras. The system uses a feature point matching algorithm to establish multiple sets of image coordinate correspondences within these overlapping areas, i.e., the coordinates of the same ground feature point in image A and image B. Given the intrinsic parameters of the two cameras and the ground coordinates of the feature points, this correspondence contains information about the relative pose of the two cameras. Using epipolar geometry or essential matrix solving algorithms in multi-view geometry, the system can calculate the initial rotation matrix and translation vector between adjacent cameras from multiple sets of such matching points. By traversing the four cameras (front, back, left, and right), the initial extrinsic parameters between all adjacent camera pairs can be calculated sequentially, and then linked to obtain the preliminary pose map of the entire surround-view camera network.

[0060] Finally, global joint optimization is performed to improve the overall optimality and consistency of the parameters. The intrinsic and initial extrinsic parameters calculated step-by-step above may have accumulated errors, leading to gaps or misalignments when stitching panoramic images. To address this issue, the system employs a bundle adjustment algorithm for global optimization. This algorithm treats all surround-view camera intrinsic parameters, all camera pose extrinsic parameters relative to the vehicle body, and the 3D coordinates of all observed ground feature points as optimizable variables, constructing a large nonlinear least-squares problem. The optimization objective is to minimize the sum of reprojection errors of all observations, i.e., the difference between the actual coordinates of each feature point in each observed camera image and the coordinates projected based on the current parameter estimates. Solved using an efficient optimization library, bundle adjustment can simultaneously correct the intrinsic and extrinsic parameters of all cameras, and even refine the 3D positions of feature points, thus obtaining a globally consistent optimal solution. The optimized output of updated intrinsic parameters for each camera and the precise relative pose relationships between cameras together constitute a high-quality third extrinsic parameter result, ensuring that the surround-view system can generate geometrically continuous and color-uniform bird's-eye view panoramic images.

[0061] Step S40: Based on the statistical fusion of multi-location data and the cross-validation of geometric constraints between sensors, the intrinsic and extrinsic parameters of multiple sensors are fused and optimized to generate the autonomous driving perception calibration parameter set of the vehicle to be calibrated. The sensors include surround-view cameras, forward-view cameras, millimeter-wave radar, and lidar.

[0062] This step involves globally optimizing and fusing the preliminary calibration parameters calculated in the preceding steps to generate a final, unified, and high-precision set of autonomous driving perception calibration parameters. Its core data processing logic involves constructing a global optimization model that includes all sensor parameters and all observation data. This model utilizes the statistical redundancy of multi-location data and the spatial geometric constraints between sensors for cross-validation and joint optimization to improve the overall consistency and accuracy of the calibration.

[0063] The preliminary calculations of intrinsic and extrinsic parameters are based on independent solutions derived from local data, which may contain accumulated errors or slight inconsistencies. For example, the extrinsic parameters of the forward-looking camera and the LiDAR, and the extrinsic parameters of the forward-looking camera and the millimeter-wave radar, should theoretically constrain the relative relationship between the LiDAR and the millimeter-wave radar indirectly through the mediation of the forward-looking camera. However, in independent calculations, this cross-sensor constraint was not utilized.

[0064] Therefore, this step first uses the initial intrinsic and extrinsic parameters of all sensors as initial values. Then, a unified cost function is constructed, which integrates all observation data from all locations and all sensors. This data includes: corner point observations of multiple checkerboard grids by forward-looking cameras at multiple locations; 3D point observations of the same set of checkerboard grid corner points by lidar at multiple locations; distance and angle observations of multiple corner reflectors by millimeter-wave radar at multiple locations; and observations of feature points of the ground projection pattern by each surround-view camera at the initial location.

[0065] The optimization model uses all sensor parameters to be optimized as variables, aiming to minimize the sum of reprojection errors or prediction errors of all the aforementioned observation data. For example, for a checkerboard corner point, its error can be calculated as: the difference between the 3D point transformed to the camera coordinate system using the extrinsic parameters of the LiDAR in the current optimization, then projected onto the image plane using the intrinsic parameters of the camera in the current optimization, and the pixel coordinates actually observed by the camera. Simultaneously, the observations from the surround-view camera are also included in the optimization, constraining its relationship with the vehicle body coordinate system.

[0066] Through this global optimization, the parameters of each sensor are constrained not only by its own directly observed data but also by the cross-constraints of indirect observation data from other sensors. Multi-location data provides multiple observations of the same geometric relationship, allowing the optimization process to reduce the impact of random noise through statistical fusion. Furthermore, the spatial relationship between the radar corner reflector and the checkerboard pattern fixed by the integrated calibration object provides a strong internal geometric constraint, further locking down the relative relationships between the parameters of different sensors.

[0067] Finally, by executing nonlinear optimization algorithms such as large-scale beam adjustment, the system can output a set of fused and optimized calibration parameters. This parameter set ensures that the internal parameters of all forward-looking cameras, surround-view cameras, millimeter-wave radar, and lidar on the vehicle, as well as their external transformation relationships, achieve global optimal consistency in a unified world coordinate system or vehicle coordinate system, thereby providing an accurate and reliable spatial reference for the autonomous driving perception system.

[0068] Furthermore, you can also view Figure 2 , Figure 2 This is a detailed process diagram based on step S10 in the first embodiment. Figure 2 The step of controlling the vehicle to be calibrated to park in an initial position relative to at least three integrated calibration objects, and collecting the relative position calibration data of the integrated calibration objects through a surround-view camera, includes S11~13: Step S11: Control the surround-view camera to acquire an image containing the surround-view calibration pattern; Step S12: Identify each color difference feature point in the image and obtain the coordinates of each feature point in the corresponding panoramic camera image coordinate system; Step S13: Use the coordinates as the relative position calibration data.

[0069] This embodiment details the specific implementation process for acquiring relative position data of a ground projection calibration pattern using a surround-view camera. Its core lies in the automated processing of the acquired images, accurately identifying and locating pre-designed color difference feature points within the pattern, and ultimately outputting standardized coordinate data. The entire process relies on image processing and computer vision technologies to achieve the conversion from raw images to usable calibration data.

[0070] First, the surround-view camera system acquires images containing the complete projected pattern. Once the vehicle is stably parked in its initial position, the calibration system sends a command to the onboard controller, simultaneously triggering the surround-view cameras installed in the front, rear, left, and right directions to capture images. To ensure the accuracy and consistency of subsequent processing, the system automatically adjusts the camera's shooting parameters before acquisition, such as setting appropriate exposure time and gain to avoid overexposure or underexposure, and ensures strict temporal synchronization of all cameras to capture the state of the calibration scene at the same moment. The acquired raw images are temporarily stored in high-resolution format, providing a clear input source for subsequent feature point recognition.

[0071] Next, in the crucial data processing stage, the goal is to automatically identify all color difference feature points from the acquired images and calculate their precise coordinates. The processing follows a systematic image analysis workflow. First, each raw image undergoes preprocessing, including noise reduction and contrast enhancement, to improve image quality and highlight the calibration pattern. Then, the image is converted from the common RGB color space to either HSV or Lab color space. This conversion is necessary because the HSV space separates color, saturation, and brightness, while the Lab space better aligns with human color perception; both can more effectively distinguish different feature points from the background and other points based on color differences.

[0072] Based on the transformed image, the system applies a preset color threshold to segment the feature point region for each specific color, generating a corresponding binary image. Then, morphological processing methods are used, such as opening operations to remove minor noise and closing operations to fill tiny holes, thereby optimizing the shape of the segmented feature point region. After obtaining a clear binary region, connected component analysis is performed, marking interconnected pixels as the same independent feature point target, and assigning a temporary number to each target.

[0073] For each identified independent feature point region, its coordinates in the corresponding panoramic camera image coordinate system are calculated. Typically, these coordinates are obtained by calculating the centroid of the connected region, in pixels. To achieve sub-pixel level positioning accuracy, the system further refines the initially calculated pixel-level coordinates using the grayscale centroid method or moment-based algorithms. Finally, the system records the precise image coordinates and color or number information for each successfully identified feature point.

[0074] Finally, the results obtained from the above processing are standardized and organized to form the final relative position calibration data. The system encapsulates and stores the coordinates of all feature points identified by each surround-view camera, as well as the correspondence between these points and feature points in the preset calibration pattern template, according to an agreed data structure. This dataset constitutes the basic observation data required for the calibration of the surround-view camera system. Its quality is automatically evaluated through indicators such as feature point detection rate and coordinate positioning consistency to ensure that it can be used for subsequent camera parameter calculations.

[0075] Furthermore, you can also view Figure 3 , Figure 3 This is a detailed process diagram based on step S20 in the first embodiment. Figure 3 The steps of controlling the vehicle to be calibrated to sequentially stop at multiple calibration positions, and at each stopping position, collecting radar calibration data corresponding to the at least three integrated calibration objects through a forward-looking camera, millimeter-wave radar, and lidar respectively, include S21~23: Step S21: Control the vehicle to be calibrated to stop sequentially at the initial position, the first forward position and the second forward position relative to the three preset integrated calibration objects; Step S22: At each parking position, the calibration data corresponding to the vehicle to be calibrated and the integrated calibration object are collected by the forward-looking camera, millimeter-wave radar and lidar respectively. Step S23: Generate the radar calibration data based on the calibration data.

[0076] This embodiment clarifies the specific execution scheme for multi-location data acquisition by the vehicle. Its core lies in controlling the vehicle to stop sequentially at three locations at different distances, collecting data from the same set of fixed calibration objects from multiple observation perspectives. This yields rich geometric constraint information, laying the data foundation for subsequent high-precision calibration calculations. This process ensures the systematic nature of the data acquisition and the rationality of its spatial distribution.

[0077] Three stopping positions were specifically defined. The vehicle first stops at the initial position, which is the first observation point relative to the three integrated calibration objects placed in front. After data acquisition at this position, the system controls the vehicle to slowly move forward a predetermined distance in a straight line, for example, 1.5 meters, to reach the first forward position and remain stable. Subsequently, the vehicle moves forward again by the same or a different predetermined distance to reach the second forward position. Throughout the movement, the vehicle is required to strictly maintain its lateral position and heading angle to ensure that at each new position, all integrated calibration objects remain completely within the effective detection field of view of the forward-looking camera, millimeter-wave radar, and lidar. These three positions constitute a longitudinally aligned observation baseline.

[0078] Furthermore, the system specifies the coordinated data acquisition from multiple sensors at each parking position. Whenever the vehicle comes to a complete stop at a designated location, the calibration system sends a synchronous acquisition command to the vehicle's sensor systems. The forward-facing camera is triggered to capture a high-resolution image, which should contain at least one or more integrated calibration objects; the millimeter-wave radar is activated to scan, receiving echo signals from the corner reflectors; and the lidar simultaneously performs one or more scans to acquire a dense point cloud containing the outline of the calibration objects. To ensure the temporal correlation between data from different sensors, the acquisition commands are sent synchronously, and all acquired data is marked with a unified timestamp or frame number. This step acquires the raw observation data from each sensor.

[0079] The raw observation data is initially processed and packaged to generate structured radar calibration data. For each parking location, the system integrates the raw data from the forward-looking camera, millimeter-wave radar, and lidar, along with the location's identification information, into a single data packet. Specifically, this data packet contains: the raw image acquired by the forward-looking camera or preliminarily extracted feature information; the raw point data output by the millimeter-wave radar or the preliminarily filtered signal; and the raw point cloud data provided by the lidar. These data collectively describe the observation results of each sensor on the fixed set of calibration objects ahead under the current vehicle pose. Three such data packets will be generated for the three parking locations, collectively forming a complete radar calibration dataset for subsequent calibration calculations. This dataset contains the multi-view geometric relationships resulting from changes in vehicle pose.

[0080] In addition, based on the above Figure 3 The content of step S22 shown, which is the step of collecting calibration data corresponding to the vehicle to be calibrated and the integrated calibration object at each parking position through the forward-looking camera, millimeter-wave radar and lidar respectively, includes S22-1 to S22-4: Step S22-1: The forward-looking camera acquires an image containing the integrated calibration object, and extracts the two-dimensional pixel coordinates of multiple corner points on the checkerboard calibration board and the two-dimensional pixel coordinates of the radar corner reflector from the image as the first calibration data; Step S22-2: Collect point cloud data of the integrated calibration object using the lidar, and extract the three-dimensional spatial coordinates corresponding to the multiple corner points on the chessboard calibration board from the point cloud data as the second calibration data; Step S22-3: Collect radar echo data of the integrated calibration object using the millimeter-wave radar, and extract the two-dimensional spatial coordinates corresponding to the radar corner reflector from the radar echo data as the third calibration data; Step S22-4: Associate and store the first calibration data, the second calibration data, and the third calibration data as radar calibration data at the current docking position.

[0081] The system accurately locates and positions the checkerboard calibration board within the entire image captured by the forward-looking camera. First, the image is preprocessed, including Gaussian filtering for noise reduction and histogram equalization to enhance contrast. Then, leveraging prior knowledge of the checkerboard pattern's regular alternation of black and white and strong corner features, an object detection algorithm is employed. This might use traditional computer vision methods, such as edge detection combined with Hough transform to find quadrilateral regions and verifying them using the periodicity of the black and white squares; or a deep learning-based object detection model to directly output the bounding box position of the checkerboard calibration board in the image. After successful localization, the system extracts the region of interest containing the checkerboard pattern for further fine-grained feature extraction.

[0082] Precise feature point coordinate extraction is performed within the aforementioned checkerboard area. This step is divided into two parallel or sequential subtasks. The first subtask is to extract the coordinates of the interior corner points of the checkerboard. The system applies corner detection algorithms, such as the Harris corner detector or the Shi-Tomasi corner detector, within the checkerboard area to initially locate candidate corner positions. Due to insufficient pixel-level coordinate accuracy, the system further employs sub-pixel-level corner refinement algorithms, such as iteratively solving using image grayscale gradients, ultimately obtaining high-precision two-dimensional pixel coordinates for at least three, typically all, interior corner points. The second subtask is to identify and extract the position coordinates of the radar corner reflector in the image. Based on the fixed relative position of the corner reflector within the integrated calibration object, the system estimates its approximate area of ​​appearance in the image. Then, combining the metallic reflective characteristics, specific colors, or triangular geometric features of the corner reflector, the system determines the two-dimensional pixel coordinates of its centroid in the image through feature matching or contour analysis. The extracted two-dimensional pixel coordinates of the checkerboard corner points and the corner reflector together constitute the first calibration data.

[0083] The system acquires 3D point cloud data of the integrated calibration object using LiDAR. Since LiDAR directly outputs 3D spatial coordinates, the system needs to identify and accurately locate 3D spatial points corresponding to multiple corner points on the checkerboard calibration board from the dense point cloud. First, the raw point cloud is preprocessed, including outlier filtering and distance- or intensity-based threshold segmentation, to initially separate point cloud clusters that may belong to the calibration object. Then, using prior geometric distribution of the calibration object in space (e.g., three integrated calibration objects in a specific geometric layout) and the geometric features of the checkerboard plane, a subset of the point cloud corresponding to each checkerboard calibration board is segmented from the point cloud. Based on this, a plane fitting algorithm (such as RANSAC) is used to fit the plane containing the checkerboard, and the 3D spatial coordinates of multiple interior corner points on the checkerboard are accurately located through point cloud edge extraction or intensity abrupt change analysis. These coordinates, referenced to the radar coordinate system, typically achieve centimeter-level or even millimeter-level accuracy, constituting the second calibration data.

[0084] The system acquires radar echo data from the integrated calibration object using millimeter-wave radar. The millimeter-wave radar outputs the target's range, azimuth, and elevation angles (or only range and azimuth) in polar coordinates, typically provided as point traces or a target list. The system needs to extract the two-dimensional spatial coordinates corresponding to the radar corner reflectors from the radar echo data. First, utilizing the high radar cross-section of the corner reflectors, constant false alarm rate (CFAR) detection or threshold segmentation is used to filter out radar target points that may belong to the corner reflectors from the raw echoes. Then, combining the known spatial location of the integrated calibration object and the vehicle's current parking position, multi-target association and tracking are performed to eliminate false targets introduced by environmental clutter. For valid targets confirmed to belong to the corner reflectors, the system records their range, azimuth, and elevation angles in polar coordinates and converts them into two-dimensional spatial coordinates in a Cartesian coordinate system with the millimeter-wave radar as the origin using a coordinate transformation formula. The conversion process requires the application of the radar's internal calibration parameters to ensure accuracy. These final two-dimensional spatial coordinates constitute the third calibration data.

[0085] The three types of calibration data extracted above are associated and stored according to the current parking position and calibration object number. The system establishes a data record for each parking position, which includes: the first calibration data extracted from the image at that position (checkerboard corner points and corner reflector pixel coordinates), the second calibration data extracted from the lidar point cloud (checkerboard corner point three-dimensional coordinates), and the third calibration data extracted from the millimeter-wave radar echo (corner reflector two-dimensional coordinates). To ensure data consistency, all coordinates must be associated with their respective integrated calibration object numbers (e.g., #1, #2, #3), and the identifier of the vehicle's current parking position (e.g., initial position, first forward position, second forward position) must be recorded. In this way, heterogeneous observation data from the same physical calibration object, at the same time, and from different sensors are integrated into a complete radar calibration data unit, which can be used for multi-sensor internal and external parameter joint calculation in subsequent step S30.

[0086] Furthermore, you can also view Figure 4 , Figure 4 This is a detailed process diagram based on step S30 in the first embodiment. Figure 4 The step of calculating the intrinsic parameters of the forward-looking camera based on the relative position calibration data and multiple radar calibration data includes S31~33: Step S31: Obtain the two-dimensional pixel coordinates of multiple corner points on the chessboard calibration board extracted from the relative position calibration data at multiple docking positions; Step S32: Establish multiple sets of point correspondences based on the two-dimensional pixel coordinates and the known three-dimensional coordinates of the corner points in the world coordinate system; Step S33: Based on the correspondence of the multiple sets of points, the intrinsic parameter matrix and distortion coefficient of the forward-looking camera are solved by a nonlinear optimization algorithm to obtain the intrinsic parameters of the forward-looking camera.

[0087] This embodiment details the specific implementation process for calculating the internal parameters of the forward-looking camera. Its core data processing logic involves using image observation data collected from multiple vehicle locations against a fixed checkerboard calibration board to construct a precise correspondence set between two-dimensional image points and three-dimensional spatial points. Then, through an optimization method that minimizes reprojection error, it solves for the internal parameter matrix and distortion coefficients that most accurately describe the camera's imaging geometry. This process relies solely on the forward-looking camera's own image data and does not involve any millimeter-wave radar or lidar measurement information, corresponding to the specific implementation of steps S31 to S33.

[0088] During the data preparation phase, visual feature information related to the forward-looking camera is extracted from the collected and stored first calibration data. The system iterates through each first calibration data record generated in the previous steps. This record corresponds to a specific vehicle parking position and contains only the image feature data of the forward-looking camera at that position, which has been pre-processed. Specifically, this step needs to extract the two-dimensional pixel coordinates of multiple interior corner points accurately detected in the corresponding image of the checkerboard calibration board on each integrated calibration object. These corner coordinates may have been obtained in previous processing using a sub-pixel precision algorithm and are saved along with the calibration object number and corner index. The task of this step is to gather the image coordinates of all valid corner points at all positions and on all calibration boards to form a large-scale set of two-dimensional observation points. These observation points constitute the sole input data for subsequent calibration solutions.

[0089] Establishing a connection between observed values ​​and the theoretical model involves constructing corresponding point pairs for calibration and solution. The system assigns a known three-dimensional world coordinate to each extracted two-dimensional pixel coordinate point. These three-dimensional coordinates originate from the precise physical dimensions of the checkerboard calibration board and a predefined world coordinate system. Typically, a corner point of a calibration board can be set as the origin of the world coordinate system, and the plane of the calibration board can be used as the XY plane. Based on the actual side length of each square in the checkerboard, the three-dimensional coordinates of each interior corner point in the local coordinate system of the calibration board can be calculated. Combined with the placement and orientation of the integrated calibration object in the global calibration space, these local coordinates can be transformed into a unified, fixed world coordinate system. Through the mapping between calibration object numbers and corner point indices, the system pairs the extracted two-dimensional pixel points with the calculated three-dimensional world coordinate points one-to-one, thereby establishing dozens or even hundreds of reliable two-dimensional-three-dimensional point correspondences. These correspondences constitute the observational basis of the calibration equations.

[0090] The system solves for the parameters based on the established mathematical model. Camera intrinsic parameter calibration is typically based on a pinhole camera model, taking into account lens distortion. The system substitutes all the above point correspondences into the camera imaging equation, which contains the intrinsic parameter matrix and distortion coefficient vector to be determined. The intrinsic parameter matrix mainly includes the focal length and principal point coordinates, while the distortion coefficients describe radial and tangential distortion. The solution process is a nonlinear optimization problem, the goal of which is to find a set of parameter values ​​that minimizes the sum of the positional errors between the calculated projected points and the actual observation points extracted in step S31 when the 3D world points are projected back onto the image plane through this set of parameters. The system uses a nonlinear least squares optimization algorithm to solve this problem, such as the Levenberg-Marquardt algorithm. The optimization process requires a reasonable set of initial parameter values, which can be roughly estimated using linear methods. In the iterative optimization, the algorithm continuously adjusts the values ​​of the intrinsic parameter matrix and distortion coefficients until the reprojection error converges to a minimum. Finally, the system outputs the optimized intrinsic parameter matrix and distortion coefficient vector, which together define the internal imaging parameters of the forward-looking camera, completing its intrinsic parameter calibration.

[0091] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the sensor automatic calibration method based on autonomous driving in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0092] This application provides an automatic sensor calibration device based on autonomous driving. The automatic sensor calibration device based on autonomous driving includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the automatic sensor calibration method based on autonomous driving in the above embodiment 1.

[0093] The following is for reference. Figure 5 This document illustrates a structural schematic diagram of an automatic sensor calibration device suitable for implementing embodiments of this application based on autonomous driving. The automatic sensor calibration device based on autonomous driving in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The illustrated sensor calibration device based on autonomous driving is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0094] like Figure 5As shown, the automatic sensor calibration device based on autonomous driving may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the automatic sensor calibration device based on autonomous driving. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the autonomous driving-based sensor calibration device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows an autonomous driving-based sensor calibration device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0095] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0096] The automatic sensor calibration device based on autonomous driving provided in this application, employing the automatic sensor calibration method based on autonomous driving described in the above embodiments, can solve the technical problems of cumbersome and inefficient multi-sensor calibration processes in existing autonomous driving systems, and the difficulty in ensuring consistency between parameters. Compared with the prior art, the beneficial effects of the automatic sensor calibration device based on autonomous driving provided in this application are the same as those of the automatic sensor calibration method based on autonomous driving provided in the above embodiments, and other technical features in this automatic sensor calibration device based on autonomous driving are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0097] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0098] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0099] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the automatic sensor calibration method based on autonomous driving in the above embodiments.

[0100] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0101] The aforementioned computer-readable storage medium may be included in an automatic sensor calibration device based on autonomous driving; or it may exist independently and not assembled into an automatic sensor calibration device based on autonomous driving.

[0102] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the autonomous driving-based sensor automatic calibration device, enable the autonomous driving-based sensor automatic calibration device to implement the technical content of the above-described embodiment of the autonomous driving-based sensor automatic calibration method.

[0103] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0106] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described automatic sensor calibration method based on autonomous driving. This solves the technical problems of cumbersome and inefficient multi-sensor calibration processes in existing autonomous driving systems, and the difficulty in ensuring consistency between parameters. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the automatic sensor calibration method based on autonomous driving provided in the above embodiments, and will not be repeated here.

Claims

1. A method for automatic sensor calibration based on autonomous driving, characterized in that, The automatic sensor calibration method based on autonomous driving includes the following steps: The vehicle to be calibrated is controlled to park at an initial position relative to at least three integrated calibration objects. The relative position calibration data of the integrated calibration objects are collected by a surround-view camera. A surround-view calibration pattern containing color difference feature points and generated by projection is set on the ground of the calibration space. The vehicle to be calibrated is controlled to stop sequentially at multiple calibration positions. At each stopping position, radar calibration data corresponding to the at least three integrated calibration objects are collected by a forward-looking camera, millimeter-wave radar, and lidar, respectively. At least three preset integrated calibration objects are arranged in the calibration space. The preset integrated calibration objects include a radar corner reflector and a checkerboard calibration plate fixedly connected to the same support structure, and the at least three preset integrated calibration objects are geometrically distributed in space. Based on the relative position calibration data and multiple radar calibration data, calculate the intrinsic parameters of the forward-looking camera, the first extrinsic parameter between the forward-looking camera and the millimeter-wave radar, the second extrinsic parameter between the forward-looking camera and the lidar, and the intrinsic parameters of each surround-view camera and the third extrinsic parameter between the surround-view cameras. Based on statistical fusion of multi-location data and cross-validation of geometric constraints among sensors, the intrinsic and extrinsic parameters of multiple sensors are fused and optimized to generate an autonomous driving perception calibration parameter set for the vehicle to be calibrated. The sensors include surround-view cameras, forward-view cameras, millimeter-wave radar, and lidar.

2. The automatic sensor calibration method based on autonomous driving as described in claim 1, characterized in that, The step of controlling the vehicle to be calibrated to park in an initial position relative to at least three integrated calibration objects, and collecting the relative position calibration data of the integrated calibration objects through a surround-view camera, includes: Control the surround-view camera to acquire an image containing the surround-view calibration pattern; Identify each color difference feature point in the image and obtain the coordinates of each feature point in the corresponding panoramic camera image coordinate system; The coordinates are used as the relative position calibration data.

3. The automatic sensor calibration method based on autonomous driving as described in claim 1, characterized in that, The steps of controlling the vehicle to be calibrated to sequentially stop at multiple calibration positions, and at each stopping position, collecting radar calibration data corresponding to the at least three integrated calibration objects through a forward-looking camera, millimeter-wave radar, and lidar respectively, include: The vehicle to be calibrated is controlled to sequentially stop at the initial position, the first forward position, and the second forward position relative to the three preset integrated calibration objects; At each stop, calibration data corresponding to the vehicle to be calibrated and the integrated calibration object are collected by the forward-looking camera, millimeter-wave radar and lidar respectively. The radar calibration data is generated based on the calibration data.

4. The automatic sensor calibration method based on autonomous driving as described in claim 3, characterized in that, The step of collecting calibration data corresponding to the vehicle to be calibrated and the integrated calibration object at each parking position using the forward-looking camera, millimeter-wave radar, and lidar respectively includes: The forward-looking camera acquires an image containing the integrated calibration object, and extracts the two-dimensional pixel coordinates of multiple corner points on the checkerboard calibration board and the two-dimensional pixel coordinates of the radar corner reflector from the image as the first calibration data. The point cloud data of the integrated calibration object is collected by the lidar, and the three-dimensional spatial coordinates corresponding to the multiple corner points of the chessboard calibration board are extracted from the point cloud data as the second calibration data. The millimeter-wave radar collects radar echo data of the integrated calibration object, and extracts two-dimensional spatial coordinates corresponding to the radar corner reflector from the radar echo data as third calibration data. The first calibration data, the second calibration data, and the third calibration data are associated and stored as radar calibration data at the current docking position.

5. The automatic sensor calibration method based on autonomous driving as described in claim 1, characterized in that, The step of calculating the intrinsic parameters of the forward-looking camera based on the relative position calibration data and multiple radar calibration data includes: Obtain the two-dimensional pixel coordinates of multiple corner points on the chessboard calibration board extracted from the relative position calibration data at multiple docking positions; Establish multiple sets of point correspondences based on the two-dimensional pixel coordinates and the known three-dimensional coordinates of the corner points in the world coordinate system; Based on the aforementioned multiple sets of point correspondences, the intrinsic parameter matrix and distortion coefficients of the forward-looking camera are solved using a nonlinear optimization algorithm to obtain the intrinsic parameters of the forward-looking camera.

6. The automatic sensor calibration method based on autonomous driving as described in claim 1, characterized in that, The step of calculating the first extrinsic parameter between the forward-looking camera and the millimeter-wave radar based on the relative position calibration data and multiple radar calibration data includes: Extract the two-dimensional pixel coordinates of each radar corner reflector from the image captured by the forward-looking camera; The two-dimensional spatial coordinates of each radar corner reflector detected by the millimeter-wave radar are obtained from the radar calibration data. Match the two-dimensional pixel coordinates of each radar corner reflector with its corresponding two-dimensional spatial coordinates to construct multiple two-dimensional point pairs; Based on multiple two-dimensional point pairs, a nonlinear optimization algorithm based on a perspective projection model and a robust RANSAC estimation strategy are used to solve for the rotation matrix and translation vector between the forward-looking camera and the millimeter-wave radar, thereby obtaining the first extrinsic parameters.

7. The automatic sensor calibration method based on autonomous driving as described in claim 1, characterized in that, The step of calculating the second extrinsic parameter between the forward-looking camera and the lidar based on the relative position calibration data and multiple radar calibration data includes: Extract the two-dimensional pixel coordinates of multiple corner points on the checkerboard calibration board from the image captured by the forward-looking camera; Extract the three-dimensional point cloud coordinates corresponding to the same multiple corner points in the image from the radar calibration data collected by the lidar; Match the two-dimensional pixel coordinates of each corner point with the corresponding three-dimensional point cloud coordinates to construct multiple sets of point correspondences; Based on the multiple sets of point correspondences, a point cloud and image registration algorithm is used to iteratively optimize and solve the rotation matrix and translation vector between the forward-looking camera and the lidar, thereby obtaining the second extrinsic parameter.

8. The automatic sensor calibration method based on autonomous driving as described in claim 1, characterized in that, The step of calculating the intrinsic parameters of each surround-view camera and the third extrinsic parameter between surround-view cameras based on the relative position calibration data and multiple radar calibration data includes: Based on the image coordinates of each color difference feature point in the relative position calibration data and their known coordinates in the world coordinate system, the intrinsic parameters of each surround-view camera are solved respectively. Establish multiple sets of image coordinate correspondences by using observation data of the same feature point within the overlapping field of view of adjacent panoramic cameras; Based on the correspondence of multiple sets of image coordinates and the intrinsic parameters of each surround view camera, the relative rotation and translation between adjacent surround view cameras are calculated using a multi-view geometric algorithm as initial extrinsic parameters; A beam adjustment algorithm is used to jointly optimize the intrinsic parameters of all surround-view cameras and the initial extrinsic parameters to obtain the optimized third extrinsic parameters and the updated surround-view camera intrinsic parameters.

9. An automatic sensor calibration device based on autonomous driving, characterized in that, The automatic sensor calibration device based on autonomous driving stores a computer program, which, when executed by a processor, implements the automatic sensor calibration method based on autonomous driving as described in any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the automatic sensor calibration method based on autonomous driving as described in any one of claims 1-8.

Images