A method, system, device and storage medium for calibrating vehicle-mounted multi-sensor systems
By arranging multiple target facades longitudinally on the vehicle and using a joint calibration method, the problem of insufficient longitudinal positioning accuracy of telephoto cameras is solved, and the accuracy and efficiency of multi-sensor calibration are improved, providing a high-precision vehicle-mounted multi-sensor calibration solution.
Patent Information
- Application Number
- CN202611414377.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-11
- Publication Date
- 2026-11-17
AI Technical Summary
In existing vehicle-mounted multi-sensor calibration fields, the longitudinal positioning accuracy of telephoto cameras is limited, making it difficult to meet the high-precision calibration requirements of advanced intelligent driving scenarios. Furthermore, the consistency and efficiency of multi-sensor calibration results need to be improved.
A method and system for calibrating vehicle-mounted multi-sensor sensors is designed. By arranging multiple target surfaces, including a vertical reference surface, in the longitudinal direction of the vehicle, and combining a base for surveying instruments and wheel positioning fixtures, the system enables joint calibration of cameras and lidar. The system improves calibration accuracy and efficiency by utilizing the longitudinally layered target layout and the common reference surface.
It improves the longitudinal positioning accuracy of telephoto cameras and the benchmark consistency of multi-sensor calibration, enhances the accuracy and operational efficiency of vehicle-mounted multi-sensor extrinsic parameter calibration, and solves the problems of insufficient longitudinal positioning accuracy and inconsistent calibration results in existing technologies.
Smart Images

Figure CN123118768A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multi-sensor calibration technology, and more specifically, to a vehicle-mounted multi-sensor calibration method, system, device, and storage medium. Background Technology
[0002] With the iterative upgrade of intelligent driving technology, multi-sensor fusion perception, including vehicle cameras and LiDAR, has become a core configuration of intelligent vehicle environmental perception systems. The extrinsic parameter calibration accuracy between sensors directly determines the accuracy and reliability of multi-sensor data fusion. As the core infrastructure for mass calibration of sensors in mass-produced vehicles, the target layout and benchmark setting scheme of the whole-vehicle calibration field are key factors affecting calibration accuracy and operational efficiency.
[0003] The commonly used vehicle-mounted multi-sensor calibration field usually sets up a vertical calibration wall at a fixed distance in front of the vehicle. The calibration wall has an array of coded targets arranged on its surface. During the calibration operation, the vehicle to be calibrated is parked directly in front of the calibration wall, and the coded target images on the wall are acquired by the vehicle-mounted camera. The camera's extrinsic parameters are then calculated.
[0004] This single-distance target arrangement scheme has no distribution span of targets in the longitudinal direction of the camera's optical axis. It cannot provide geometric constraints in the longitudinal depth direction for telephoto cameras with their optical axis arranged along the vehicle's longitudinal direction, resulting in limited longitudinal positioning accuracy of the telephoto cameras and making it difficult to meet the high-precision calibration requirements of advanced intelligent driving scenarios. Summary of the Invention
[0005] In view of this, the purpose of this application is to provide a method, system, device and storage medium for calibrating vehicle-mounted multi-sensor extrinsic parameters, which can improve the accuracy and efficiency of calibrating vehicle-mounted multi-sensor extrinsic parameters.
[0006] In a first aspect, embodiments of this application provide a method for calibrating vehicle-mounted multi-sensor systems, the method comprising: A calibration field is provided, which includes a vehicle parking area, at least three target facades arranged at intervals along the longitudinal direction of the vehicle, a near-field target group arranged in the near-field area, a wheel positioning fixture with marking points set on the ground of the vehicle parking area, and a surveying instrument mounting base set on the side of the vehicle parking area, wherein the target facade furthest away is the vertical reference facade. The vehicle to be calibrated is parked in the vehicle parking area, and each wheel is aligned with the wheel positioning fixture. A surveying instrument is set up on the surveying instrument mounting base to determine the coordinates of the coded targets on each of the target elevations, the coordinates of each target in the near-field target group, and the coordinates of each of the marker points. Acquire coded target images on each of the target elevations captured by the camera on the vehicle; Acquire point cloud data, including at least the vertical reference elevation, collected by the lidar on the vehicle; External parameter calibration of the telephoto camera on the vehicle was performed using multiple target facades at different longitudinal distances. Furthermore, the vertical reference elevation is used simultaneously as both a camera reference and a lidar reference to perform joint calibration of the camera and lidar on the vehicle.
[0007] Optionally, the step of using the vertical reference elevation as both a camera reference and a lidar reference to jointly calibrate the camera and lidar on the vehicle includes: Extract the image information of the coded target on the vertical reference surface to obtain the target feature points in the camera coordinate system; Extract the point cloud data of the lidar on the vertical reference surface, perform plane fitting on the point cloud data, and obtain the plane equation in the radar coordinate system; Using the global coordinates of each target point on the vertical reference surface measured by the surveying instrument as a common reference, the target feature points in the camera coordinate system and the plane equations in the radar coordinate system are unified to the global coordinate system to construct a joint optimization objective function. The joint optimization objective function is iteratively optimized, and the extrinsic parameters of the camera and the lidar are solved simultaneously.
[0008] Optionally, the external parameter calibration of the telephoto camera on the vehicle using multiple target facades at different longitudinal distances includes: Extract the pixel coordinates of coded targets located on different target facades in the images captured by the camera; Based on the three-dimensional coordinates of the centers of each coded target in the global coordinate system determined by the surveying instrument, spatial point sets at different depth levels are established. The extrinsic parameters of the telephoto camera are calculated by using the longitudinal coordinate difference of each target along the optical axis of the camera as a geometric constraint.
[0009] Optionally, the distance from the vertical reference facade to the vehicle parking area is greater than the distance from the other target facades to the vehicle parking area, and there is no obstruction between the vertical reference facade and the other target facades; In the point cloud of the lidar, the farthest vertical plane is identified as the vertical reference elevation, to distinguish it from the other target elevations nearby.
[0010] Optionally, the near-field target group includes ground targets laid on the ground; The normal direction of the plane where the ground target is located is perpendicular to the normal direction of the target's elevation. The method further includes: Acquire images of the ground target captured by the fisheye camera on the vehicle; The near-field extrinsic parameters of the filler fisheye camera are calibrated using the geometric relationship between the ground target and the perpendicularity of the normal direction.
[0011] Optionally, the near-field target group also includes a combined target for downward-looking radar, which is formed by a ground panel and a vertical panel fixedly connected at a defined angle; The method further includes: Acquire point cloud data of the ground panel and the vertical panel collected by the downward-facing blind spot fill lidar on the vehicle; The near-field extrinsic parameters of the downward-looking lidar are calibrated using the included angle constraint of the two sides of the combined target of the downward-looking radar.
[0012] Optionally, the wheel positioning fixture includes ground markers; The ground markers are permanently fixed to the ground; The marker point is located on the ground marker; The height deviation between the marker point and the ground shall not exceed 5 mm.
[0013] Optionally, the encoding target is an AprilTag encoding target; The coded targets on each target facade are arranged in consecutive numbered intervals, with one numbered interval corresponding to one target facade; The corresponding target facade is distinguished by identifying the number range of the coded target.
[0014] Optionally, for multi-axle vehicles with two or more wheels on the same side, the ground markings of each wheel on the same side form a group, and the ground markings in the group are arranged along the longitudinal direction of the vehicle. The ground markers on the left and right sides are arranged in pairs, and the two ground markers in the same pair are aligned in the longitudinal direction of the vehicle. The wheel reference position on that side is determined by the combined mapping coordinates of multiple ground markers on the same side, and the longitudinal axis of the vehicle is established.
[0015] Secondly, embodiments of this application provide an in-vehicle multi-sensor calibration system, including: Vehicle parking area; A multi-layered target facade group consists of at least three target facades arranged sequentially at intervals along the longitudinal direction of the vehicle. Each target facade is perpendicular to the longitudinal direction of the vehicle, and the distance from each target facade to the vehicle parking area increases sequentially. Among them, the target facade furthest away is the vertical reference facade, which is a single, flat, and continuous vertical plane, and each target facade is fixed with an array of coded targets. Near-field target groups are arranged on the ground and / or at low positions on both sides of the vehicle parking area; The near-field target group includes ground targets and / or a combined target for downward-looking radar consisting of a ground panel and a vertical panel fixedly connected at a defined angle. A wheel positioning fixture is installed on the ground in the vehicle parking area, including ground markers located at the ground contact positions of each wheel and permanently fixed to the ground, as well as marking points on each ground marker for aiming at by surveying instruments. The height deviation between the marker and the ground shall not exceed 5 mm; The surveying instrument mounting base is located on the side of the vehicle parking area. The position of the surveying instrument mounting base allows the surveying instrument to have a clear view of the entire target facade, the near-field target group, and the marking points of the wheel positioning fixture in one setup. And a controller, the controller being configured to perform the steps of the vehicle-mounted multi-sensor calibration method described in any of the alternative embodiments of the first aspect above.
[0016] The technical solution provided in this application includes, but is not limited to, the following beneficial effects: A calibration field is provided, comprising a vehicle parking area, at least three target facades arranged at intervals along the longitudinal direction of the vehicle, a near-field target group arranged in the near-field area, wheel alignment fixtures with markings on the ground of the vehicle parking area, and a surveying instrument mounting base located to the side of the vehicle parking area. The target facade furthest away is the vertical reference facade. This step, by constructing an integrated calibration field environment including multiple target layers, near-field targets, positioning fixtures, and a surveying base, provides a complete spatial reference carrier for vehicle-mounted multi-sensor calibration, covering the sensor observation range of the mid-far field and near field, and providing a standardized field foundation for subsequent calibration procedures.
[0017] The vehicle to be calibrated is parked in the vehicle parking area, with each wheel engaging with the wheel positioning fixture. This step, through the alignment and docking of the wheels with the positioning fixture, ensures that the vehicle's parking position is consistent in each calibration operation, making the vehicle body reference reproducible. It avoids interference from random differences in the vehicle's parking position on the calibration results, thus guaranteeing the consistency of the calibration results.
[0018] A surveying instrument is mounted on the surveying instrument mounting base to determine the coordinates of the coded targets on each target elevation, the coordinates of each target in the near-field target group, and the coordinates of each marker point. This step, relying on a fixed surveying instrument mounting base, allows the surveying instrument to complete the coordinate acquisition of all targets and marker points in a single setup, eliminating the need for multiple relocations and avoiding transmission errors caused by multi-coordinate system conversions. Furthermore, all reference coordinates are unified under the same coordinate system, ensuring the accuracy and consistency of the reference data.
[0019] The process involves acquiring coded target images from the camera on the vehicle, focusing on each of the target elevations. This step, by acquiring coded target images from elevations at different longitudinal distances, yields target image feature data at multiple depth levels. This provides multiple sets of corresponding matching points for camera extrinsic parameter calculation, supporting the solution of camera parameters under multiple constraints.
[0020] The process involves acquiring point cloud data, including at least the vertical reference elevation, collected by the LiDAR on the vehicle. This step, by acquiring the point cloud data of the vertical reference elevation, allows for the extraction of planar geometric features in the LiDAR coordinate system, providing a stable reference for the calibration of the LiDAR's extrinsic parameters. The flatness and continuity of the vertical reference elevation also ensure the accuracy of the point cloud planar fitting.
[0021] The extrinsic parameters of the telephoto camera on the vehicle are calibrated using multiple target facades at different longitudinal distances. This step relies on the depth geometric baseline formed by multiple target facades distributed along the longitudinal direction of the vehicle to provide calibration constraints for the optical axis of the telephoto camera, which can effectively improve the positioning and calibration accuracy of the telephoto camera in the longitudinal direction and improve the problem of insufficient longitudinal calibration accuracy of the telephoto camera under single-distance targets.
[0022] Furthermore, the vertical reference plane is used simultaneously as both the camera reference and the lidar reference to perform joint calibration of the camera and lidar on the vehicle. This step uses the same vertical reference plane as the common calibration reference for both the camera and lidar, eliminating the need for additional reference transfer structures and avoiding additional errors introduced during the reference transfer process. This effectively improves the consistency of the extrinsic parameter calibration results for the camera and lidar, ensuring the accuracy of multi-sensor data fusion.
[0023] In summary, this vehicle-mounted multi-sensor calibration method improves the longitudinal calibration accuracy of telephoto cameras and the consistency of multi-sensor calibration benchmarks simultaneously through a longitudinally layered target layout, a reuse design of common reference elevations, and a single-setup surveying mode. Overall, it enhances the accuracy and operational efficiency of vehicle-mounted multi-sensor extrinsic parameter calibration.
[0024] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1The flowchart of an on-board multi-sensor calibration method provided in Embodiment 1 of this application is shown; Figure 2 This shows a top view of the overall layout of a vehicle-mounted multi-sensor calibration field provided in Embodiment 1 of this application; Figure 3 A side view schematic diagram of a calibration field provided in Embodiment 1 of this application is shown; Figure 4 This illustration shows a schematic diagram of a wheel positioning fixture structure provided in Embodiment 1 of this application; Figure 5 This is a top view schematic diagram showing the coverage relationship between a target array and the field of view of an adjacent camera according to Embodiment 1 of this application; Figure 6 This paper shows a comparison diagram of the longitudinal layered arrangement and effect of a target facade provided in Embodiment 1 of this application; Figure 7 A flowchart of a joint calibration method provided in Embodiment 1 of this application is shown; Figure 8 This illustration shows a front view of a farthest reference elevation provided in Embodiment 1 of this application; Figure 9 A flowchart of an external parameter calibration method provided in Embodiment 1 of this application is shown; Figure 10 This shows a near-field real-world image of a calibration field provided in Embodiment 1 of this application; Figure 11 A flowchart of a near-field extrinsic parameter calibration method provided in Embodiment 1 of this application is shown; Figure 12 This paper shows a three-dimensional structural schematic diagram of a combined target for downward-looking radar provided in Embodiment 1 of this application; Figure 13 A flowchart of the second near-field extrinsic parameter calibration method provided in Embodiment 1 of this application is shown; Figure 14 This figure shows a partial cross-sectional schematic diagram of a wheel positioning fixture provided in Embodiment 1 of this application.
[0027] Attached label: 1-Vehicle parking area; 2-Near target elevation; 3-Middle target elevation; 4-Vertical reference elevation; 5-Ground near-field target; 6-Wheel positioning fixture; 7-Combined target for downward-looking radar; 8-Surveying instrument mounting base; 9-Ground marker; 10-Identification point for surveying instrument aiming; 11-Ground panel; 12-Vertical panel; 13-Definition angle between two panels; 61-Wheel positioning marker; 62-Multiple wheel marker group on the same side. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0029] Example 1 To facilitate understanding of this application, the following is combined with... Figure 1 The flowchart of the vehicle-mounted multi-sensor calibration method provided in Embodiment 1 of this application illustrates Embodiment 1 in detail.
[0030] See Figure 1 As shown, Figure 1 A flowchart of a vehicle-mounted multi-sensor calibration method according to Embodiment 1 of this application is shown, wherein the method includes steps S101 to S107: Specifically, this method is applied to the external parameter calibration scenario of a commercial vehicle equipped with 10 cameras and 6 LiDARs. After the vehicle enters the dedicated calibration field, the mapping instrument sets up the three-dimensional coordinates of each target and wheel reference point in the calibration field at once. Each sensor completes the external parameter calibration and delivers the calibration results accordingly. The structure of the calibration field directly determines the geometric conditions that can be achieved during calibration. The seven steps of this method sequentially cover site provision, vehicle parking, ground truth measurement, image acquisition, point cloud acquisition, individual calibration of the telephoto camera, and joint calibration of the camera and LiDAR, forming a complete closed-loop process from site preparation to multi-sensor joint calibration.
[0031] S101: Provide a calibration field, the calibration field including a vehicle parking area, at least three target facades arranged sequentially at intervals along the longitudinal direction of the vehicle, a near-field target group arranged in the near-field area, a wheel positioning fixture with marking points set on the ground of the vehicle parking area, and a surveying instrument mounting base set on the side of the vehicle parking area, wherein the target facade furthest away is a vertical reference facade.
[0032] Specifically, this calibration field is a dedicated calibration site for the external parameters of a commercial vehicle that is compatible with 10 cameras and 6 LiDARs. The vehicle parking area is located in the middle of the calibration field and is for vehicles to be calibrated to park. Its size must meet the space requirements for the commercial vehicle to drive in and park. The ground must be flat. A total station is used to complete the height measurement. The ground height error is controlled within a few millimeters before it is put into use.
[0033] See Figure 2 As shown, Figure 2 This diagram illustrates the overall top-down layout of a vehicle-mounted multi-sensor calibration field provided in Embodiment 1 of this application, fully showcasing the planar layout and relative positions of the various functional areas and facilities within the field. The central area of the calibration field is the vehicle parking area 1, within which the parking positions of the vehicles to be calibrated are indicated. Wheel positioning fixtures 6 are installed at the ground contact points of each vehicle wheel to determine and reproduce the planar positions of the wheel ground contact reference points. Three target facades are arranged sequentially from near to far along the longitudinal front of the vehicle: a near target facade 2, a middle target facade 3, and a vertical reference facade 4 (the farthest target facade). The distances from each target facade to the parking area increase sequentially; in this embodiment, these are 18.7 / 27.8 / 42.5m. Corresponding distance scales are provided below the diagram, labeled 0, 18.7m, 27.8m, and 42.5m respectively. This longitudinally layered layout provides a geometric baseline in the depth direction for the telephoto camera whose optical axis runs longitudinally along the vehicle. Ground-based near-field targets 5 are laid in front of the vehicle parking area, covering the vicinity of the vehicles to meet the observation requirements of blind spot sensors. A combined target 7 for downward-looking radar is arranged nearby on the side of the parking area, providing a calibration reference for the downward-looking blind spot lidar. A surveying instrument mounting base 8 is located on the side of the vehicle parking area, enabling the simultaneous installation of all targets and wheel markers within a single line of sight. This ensures that the true values of the targets and the wheel reference points are obtained in the same measurement and coordinate system, avoiding additional errors introduced by coordinate system transfer.
[0034] Specifically, at least three target facades are sequentially spaced along the longitudinal front of the vehicle, each perpendicular to the vehicle's longitudinal direction, with the distance to the parking area increasing sequentially, forming a longitudinally layered structure. In this embodiment, the positions of each target facade in the vehicle coordinate system are approximately 18.7m, 27.8m, and 42.5m, respectively. The actual target distance used by each camera can range from 12.1m to 36.2m, with a depth span of approximately 24m. This longitudinally layered structure allows the targets to have an expansion along the camera's optical axis, forming a longitudinal geometric baseline. For a telephoto camera whose optical axis coincides with the vehicle's longitudinal direction, this baseline is the geometric condition upon which its positioning in that direction depends.
[0035] Specifically, the near-field target group covers the ground and low-lying spaces near the vehicle, matching the observation range of blind-spot sensors. This includes ground targets laid on the ground and combined targets for downward-looking radar, consisting of a ground panel and vertical panels. The observation areas of the wide-field-of-view blind-spot camera and the downward-looking blind-spot lidar are concentrated on the ground and low-lying spaces near the vehicle, while the targets in the existing calibration field are all placed on distant vertical surfaces, leaving this area uncovered. The near-field target group is designed to fill this gap.
[0036] Specifically, the wheel alignment fixture is used to fix and reproduce the planar position of the wheel contact reference point. It includes ground markers located at each wheel contact point and markings on them for the surveying instrument to aim at. Markers on multiple wheels on the same side form a set to accommodate dual-rear-axle commercial vehicles. The vehicle coordinate system needs to be established based on the wheel contact point. The wheel alignment fixture eliminates the need to relocate the wheel contact point during each calibration, ensuring good consistency and solving the problem of the lack of wheel contact position positioning structures in existing calibration fields.
[0037] Specifically, the surveying instrument's mounting base is located to the side of the parking area. This position allows the instrument to have a clear view of all target elevations, near-field targets, and the wheel positioning fixture's markers in a single setup. "Single setup" means the instrument base is fixed in place, and all targets are observed by rotating the instrument lens 360°; it does not mean all targets are statically positioned in the same field of view without the lens rotating. The placement of targets has already considered the issue of vehicle body obstruction after parking; targets are not placed in areas where the vehicle body would obstruct the view, thus avoiding observational obstruction. This positioning condition ensures that the target true values and wheel reference points are obtained in the same measurement, within the same coordinate system, without introducing additional coordinate system transmission errors between them.
[0038] Specifically, the vertical reference elevation is the farthest elevation in the multi-layered target elevation group. It is a single, flat, and continuous vertical plane with a definite width in the horizontal direction. In this embodiment, it is 7.995m. This value is a measured value obtained from the actual physical wall dimensions of the existing calibration room and is not a rigid limitation of the patent. This elevation can serve as a shared reference for both the camera and the lidar. The target is attached to this elevation, and the target coordinates implicitly contain the spatial orientation of the elevation. Thus, the same elevation is simultaneously a pattern reference usable by the camera and a planar reference usable by the lidar, and both share the same set of mapping true values.
[0039] See Figure 3 As shown, Figure 3This diagram illustrates a side view of a calibration field provided in Embodiment 1 of this application. The diagram fully presents the vertical layout and spatial relationship of the vehicle, multi-layered target facades, near-field targets, and combined targets. The vehicle is on the left, with a downward-looking radar mounted at its front. The line of sight from the downward-looking lidar at the front of the vehicle to the reference facade is unobstructed, allowing for complete acquisition of point cloud data from the furthest reference facade, number 4. Target facades numbered 2, 3, and 4 are arranged sequentially from near to far along the longitudinal front of the vehicle. The distances from each facade to the vehicle's parking position are 18.7m, 27.8m, and 42.5m, respectively. The height of each facade and the vertical distribution of the targets ensure that sufficient targets are present within the fields of view of the 7-channel pinhole cameras and 3-channel filler fisheye cameras; near-field targets on the ground cover the adjacent area. The ground-based near-field target 5 covers the area near the vehicle, filling the near-field observation gap that cannot be covered by the distant vertical targets; the combined target 7 for the downward-looking radar arranged near the side of the vehicle is within the field of view of the downward-looking blind spot filler lidar, which can provide a stable structural reference for the calibration of the near-field lidar.
[0040] S102: Park the vehicle to be calibrated in the vehicle parking area, so that each wheel is engaged with the wheel positioning fixture.
[0041] Specifically, after the vehicle is driven into the calibration area, the ground contact position of each wheel must be aligned with the ground markers on the wheel positioning fixture to ensure that the reference position of the wheels is consistent for each calibration, providing a unified reference origin for the vehicle coordinate system. When parking the vehicle, it is necessary to ensure that each wheel is accurately positioned above the corresponding marker to avoid deviations in the vehicle coordinate system reference due to parking position errors.
[0042] Specifically, for the structure of dual rear axle commercial vehicles with dual rear wheels on the same side, the two rear wheels on the same side need to be mapped to two markers in the same group on the same side. The markers in the group are arranged along the longitudinal direction of the vehicle to jointly determine the reference position of the wheel on that side. The markers on the left and right sides are arranged in pairs, and the two markers in the same pair are aligned in the longitudinal direction of the vehicle to ensure that the left and right wheel positions are symmetrical after the vehicle is parked, which is convenient for establishing an accurate longitudinal axis of the vehicle body.
[0043] See Figure 4 As shown, Figure 4This diagram illustrates a wheel positioning fixture structure according to Embodiment 1 of this application, divided into a top view and a partial cross-sectional view. This diagram shows a wheel positioning fixture structure adapted to dual-rear-axle vehicles, divided into a top view layout and a partial cross-sectional view of the marking components. The upper part is a top view (dual-rear-axle vehicle) of the wheel positioning fixture, presenting the overall layout of the marking components within the vehicle parking area. 61 represents wheel positioning marking components, distributed corresponding to the ground contact positions of each wheel; two rear wheel marking components on the same side together form a multi-wheel marking point group 62 on the same side, arranged longitudinally along the vehicle, adapting to the multi-axle wheel structure of dual-rear-axle commercial vehicles; each marking component has a center point for aiming at by surveying instruments. The overall layout follows the principle of left-right pairing and longitudinal alignment; the two marking components in the same pair are aligned longitudinally along the vehicle, ensuring symmetrical left-right positions after the vehicle is parked. The lower part is a partial cross-sectional view of the marking components, showing the installation and mating relationship between the wheels and the ground marking components. The design follows the principle of minimizing the height of the marker points. The markers are fixed to the ground to determine and reproduce the planar position of the wheel contact reference point. The smaller the height of the marker points, the smaller the positional deviation caused by their own height, which can effectively reduce the impact of height error on the calibration benchmark.
[0044] S103: Set up a surveying instrument on the surveying instrument mounting base to determine the coordinates of the coded targets on each of the target elevations, the coordinates of each target in the near-field target group, and the coordinates of each of the marker points.
[0045] Specifically, the surveying instrument uses a high-precision total station. After being set up once on a fixed base, the instrument lens rotates 360° to complete the observation of all targets and marker points, eliminating the need for secondary station relocation. When setting up the targets, the problem of vehicle body obstruction after parking has been considered in advance. Targets are not set up in areas where the vehicle body will block the view, thus avoiding observation obstruction and ensuring that all measurements can be completed in one setup.
[0046] Specifically, the measurements included the three-dimensional coordinates of the coded target centers on each target elevation, the three-dimensional coordinates of near-field ground targets and combined targets, and the three-dimensional coordinates of the wheel alignment fixture markings. All coordinates were unified to the same mapping coordinate system, achieving sub-millimeter accuracy. The ground needed to be flat, and a total station was used to complete the height measurement. The ground height error was controlled within a few millimeters before the system was put into use, ensuring the accuracy of the reference surface for coordinate measurements.
[0047] S104: Acquire coded target images on each of the target elevations captured by the camera on the vehicle.
[0048] Specifically, the vehicle is equipped with seven pinhole cameras, three fisheye cameras for blind spot detection, and telephoto cameras. The telephoto cameras include two types: 30° ultra-telephoto and 60° telephoto. Each camera acquires coded target images within its corresponding field of view. For pinhole and fisheye cameras with large field of view, a single-distance target arrangement is sufficient. However, for the telephoto cameras whose optical axis coincides with the longitudinal direction of the vehicle, a longitudinally layered target arrangement is required to provide a geometric baseline in the depth direction.
[0049] Specifically, the targets are arranged in a row and column array on the elevation. The row and column spacing ensures that there are sufficient targets within the field of view of all 7 pinhole cameras and 3 filler fisheye cameras on the vehicle, and targets are also present in the overlapping areas of the fields of view of adjacent cameras. The common targets in the overlapping areas are the structural condition upon which cross-camera calibration depends. If there are no targets in the overlapping areas, the calibration cannot be performed structurally. This arrangement design provides the necessary structural basis for calibration consistency calibration among multiple cameras on the vehicle.
[0050] See Figure 5 As shown, Figure 5 This figure shows a top view of the coverage relationship between a target array and the field of view of adjacent cameras provided in Embodiment 1 of this application. This figure is a top view of the coverage relationship between the target array and the field of view of adjacent cameras, presenting the spatial matching design of the field of view of vehicle-mounted multi-cameras and the target array.
[0051] The vehicle on the left is equipped with camera A and camera B, and a target array is arranged vertically on the right. Camera A corresponds to the field of view of camera A, and camera B corresponds to the field of view of camera B. The fields of view of the two adjacent cameras have a common overlapping area.
[0052] The targets are arranged in a row and column array with specially designed row and column spacing, so that each camera's field of view has a sufficient number of targets, and the overlapping areas of the fields of view of adjacent cameras also have targets, providing the necessary structural conditions for cross-camera consistency verification.
[0053] S105: Acquire point cloud data collected by the lidar on the vehicle, including at least the vertical reference elevation.
[0054] Specifically, the vehicle-mounted lidar includes a main lidar and a downward-looking blind-spot lidar. The collected point cloud data must completely cover the farthest vertical reference elevation, while also covering the other target elevations and near-field combined targets. The measured number of points on the farthest reference elevation by the lidar is 1460 (T001) / 1477 (T002), and the point cloud density is sufficient to support high-precision planar fitting.
[0055] Specifically, the vertical reference facade is significantly farther away than the other facades (42.5m in this embodiment, compared to 18.7m and 27.8m for the others), and there are no obstructions between them. This allows the downward-looking lidar at the front of the vehicle to have a complete line of sight to this facade. The lidar measured the centroid distance of this facade to be 42.12m. This facade becomes the farthest vertical plane in the lidar's field of view, thus distinguishing it from the other parallel facades nearby. This distinguishability is purely provided by the spatial layout.
[0056] S106: Using multiple target facades at different longitudinal distances, external parameter calibration is performed on the telephoto camera on the vehicle.
[0057] Specifically, multiple target facades are arranged in layers along the longitudinal direction of the vehicle, creating a depth projection of the targets along the optical axis of the telephoto camera, forming a longitudinal geometric baseline. For the telephoto camera whose optical axis coincides with the longitudinal direction of the vehicle, this baseline is the geometric condition upon which its positioning in that direction depends. Existing calibration fields often concentrate calibration plates or target arrays on a wall or screen at a fixed distance in front of the vehicle. The distance from each target to the camera is basically the same, and the targets have no projection along the optical axis, lacking a geometric baseline for positioning in that direction. The longitudinally layered structure of this scheme is designed to address this deficiency.
[0058] Specifically, by constraining the target coordinates at different depth levels to calculate the extrinsic parameters of the telephoto camera, the longitudinal positioning accuracy can be significantly improved compared to the calibration method using a single distance target. Actual measurements show that the standard deviation of the longitudinal positioning of the 30° ultra-telephoto camera in a four-layer arrangement is 0.72 mm. This value increases to 2.17 mm when degenerated to a single-layer arrangement, and further deteriorates to 25 mm and 56 mm in the lateral and vertical directions, respectively. The structural layering directly leads to a significant difference in positioning conditions.
[0059] See Figure 6 As shown, Figure 6 This paper presents a comparison diagram of the longitudinal layered arrangement and effect of a target facade provided in Embodiment 1 of this application. The diagram is divided into upper and lower parts to show the differences between the two schemes and the comparison of calibration accuracy.
[0060] The upper part is the scheme of this application. The target facade is layered along the longitudinal direction of the vehicle, and the target has an expansion amount in the optical axis direction. The optical axis of the telephoto camera coincides with the longitudinal direction of the vehicle. Multiple target facades are distributed longitudinally in front of it. The distances from each target to the camera are 12.1m, 21.2m, 24.0m, and 36.2m, respectively, forming a longitudinal depth baseline with a span of about 24m, which provides geometric conditions for the longitudinal positioning of the telephoto camera. The farthest reference facade is a single flat continuous structure, which also serves as a reference for the lidar.
[0061] The lower half represents the existing technical solution, where the target is concentrated at a single distance, with no expansion in the optical axis direction. The target is only placed at the same distance position and lacks a geometric baseline in the longitudinal depth direction.
[0062] Structural effect comparison: When the four-layer arrangement is used, the standard deviation of the longitudinal positioning form of the telephoto camera is 0.72 mm; when only the farthest single layer is used, it degrades to 2.17 mm, and the lateral and height directions deteriorate to 25 mm / 56 mm.
[0063] S107: And, using the vertical reference elevation as both a camera reference and a lidar reference, the camera and lidar on the vehicle are jointly calibrated.
[0064] Specifically, the coded target on the vertical reference facade can be identified by the camera and its feature points extracted. Its flat and continuous planar structure can be fitted by the lidar to obtain the plane equation through point cloud fitting. The number of measured internal points on this facade by the lidar is 1460 (T001) / 1477 (T002), and the RMS (Root Mean Square) of the plane fitting is 0.0253m (T001) / 0.0269m (T002). The angle between the normals fitted to the left and right halves of the facade is only 0.019°~0.166°, indicating that the entire facade is indeed on the same plane, and the fitted plane can accurately represent the overall spatial orientation of the facade.
[0065] Specifically, both cameras share the same set of global coordinate true values measured by surveying instruments. The target is attached to the facade, and the target coordinates implicitly contain the spatial orientation of the facade. There is no need for additional reference transfer between two independent calibration facilities, enabling unified joint calculation of the extrinsic parameters of the camera and lidar. The calibration results of the two types of sensors are naturally under the same reference. The targets in the existing calibration field are patterns designed for the camera, which the lidar cannot recognize. The ordinary walls in the calibration field do not have surveyed true values, and many walls are parallel and similar in shape, making them unsuitable as references. Therefore, the camera and lidar usually require their own calibration facilities, and the reference relationship between the two facilities requires additional calibration and transfer. The shared reference facade structure of this solution is designed to address this deficiency.
[0066] In an optional implementation, see Figure 7 As shown, Figure 7 The flowchart of a joint calibration method provided in Embodiment 1 of this application is shown, wherein the joint calibration of the camera and lidar on the vehicle using the vertical reference elevation as both a camera reference and a lidar reference includes steps S701 to S704: Specifically, this joint calibration method uses the vertical reference plane as a common reference plane. It extracts the target feature points from the camera side and the plane equations from the radar side, and then constructs a joint optimization objective function using global coordinates as a common reference. Finally, it solves for the extrinsic parameters of both types of sensors simultaneously. This method does not require building a separate calibration facility for the lidar, nor does it require transferring the reference between the two facilities, ensuring that the calibration results of both types of sensors are under the same reference.
[0067] S701: Extract the image information of the coded target on the vertical reference surface to obtain the target feature points in the camera coordinate system.
[0068] Specifically, the coded target uses the 361 family AprilTag (AprilTag is a coded reference mark used for visual positioning and camera pose calculation), which belongs to the Yougou code category and has good recognition accuracy and robustness. The corner points or center pixel coordinates of the target are extracted through image recognition algorithms, and the corresponding two-dimensional feature points in the camera coordinate system are obtained. This encoding type is only an example of the implementation scheme and is not the only optional encoding form.
[0069] Specifically, the coded targets on the vertical reference facade correspond to consecutive numbering intervals, which in this embodiment are 134–159. The facade to which a target belongs can be identified by its number, avoiding confusion with targets on nearby facades. The targets on each target facade are divided into consecutive numbering intervals, with one numbering interval corresponding to one target facade. The target facade to which a target belongs can be distinguished by identifying the numbering interval of the coded target.
[0070] See Figure 8 As shown, Figure 8 This diagram shows a frontal view of the furthest reference facade provided in Embodiment 1 of this application, illustrating the structural features and target arrangement of the reference facade. This facade is the fourth facade, the furthest in a multi-layered target facade group, and is a single, flat, and continuous vertical plane. Encoded targets are fixed in a three-row array on the facade, numbered 134-159 in a continuous interval. The diagram is marked with "Horizontal width 7.995m (determined by the target's mapping coordinates, constituting the extraction window width on the lidar side)," and at the bottom, it is noted that "The continuous interval numbered 134-159 corresponds to this facade; the three-dimensional coordinates of the target center are determined by a surveying instrument, and the target coordinates implicitly represent the spatial orientation of this facade." The horizontal width of the facade is 7.995m, which is determined by the mapping coordinates of the target on it, forming the width of the planar extraction window on the lidar side. The three-dimensional coordinates of the target center are determined by the mapping instrument. Since the target is attached to the facade, the target coordinates implicitly contain the spatial orientation of the facade, so that the same facade can be used as both a pattern reference for the camera and a planar reference for the lidar. The two types of sensors share the same set of mapping true values, without the need for additional reference transfer.
[0071] S702: Extract the point cloud data of the lidar on the vertical reference surface, perform plane fitting on the point cloud data, and obtain the plane equation in the radar coordinate system.
[0072] Specifically, a subset of the point cloud corresponding to the farthest vertical plane is selected from the lidar point cloud, and a plane fitting algorithm is used to obtain the planar expression of the facade in the radar coordinate system. Since the vertical reference facade is a single flat and continuous plane, the angle between the normals fitted to the left and right halves of the facade is only 0.019°~0.166°, indicating that the entire facade is indeed on the same plane, and the fitted plane can accurately represent the overall spatial orientation of the facade.
[0073] Specifically, the target facades near the calibration field are mostly composed of multiple faces with different orientations, and they deviate from the reference facade by about 0.49°. Confusing the two will introduce a reference deviation of a corresponding magnitude. By filtering the point cloud by distance threshold and verifying the plane normal, the farthest vertical reference facade can be accurately located, avoiding confusion with the nearby parallel facades. This is a distinguishability provided purely by the spatial layout.
[0074] S703: Using the global coordinates of each target point on the vertical reference surface measured by the surveying instrument as a common reference, the target feature points in the camera coordinate system and the plane equations in the radar coordinate system are unified to the global coordinate system to construct a joint optimization objective function.
[0075] Specifically, the global coordinates are pre-determined by surveying instruments, encompassing both the true positional values of the target pattern and the true geometric values of the plane. All coordinates are obtained in the same measurement and within the same coordinate system, without introducing additional coordinate system transmission errors. The surveying instruments used are high-precision total stations, achieving sub-millimeter level coordinate accuracy, providing high-precision benchmark constraints for joint optimization.
[0076] Specifically, the feature point reprojection error on the camera side and the plane fitting error on the radar side are both incorporated into the same objective function. Constraints are established using the global coordinate system as a unified benchmark, allowing the calibration errors of the two types of sensors to be jointly optimized. The target is attached to the reference elevation, and the target coordinates implicitly contain the spatial orientation of the elevation. Therefore, the point constraints on the camera side and the plane constraints on the radar side essentially share the same geometric benchmark.
[0077] S704: Iteratively optimize the joint optimization objective function, and simultaneously solve for the extrinsic parameters of the camera and the lidar.
[0078] Specifically, by iteratively minimizing the total error of the objective function using a nonlinear optimization algorithm, the extrinsic parameters of the camera and the lidar relative to the global coordinate system are simultaneously calculated, indirectly obtaining the extrinsic parameter relationship between the camera and the lidar. This joint calibration method ensures that the calibration results of the two types of sensors are naturally under the same reference, avoiding the additional errors introduced by separate calibration and subsequent reference transfer.
[0079] Specifically, this joint calibration method eliminates the need for separate calibration facilities for lidar and the need to transfer references between the two facilities, ensuring that the calibration results of both types of sensors are under the same reference. In existing technologies, cameras and lidar typically require their own calibration facilities, and the reference relationship between the two facilities requires additional calibration and transfer. This solution, through a structural design that shares a reference facade, fundamentally eliminates this additional transfer step.
[0080] In an optional implementation, see Figure 9 As shown, Figure 9 The flowchart of an extrinsic parameter calibration method provided in Embodiment 1 of this application is shown, wherein the extrinsic parameter calibration of a telephoto camera on a vehicle using multiple target facades at different longitudinal distances includes steps S901-S903: Specifically, this extrinsic parameter calibration method is designed for telephoto cameras whose optical axes coincide with the longitudinal direction of the vehicle. It utilizes the depth span formed longitudinally by multiple target facades as a geometric baseline, and calculates the camera's extrinsic parameters through the correspondence between pixel coordinates and global 3D coordinates. This project is equipped with two types of telephoto cameras: a 30° super telephoto and a 60° telephoto, whose optical axes coincide with the longitudinal direction of the vehicle and are strongly dependent on the longitudinal geometric baseline.
[0081] S901: Extract the pixel coordinates of the coded targets located on different target facades in the images captured by the camera.
[0082] Specifically, different target facades correspond to different AprilTag number ranges. In this embodiment, the nearby facades correspond to numbers 120–126, the intermediate facades to 127–133, and the farthest reference facades to 134–159. The facade level to which the target belongs can be distinguished by identifying the coded number. After the camera acquires the image, the number of each target is obtained by decoding. Based on the range to which the number belongs, it can be determined which vertical facade level the target is located on, without the need for additional facade recognition steps.
[0083] Specifically, the center pixel coordinates of the targets on the near, middle, and far elevation surfaces are extracted to fully cover the entire field of view of the telephoto camera. In this embodiment, a total of 145 coded targets are arranged, with numbers ranging from 102 to 265. The targets are arranged in a row and column array on the elevation surface, and the row and column spacing ensures that there are enough targets within the field of view of the telephoto camera to meet the calibration sampling requirements.
[0084] S902: Based on the three-dimensional coordinates of the centers of each coded target in the global coordinate system determined by the surveying instrument, establish spatial point sets at different depth levels.
[0085] Specifically, the three-dimensional coordinates of each coded target are obtained in a single measurement by surveying instruments, with coordinate accuracy reaching sub-millimeter level. Ground height errors are controlled within a few millimeters before the target is put into use. All coordinates are obtained in the same measurement and under the same coordinate system to avoid additional errors introduced by coordinate system transfer.
[0086] Specifically, the spatial coordinate point sets are grouped according to the distance of the target to the facade, forming multiple depth levels. The depth span of the point sets in the longitudinal direction of the vehicle can reach approximately 24m, and the actual target distance span used by each camera ranges from 12.1m to 36.2m. The spatial point sets at different depth levels provide the core geometric constraints for the longitudinal positioning of the telephoto camera, allowing the target to have an expansion amount in the optical axis direction.
[0087] S903: Using the longitudinal coordinate difference of each target in the direction of the camera's optical axis as a geometric constraint, calculate the extrinsic parameters of the telephoto camera.
[0088] Specifically, the depth baseline formed by the longitudinal coordinate difference provides the core geometric constraint for the longitudinal positioning of the telephoto camera. Combined with the lateral and height coordinate constraints of the target, the extrinsic parameters of the telephoto camera are obtained through the camera pose calculation algorithm. For a telephoto camera whose optical axis coincides with the longitudinal direction of the vehicle, this longitudinal baseline is the geometric condition upon which its positioning in this direction depends. In existing technologies, this baseline is lacking when the target is concentrated at a single distance.
[0089] Specifically, under this calibration method, the standard deviation of the longitudinal positioning of the 30° super telephoto camera can reach 0.72mm, which is far superior to the calibration accuracy of a single-distance target. When the target is concentrated at a single layer, the longitudinal standard deviation increases to 2.17mm, and the lateral and height directions deteriorate further to 25mm and 56mm, respectively. The layering of the structure directly leads to a difference in the magnitude of the positioning conditions. When the target is concentrated at a single distance, the longitudinal standard deviation deteriorates by about 3 times, and the lateral and height directions deteriorate further to the tens of millimeters.
[0090] In an optional implementation, the distance from the vertical reference facade to the vehicle parking area is greater than the distance from the other target facades to the vehicle parking area, and there is no obstruction between the vertical reference facade and the other target facades.
[0091] Specifically, in this embodiment, the distance from the vertical reference facade to the parking area is approximately 42.5m, and the distance to the centroid of this facade measured by the lidar is 42.12m. The distances to the other target facades are approximately 18.7m and 27.8m, showing a significant difference. This location feature, "significantly farther than the other facades," makes it the farthest vertical plane in the lidar's field of view, thus distinguishing it from the other nearby parallel facades.
[0092] Specifically, there are no buildings, equipment, or other obstructions between the vertical reference facade and the other target facades, ensuring that the downward-looking lidar at the front of the vehicle has a complete line of sight to that facade, while also making that facade the farthest vertical plane in the lidar point cloud. The orientation of the farthest reference facade differs from that of the nearest parallel facade by 0.49°. Confusing the two would introduce a corresponding order of magnitude of reference deviation, which is precisely why the reference facade is required to be "significantly farther away from the other facades."
[0093] In the point cloud of the lidar, the farthest vertical plane is identified as the vertical reference elevation, to distinguish it from the other target elevations nearby.
[0094] Specifically, the target facades near the calibration field are mostly composed of multiple faces with different orientations, and deviate from the reference facade by about 0.49°. Confusing the two will introduce a reference deviation of a corresponding magnitude. The facades near the calibration field are composed of multiple faces with different orientations, and the included angle is significantly larger under the same measurement. In contrast, the included angle of the normals fitted to the left and right halves of the farthest reference facade is only 0.019°~0.166°, indicating that the entire facade is indeed on the same plane.
[0095] Specifically, by filtering point cloud distance thresholds and verifying planar normals, the farthest vertical reference facade can be accurately located, avoiding confusion with nearby parallel facades. This distinguishability is purely provided by spatial layout. This distinguishability does not rely on target patterns or additional markings; it can be achieved solely through the spatial location characteristics of the facade, simplifying the reference identification process on the lidar side.
[0096] In an alternative implementation, the near-field target group includes ground targets laid on the ground.
[0097] Specifically, ground targets are laid or affixed to the ground in the vehicle parking area and its surrounding areas, located at the lower edge of the pinhole camera's field of view and within the field of view of the fill-in fisheye camera, to complete the calibration target coverage in the near-field area. The plane on which the ground targets are located is a horizontal ground surface, with its normal direction along the vertical direction, perpendicular to the normal direction of the vertical targets (along the vehicle's longitudinal direction).
[0098] Specifically, the observation area of the wide-field-of-view fill-in-the-blank camera is concentrated on the ground and low-lying areas near the vehicle. In the existing calibration field, all targets are placed on distant vertical surfaces, leaving this area uncovered. The ground targets are designed to fill this gap. The fill-in-the-blank fisheye camera has a large field of view, concentrating its observation area on the ground near the vehicle. The ground targets are distributed within this field of view and can be completely captured by the fisheye camera, providing characteristic data for near-field camera calibration.
[0099] The normal direction of the plane on which the ground target is located is perpendicular to the normal direction of the vertical surface of the target.
[0100] Specifically, the target's elevation is a vertical plane with its normal along the vehicle's longitudinal direction; the ground target is located on a horizontal surface with its normal along the vertical direction; the normal directions of the two targets are perpendicular to each other, forming a complementary geometric structure. This complementarity in the normal direction is a geometric condition that makes the two types of targets structurally complementary, providing calibration constraints for the camera in different directions.
[0101] Specifically, by combining the longitudinal constraints of the facade target and the vertical constraints of the ground target, multi-directional geometric constraints are formed to solve the extrinsic parameters of the fill-in fisheye camera, ensuring the calibration accuracy of the near-field area and filling the gap where the distant facade target cannot cover the near field. The normals of the near-field target and the facade target are perpendicular to each other, forming a geometrically complementary coverage that covers the near-field observation area of the fill-in camera and the downward-looking radar.
[0102] See Figure 10 As shown, Figure 10 This image illustrates a near-field real-world calibration scene provided in Embodiment 1 of this application, captured by a vehicle-mounted telephoto camera. The image shows multiple near-field coded targets (AprilTags) laid on the ground, within the near-field observation range of the fill-in fisheye camera and the downward-looking lidar. In the distance, multiple layers of vertical coded targets are arranged on walls and pillars, distributed longitudinally at different distances, covering the camera's mid-to-long-range field of view. The location is an indoor calibration workshop equipped with load-bearing pillars, industrial fans, lighting fixtures, and overhead crane rails. The horizontal plane containing the ground targets and the vertical plane containing the vertical targets have their normals perpendicular to each other, forming a complementary coverage in geometry, jointly meeting the calibration requirements of multiple types and multiple field-of-view sensors in the vehicle.
[0103] See Figure 11 As shown, Figure 11 The flowchart of a near-field extrinsic parameter calibration method provided in Embodiment 1 of this application is shown, wherein the method further includes steps S1101~S1102: Specifically, this near-field extrinsic parameter calibration method is designed for a blind-spot fisheye camera on a vehicle. It utilizes a ground target laid on the ground and its geometric relationship with the normal of a vertical target to achieve near-field camera extrinsic parameter calibration. The observation area of the blind-spot fisheye camera is concentrated on the ground and low-ceilinged space near the vehicle; the distant vertical target cannot cover this area. The ground target provides the necessary characteristic basis for near-field calibration.
[0104] S1101: Acquire the image of the ground target captured by the fisheye camera on the vehicle.
[0105] Specifically, the filler fisheye camera has a large field of view, and its observation area is concentrated on the ground near the vehicle. Ground targets are distributed within this field of view and can be completely captured by the fisheye camera, providing characteristic data for near-field camera calibration. The ground targets are laid or affixed to the ground in the vehicle parking area and its extended areas in front of and behind it, located at the lower edge of the pinhole camera's field of view and within the field of view of the filler fisheye camera.
[0106] S1102: Using the geometric relationship that the ground target and the normal direction are perpendicular to each other, perform near-field extrinsic parameter calibration on the blind spot fisheye camera.
[0107] Specifically, by combining the longitudinal constraints of the facade target and the vertical constraints of the ground target, multi-directional geometric constraints are formed to solve the extrinsic parameters of the filler fisheye camera, ensuring calibration accuracy in the near-field region and filling the gap where distant facade targets cannot cover the near-field. The normal direction of the plane where the ground target is located is perpendicular to the normal direction of the target facade; this complementarity in normal direction is the geometric condition that makes the two types of targets structurally complementary.
[0108] In an alternative implementation, the near-field target group further includes a combined target for downward-looking radar, consisting of a ground panel and a vertical panel fixedly connected at a defined angle.
[0109] Specifically, the combined targets are positioned on both sides and near the rear of the vehicle. The ground panel is flush with the ground, while the vertical panel is perpendicular to the ground, with the included angle fixed at 90°. Both panels are within the field of view of the downward-looking blind-filling lidar. The different normal directions of the two panels ensure that the nearby area has an observable planar structure in two mutually perpendicular directions, providing multi-directional planar constraints for the downward-looking blind-filling lidar.
[0110] Specifically, the observation area of the downward-looking blind-filling lidar is concentrated on the ground and low-ceilinged space near the vehicle. The distant vertical targets in the existing calibration field cannot cover this area. The combined target is specifically designed for the near-field calibration of the downward-looking blind-filling lidar. During calibration, this fixed angle is used as a geometric constraint. By combining the point cloud plane fitting results of the two plates, the extrinsic parameters of the downward-looking blind-filling lidar can be calculated, achieving accurate calibration of the lidar in the near-field area.
[0111] See Figure 12 As shown, Figure 12 This diagram illustrates a three-dimensional structure of a combined target for a downward-looking radar according to Embodiment 1 of this application. 11 is a ground panel, 12 is a vertical panel, and 13 is the defined angle between the two panels. The ground panel and the vertical panel are fixedly connected as a single unit, and the defined angle α formed by them is 90° in this embodiment. The normal directions of the two panels are different, providing observable planar structures in two mutually perpendicular directions near the vehicle, thus providing a geometric constraint reference for the near-field extrinsic parameter calibration of the downward-looking blind-spot lidar.
[0112] See Figure 13 As shown, Figure 13 A flowchart of the second near-field extrinsic parameter calibration method provided in Embodiment 1 of this application is shown, wherein the method further includes steps S1301~S1302: Specifically, this near-field extrinsic parameter calibration method is designed for a downward-looking blind-filling lidar on a vehicle. It utilizes the ground panel and vertical panel of the combined target and their fixed angle constraints to achieve lidar extrinsic parameter calibration in the near-field region. The observation direction of the downward-looking blind-filling lidar is towards the lower and lower sides of the vehicle, and the combined target is arranged on both sides and near the rear of the vehicle, within its field of view.
[0113] S1301: Acquire point cloud data of the ground panel and the vertical panel collected by the downward-facing blind spot lidar on the vehicle.
[0114] Specifically, the downward-looking blind spot fill lidar observes from below and to the side below the vehicle, simultaneously acquiring point clouds from both the ground panel and the vertical panel of the combined target. The point clouds of the two panels can be segmented and extracted based on differences in spatial position and normal direction. The ground panel is attached to the ground, while the vertical panel is perpendicular to the ground; the two differ significantly in spatial position and normal direction, facilitating point cloud segmentation.
[0115] S1302: The near-field extrinsic parameter calibration of the downward-looking lidar is performed using the included angle constraint of the two sides of the combined target of the downward-looking radar.
[0116] Specifically, the included angle between the two plates of the combined target is a known fixed value, which serves as a geometric constraint. Combined with the point cloud plane fitting results of the two plates, the extrinsic parameters of the down-view blind-filling lidar are calculated, achieving accurate calibration of the lidar in the near-field region. The different normal directions of the two plates allow the nearby region to have observable planar structures in two mutually perpendicular directions simultaneously, providing multi-directional constraints for extrinsic parameter calculation.
[0117] In an alternative implementation, the wheel positioning fixture includes ground markers.
[0118] Specifically, the ground markers are reflective patches adapted to the high reflectivity measurement requirements of total stations, located at the contact points of each wheel, used to determine and reproduce the planar position of the wheel contact reference point. Existing calibration room conditions are simple, lacking a fixed standard structure, and reflective patches are mainly used on-site, without any special customized hardware. This solution uses permanently fixed reflective patches as ground markers, forming a standardized wheel positioning structure.
[0119] Specifically, the vehicle coordinate system needs to be established based on the wheel contact points. Ground markers ensure that the position of the wheel contact points does not need to be re-found in each calibration, resulting in good consistency and solving the problem of the lack of wheel contact point positioning structures in existing calibration fields. For commercial vehicles with dual rear axles, there is an even greater lack of a unified structural convention for the reference positions of the two rear wheels on the same side. The markers for multiple wheels on the same side forming a set are designed to solve this problem.
[0120] The ground marker is permanently fixed to the ground.
[0121] Specifically, the ground markers are embedded and fixed to the calibration field surface, making them immovable and unremovable. Their position remains unchanged over time, avoiding the need to relocate the wheel contact point for each calibration and eliminating differences in reference position between different calibrations. This permanent installation method ensures the reproducibility of the wheel reference position, making the vehicle coordinate system reference consistent for each calibration.
[0122] The marker point is located on the ground marker.
[0123] Specifically, the marker point is a characteristic point on the ground marker for the surveying instrument to aim at. It is a representative point for the wheel contact reference point, and its position corresponds to the center of the wheel contact point. The marker point uses reflective material adapted to the high reflectivity measurement requirements of the total station, ensuring that the surveying instrument can accurately aim at and determine its three-dimensional coordinates.
[0124] The height deviation between the marker point and the ground shall not exceed 5 mm.
[0125] Specifically, the markers are positioned as close to the ground as possible in the vertical direction, with a height deviation controlled within 5 millimeters, and no algorithmic compensation is applied to this height deviation. The design simplifies the calibration process by ensuring the markers are as close to the ground as possible with a height deviation controlled within 5 millimeters and by not applying algorithmic compensation to this height deviation.
[0126] Specifically, the z=0 plane of the vehicle coordinate system is taken from the ground rather than the height of the marker point itself. This avoids the height deviation causing a uniform shift in the height direction of each camera when it is incorporated into the vehicle coordinate system origin. The height of the marker point is directly entered into the z-coordinate of the wheel reference point. If this height were incorporated into the origin of the vehicle coordinate system, it would cause a uniform systematic shift in the height reported by each camera. Therefore, the z=0 plane of the vehicle coordinate system is taken from the ground rather than the height of the wheel point itself.
[0127] See Figure 14 As shown, Figure 14 This figure shows a partial cross-sectional view of a wheel positioning fixture provided in Embodiment 1 of this application, illustrating the installation structure and height design logic of the marker on the calibration field surface. 9 is the ground marker, installed on the calibration field surface using an embedded fixing method; 10 is the target point for the surveying instrument, located on the top surface of the ground marker. The design follows the principle of minimizing the height h of the target point from the ground, thereby reducing the positional deviation caused by the height of the target point itself and ensuring the positioning accuracy of the wheel contact reference point.
[0128] In an optional implementation, the encoded target is an AprilTag encoded target.
[0129] Specifically, the coded target belongs to the 361 family of coding systems, specifically the Yougou code category, and possesses good recognition accuracy and robustness. This coding type is merely an example of an implementation plan and is not the only optional coding form; other coded targets with similar recognition performance can also be applied.
[0130] Specifically, the actual on-site mounting numbers start from 100, and the specific numbers can be adjusted according to the actual layout. In this embodiment, a total of 145 coded targets are arranged, with the numbering range from 102 to 265. The coded targets on each target facade are arranged in sections according to continuous numbering intervals.
[0131] The coded targets on each target facade are arranged in consecutive numbered intervals, with one numbered interval corresponding to one target facade.
[0132] Specifically, this embodiment deploys 145 coded targets, numbered from 102 to 265. Nearby facades correspond to numbers 120–126, intermediate facades to 127–133, and the furthest reference facades to 134–159. The facades to which the targets belong can be quickly identified through these numbering intervals. Each numbering interval corresponds to one target facade, allowing the correspondence between targets and facades to be directly determined by the coded number, eliminating the need for additional spatial location determination.
[0133] The corresponding target facade is distinguished by identifying the number range of the coded target.
[0134] Specifically, after the camera acquires images, it decodes each target to obtain its number. Based on the interval to which the number belongs, it can be determined which vertical elevation level the target is located on, eliminating the need for additional elevation recognition steps. This zoning method simplifies the determination of the correspondence between targets and elevations, improving the automation and reliability of the calibration process.
[0135] In an optional implementation, for a multi-axle vehicle with two or more wheels on the same side, the ground markings of each wheel on the same side form a group, and the ground markings in the group are arranged longitudinally along the vehicle.
[0136] Specifically, for the dual-rear-axle commercial vehicle structure with two rear wheels on the same side, the ground marking points corresponding to the two rear wheels on the same side are set as a group and arranged longitudinally along the vehicle to match the wheel spacing of the dual rear axles, thus adapting to the calibration requirements of multi-axle vehicles. For commercial vehicles with dual rear axles, the lack of a unified structural convention for the reference positions of the two rear wheels on the same side is one of the shortcomings of existing technology. The set of marking points for multiple wheels on the same side is designed to solve this problem.
[0137] Specifically, the markers within the group are arranged longitudinally along the vehicle to jointly determine the reference position of the wheel on that side. By combining the survey coordinates of multiple markers within the same group on the same side, the wheel baseline on that side can be fitted. The wheel reference position on that side is determined by the survey coordinates of multiple ground markers on the same side, and the longitudinal axis of the vehicle is established. The baselines on both sides jointly determine the longitudinal axis of the vehicle, and a unified vehicle coordinate system is established based on this.
[0138] The ground markers on the left and right sides are arranged in pairs, and the two ground markers in the same pair are aligned in the longitudinal direction of the vehicle.
[0139] Specifically, the markers on the left and right sides of the vehicle correspond one-to-one, and the longitudinal coordinates of the same pair of markers are basically consistent, ensuring that the left and right wheel positions are symmetrical after the vehicle is parked, which facilitates the establishment of an accurate longitudinal axis of the vehicle body. The paired arrangement makes the reference positions on the left and right sides of the vehicle symmetrical, providing symmetrical benchmark conditions for the establishment of the vehicle coordinate system.
[0140] The wheel reference position on that side is determined by the combined mapping coordinates of multiple ground markers on the same side, and the longitudinal axis of the vehicle is established.
[0141] Specifically, by combining the survey coordinates of multiple marker points within the same side group, the wheel baseline for that side can be fitted. The baselines on both sides together determine the longitudinal axis of the vehicle, and a unified vehicle coordinate system is established based on this. The wheel positioning fixture makes the determination of the vehicle baseline reproducible and is suitable for multi-axle commercial vehicles. The markers are fixed to the ground, and the same set of reference positions is used for each calibration. The set of marker points for multiple wheels on the same side is suitable for dual rear axle models.
[0142] Example 2 Based on the same technical approach, this application also provides an in-vehicle multi-sensor calibration system, which includes: Vehicle parking area.
[0143] Specifically, the vehicle parking area is located in the center of the calibration field, with dimensions adapted to the needs of commercial vehicles entering and parking. The ground has been calibrated for flatness, and the ground height error is controlled within millimeters, providing a unified parking reference position for vehicles to be calibrated.
[0144] A multi-layered target facade group consists of at least three target facades arranged sequentially at intervals along the longitudinal direction of the vehicle. Each target facade is perpendicular to the longitudinal direction of the vehicle, and the distance from each target facade to the vehicle parking area increases sequentially.
[0145] Specifically, in this embodiment, three target facades are set at distances of 18.7m, 27.8m, and 42.5m from the vehicle parking area, respectively. They form a layered arrangement along the longitudinal direction of the vehicle, so that the targets have depth expansion in the direction of the camera optical axis, providing a longitudinal depth geometric baseline for the telephoto camera with the optical axis along the longitudinal direction of the vehicle, and improving the longitudinal positioning accuracy.
[0146] Among them, the target facade furthest away is the vertical reference facade, which is a single, flat, and continuous vertical plane, and each target facade is fixed with an array of coded targets.
[0147] Specifically, the vertical reference facade has a horizontal width of 7.995m, and the coded targets on the facade are arranged in three rows, corresponding to consecutive numbering intervals. This facade can serve as both the pattern reference for the camera and the planar reference for the lidar. The two types of sensors share the same set of mapping true values, and no additional reference transfer is required.
[0148] Near-field target groups are arranged on the ground and / or at low positions on both sides of the vehicle parking area.
[0149] Specifically, the near-field target group covers the ground and low-lying space near the vehicle, matching the near-field observation range of the fill-in fisheye camera and the downward-looking fill-in lidar, filling the gaps in the near-field area that cannot be covered by distant vertical targets, and achieving calibration coverage of the entire field of view.
[0150] The near-field target group includes ground targets and / or combined targets for downward-looking radar, which are formed by a ground panel and a vertical panel fixedly connected at a defined angle.
[0151] Specifically, the ground target is laid on the ground around the vehicle parking area, and the plane normal is perpendicular to the vertical target normal, providing multi-directional geometric constraints for the blind spot camera; the angle between the two plates of the combined target for the downward-looking radar is 90° in this embodiment, and the normal directions of the two plates are different, which can provide a plane calibration reference for the downward-looking blind spot lidar in two mutually perpendicular directions.
[0152] A wheel positioning fixture is installed on the ground in the vehicle parking area, including ground markers located at the ground contact points of each wheel and permanently fixed to the ground, as well as marking points on each ground marker for aiming at by surveying instruments.
[0153] Specifically, the ground markers are installed on the calibration field surface in an embedded and fixed manner, and their positions remain unchanged for a long time, which can reproduce the planar position of the wheel ground contact reference point. For dual rear axle vehicles, the markers of the two rear wheels on the same side form a multi-wheel marker point group on the same side, which is arranged along the longitudinal direction of the vehicle to meet the wheel positioning requirements of multi-axle vehicles.
[0154] The height deviation between the marker point and the ground shall not exceed 5 mm.
[0155] Specifically, the height of the marker point is designed to be as close to the ground as possible, with a height deviation controlled within 5 millimeters. This height is directly entered into the z-coordinate of the wheel reference point. The z=0 plane of the vehicle coordinate system is taken from the ground rather than the height of the marker point itself, so as to avoid the height deviation causing the same amount of offset in the height direction of each camera after being introduced into the vehicle system origin.
[0156] The surveying instrument mounting base is located on the side of the vehicle parking area. The position of the surveying instrument mounting base allows the surveying instrument to have a clear view of the entire target facade, the near-field target group, and the marking points of the wheel positioning fixture in one setup.
[0157] Specifically, once the surveying instrument is set up on the fixed base, it can complete the observation of all targets and markers by rotating the lens 360°, without the need for a second relocation; the coordinates of all targets and markers are obtained in the same measurement and in the same coordinate system, avoiding additional errors introduced by coordinate system transfer.
[0158] And a controller, which is used to execute the steps of the vehicle-mounted multi-sensor calibration method shown in Embodiment 1 above.
[0159] Specifically, the controller can receive target images collected by the camera and point cloud data collected by the lidar, call the pre-determined true global coordinates of the target, and sequentially complete processes such as individual extrinsic parameter calibration of the telephoto camera, joint calibration of the camera and lidar, and calibration of the near-field blind spot sensor, and output the extrinsic parameter calibration results of the vehicle's multiple sensors.
[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the system and its specific working process described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0161] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.
[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0163] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0164] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0165] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.
Claims
1. A method for calibrating vehicle-mounted multi-sensor systems, characterized in that, Includes the following steps: A calibration field is provided, which includes a vehicle parking area, at least three target facades arranged at intervals along the longitudinal direction of the vehicle, a near-field target group arranged in the near-field area, a wheel positioning fixture with marking points set on the ground of the vehicle parking area, and a surveying instrument mounting base set on the side of the vehicle parking area, wherein the target facade furthest away is the vertical reference facade. The vehicle to be calibrated is parked in the vehicle parking area, and each wheel is aligned with the wheel positioning fixture. A surveying instrument is set up on the surveying instrument mounting base to determine the coordinates of the coded targets on each of the target elevations, the coordinates of each target in the near-field target group, and the coordinates of each of the marker points. Acquire coded target images on each of the target elevations captured by the camera on the vehicle; Acquire point cloud data, including at least the vertical reference elevation, collected by the lidar on the vehicle; External parameter calibration of the telephoto camera on the vehicle was performed using multiple target facades at different longitudinal distances. Furthermore, the vertical reference elevation is used simultaneously as both a camera reference and a lidar reference to perform joint calibration of the camera and lidar on the vehicle.
2. The vehicle-mounted multi-sensor calibration method according to claim 1, characterized in that, The method of using the vertical reference plane as both a camera reference and a lidar reference to jointly calibrate the camera and lidar on the vehicle includes: Extract the image information of the coded target on the vertical reference surface to obtain the target feature points in the camera coordinate system; Extract the point cloud data of the lidar on the vertical reference surface, perform plane fitting on the point cloud data, and obtain the plane equation in the radar coordinate system; Using the global coordinates of each target point on the vertical reference surface measured by the surveying instrument as a common reference, the target feature points in the camera coordinate system and the plane equations in the radar coordinate system are unified to the global coordinate system to construct a joint optimization objective function. The joint optimization objective function is iteratively optimized, and the extrinsic parameters of the camera and the lidar are solved simultaneously.
3. The vehicle-mounted multi-sensor calibration method according to claim 1, characterized in that, The method of calibrating the extrinsic parameters of the telephoto camera on the vehicle using multiple target facades at different longitudinal distances includes: Extract the pixel coordinates of coded targets located on different target facades in the images captured by the camera; Based on the three-dimensional coordinates of the centers of each coded target in the global coordinate system determined by the surveying instrument, spatial point sets at different depth levels are established. The extrinsic parameters of the telephoto camera are calculated by using the longitudinal coordinate difference of each target along the optical axis of the camera as a geometric constraint.
4. The vehicle-mounted multi-sensor calibration method according to claim 1, characterized in that, The distance from the vertical reference elevation to the vehicle parking area is greater than the distance from the other target elevations to the vehicle parking area, and there is no obstruction between the vertical reference elevation and the other target elevations; In the point cloud of the lidar, the farthest vertical plane is identified as the vertical reference elevation, to distinguish it from the other target elevations nearby.
5. The vehicle-mounted multi-sensor calibration method according to claim 1, wherein The near-field target group includes ground targets laid on the ground; The normal direction of the plane where the ground target is located is perpendicular to the normal direction of the target's elevation. The method further includes: Acquire images of the ground target captured by the fisheye camera on the vehicle; The near-field extrinsic parameters of the filler fisheye camera are calibrated using the geometric relationship between the ground target and the perpendicularity of the normal direction.
6. The vehicle-mounted multi-sensor calibration method according to claim 1, wherein, The near-field target group also includes a combined target for downward-looking radar, which is formed by a ground panel and a vertical panel fixedly connected at a defined angle. The method further includes: Acquire point cloud data of the ground panel and the vertical panel collected by the downward-facing blind spot fill lidar on the vehicle; The near-field extrinsic parameters of the downward-looking lidar are calibrated using the included angle constraint of the two sides of the combined target of the downward-looking radar.
7. The vehicle-mounted multi-sensor calibration method according to claim 1, wherein The wheel positioning fixture includes ground markers; The ground markers are permanently fixed to the ground; The marker point is located on the ground marker; The height deviation between the marker point and the ground shall not exceed 5 mm.
8. The vehicle-mounted multi-sensor calibration method according to claim 1, wherein, The encoded target is an AprilTag encoded target; The coded targets on each target facade are arranged in consecutive numbered intervals, with one numbered interval corresponding to one target facade; The corresponding target facade is distinguished by identifying the number range of the coded target.
9. The vehicle-mounted multi-sensor calibration method according to claim 7, wherein For multi-axle vehicles with two or more wheels on the same side, the ground markings of each wheel on the same side form a group, and the ground markings in the group are arranged along the longitudinal direction of the vehicle. The ground markers on the left and right sides are arranged in pairs, and the two ground markers in the same pair are aligned in the longitudinal direction of the vehicle. The wheel reference position on that side is determined by the combined mapping coordinates of multiple ground markers on the same side, and the longitudinal axis of the vehicle is established.
10. A vehicle-mounted multi-sensor calibration system, comprising: include: Vehicle parking area; A multi-layered target facade group consists of at least three target facades arranged sequentially at intervals along the longitudinal direction of the vehicle. Each target facade is perpendicular to the longitudinal direction of the vehicle, and the distance from each target facade to the vehicle parking area increases sequentially. Among them, the target facade furthest away is the vertical reference facade, which is a single, flat, and continuous vertical plane, and each target facade is fixed with an array of coded targets. Near-field target groups are arranged on the ground and / or at low positions on both sides of the vehicle parking area; The near-field target group includes ground targets and / or a combined target for downward-looking radar consisting of a ground panel and a vertical panel fixedly connected at a defined angle. A wheel positioning fixture is installed on the ground in the vehicle parking area, including ground markers located at the ground contact positions of each wheel and permanently fixed to the ground, as well as marking points on each ground marker for aiming at by surveying instruments. The height deviation between the marker and the ground shall not exceed 5 mm; The surveying instrument mounting base is located on the side of the vehicle parking area. The position of the surveying instrument mounting base allows the surveying instrument to have a clear view of the entire target facade, the near-field target group, and the marking points of the wheel positioning fixture in one setup. And a controller for performing the method as described in any one of claims 1 to 9.
