A multi-layer extrinsic parameter collaborative calibration system and method for near-electric vehicle multi-source perception
By employing a multi-layer recursive calibration and local extrinsic parameter compensation method, the problem of large fusion error between conductor point cloud and image in existing technologies is solved, achieving high-precision extrinsic parameter collaborative calibration and improving the perception accuracy and reliability of near-electric vehicles in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN NARI LIABILITY OF STATE GRID ELECTRIC POWER RES INST
- Filing Date
- 2025-06-19
- Publication Date
- 2026-07-21
AI Technical Summary
Existing camera and lidar extrinsic parameter calibration methods suffer from large local area fusion errors in power operations, making it difficult to meet the requirements of conductor point cloud and image fusion in complex scenarios, which leads to difficulties in ensuring the safety of operation vehicles.
A method combining multi-layer recursive calibration and local extrinsic parameter compensation is adopted. The feature acquisition module collects image features and point cloud features, the multi-layer calibration module calculates the extrinsic parameter matrix and optimizes the error layer by layer, and the extrinsic parameter compensation module fits the extrinsic parameter compensation function to achieve high-precision extrinsic parameter collaborative calibration.
It achieves high-precision and robust external parameter collaborative calibration between the camera and the lidar, improves the accuracy and reliability of perception fusion of near-electric vehicles in dynamic scenes, enhances the fault tolerance of the system, and adapts to the multi-source perception requirements in complex environments.
Smart Images
Figure CN120747244B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of near-electric operation scenario perception, specifically to a multi-layer extrinsic parameter collaborative calibration system and method for multi-source perception of near-electric vehicles. Background Technology
[0002] During power construction work, vehicles inevitably come into close contact with power lines, posing a risk of electric shock. Traditional manual monitoring has a limited field of view and struggles to accurately determine the distance between the vehicle and the power line equipment, making it difficult to guarantee the safety of the vehicles.
[0003] To improve the safety of power line operations using vehicles, the current main method is to use fused data from cameras and lidar to measure the distance between the vehicle and the power line in real time. Existing camera and lidar extrinsic parameter calibration methods ensure a certain level of accuracy in the overall fusion of images and point clouds, but the fusion error is large in some local areas. This can lead to significant errors in the fusion of the power line point cloud and the image, making it difficult to meet the requirements for image and point cloud fusion of power lines in complex power operation scenarios. Summary of the Invention
[0004] The purpose of this invention is to provide a multi-layer extrinsic parameter collaborative calibration system and method for multi-source perception of near-electric vehicles, which solves the problem that the point cloud and image fusion of conductors in existing power operations may have large errors, making it difficult to meet the requirements of conductor image and point cloud fusion in complex power operation scenarios.
[0005] To achieve this objective, the present invention provides a multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles, comprising: The feature acquisition module is used to collect image features and point cloud features from the camera and lidar, and the image features and point cloud features constitute the initial calibration area; The multi-layer calibration module is used to calculate the extrinsic matrix of the initial region to be calibrated using image features and point cloud features in the initial region to be calibrated; the point cloud features are projected onto the image plane using the extrinsic matrix of the initial region to be calibrated to obtain the projected features of the initial region to be calibrated; the initial region to be calibrated is divided into several sub-regions; the image features, point cloud features, and projected features corresponding to each sub-region of the initial region to be calibrated are obtained; the error between the projected features of each sub-region and the corresponding image features is calculated; if the error does not exceed a preset error threshold, it is marked as a calibrated region; otherwise, it is recorded as a region to be calibrated; it is determined whether the pixel size and region length of the region to be calibrated meet the preset pixel size and region length; and the regions to be calibrated that meet the preset pixel size and region length are saved in the minimum set of regions to be calibrated.
[0006] The extrinsic parameter compensation module is used to fit the extrinsic parameter compensation function of the corresponding minimum region to be calibrated by utilizing the coordinate deviation between each projection feature of the minimum region to be calibrated and the corresponding image feature, thereby obtaining the extrinsic parameter compensation function set of the minimum region to be calibrated.
[0007] The beneficial effects of this invention are as follows: 1. This invention combines multi-layer recursive calibration with local extrinsic parameter compensation of the initial calibration region composed of collected point cloud features and image features. This achieves high-precision and robust extrinsic parameter collaborative calibration between the camera and the lidar. It can automatically identify and optimize regions with large errors, improve the overall calibration accuracy layer by layer, and finally construct the extrinsic parameter compensation function for each minimum calibration region. This effectively adapts to the multi-source perception requirements in complex environments and improves the accuracy and reliability of perception fusion for near-electric vehicles in dynamic scenes.
[0008] 2. By adopting multi-layer recursive calibration, even if there is a large error in the initial global calibration, the system can continuously approach the optimal solution through layer-by-layer optimization, thereby enhancing the fault tolerance of the system.
[0009] 3. By calculating the pixel error between the image features and the projected point cloud features in each sub-region of the area to be calibrated, the root mean square error of the sub-region is obtained. Regions with errors exceeding the limit are identified and recalibrated, which can effectively cope with environmental noise.
[0010] 4. By establishing an independent linear compensation function for the smallest calibrated region where the error exceeds the limit but cannot be further divided, it is possible to capture small-range external parameter drift caused by factors such as mechanical installation deviation and temperature change.
[0011] 5. Divide the point cloud features into multiple intervals along the image depth direction to achieve independent calibration and compensation for regions at different distances in three-dimensional space, which is suitable for near-electric vehicle perception needs in complex scenarios. Attached Figure Description
[0012] Figure 1 This is a flowchart illustrating the module operation of a multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles according to the present invention. Figure 2 This is a flowchart of a multi-layer extrinsic parameter collaborative calibration process for multi-source sensing of near-electric vehicles according to the present invention. Detailed Implementation
[0013] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1: like Figures 1-2 As shown, a multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles includes: The feature acquisition module is used to collect image features and point cloud features from the camera and lidar, and the image features and point cloud features constitute the initial calibration area; The multi-layer calibration module is used to calculate the extrinsic matrix of the initial region to be calibrated using image features and point cloud features in the initial region to be calibrated; the point cloud features are projected onto the image plane using the extrinsic matrix of the initial region to be calibrated to obtain the projected features of the initial region to be calibrated; the initial region to be calibrated is divided into several sub-regions; the image features, point cloud features, and projected features corresponding to each sub-region of the initial region to be calibrated are obtained; the error between the projected features of each sub-region and the corresponding image features is calculated; if the error does not exceed a preset error threshold, it is marked as a calibrated region; otherwise, it is recorded as a region to be calibrated; it is determined whether the pixel size and region length of the region to be calibrated meet the preset pixel size and region length; and the regions to be calibrated that meet the preset pixel size and region length are saved in the minimum set of regions to be calibrated.
[0014] The extrinsic parameter compensation module is used to fit the extrinsic parameter compensation function of the corresponding minimum region to be calibrated by utilizing the coordinate deviation between each projection feature of the minimum region to be calibrated and the corresponding image feature, thereby obtaining the extrinsic parameter compensation function set of the minimum region to be calibrated.
[0015] The method for constructing the initial calibration region is to use the physical area covered by all acquired image features and corresponding point cloud features as the initial calibration region.
[0016] In the above technical solution, when the pixel size and region length of the region to be calibrated do not meet the preset pixel size and region length, the following recalibration process is performed: The extrinsic parameter matrix of the region to be calibrated is calculated using the image features and point cloud features within the region to be calibrated; the point cloud features are projected onto the image plane using the extrinsic parameter matrix of the region to be calibrated to obtain the projected features of the region to be calibrated; the region to be calibrated is divided into several sub-regions, and the image features, point cloud features, and projected features corresponding to each sub-region are obtained; the error between the projected features of each sub-region and the corresponding image features is calculated; if the error does not exceed... If a region has a preset error threshold, it is marked as a calibrated region; otherwise, it is marked as a new region to be calibrated. The pixel size and region length of the new region to be calibrated are then determined to meet the preset pixel size and region length. Regions that meet the preset pixel size and region length are saved in the minimum set of regions to be calibrated. If the pixel size and region length of the new region to be calibrated do not meet the preset pixel size and region length, the new region to be calibrated is used as a region to be calibrated again, and the above recalibration process is repeated until all regions to be calibrated meet the preset pixel size and region length.
[0017] This invention employs a hierarchical calibration process, using acquired image features and point cloud features to form an initial calibration region. First, the calibration region is calibrated and divided into several sub-regions. By calculating the pixel error between the projection features of the point cloud features in each sub-region onto the image and the corresponding image features, sub-regions conforming to the root mean square error (RMSE) are identified as calibrated regions. Sub-regions not conforming to the RMSE are then used as the remaining calibration region for hierarchical calibration. Spatial subdivision is constrained by pixel size and the length of the subdivided sub-regions. Regions that cannot be further subdivided are designated as the minimum calibration region for local error compensation. This invention utilizes recursive spatial subdivision, calibrating only regions with errors exceeding the limit, significantly reducing computational redundancy. The termination thresholds for pixel size and region length ensure convergence at a reasonable scale, balancing accuracy and efficiency. Finally, an extrinsic parameter compensation function is fitted to the coordinate deviation of the minimum calibration region, achieving spatially adaptive error correction and effectively addressing the shortcomings of traditional calibration methods that suffer from local inaccuracies in complex environments.
[0018] The feature acquisition module in the above technical solution includes: The camera and LiDAR are fixed on a gimbal, and the feature targets are placed at different distances within the field of view of the camera and LiDAR. The positions of the feature targets are recorded by the camera and LiDAR to obtain image features and point cloud features. : ; in, Any pair of feature points within , and For the corresponding image feature points and point cloud feature points, Indicates the first Group characteristics.
[0019] In the above technical solution, the method for obtaining the projection features in the region to be calibrated in the multi-layer calibration module is as follows: Save all point cloud features and image features in the region to be calibrated to a feature set. P In the middle, using feature sets P The camera and lidar are calibrated to obtain the extrinsic parameter matrix. P Represented as: ; in, This represents the set of image feature points within the region to be calibrated. This is represented as a set of feature points in the point cloud within the region to be calibrated; Based on the extrinsic parameter matrix, the point cloud features within the region to be calibrated are transformed into pixels on the image, thus obtaining the projection feature point set of the point cloud features in the region to be calibrated onto the image. , Represented as: ; in, express Projected feature points within, any point inside and and The points in the middle correspond; The calibration method uses the PnP algorithm.
[0020] Using a pinhole camera model The point cloud features in the point set are projected onto a 2D image to obtain a preliminary extrinsic parameter matrix, based on... Point cloud features in point sets and Image features from the point set are used to initially solve for an approximate projection matrix using methods such as least squares. Finally, an iterative optimization algorithm is employed to optimize the projection matrix. In each iteration, based on the current extrinsic parameter estimates, the projected feature positions of the point cloud features onto the image are calculated using a pinhole camera model. The positions of the corresponding image features are compared, and the error is calculated. The gradient of the optimization variable (extrinsic parameter) is calculated based on the error, and the estimated value of the extrinsic parameter is updated. This process is repeated until the error converges to a certain level, that is, the set accuracy requirement is met, and the final extrinsic parameter matrix is obtained.
[0021] Based on the extrinsic parameter matrix The point cloud features are transformed from the LiDAR coordinate system to the camera coordinate system. Then, using the known camera intrinsic parameter matrix, the point cloud in the camera coordinate system is projected onto the planar coordinate system of the image, resulting in... .
[0022] In the above technical solution, the method for dividing the region to be calibrated into several sub-regions and obtaining the image features, point cloud features, and projection feature sets corresponding to each sub-region of the region to be calibrated in the multi-layer calibration module is as follows: Divide all corresponding point cloud features within the region to be calibrated into equal intervals along the image depth direction. h Divide all image features into intervals. m OK n Columns, obtaining several sub-regions and the feature set within the sub-region is Represented as: ; in, For the first in the region u Line number v Liede A set of features of a sub-region at a spatial location; , This represents the set of image feature points within a sub-region. This is represented as a set of point cloud feature points within a sub-region; It is represented as the set of projected feature points of the sub-region.
[0023] The division h Each interval can be determined by step size. step D Divide along the image depth direction h Equal intervals.
[0024] In the above technical solution, the method for calculating the error between the projection features and corresponding image features of each sub-region in the region to be calibrated in the multi-layer calibration module, and using the error to divide the calibrated region and the new region to be calibrated, is as follows: Calculate the pixel error between the projection features of each point cloud feature and the corresponding image feature in several sub-regions of the region to be calibrated, and calculate the root mean square error of all pixel errors in each sub-region based on the pixel error. Save the root mean square error of each sub-region in the region to be calibrated to the root mean square error set D of the region to be calibrated. ; Wherein, any root mean square error For the first u Line number v Column space The root mean square error of pixel errors between the projected features of all point cloud features and the corresponding image features in a sub-region of a spatial location; Set preset error The root mean square error of each sub-region within D is compared with the preset error. A one-to-one comparison is performed to determine the root mean square error of each sub-region within the calibration area. The sub-region is denoted as the region to be calibrated, and the root mean square error is... The sub-regions are denoted as the labeled regions.
[0025] This invention utilizes root mean square error to dynamically filter sub-regions, avoiding the averaging effect of global calibration, and accurately locates local nonlinear deviations caused by mechanical vibration or assembly deformation, making it particularly suitable for large field-of-view scenarios.
[0026] The above-mentioned acquisition of projection features, image features, point cloud features and projection feature sets corresponding to each sub-region of the region to be calibrated, and the division of the calibrated region and the new region to be calibrated, wherein the region to be calibrated is the initial region to be calibrated and the region to be calibrated that is gradually layered from the initial region to be calibrated.
[0027] In the above technical solution, the method for determining whether the pixel size and region length of the region to be calibrated meet the preset pixel size and region length is as follows: After comparing the root mean square error of each sub-region of the region to be calibrated with a preset error threshold, the following judgments are made on all the regions to be calibrated: Determine whether the region to be calibrated satisfies the condition that the pixel size is less than the pixel threshold. And the length of the region to be calibrated is less than the length threshold. ; If the requirements are not met, the calibration process will be repeated for the area to be calibrated. If the conditions are met, then all regions to be calibrated that meet the conditions are stored in the minimum set of regions to be calibrated. Inside.
[0028] If the area to be calibrated has been divided i The second time indicates that the region to be calibrated is the [number]th [number]. i The initial calibration region is the first layer. After dividing the initial calibration region into several sub-regions, the new calibration region obtained by utilizing the root mean square error of each sub-region is the second layer calibration region. Therefore, the initial calibration region is... i After the first partition, use the... i The layer to be calibrated region obtains the first i+1 The process for calibrating the layer's uncalibrated area is as follows: The first i All point cloud features and image features in the area to be calibrated are saved to a feature set. The feature set is used to calibrate the camera and LiDAR to obtain the extrinsic parameter matrix; the first... i Layer j The area to be calibrated is represented as Area to be calibrated The feature set of point cloud features and image features is Obtain the area to be calibrated The extrinsic parameter matrix is , Represented as: ; in, Indicates the area to be calibrated Image feature point set within, Indicates the area to be calibrated. The feature point set of the point cloud within; Based on the extrinsic parameter matrices of each region to be calibrated, the point cloud features within the region to be calibrated are transformed into pixels on the image, thus obtaining the projection features of the point cloud features in the region to be calibrated onto the image. Feature projection within Represented as: ; in, express Projected feature points, any point inside and and The image feature points and point cloud feature points correspond to each other; Using spatial subdivision to divide the first i The calibration region of each layer is divided into several sub-regions, and the feature sets of each sub-region within the calibration region are obtained; the calibration region The division obtained by the first u Line number v The first of the columns g Subregions at spatial locations are represented as The feature set of the corresponding sub-region is : ; in, Area to be calibrated Inner u Line number v Liede The feature set of a sub-region at a spatial location; , sub-region Image feature point set within, sub-region The feature point set of the point cloud within; sub-region The image feature point set and the projection point set corresponding to the point cloud feature point set; The division h Each interval can be determined by step size. step D Divide along the image depth direction h There are equidistant intervals, where the step size is... step D The value decreases dynamically as the number of layers increases.
[0029] By utilizing the pixel error between the image feature point set and the projection point set in the feature set of each sub-region, the root mean square error of each sub-region is obtained and saved to the root mean square error set of the region to be calibrated. In the middle, it won the first i The root mean square error set of all regions to be calibrated in the layer. Represented as: ; Wherein, any root mean square error Area to be calibrated Inner u Line number v Column space The projection feature point set of each region With the corresponding image feature point set The root mean square error of the pixel error between them; The root mean square error within the root mean square error set of all regions to be calibrated is compared with the preset error. Compare the root mean square error The subregion corresponding to the root mean square error is denoted as the first subregion. i+1 The calibration region of the layer, root mean square error The sub-region corresponding to the root mean square error is denoted as the calibrated region. Determine the obtained first i+1 Does the area to be calibrated in the layer satisfy the condition that the pixel size is less than the pixel threshold? And the length of the region to be calibrated is less than the length threshold. ; If not satisfied, the first i+1 The calibration process is repeated for the area to be calibrated. If satisfied, then all satisfying conditions will be... i+1 The region to be calibrated is stored in the smallest set of regions to be calibrated. Inside.
[0030] In the above technical solution, the external parameter compensation module includes: Set the smallest region to be calibrated The extrinsic parameters of the smallest region to be calibrated are compensated. The point cloud features and image features in the smallest region to be calibrated are calibrated using the calibration module to obtain the extrinsic parameter matrix of the smallest region to be calibrated. The point cloud features in the smallest region to be calibrated are converted into projection features on the image using the extrinsic parameter matrix of the smallest region to be calibrated. Construct the extrinsic compensation function for the smallest region to be calibrated: ; in, and , For points within the projected features, These are the image feature points corresponding to the points within the projection features; The parameters in the external parameter compensation function are obtained by using the multiple linear regression method. , , , , , By performing fitting, the extrinsic compensation function of the minimum region to be calibrated is obtained. ; Set the smallest region to be calibrated External parameter compensation is performed on all the smallest regions to be calibrated to obtain the smallest region to be calibrated. The set of extrinsic compensation functions.
[0031] Example 2: A multi-layer extrinsic parameter collaborative calibration method for multi-source sensing of near-electric vehicles includes the following steps: Step 1: Collect image features and point cloud features from the camera and lidar, which constitute the initial calibration area; Step 2: Calculate the extrinsic matrix of the initial region to be calibrated using the image features and point cloud features of the initial region to be calibrated; project the point cloud features onto the image plane using the extrinsic matrix of the initial region to be calibrated to obtain the projected features of the initial region to be calibrated; divide the initial region to be calibrated into several sub-regions; obtain the image features, point cloud features, and projected features corresponding to each sub-region of the initial region to be calibrated; calculate the error between the projected features of each sub-region and the corresponding image features; if the error does not exceed a preset error threshold, mark it as a calibrated region; otherwise, mark it as a region to be calibrated; determine whether the pixel size and region length of the region to be calibrated meet the preset pixel size and region length; save the regions to be calibrated that meet the preset pixel size and region length into the minimum set of regions to be calibrated.
[0032] Step 3: Using the coordinate deviations between each projection feature of the minimum uncalibrated region set and the corresponding image features, fit the corresponding extrinsic compensation function of the minimum uncalibrated region set to obtain the extrinsic compensation function set of the minimum uncalibrated region set.
[0033] In step 1, key information (image features and point cloud features) is extracted from the camera and LiDAR sensors, forming the initial calibration area. This provides the foundational data for the entire calibration process. By selecting feature points instead of the original massive dataset, the computational complexity of subsequent steps is significantly reduced, focusing on representative spatial locations.
[0034] In step 2, an initial extrinsic parameter matrix is calculated based on global features and projected to evaluate the initial accuracy. The region with excessive error is recursively subdivided into smaller sub-regions, and the local extrinsic parameter matrix is recalculated and the projection error is evaluated in each sub-region. This process iterates continuously, only delving into sub-regions with excessive error, until the sub-region size (pixels and physical) reaches a preset minimum threshold, ultimately outputting all the smallest calibrated regions.
[0035] It achieves precise positioning and efficient computation, avoiding the "averaging effect" of a single global calibration parameter on local distortions. By recursively optimizing only in areas with large errors, unnecessary computation is significantly reduced (skipping calibrated areas). Spatial subdivision and size thresholding ensure that the optimization is practically meaningful at both the physical and pixel levels, capturing subtle distortions while preventing overfitting (stopping when the threshold is reached), thus greatly improving the balance between calibration accuracy and efficiency.
[0036] In step 3, for the smallest uncalibrated regions identified in step 2 whose errors cannot be eliminated through further calibration, this step calculates the coordinate deviation between the projected feature points and the real image feature points within these regions. Using this localized deviation data, an independent extrinsic compensation function is fitted to each smallest uncalibrated region, ultimately resulting in a set of compensation functions. This provides high-precision local correction capabilities; by fitting extrinsic compensation functions to specific regions with residual nonlinear errors, it can effectively correct complex and non-uniform spatial deformations. This compensation set significantly improves the overall accuracy and robustness of the entire sensor fusion system in complex real-world environments and is a key step in achieving the final high-precision fusion.
[0037] Example 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.
[0038] Example 4: A computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.
[0039] The contents not described in detail in this specification are existing technologies known to those skilled in the art.
Claims
1. A multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles, characterized in that, include: The feature acquisition module is used to collect image features and point cloud features from the camera and lidar, and the image features and point cloud features constitute the initial calibration area; The multi-layer calibration module is used to calculate the extrinsic matrix of the initial region to be calibrated using image features and point cloud features in the initial region to be calibrated; the point cloud features are projected onto the image plane using the extrinsic matrix of the initial region to be calibrated to obtain the projection features of the initial region to be calibrated; the initial region to be calibrated is divided into several sub-regions; the image features, point cloud features, and projection features corresponding to each sub-region of the initial region to be calibrated are obtained; the error between the projection features of each sub-region and the corresponding image features is calculated; if the error does not exceed a preset error threshold, it is marked as a calibrated region; otherwise, it is recorded as a region to be calibrated; it is determined whether the pixel size and region length of the region to be calibrated meet the preset pixel size and region length; the regions to be calibrated that meet the preset pixel size and region length are saved in the minimum set of regions to be calibrated. The multi-layer calibration module is further configured to perform the following recalibration process when the pixel size and region length of the region to be calibrated do not meet the preset pixel size and region length: calculate the extrinsic parameter matrix of the region to be calibrated using image features and point cloud features within the region to be calibrated; project the point cloud features onto the image plane using the extrinsic parameter matrix of the region to be calibrated to obtain the projected features of the region to be calibrated; divide the region to be calibrated into several sub-regions; obtain the image features, point cloud features, and projected features corresponding to each sub-region of the region to be calibrated; and calculate the error between the projected features of each sub-region and the corresponding image features. If the error does not exceed the preset error threshold, it is marked as a calibrated area; otherwise, it is recorded as a new area to be calibrated. It is then determined whether the pixel size and region length of the new area to be calibrated meet the preset pixel size and region length. Areas to be calibrated that meet the preset pixel size and region length are saved in the minimum set of areas to be calibrated. If the pixel size and region length of the new area to be calibrated do not meet the preset pixel size and region length, the new area to be calibrated is used as an area to be calibrated and the above recalibration process is repeated until the pixel size and region length of all areas to be calibrated meet the preset pixel size and region length.
2. The multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles according to claim 1, characterized in that, The feature acquisition module includes: The feature targets are placed at different distances within the field of view of the camera and LiDAR, and their positions are recorded by the camera and LiDAR to obtain image features and point cloud features. : ； in, Any pair of feature points within , and For the corresponding image feature points and point cloud feature points, Indicates the first Group characteristics.
3. The multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles according to claim 2, characterized in that, In the multi-layer calibration module, the method for obtaining the projection features in the region to be calibrated is as follows: Save all point cloud features and image features in the region to be calibrated to a feature set. P In the middle, using feature sets P The camera and lidar are calibrated to obtain the extrinsic parameter matrix. P Represented as: ； in, This represents the set of image feature points within the region to be calibrated. This is represented as a set of feature points in the point cloud within the region to be calibrated; Based on the extrinsic parameter matrix, the point cloud features within the region to be calibrated are transformed into pixels on the image, thus obtaining the projection feature point set of the point cloud features in the region to be calibrated onto the image. , Represented as: ； in, express Projected feature points within, any point inside and and The points in the middle correspond; The calibration method uses the PnP algorithm.
4. The multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles according to claim 3, characterized in that, In the multi-layer calibration module, the method for dividing the region to be calibrated into several sub-regions and obtaining the image features, point cloud features, and projection features corresponding to each sub-region of the region to be calibrated is as follows: The corresponding point cloud features within the area to be calibrated are divided at equal intervals along the image depth direction. h Divide all image features into intervals. m OK n Columns, obtaining several sub-regions and the feature set within the sub-region is Represented as: ； in, For the first in the region u Line 1 v Liede A set of features of a sub-region at a spatial location; , This represents the set of image feature points within a sub-region. This is represented as a set of feature points in a point cloud within a sub-region; It is represented as the set of projected feature points of the sub-region.
5. The multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles according to claim 4, characterized in that, In the multi-layer calibration module, the method for calculating the error between the projection features and the corresponding image features of each sub-region, and using the error to divide the calibrated region and the new region to be calibrated, is as follows: Calculate the pixel error between the projection features of each point cloud feature and the corresponding image feature in several sub-regions of the region to be calibrated, and calculate the root mean square error of all pixel errors in each sub-region based on the pixel error. Save the root mean square error of each sub-region in the region to be calibrated to the root mean square error set D of the region to be calibrated. ； Wherein, any root mean square error For the first u Line number v Column space The root mean square error of pixel errors between the projected features of all point cloud features and the corresponding image features in a sub-region of a spatial location; Set preset error The root mean square error of each sub-region within D is compared with the preset error. A one-to-one comparison is performed to determine the root mean square error of each sub-region within the calibration area. The sub-region is denoted as the region to be calibrated, and the root mean square error is... The sub-regions are denoted as the labeled regions.
6. The multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles according to claim 5, characterized in that, In the multi-layer calibration module, the method for determining whether the pixel size and region length of the region to be calibrated meet the preset pixel size and region length is as follows: After comparing the root mean square error of each sub-region of the region to be calibrated with a preset error threshold, the following judgments are made on all the regions to be calibrated: Determine whether the region to be calibrated satisfies the condition that the pixel size is less than the pixel threshold. And the length of the region to be calibrated is less than the length threshold. ; If the requirements are not met, the calibration process will be repeated for the area to be calibrated. If the conditions are met, then all regions to be calibrated that meet the conditions are stored in the minimum set of regions to be calibrated. Inside.
7. The multi-layer extrinsic parameter collaborative calibration system for multi-source sensing of near-electric vehicles according to claim 6, characterized in that, It also includes an extrinsic parameter compensation module, which is used to fit the extrinsic parameter compensation function of the corresponding minimum calibrated region by utilizing the coordinate deviation between each projection feature of the minimum calibrated region set and the corresponding image feature, thereby obtaining the extrinsic parameter compensation function set of the minimum calibrated region set. The process of obtaining the extrinsic parameter compensation function set of the minimum calibrated region set is as follows: Set the smallest region to be calibrated The extrinsic parameters of the smallest uncalibrated region are compensated, and the point cloud features and image features in the smallest uncalibrated region are used to calibrate the camera and LiDAR to obtain the extrinsic parameter matrix of the smallest uncalibrated region. The point cloud features in the smallest uncalibrated region are converted into projection features on the image using the extrinsic parameter matrix of the smallest uncalibrated region. Constructing the extrinsic function for the smallest region to be calibrated: ； in, and , For points within the projected features, These are the image feature points corresponding to the points within the projection features; The parameters in the external parameter compensation function are obtained by using the multiple linear regression method. , , , , , By performing fitting, the extrinsic compensation function of the minimum region to be calibrated is obtained. ; Set the smallest region to be calibrated External parameter compensation is performed on all the smallest regions to be calibrated to obtain the smallest region to be calibrated. The set of extrinsic compensation functions.
8. A multi-layer extrinsic parameter collaborative calibration method for multi-source sensing of near-electric vehicles, characterized in that, Includes the following steps: Image features and point cloud features are collected from the camera and lidar, and the image features and point cloud features constitute the initial calibration area; The extrinsic parameter matrix of the initial region to be calibrated is calculated using image features and point cloud features in the initial region to be calibrated. The point cloud features are projected onto the image plane using the extrinsic parameter matrix of the initial region to be calibrated to obtain the projection features of the initial region to be calibrated. The initial region to be calibrated is divided into several sub-regions, and the image features, point cloud features, and projection features corresponding to each sub-region are obtained. The error between the projection features of each sub-region and the corresponding image features is calculated. If the error does not exceed a preset error threshold, it is marked as a calibrated region; otherwise, it is marked as a region to be calibrated. It is determined whether the pixel size and region length of the region to be calibrated meet the preset pixel size and region length. The regions to be calibrated that meet the preset pixel size and region length are saved in the minimum set of regions to be calibrated. When the pixel size and region length of the region to be calibrated do not meet the preset pixel size and region length, the following recalibration process is performed: The extrinsic parameter matrix of the region to be calibrated is calculated using the image features and point cloud features within the region to be calibrated; the point cloud features are projected onto the image plane using the extrinsic parameter matrix of the region to be calibrated to obtain the projected features of the region to be calibrated; the region to be calibrated is divided into several sub-regions, and the image features, point cloud features, and projected features corresponding to each sub-region are obtained; the error between the projected features of each sub-region and the corresponding image features is calculated; if the error does not exceed... If a region has a preset error threshold, it is marked as a calibrated region; otherwise, it is marked as a new region to be calibrated. The pixel size and region length of the new region to be calibrated are then determined to meet the preset pixel size and region length. Regions that meet the preset pixel size and region length are saved in the minimum set of regions to be calibrated. If the pixel size and region length of the new region to be calibrated do not meet the preset pixel size and region length, the new region to be calibrated is used as a region to be calibrated again, and the above recalibration process is repeated until the pixel size and region length of all regions to be calibrated meet the preset pixel size and region length.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 8.