External parameter calibration method and device, and electronic device

Abstract

The application provides an external parameter calibration method and device and electronic equipment, and relates to the technical field of autonomous driving. The method comprises the following steps: acquiring multiple frames of original point clouds and calibration images; the original point clouds are point clouds of a calibration target collected by a solid-state laser radar; the relative position between the solid-state laser radar and the calibration target is fixed; the original point clouds are time-synchronized; the calibration images are images of the calibration target collected by a camera device; the original point clouds are preprocessed by point cloud fusion to generate time-domain integrated point clouds; the time-domain integrated point clouds are optimized by geometric constraints to generate target point clouds with uniform point cloud density; three-dimensional corner points of the target point clouds and two-dimensional corner points of the calibration images are determined, and the three-dimensional corner points and the two-dimensional corner points are paired to generate multiple groups of paired corner points; and target external parameters between the solid-state laser radar and the camera device are generated according to all the paired corner points. The method can improve the external parameter calibration accuracy.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, specifically relating to an external parameter calibration method, device, and electronic device. Background Technology

[0002] Unmanned mining trucks, as a typical intelligent transportation tool, are widely used in heavy industries such as mines and open-pit coal mines, mainly for transporting minerals. During transportation, the vehicles rely on the combined perception of solid-state lidar and onboard cameras to achieve transportation route planning, obstacle recognition along the way (such as scattered coal blocks and temporary construction equipment), and precise stopping at loading and unloading points.

[0003] Due to the complex environment of mining areas, including high dust levels, strong sunlight, and mechanical vibrations from vehicles, solid-state lidar is prone to point cloud sparsity due to non-repeating scanning, and its extrinsic parameters tend to shift after long-term use. When performing joint calibration of solid-state lidar and cameras, the point cloud sparsity issue caused by non-repeating scanning of the solid-state lidar affects calibration accuracy and requires further optimization. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide an external parameter calibration method, apparatus and electronic device to address the above-mentioned deficiencies in the prior art. Using the external parameter calibration method, the point cloud density can be increased and the point cloud distribution can be made more uniform, thereby improving the external parameter calibration accuracy.

[0005] In a first aspect, embodiments of this application provide an external parameter calibration method, including: Acquire multiple frames of raw point clouds and calibration images; the raw point clouds are point clouds of the calibration target acquired by the solid-state lidar; the relative positions between the solid-state lidar and the calibration target are fixed; the raw point clouds are synchronized in time; the calibration images are images of the calibration target acquired by the camera equipment. Perform point cloud fusion preprocessing on multiple frames of raw point clouds to generate temporal integrated point clouds; Geometric constraint optimization is performed on the temporal integrated point cloud to generate a target point cloud with uniform point cloud density; The three-dimensional corner points of the target point cloud and the two-dimensional corner points of the calibration image are determined, and the three-dimensional corner points and the two-dimensional corner points are paired to generate multiple sets of paired corner points; Target extrinsic parameters between the solid-state lidar and the camera device are generated based on all paired corner points.

[0006] In some embodiments of the first aspect, point cloud fusion preprocessing is performed on multiple frames of original point clouds to generate a temporally integrated point cloud, including: Denoise the original point clouds to generate corresponding denoised point clouds; Using the first frame of denoised point cloud as a reference, the denoised point clouds of all other frames are calibrated to the same coordinate system through rigid body transformation and superimposed to form a preliminary dense point cloud. Voxel grid downsampling is performed on the initial dense point cloud to generate a temporal integrated point cloud.

[0007] In some embodiments of the first aspect, the calibration target is a checkerboard target; Geometric constraint optimization is performed on the temporal integrated point cloud to generate a target point cloud with uniform point cloud density, including: Based on the actual size of the checkerboard target, remove point clouds unrelated to the checkerboard target from the temporal integrated point cloud to generate the corresponding checkerboard point cloud; Iterative plane fitting is performed on the chessboard point cloud to generate the corresponding target fitting plane; The checkerboard point cloud is projected onto the target fitting plane, and the projected checkerboard point cloud is randomly downsampled to generate a target point cloud with uniform point cloud density.

[0008] In some embodiments of the first aspect, the checkerboard target is a black and white checkerboard target; Determine the 3D corner points of the target point cloud, including: The difference in reflectance spatial distribution and geometric position between the target point cloud and the preset standard checkerboard model is calculated using a predefined reflectance similarity function; the preset standard checkerboard model is constructed based on the checkerboard parameters of the black and white checkerboard target; the checkerboard parameters include at least the binary distribution of black and white checkerboard reflectance; With the goal of minimizing the reflectivity similarity function, the optimal rigid body transformation parameters from the target point cloud to the preset standard checkerboard model are solved by a nonlinear optimization algorithm. Based on the optimal rigid body transformation parameters, the coordinates of the corner points in the preset standard checkerboard model are transformed to the coordinate system of the solid-state lidar to obtain the three-dimensional corner points.

[0009] In some embodiments of the first aspect, determining the two-dimensional corner points of the calibration image includes: The calibration image is subjected to checkerboard corner detection to identify the two-dimensional corner points at the intersection of black and white checkerboard squares in the calibration image.

[0010] In some embodiments of the first aspect, three-dimensional corner points and two-dimensional corner points are paired to generate multiple sets of paired corner points, including: Starting from a corner of a black and white checkerboard target, the two-dimensional corner points are reordered according to a preset sorting rule to generate sorted two-dimensional corner points. The sorted 2D corner points are paired with their corresponding 3D corner points to generate multiple sets of paired corner points.

[0011] In some embodiments of the first aspect, generating target extrinsic parameters between the solid-state lidar and the camera device based on all paired corner points includes: Based on all paired corner points, the initial extrinsic parameters are solved using the random sampling consensus perspective n-point algorithm, and abnormal corner points with reprojection errors exceeding the first threshold are removed. Using the set of valid corner points retained after removing abnormal corner points as input, iteratively perform perspective n-point calculation and reprojection error evaluation, and update extrinsic parameters; The iteration stops when the reprojection error is lower than the second threshold, and the extrinsic parameters at this point are output as the target extrinsic parameters.

[0012] In some embodiments of the first aspect, after generating target extrinsic parameters between the solid-state lidar and the camera device based on all paired corner points, the method further includes: The target extrinsic parameters are used to reproject all three-dimensional corner points in the solid-state lidar coordinate system onto the calibration image plane, and the average reprojection error between each three-dimensional corner point and its corresponding two-dimensional corner point is calculated. If the average reprojection error is less than or equal to the preset verification threshold, the target extrinsic parameters are determined to be valid, and the calibration is completed.

[0013] Based on the same inventive concept, in a second aspect, embodiments of this application also provide an external parameter calibration device, comprising: The acquisition module is used to acquire multiple frames of raw point clouds and calibration images; the raw point clouds are the point clouds of the calibration target acquired by the solid-state lidar; the relative positions between the solid-state lidar and the calibration target are fixed; the time is synchronized between each raw point cloud; the calibration images are the images of the calibration target acquired by the camera equipment. The preprocessing module is used to perform point cloud fusion preprocessing on multiple frames of raw point clouds to generate a temporal integrated point cloud; The optimization module is used to perform geometric constraint optimization on the temporal integrated point cloud to generate a target point cloud with uniform point cloud density. The pairing module is used to determine the 3D corner points of the target point cloud and the 2D corner points of the calibration image, and to pair the 3D corner points and 2D corner points to generate multiple sets of paired corner points; The generation module is used to generate target extrinsic parameters between the solid-state lidar and the camera device based on all paired corner points.

[0014] In some embodiments of the second aspect, the preprocessing module is specifically used for: Each original point cloud is denoised to generate a corresponding denoised point cloud. Using the first frame of denoised point cloud as a reference, all other frames of denoised point clouds are calibrated to the same coordinate system through rigid body transformation and superimposed to form a preliminary dense point cloud. The preliminary dense point cloud is then downsampled using a voxel grid to generate a temporal integrated point cloud.

[0015] In some embodiments of the second aspect, the calibration target is a checkerboard target; The optimization module is specifically used for: Based on the actual size of the checkerboard target, remove point clouds unrelated to the checkerboard target from the temporal integrated point cloud to generate the corresponding checkerboard point cloud; perform iterative plane fitting on the checkerboard point cloud to generate the corresponding target fitting plane; project the checkerboard point cloud onto the target fitting plane, and perform grid random downsampling on the projected checkerboard point cloud to generate a target point cloud with uniform point cloud density.

[0016] In some embodiments of the second aspect, the checkerboard target is a black and white checkerboard target; The pairing module is specifically used to determine the 3D corner points of the target point cloud. The difference in reflectance spatial distribution and geometric position between the target point cloud and the preset standard checkerboard model is calculated using a predefined reflectance similarity function. The preset standard checkerboard model is constructed based on the checkerboard parameters of the black and white checkerboard target. The checkerboard parameters include at least the binary distribution of reflectance of black and white squares. With the goal of minimizing the reflectance similarity function, the optimal rigid body transformation parameters from the target point cloud to the preset standard checkerboard model are solved by a nonlinear optimization algorithm. Based on the optimal rigid body transformation parameters, the coordinates of the corner points in the preset standard checkerboard model are transformed to the coordinate system of the solid-state lidar to obtain the three-dimensional corner points.

[0017] In some embodiments of the second aspect, the pairing module, when determining the two-dimensional corner points of the calibration image, is specifically used for: The calibration image is subjected to checkerboard corner detection to identify the two-dimensional corner points at the intersection of black and white checkerboard squares in the calibration image.

[0018] In some embodiments of the second aspect, when the pairing module pairs three-dimensional corner points and two-dimensional corner points to generate multiple sets of paired corner points, it is specifically used for: Starting from a corner of a black and white checkerboard target, the two-dimensional corner points are reordered according to a preset sorting rule to generate sorted two-dimensional corner points; the sorted two-dimensional corner points are then paired with their corresponding three-dimensional corner points to generate multiple sets of paired corner points.

[0019] In some embodiments of the second aspect, the generation module is specifically used for: Based on all paired corner points, the initial extrinsic parameters are solved using the random sampling consensus perspective n-point algorithm, and abnormal corner points whose reprojection errors exceed the first threshold are removed. The set of valid corner points remaining after removing abnormal corner points is used as input, and the perspective n-point calculation and reprojection error evaluation are performed iteratively, and the extrinsic parameters are updated. When the reprojection error is lower than the second threshold, the iteration stops, and the extrinsic parameters at this time are output as the target extrinsic parameters.

[0020] In some embodiments of the second aspect, the apparatus further includes: The verification module is used to reproject all three-dimensional corner points in the solid-state lidar coordinate system onto the calibration image plane using the target extrinsic parameters, and calculate the average reprojection error between each three-dimensional corner point and its corresponding two-dimensional corner point; if the average reprojection error is less than or equal to the preset verification threshold, the target extrinsic parameters are determined to be valid, and the calibration is completed.

[0021] Based on the same inventive concept, in a third aspect, embodiments of this application also provide an electronic device, including: a memory and a processor; The memory stores the instructions that the computer executes; The processor executes computer-executable instructions stored in memory to implement the extrinsic calibration method as described in any of the first aspects.

[0022] Based on the same inventive concept, in a fourth aspect, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the external parameter calibration method as described in any of the first aspects.

[0023] According to the extrinsic parameter calibration method, apparatus, and electronic device provided in the embodiments of this application, a temporal integrated point cloud is generated by performing point cloud fusion preprocessing on multiple frames of original point clouds. Furthermore, geometric constraint optimization is applied to the temporal integrated point cloud to generate a target point cloud with uniform point cloud density, thus solving the problem of sparse point clouds easily occurring in solid-state lidar due to non-repeating scanning. Simultaneously, the target extrinsic parameters between the solid-state lidar and the camera device are determined based on this uniformly dense target point cloud and the calibration image, which can improve the accuracy of extrinsic parameter calibration. Attached Figure Description

[0024] Figure 1 This illustration shows a flowchart of an external parameter calibration method provided in an embodiment of this application. Figure 2 This illustration shows another flowchart of the external parameter calibration method provided in the embodiments of this application; Figure 3 This diagram illustrates the overall flow of the external parameter calibration method provided in the embodiments of this application. Figure 4 This illustration shows a checkerboard target provided in an embodiment of this application. Figure 5 This illustration shows a flowchart of the data acquisition and preprocessing process provided in an embodiment of this application. Figure 6 This illustration shows a flowchart of corner point extraction from point clouds and images provided in an embodiment of this application; Figure 7 This diagram illustrates the verification of calibration results provided in an embodiment of this application. Figure 8The experimental results provided in the embodiments of this application are shown in the figure. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of this application, the application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0026] The features and exemplary embodiments of various aspects of this application will now be described in detail. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain this application and are not configured to limit this application. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application.

[0027] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0028] As described in the background section, due to the complex environment of mining areas, including high dust levels, strong sunlight, and mechanical vibrations from vehicles, solid-state lidar is prone to point cloud sparsity due to non-repeating scanning, and its extrinsic parameters tend to shift after long-term use. When performing joint calibration of solid-state lidar and cameras, the point cloud sparsity issue caused by non-repeating scanning of the solid-state lidar affects calibration accuracy and requires further optimization.

[0029] Example 1

[0030] The extrinsic parameter calibration method provided in this application is applied to an electronic device, which can be a computer or a device within a computer for implementing the extrinsic parameter calibration method. The computer can be a server, a server cluster, or a terminal; this application does not specifically limit this. Furthermore, the following description uses the example of the extrinsic parameter calibration method being executed by an electronic device.

[0031] like Figure 1 As shown, the external parameter calibration method provided in this application embodiment may include steps S101 to S105.

[0032] S101. Acquire multiple frames of raw point clouds and calibration images. The raw point clouds are point clouds of the calibration target acquired by the solid-state lidar. The relative positions between the solid-state lidar and the calibration target are fixed. The raw point clouds are synchronized in time. The calibration images are images of the calibration target acquired by the camera equipment.

[0033] For example, the number of frames acquired for the original point cloud can be selected according to the actual application requirements, and this embodiment does not limit this.

[0034] The fixed relative position between the solid-state lidar and the calibration target, as well as the time synchronization between the original point clouds, are prerequisites for point cloud fusion preprocessing (point cloud fusion overlay) to generate a target point cloud with uniform point cloud density distribution.

[0035] The video recording equipment can be a camera, webcam, etc.

[0036] S102. Perform point cloud fusion preprocessing on multiple frames of original point clouds to generate temporal integrated point clouds.

[0037] For example, point cloud fusion preprocessing includes denoising, point cloud overlay, downsampling, and other processes related to point cloud fusion and preprocessing. Temporal integrated point cloud is a point cloud generated by temporally overlaying and integrating multiple frames of original point clouds. Compared with the original point cloud, the temporal integrated point cloud has a higher density.

[0038] The purpose of point cloud fusion preprocessing is to improve density and accuracy by integrating point cloud data from multiple time periods, providing high-quality input for subsequent corner point extraction.

[0039] S103. Perform geometric constraint optimization on the temporal integrated point cloud to generate a target point cloud with uniform point cloud density distribution.

[0040] For example, geometric constraint optimization includes processes such as plane fitting, projection, and downsampling. This results in a target point cloud with a uniform point cloud density, providing a foundation for subsequently determining higher-precision extrinsic parameters.

[0041] S104. Determine the three-dimensional corner points of the target point cloud and the two-dimensional corner points of the calibration image, and pair the three-dimensional corner points and the two-dimensional corner points to generate multiple sets of paired corner points.

[0042] For example, 3D and 2D corner points are used for pairing and subsequent determination of extrinsic parameters. The 3D corner points of the target point cloud can be determined using currently common methods, or by employing a reflectivity similarity function and a standard checkerboard model.

[0043] S105. Generate target extrinsic parameters between the solid-state lidar and the camera device based on all paired corner points.

[0044] For example, the extrinsic parameters can be solved using the Random Sample Consensus-Perspective-n-Point (RANSAC-PnP) algorithm, which improves the efficiency of extrinsic parameter solving.

[0045] According to the extrinsic parameter calibration method provided in this application, a temporal integrated point cloud is generated by performing point cloud fusion preprocessing on multiple frames of original point clouds. Furthermore, geometric constraint optimization is applied to the temporal integrated point cloud to generate a target point cloud with uniform point cloud density, thus solving the problem of sparse point clouds easily occurring in solid-state lidar due to non-repeating scanning. Simultaneously, the target extrinsic parameters between the solid-state lidar and the camera device are determined based on this uniformly dense target point cloud and the calibration image, which can improve the accuracy of extrinsic parameter calibration.

[0046] Example 2

[0047] like Figure 2 As shown, the external parameter calibration method provided in this application embodiment is based on the external parameter calibration method provided in embodiment 1 of this application, and further describes the method, which may include steps S201 to S207.

[0048] S201. Acquire multiple frames of original point cloud and calibration image.

[0049] S202. Denoise each original point cloud and generate a corresponding denoised point cloud.

[0050] For example, statistical filtering is performed on each original point cloud individually to remove outliers and filter out points that resemble checkerboard patterns, thereby generating a denoised point cloud.

[0051] S203. Using the first frame of denoised point cloud as a reference, calibrate the denoised point clouds of all other frames to the same coordinate system through rigid body transformation, and superimpose them to form a preliminary dense point cloud.

[0052] For example, calibrating all other frame denoised point clouds to the same coordinate system through rigid body transformation is to provide a coordinate system basis for superimposing and forming an initial dense point cloud.

[0053] S204. Perform voxel grid downsampling on the initial dense point cloud to generate a temporal integrated point cloud.

[0054] For example, the initial dense point cloud is downsampled using a voxel grid to simplify redundant points (the space is divided into voxels, and each voxel retains one representative point), thereby obtaining a temporal integrated point cloud with uniform density that can clearly present the details of the checkerboard pattern.

[0055] S205. Perform geometric constraint optimization on the temporal integrated point cloud to generate a target point cloud with uniform point cloud density distribution.

[0056] In some implementations, the calibration target is a checkerboard target.

[0057] S205 can be specifically described as follows: Based on the actual size of the checkerboard target, remove point clouds unrelated to the checkerboard target from the temporal integrated point cloud to generate the corresponding checkerboard point cloud.

[0058] Iterative plane fitting is performed on the chessboard point cloud to generate the corresponding target fitting plane.

[0059] The checkerboard point cloud is projected onto the target fitting plane, and the projected checkerboard point cloud is randomly downsampled to generate a target point cloud with uniform point cloud density.

[0060] For example, S205 is an optimization step that addresses potential issues such as interference points, planar deviations, and uneven density in point clouds that may exist in non-repeated scanning. It not only extracts the calibration target point cloud cluster through technical means but also lays the foundation for subsequent accurate positioning of the calibration target's 3D (three-dimensional) corner points.

[0061] To remove point clouds unrelated to the checkerboard target from the temporal integrated point cloud, first, the Z-axis coordinate range is set according to the actual size of the checkerboard, and interference points unrelated to the target, such as the ground and supports, are eliminated, leaving a pure checkerboard point cloud.

[0062] For example, plane fitting can employ the RANSAC (Random Sample Consensus) algorithm. The RANSAC algorithm is used to progressively reduce the plane fitting threshold, iterating repeatedly to make the checkerboard point cloud as close as possible to the ideal checkerboard plane, thereby correcting the radial deviation caused by solid-state lidar ranging.

[0063] The fitted point cloud is projected onto the target fitting plane (at this point, the target fitting plane may already be close enough to the ideal plane) to further eliminate spatial position errors.

[0064] The standard RANSAC algorithm presets a fixed distance threshold when fitting a plane. All points whose distance to the candidate plane is less than this threshold are considered "interior points" of that plane.

[0065] Finally, random downsampling through the grid is used to ensure a uniform distribution of point cloud density, avoiding interference from high-density areas in subsequent feature extraction. This results in a target point cloud with clear outlines and accurate positioning, providing high-quality feature input for subsequent 3D corner estimation.

[0066] S206. Determine the three-dimensional corner points of the target point cloud and the two-dimensional corner points of the calibration image, and pair the three-dimensional corner points and the two-dimensional corner points to generate multiple sets of paired corner points.

[0067] In some implementations, the checkerboard target is a black and white checkerboard target.

[0068] In S206, the 3D corner points of the target point cloud can be determined as follows: A predefined reflectance similarity function is used to calculate the differences in reflectance spatial distribution and geometric location between the target point cloud and a preset standard checkerboard model. The preset standard checkerboard model is constructed based on the checkerboard parameters of a black and white checkerboard target. The checkerboard parameters include at least a binary distribution of reflectance for black and white squares.

[0069] With the goal of minimizing the reflectivity similarity function, the optimal rigid body transformation parameters from the target point cloud to the preset standard checkerboard model are solved by a nonlinear optimization algorithm.

[0070] Based on the optimal rigid body transformation parameters, the coordinates of the corner points in the preset standard checkerboard model are transformed to the coordinate system of the solid-state lidar to obtain the three-dimensional corner points.

[0071] For example, the core idea of ​​determining 3D corner points is based on matching the point cloud reflectance intensity distribution with a standard model, and then using nonlinear optimization to accurately estimate the corner point positions. This process takes the target point cloud as input, and the specific steps are as follows: First, based on the known checkerboard parameters (number of rows, number of columns, grid spacing), a standard checkerboard model containing a binary distribution of black and white grid reflectance (black grids have low reflectance, white grids have high reflectance) is constructed to provide a precise reference for subsequent matching.

[0072] Meanwhile, a reflectance similarity function is defined to comprehensively calculate the differences between the target point cloud and the standard checkerboard model in terms of reflectance spatial distribution and geometric position.

[0073] Subsequently, a nonlinear optimization algorithm (such as L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm)) is used to minimize the reflectivity similarity function in order to solve for the optimal rigid body transformation parameters (rotation matrix and translation vector) from the target point cloud to the standard checkerboard model, thereby achieving precise alignment between the two.

[0074] Finally, the pre-calculated corner coordinates in the standard checkerboard model are transformed to the solid-state lidar coordinate system through the inverse transformation of the aforementioned transformation parameters, resulting in 3D corner points that precisely correspond to the point cloud features, thus completing the 3D corner point determination process. This further overcomes the problem of uneven point cloud distribution in the non-repeating scanning mode of solid-state lidar, significantly improving the accuracy of joint calibration between lidar and camera equipment.

[0075] In some implementations, determining the two-dimensional corner points of the calibration image includes: The calibration image is subjected to checkerboard corner detection to identify the two-dimensional corner points at the intersection of black and white checkerboard squares in the calibration image.

[0076] For example, classic algorithms such as Harris corner detection or built-in functions of OpenCV (Open Source Computer Vision Library) can be used to identify two-dimensional corners at the boundaries of black and white checkerboard squares in an image.

[0077] In some implementations, the process of pairing three-dimensional corner points and two-dimensional corner points to generate multiple sets of paired corner points can be as follows: Starting from a corner of a black and white checkerboard target, the two-dimensional corner points are reordered according to a preset sorting rule to generate sorted two-dimensional corner points.

[0078] The sorted 2D corner points are paired with their corresponding 3D corner points to generate multiple sets of paired corner points.

[0079] For example, a corner could be the bottom left corner, the top right corner, etc. To address the issue of disordered corner point order caused by the diagonal symmetry of the checkerboard grid, the two-dimensional corner points are reordered starting from the bottom left corner of the checkerboard grid according to a preset sorting rule of "from left to right, from top to bottom". This ensures that the index order of the two-dimensional corner points is completely consistent with the three-dimensional corner points on the solid-state LiDAR side, ultimately forming a one-to-one pairing relationship between "image two-dimensional corner points - point cloud three-dimensional corner points".

[0080] S207. Generate target extrinsic parameters between the solid-state lidar and the camera device based on all paired corner points.

[0081] In some implementations, S207 may be specifically as follows: Based on all paired corner points, the initial extrinsic parameters are solved using the random sampling consensus perspective n-point algorithm, and abnormal corner points whose reprojection errors exceed the first threshold are removed.

[0082] Using the set of valid corner points retained after removing abnormal corner points as input, the system iteratively performs perspective n-point calculation and reprojection error evaluation, and updates the extrinsic parameters.

[0083] The iteration stops when the reprojection error is lower than the second threshold, and the extrinsic parameters at this point are output as the target extrinsic parameters.

[0084] For example, the purpose of this process is to solve and optimize the coordinate system transformation parameters (rotation matrix R and translation vector t) between the solid-state lidar and the camera device using the random sampling consistency perspective n-point algorithm, so as to achieve accurate correlation between the two sensors.

[0085] First, using paired corner points as input, the RANSAC-PnP algorithm (a combination of random sampling consensus and perspective n-point algorithm) is employed to randomly select a subset of corner points to construct perspective projection equations, solving for the initial extrinsic parameters. Simultaneously, outlier corner points (exterior points) with excessively large reprojection errors (i.e., exceeding the first threshold) are removed. Next, using the remaining set of valid corner points (interior points) as new input, PnP (Perspective-n-Point algorithm) is re-executed to update the extrinsic parameters and evaluate the reprojection error. This "calculation-verification-update" process is iterated repeatedly until the reprojection error is less than the second threshold. The final output is the target extrinsic parameters that allow the solid-state LiDAR's 3D corner points to be accurately mapped to the camera's 2D corner points.

[0086] In some implementations, a verification process is included after S207: The target extrinsic parameters are used to reproject all three-dimensional corner points in the solid-state lidar coordinate system onto the calibration image plane, and the average reprojection error between each three-dimensional corner point and its corresponding two-dimensional corner point is calculated.

[0087] If the average reprojection error is less than or equal to the preset verification threshold, the target extrinsic parameters are determined to be valid, and the calibration is completed.

[0088] For example, during the verification phase, all three-dimensional corner points of the solid-state lidar (including redundant corner points that were not previously included in the calculation) are reprojected onto the calibration image using the target extrinsic parameters. The overall average reprojection error is calculated, and the visual fit between the reprojected points and the two-dimensional corner points of the image is observed. If the error is within an acceptable range (usually ≤0.5 pixels) and the fit is high, the extrinsic parameters are verified to be effective, and the entire calibration process is completed.

[0089] Meanwhile, further optimization can be performed before the verification phase: In the optimization phase, the initial effective corner points are calculated based on the target extrinsic parameters, and continuous optimization is performed through a closed-loop process of "iterative recalculation of PnP - updating extrinsic parameters - evaluating error" - each time the 3D corner points are reprojected onto the calibration image using the current extrinsic parameters and the error is calculated. If the error does not reach the preset threshold, the extrinsic parameters are recalculated with the corner points with smaller errors until the error converges to the target range and the optimal extrinsic parameters are obtained.

[0090] It should be noted that each preset threshold (such as the first threshold, the second threshold, the preset verification threshold, etc.) in this embodiment can be set based on actual needs.

[0091] The extrinsic parameter calibration method in this embodiment has the following effects: 1. Improved calibration accuracy: By combining point cloud fusion preprocessing and geometric constraint optimization, noise in solid-state lidar point cloud data can be effectively eliminated, and the extraction process of three-dimensional corner points is optimized, overcoming the problem of uneven distribution of point clouds in the non-repeating scanning mode of solid-state lidar, thereby significantly improving the accuracy of joint calibration of lidar and camera.

[0092] 2. High level of automation: The calibration process is fully automated, reducing the need for manual intervention. Users only need to change the position of the calibration board, and the system can automatically perform data acquisition, corner point extraction, extrinsic parameter calculation and optimization, greatly improving calibration efficiency and reliability, and making it suitable for rapid calibration in complex environments.

[0093] 3. Enhanced robustness: Effectively addresses adverse factors such as lighting variations, dust pollution, and vibration interference that may occur in open-pit coal mines, autonomous driving, or other complex environments. Through feature refinement and noise removal, the calibration results maintain high stability under varying environments, ensuring rapid and accurate acquisition of external parameter data under complex operating conditions.

[0094] 4. High adaptability and wide applicability: Not only is it suitable for the joint calibration of solid-state LiDAR and camera systems, but it also has strong universality, and can be widely used in multiple fields such as autonomous driving, robot perception, and 3D mapping. Through flexible algorithm design, it can support different types of solid-state LiDAR and camera sensor combinations to meet the needs of different environments and applications.

[0095] 5. Optimized computational efficiency: Extrinsic parameters are accurately calculated using a nonlinear optimization algorithm, and the calibration results are further optimized by minimizing the reprojection error, ensuring high-precision extrinsic parameter data is obtained in a short time. Simultaneously, the optimized algorithm design reduces computational complexity and improves the execution efficiency of the calibration process, making it suitable for real-time or high-efficiency applications.

[0096] 6. Supports high-precision multi-sensor data fusion: This calibration method provides high-precision and stable extrinsic parameter data for multi-sensor data fusion, ensuring accurate registration of LiDAR and camera data. Through the method in this embodiment, efficient and accurate environmental perception and navigation can be achieved, improving the safety and reliability of intelligent systems such as autonomous driving and robotics.

[0097] To better understand the external parameter calibration method provided in the embodiments of this application, more specific embodiments will be used for illustration below.

[0098] This embodiment improves the ability to extract effective information from non-uniformly distributed point clouds through temporal domain integration and feature refinement (i.e., the content of S202 to S205), and uses reflection intensity distribution to estimate 3D (three-dimensional) corner points. Combined with 2D (two-dimensional) corner points in the image, the extrinsic parameter matrix is ​​solved using the RANSAC algorithm and the PnP method, achieving high-precision automated calibration.

[0099] A specific case is as follows: A certain open-pit coal mine has deployed multiple driverless mining trucks for raw coal transportation. The vehicles rely on the joint perception of solid-state LiDAR and onboard cameras to achieve transportation route planning, obstacle identification along the way (such as scattered coal blocks, temporary construction equipment), and precise parking at loading and unloading points (with a parking deviation of ≤10cm). Due to the complex environment of the mining area, which is characterized by high dust, strong light, and mechanical vibration from vehicle movement, solid-state LiDAR is prone to sparse point clouds due to non-repeating scanning. Furthermore, the external parameters of the equipment are prone to shift after long-term use, requiring at least one LiDAR-camera joint calibration per month.

[0100] Traditional calibration methods use ordinary white checkerboard targets, which have three major drawbacks: First, the targets are prone to dust accumulation and obstruction, requiring frequent manual wiping, with each cleaning and redeployment taking over 15 minutes. Second, the targets reflect light severely under strong light, resulting in a corner point recognition success rate of less than 60%, often leading to calibration interruptions. Third, the impact of vibration on solid-state LiDAR point clouds is not considered, resulting in a final registration error often exceeding 0.8 pixels, which cannot meet the perception requirements of autonomous driving, and a single calibration process takes over 2.5 hours, severely impacting mining truck operation time.

[0101] Therefore, the external parameter calibration method proposed in this embodiment solves the calibration problem in the complex environment of the mining area through "anti-interference target design + targeted algorithm optimization", ensuring that the joint calibration error is ≤0.4 pixels and the calibration time is controlled within 1 hour, which is suitable for the efficient operation rhythm of mining trucks.

[0102] like Figure 3 As shown, the overall process includes: Step S301: Data acquisition.

[0103] Step S302: Acquire multiple frames of SSL (Solid-State LiDAR) point cloud and corresponding camera images.

[0104] Step S303: SSL point cloud preprocessing.

[0105] Step S304: Single-frame statistical filtering - temporal stacking - segmentation and filtering of checkerboard point cloud.

[0106] Step S305: Point cloud feature optimization.

[0107] Step S306: Remove interference points - RANSAC plane fitting - projection downsampling.

[0108] Step S307: 3D corner estimation.

[0109] Step S308: Construct a standard chessboard model - L-BFGS optimization - inverse transformation to obtain 3D corner points C3D.

[0110] Step S309: 2D corner point extraction.

[0111] Step S310: Image corner detection - reordering to obtain C2D.

[0112] Step S311: External parameter calibration.

[0113] Step S312: RANSAC-Pnp calculates initial extrinsic parameters - iteratively optimizes errors - outputs final extrinsic parameters.

[0114] The specific case process is as follows: Equipment and target deployment

[0115] Hardware selection: The unmanned mining truck is equipped with a solid-state LiDAR (solid-state LiDAR, ranging range 0.3-450m, dust resistance rating IP67) and an 8-megapixel vehicle-mounted camera (focal length 16mm, with multi-layer anti-glare coating, frame rate 30fps). Both are fixed to the front of the cab top with a special bracket with shock-absorbing pads (20cm spacing to ensure no relative displacement), with the lens facing the vehicle's forward driving path.

[0116] Target optimization: employing methods such as Figure 4 The 1000mm×700mm anti-reflective and dustproof checkerboard target has black and white grids measuring 100mm×100mm. The grid surface is coated with a matte silicon carbide coating (reflectivity ≤15%). A detachable dust cover (dustproof rating IP65) is installed on the edge of the target. After the mining truck stops, the calibration plate is placed 12m away from the target (avoiding the risk of near-distance distortion and far-distance dust obstruction). This completes the deployment of the equipment and the calibration plate.

[0117] Data collection: After deploying the equipment and targets, data collection began. Specifically, the calibration operation was carried out in the mining area between 6:30 and 7:30 in the morning (when dust concentration is low and the light is soft). The mining truck was precisely parked 12m in front of the target, and the solid-state lidar and camera were simultaneously turned on through the vehicle's CAN (Controller Area Network) bus.

[0118] The system collects 30 seconds of data (600 point cloud frames from the solid-state lidar at a frame rate of 20Hz, and 900 images from the camera at a frame rate of 30fps). The system automatically selects 400 point cloud frames and 600 images of the target that are unobstructed and non-reflective for subsequent processing.

[0119] Point cloud feature refinement (excluding the effects of dust accumulation and planar deviation): After data collection, the collected data undergoes filtering and feature extraction. For example... Figure 5 As shown, data acquisition and preprocessing can be specifically performed as follows: Step S401: Data Acquisition.

[0120] Step S402: Fix SSL+ camera, camera intrinsic parameters are known.

[0121] Step S403: Arrange 3-5 target positions on a checkerboard pattern and simultaneously collect: multi-frame SSL point cloud + camera images.

[0122] Step S404: SSL point cloud preprocessing.

[0123] Step S405: Single-frame statistical filtering - temporal stacking - segmentation and filtering to obtain a chessboard point cloud.

[0124] Step S406: Point cloud feature optimization.

[0125] Step S407: Remove interference points - RANSAC plane fitting - radial projection - network downsampling.

[0126] Step S408: Output: Optimized point cloud (i.e. the aforementioned target point cloud) + synchronized camera image (for corner point extraction).

[0127] The process begins by setting the Z-axis coordinate range based on the actual dimensions of the checkerboard pattern, eliminating interference points unrelated to the target, such as those on the ground and supports, and retaining a clean checkerboard point cloud. Next, the RANSAC algorithm is used to progressively reduce the plane fitting threshold, iterating repeatedly to make the point cloud as close as possible to the ideal checkerboard plane, correcting the radial deviation caused by solid-state LiDAR ranging. Then, the fitted point cloud is projected onto the ideal plane to further eliminate spatial position errors. Finally, random downsampling of the grid ensures a uniform distribution of point cloud density, avoiding interference from high-density areas in subsequent feature extraction, ultimately resulting in an optimized point cloud with clear contours and accurate positioning, providing high-quality feature input for subsequent 3D corner estimation.

[0128] Interference point removal: Based on the actual size of the target (1000mm×700mm), set the Z-axis coordinate range (base height ±100mm), automatically identify and remove the point cloud corresponding to a small amount of dust accumulation on the target surface (dust accumulation points ≤3%), and retain the pure checkerboard point cloud.

[0129] Plane fitting correction: The RANSAC algorithm (300 iterations, distance threshold 1.2mm) is used to gradually reduce the plane fitting error and correct the radial ranging deviation caused by strong light reflection of solid-state lidar (deviation correction amount ≤0.8mm), so that the point cloud is close to the ideal checkerboard plane.

[0130] Point cloud projection and downsampling: The fitted point cloud is projected onto an ideal plane to eliminate spatial position errors. Then, random downsampling through a grid (grid size 2mm) ensures uniform point cloud density (5 points per square centimeter), providing high-quality input for 3D corner estimation.

[0131] 3D Corner Estimation (Accurate Extraction of Corner Points in Solid-State LiDAR Coordinate System)

[0132] After refining the point cloud features, the next step is to estimate the 3D corner points of the data. First, based on the known checkerboard parameters (number of rows, columns, and grid spacing), a standard checkerboard model is constructed, containing a binary distribution of reflectivity (low reflectivity for black squares, high reflectivity for white squares), providing a precise reference for subsequent matching. Next, a reflectivity similarity function is defined, comprehensively calculating the differences between the point cloud and the standard model in reflectivity spatial distribution and geometric position, constructing an optimization objective function. Then, the L-BFGS nonlinear optimization algorithm is used to minimize this similarity function to solve for the optimal rigid body transformation parameters (rotation matrix and translation vector) from the point cloud to the standard model, achieving precise alignment between the two. Finally, the pre-calculated precise corner point coordinates in the standard model are transformed to the solid-state LiDAR coordinate system through the inverse transformation of the above parameters, obtaining the 3D corner points that precisely correspond to the point cloud features, completing the estimation process.

[0133] Standard model construction: Based on the anti-reflective checkerboard grid parameters (13×13 grid, 80mm size), a binary distribution standard model of "black grid low reflection, white grid high reflection" is constructed, and the accuracy of the model corner coordinates is controlled within ±0.1mm.

[0134] Similarity function calculation: Define a reflectance similarity function, compare the reflectance distribution and geometric position differences between the optimized point cloud and the standard model, and weaken the local reflection fluctuations caused by strong light through weighted calculation (the fluctuation impact is reduced to within 5%).

[0135] Nonlinear optimization and corner transformation: The similarity function is minimized using the L-BFGS algorithm (iteration step size 0.005, convergence threshold 1e-7) to solve for the rigid body transformation parameters (rotation matrix R, translation vector t) from the point cloud to the standard model. Then, the coordinates of multiple precise corner points of the standard model are transformed to the solid-state LiDAR coordinate system through inverse transformation to obtain a 3D corner point set (corner point coordinate error ≤ 0.3mm).

[0136] 2D Corner Extraction and Matching (Solving Image Reflections and Disordered Layout)

[0137] like Figure 6 As shown, the process for optimizing corner-to-corner pairing can be as follows: Step S501: Input data.

[0138] Step S502: SSL Optimization of Point Cloud .

[0139] Step S503: 3D corner point extraction (SSL side).

[0140] Step S504: Construct a standard checkerboard model S containing reflectivity - C.

[0141] Step S505: Define the reflectance similarity function L and evaluate... With S - C-position posture difference.

[0142] Step S506: L-BFGS optimization solution for transformation parameters (R, t).

[0143] Step S507: Standard Corner C - std inverter, obtain SSL 3D corner point C3D.

[0144] Step S508: Output: 3D corner point C3D (SSL coordinate system)

[0145] Step S509: Synchronize camera images.

[0146] Step S510: 2D corner point extraction (camera side).

[0147] Step S511: Chessboard corner detection algorithm, initially extracting 2D corners.

[0148] Step S512: Reorder from the bottom left index to eliminate symmetry ambiguity and obtain C2D.

[0149] Step S513: Output: 2D corner point C2D (camera image coordinate system).

[0150] Step S514: For subsequent external parameter calibration: C3D and C2D correspond one-to-one.

[0151] First, checkerboard corner detection is performed on the synchronously acquired camera images. Using classic algorithms such as Harris corner detection or OpenCV's built-in functions, the initial 2D corners at the boundaries of the black and white checkerboard squares in the image are identified. Next, to address the issue of corner point order disorder caused by the diagonal symmetry of the checkerboard, the initial corners are reordered starting from the bottom left corner of the checkerboard, following the rule of "from left to right, from top to bottom." This ensures that the index order of the 2D corners is completely consistent with the 3D corners on the solid-state LiDAR side, ultimately forming a one-to-one pairing relationship between "image 2D corners - point cloud 3D corners." The specific steps are as follows: Image preprocessing and corner detection: For 600 frames of camera images, grayscale conversion and Gaussian filtering (kernel size 7×7) are first performed to suppress strong light reflection noise. Then, an improved Harris corner detection algorithm (window size 5×5, response threshold 0.02) is used to extract multiple initial 2D corners from each frame, improving the corner recognition success rate to 98%.

[0152] Corner Reordering: Starting from the bottom left corner of the chessboard, the initial 2D corners are reordered according to the rule of "from left to right and from top to bottom", automatically correcting the disorder of local corner order caused by reflection (100% accuracy of sorting correction).

[0153] Corner point pairing: Through precise matching of timestamps, "solid-state LiDAR 3D corner points - camera 2D corner points" at the same moment are matched one-to-one to form multiple pairs of paired corner points. The pairing error is ≤0.5 pixels, providing reliable input for external parameter calculation.

[0154] Extrinsic parameter calculation (solving for solid-state lidar-camera coordinate system transformation parameters)

[0155] First, using paired corner points as input, the RANSAC-PnP algorithm (random sampling consistency combined with perspective n-point algorithm) is employed to randomly select a subset of corner points to construct perspective projection equations, solving for the initial extrinsic parameters. Simultaneously, outlier corner points (exterior points) with excessively large reprojection errors are removed. Next, using the remaining valid corner points (interior points) as new input, PnP calculation is re-executed to update the extrinsic parameters and evaluate the reprojection error. This "calculation-verification-update" process is iterated repeatedly until the reprojection error is less than a preset threshold. The final output is the optimal extrinsic parameters that allow the solid-state LiDAR 3D corner points to be accurately mapped to the camera's 2D corner points. The steps are as follows: Initial extrinsic parameter solution: The RANSAC-PnP algorithm (800 sampling times, reprojection error threshold of 2 pixels) is used to randomly select 120 pairs of paired corner points to construct perspective projection equations and solve the initial extrinsic parameters (rotation matrix R, translation vector t). At the same time, 2 sets of abnormal corner points caused by reflection residue are removed (abnormal point removal rate of 100%).

[0156] Iterative optimization of extrinsic parameters: Using the remaining multiple sets of valid corner points as input, repeatedly execute the process of "recalculate PnP - update extrinsic parameters - evaluate error" (150 iterations) until the reprojection error stabilizes at 0.35 pixels (≤ preset threshold 0.4 pixels) to obtain the optimal extrinsic parameters.

[0157] Optimization and verification (ensuring calibration results are suitable for mining operations)

[0158] After the above series of steps, the external parameter calibration of the LiDAR and camera was successfully completed. Figure 7 , Figure 8 The images show the point cloud reprojection onto the image surface and the point cloud shading effect, respectively. The final verification step is as follows: Error verification: Using the optimal extrinsic parameters, all solid-state LiDAR 3D corner points were reprojected onto the camera image. The average reprojection error of 600 frames was 0.35 pixels. The visual fit between the reprojected points and 2D corner points was ≥99.5%, which meets the perception accuracy requirements of autonomous driving.

[0159] On-site testing: The calibrated mining truck was put into raw coal transportation and stripping layer transportation operations to test its core performance: 1. Obstacle recognition, mainly identifying operating vehicles, retaining walls, water trucks, and on-site dispatch vehicles; the test results were satisfactory. 2. Loading and unloading point parking (parking deviation ≤8cm, better than the 10cm operational requirement). After 30 days of continuous operation, the calibration error was periodically checked, with an offset ≤0.07 pixels, verifying stable and reliable results.

[0160] This embodiment's method overcomes the problems of uneven point cloud distribution and ranging errors in solid-state LiDAR by employing precise extrinsic parameter calibration and nonlinear optimization algorithms, achieving high-precision registration between the solid-state LiDAR and the camera. Simultaneously, it reduces manual intervention through an automated calibration process, improving both efficiency and accuracy. Furthermore, it provides reliable multi-sensor data fusion technology for scenarios such as autonomous driving and robot perception, ensuring effective fusion of solid-state LiDAR and camera data within the same coordinate system, thereby enhancing the system's perception and navigation capabilities.

[0161] The method in this embodiment solves the problems of poor accuracy, low automation, and slow processing efficiency of traditional calibration methods in non-repeating scanning environments. At the same time, it improves the automation and efficiency of the calibration process by optimizing the algorithm, ensuring that accurate extrinsic parameters can be quickly obtained even in complex environments, and providing reliable sensor data fusion support for applications such as autonomous driving and robot perception.

[0162] It is understood that the various method embodiments mentioned above in this application can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this application will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0163] Example 3

[0164] The external parameter calibration device provided in this application embodiment is located in an electronic device, and the external parameter calibration device may include: The acquisition module is used to acquire multiple frames of raw point clouds and calibration images. The raw point clouds are the point clouds of the calibration target acquired by the solid-state lidar. The relative positions between the solid-state lidar and the calibration target are fixed. The raw point clouds are synchronized in time. The calibration images are images of the calibration target acquired by the camera equipment.

[0165] The preprocessing module is used to perform point cloud fusion preprocessing on multiple frames of raw point clouds to generate a temporal integrated point cloud.

[0166] The optimization module is used to perform geometric constraint optimization on the temporal integrated point cloud to generate a target point cloud with uniform point cloud density.

[0167] The pairing module is used to determine the 3D corner points of the target point cloud and the 2D corner points of the calibration image, and to pair the 3D corner points and 2D corner points to generate multiple sets of paired corner points.

[0168] The generation module is used to generate target extrinsic parameters between the solid-state lidar and the camera device based on all paired corner points.

[0169] In some implementations, the preprocessing module is specifically used for: Each original point cloud is denoised to generate a corresponding denoised point cloud. Using the first frame's denoised point cloud as a reference, all other frames' denoised point clouds are calibrated to the same coordinate system through rigid body transformation and superimposed to form a preliminary dense point cloud. The preliminary dense point cloud is then downsampled using a voxel grid to generate a temporal integrated point cloud.

[0170] In some implementations, the calibration target is a checkerboard target.

[0171] The optimization module is specifically used for: Based on the actual size of the checkerboard target, remove point clouds unrelated to the checkerboard target from the temporal integrated point cloud to generate a corresponding checkerboard point cloud. Iterative plane fitting is performed on the checkerboard point cloud to generate a corresponding target fitting plane. The checkerboard point cloud is projected onto the target fitting plane, and the projected checkerboard point cloud is randomly downsampled to generate a target point cloud with uniform point density.

[0172] In some implementations, the checkerboard target is a black and white checkerboard target.

[0173] The pairing module is specifically used to determine the 3D corner points of the target point cloud. A predefined reflectance similarity function is used to calculate the differences in reflectance spatial distribution and geometric position between the target point cloud and a preset standard checkerboard model. The preset standard checkerboard model is constructed based on the checkerboard parameters of a black and white checkerboard target. The checkerboard parameters include at least a binary distribution of reflectance for black and white squares. With the objective of minimizing the reflectance similarity function, a nonlinear optimization algorithm is used to solve for the optimal rigid body transformation parameters from the target point cloud to the preset standard checkerboard model. Based on the optimal rigid body transformation parameters, the corner coordinates in the preset standard checkerboard model are transformed to the solid-state lidar coordinate system to obtain the three-dimensional corner points.

[0174] In some implementations, the pairing module, when determining the two-dimensional corner points of the calibration image, is specifically used for: The calibration image is subjected to checkerboard corner detection to identify the two-dimensional corner points at the intersection of black and white checkerboard squares in the calibration image.

[0175] In some implementations, when the pairing module pairs three-dimensional corner points and two-dimensional corner points to generate multiple sets of paired corner points, it is specifically used for: Starting from a corner of a black and white checkerboard target, the two-dimensional corner points are reordered according to a preset sorting rule to generate sorted two-dimensional corner points. The sorted two-dimensional corner points are then paired with their corresponding three-dimensional corner points to generate multiple sets of paired corner points.

[0176] In some implementations, the generation module is specifically used for: Based on all paired corner points, the initial extrinsic parameters are solved using a random sampling consensus perspective n-point algorithm, and outlier corner points whose reprojection errors exceed a first threshold are removed. Using the set of valid corner points remaining after removing outlier points as input, the perspective n-point calculation and reprojection error evaluation are iteratively performed, and the extrinsic parameters are updated. Iteration stops when the reprojection error falls below a second threshold, and the extrinsic parameters at this point are output as the target extrinsic parameters.

[0177] In some embodiments, the external parameter calibration device further includes: The verification module is used to reproject all three-dimensional corner points in the solid-state lidar coordinate system onto the calibration image plane using the target extrinsic parameters, and calculate the average reprojection error between each three-dimensional corner point and its corresponding two-dimensional corner point. If the average reprojection error is less than or equal to a preset verification threshold, the target extrinsic parameters are determined to be valid, and the calibration is completed.

[0178] The external parameter calibration device provided in this application has the beneficial effects and implementation methods of the external parameter calibration methods provided in Embodiments 1 and 2 of this application. For details, please refer to the specific descriptions of the external parameter calibration methods in Embodiments 1 and 2 above. This embodiment will not repeat them here.

[0179] Example 4

[0180] This application also provides an electronic device, which is intended to be various forms of devices with data processing capabilities, such as workbenches, servers, terminals, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0181] This electronic device includes a processor and memory. The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor processes instructions that execute within the electronic device.

[0182] The memory is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause at least one processor to perform the extrinsic parameter calibration method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the extrinsic parameter calibration method provided in this application.

[0183] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the extrinsic parameter calibration method in the embodiments of this application. The processor executes various functional applications and data processing of the electronic device by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the extrinsic parameter calibration method in the above method embodiments.

[0184] The electronic device provided in this application has the beneficial effects and implementation methods of the external parameter calibration method provided in Embodiments 1 and 2 of this application. For details, please refer to the specific description of the external parameter calibration method in Embodiments 1 and 2 above. This embodiment will not repeat the description here.

[0185] Example 5

[0186] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the external parameter calibration method in Embodiment 1 or Embodiment 2 above.

[0187] The computer-readable storage medium provided in this application embodiment has the beneficial effects and implementation methods of the external parameter calibration methods of Embodiment 1 and Embodiment 2 of this application. For details, please refer to the specific description of the external parameter calibration methods in Embodiment 1 and Embodiment 2 above. This embodiment will not repeat the description here.

[0188] As is known to those skilled in the art, computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0189] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0190] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of this application, and this application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this application, and these modifications and improvements are also considered to be within the scope of protection of this application.

Claims

1. A method for calibrating external parameters, characterized in that, include: Multiple frames of raw point clouds and calibration images are acquired; the raw point clouds are point clouds of the calibration target acquired by solid-state lidar; the relative position between the solid-state lidar and the calibration target is fixed; the raw point clouds are synchronized in time; the calibration images are images of the calibration target acquired by a camera device. Perform point cloud fusion preprocessing on multiple frames of raw point clouds to generate temporal integrated point clouds; Geometric constraint optimization is performed on the temporal integrated point cloud to generate a target point cloud with uniform point cloud density. The three-dimensional corner points of the target point cloud and the two-dimensional corner points of the calibration image are determined, and the three-dimensional corner points and the two-dimensional corner points are paired to generate multiple sets of paired corner points; Target extrinsic parameters between the solid-state lidar and the camera device are generated based on all the said paired corner points.

2. The method according to claim 1, characterized in that, The step of performing point cloud fusion preprocessing on multiple frames of original point clouds to generate a temporal integrated point cloud includes: The original point clouds are denoised to generate corresponding denoised point clouds. Using the first frame of denoised point cloud as a reference, the denoised point clouds of all other frames are calibrated to the same coordinate system through rigid body transformation and superimposed to form a preliminary dense point cloud. The initial dense point cloud is downsampled using a voxel grid to generate the temporal integrated point cloud.

3. The method according to claim 1, characterized in that, The calibration target is a checkerboard target; The step of performing geometric constraint optimization on the temporal integrated point cloud to generate a target point cloud with uniform point cloud density includes: Based on the actual size of the chessboard target, remove the point cloud unrelated to the chessboard target from the temporal integrated point cloud to generate the corresponding chessboard point cloud; Iterative plane fitting is performed on the chessboard point cloud to generate the corresponding target fitting plane; The chessboard point cloud is projected onto the target fitting plane, and the projected chessboard point cloud is randomly downsampled to generate a target point cloud with uniform point cloud density.

4. The method according to claim 3, characterized in that, The checkerboard target is a black and white checkerboard target; Determining the three-dimensional corner points of the target point cloud includes: The difference in reflectance spatial distribution and geometric position between the target point cloud and the preset standard checkerboard model is calculated using a predefined reflectance similarity function; the preset standard checkerboard model is constructed based on the checkerboard parameters of the black and white checkerboard target; the checkerboard parameters include at least a binary distribution of black and white checkerboard reflectance; With the goal of minimizing the reflectivity similarity function, the optimal rigid body transformation parameters from the target point cloud to the preset standard checkerboard model are solved by a nonlinear optimization algorithm. Based on the optimal rigid body transformation parameters, the coordinates of the corner points in the preset standard checkerboard model are transformed to the solid-state lidar coordinate system to obtain the three-dimensional corner points.

5. The method according to claim 4, characterized in that, Determining the two-dimensional corner points of the calibration image includes: The calibration image is subjected to checkerboard corner detection to identify the two-dimensional corner points at the intersection of black and white checkerboard squares in the calibration image.

6. The method according to claim 5, characterized in that, The process of pairing the three-dimensional corner points and the two-dimensional corner points to generate multiple sets of paired corner points includes: Starting from a corner of the black and white checkerboard target, the two-dimensional corner points are reordered according to a preset sorting rule to generate sorted two-dimensional corner points. The sorted two-dimensional corner points are paired with the corresponding three-dimensional corner points to generate multiple sets of paired corner points.

7. The method according to claim 1, characterized in that, The step of generating target extrinsic parameters between the solid-state lidar and the camera device based on all the paired corner points includes: Based on all the paired corner points, the initial extrinsic parameters are solved using the random sampling consensus perspective n-point algorithm, and abnormal corner points whose reprojection errors exceed the first threshold are removed. Using the set of valid corner points remaining after removing the abnormal corner points as input, iteratively perform perspective n-point calculation and reprojection error evaluation, and update the extrinsic parameters; The iteration stops when the reprojection error is lower than the second threshold, and the extrinsic parameters at this time are output as the target extrinsic parameters.

8. The method according to claim 7, characterized in that, After generating the target extrinsic parameters between the solid-state lidar and the camera device based on all the paired corner points, the method further includes: Using the target extrinsic parameters, all the three-dimensional corner points in the solid-state lidar coordinate system are reprojected onto the calibration image plane, and the average reprojection error between each three-dimensional corner point and the corresponding two-dimensional corner point is calculated. If the average reprojection error is less than or equal to the preset verification threshold, the target extrinsic parameter is determined to be valid, and the calibration is completed.

9. An external parameter calibration device, characterized in that, include: The acquisition module is used to acquire multiple frames of raw point clouds and calibration images; the raw point clouds are point clouds of the calibration target acquired by solid-state lidar; the relative position between the solid-state lidar and the calibration target is fixed; the raw point clouds are synchronized in time; the calibration images are images of the calibration target acquired by a camera device. The preprocessing module is used to perform point cloud fusion preprocessing on multiple frames of raw point clouds to generate a temporal integrated point cloud; The optimization module is used to perform geometric constraint optimization on the time-domain integrated point cloud to generate a target point cloud with uniform point cloud density. The pairing module is used to determine the three-dimensional corner points of the target point cloud and the two-dimensional corner points of the calibration image, and to pair the three-dimensional corner points and the two-dimensional corner points to generate multiple sets of paired corner points; A generation module is used to generate target extrinsic parameters between the solid-state lidar and the camera device based on all the paired corner points.

10. An electronic device, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the external parameter calibration method as described in any one of claims 1 to 8.

Images