Calibration method and device for laser radar and camera, and medium

Abstract

The invention provides a laser radar and camera calibration method and device and a medium, and the method comprises the steps: obtaining a camera projection parameter of a camera and an image sequence collected by the camera, and obtaining a radar point cloud frame sequence collected by a laser radar and the pose of the laser radar; according to the image sequence and the pose of the laser radar, the delay of the laser radar relative to the camera and the first relative rotation amount in the external parameters are obtained through calibration, and the first relative translation amount in the initial external parameters is determined; determining the pose of the camera according to the delay and the first relative rotation amount, and determining a visual point cloud corresponding to the camera according to the pose of the camera; and according to the visual point cloud, the camera projection parameters, the pose of the camera, the external parameters, the delay, the radar point cloud frame sequence and the pose of the laser radar, performing optimization processing on the visual point cloud, the pose of the camera and the external parameters to obtain a target visual point cloud, the target pose of the camera and the target external parameters so as to improve the calibration precision.

Description

Technical Field

[0001] The present application relates to the field of calibration technology, and more specifically, to a calibration method, device and medium for a laser radar and a camera. Background Art

[0002] Systems based on visual simultaneous localization and mapping (SLAM) can output point clouds with color attributes, but visual SLAM systems have the disadvantage of outputting point clouds that are too sparse. Lidar systems can output dense point clouds, but Lidar systems cannot output point clouds with color attributes. In scene reconstruction tasks, the structural information of the scene must be reconstructed while the color of the scene must be restored, which requires dense point clouds with color attributes, so visual Lidar systems were created.

[0003] The fusion of the vision system and the Lidar system requires the calibration of the external parameter information of the camera and the Lidar. At present, the external parameter calibration methods of the vision system and the Lidar system include motion-based calibration method and scene-based calibration method. The motion-based calibration method can calibrate the relative rotation and delay information of the vision and Lidar systems, but it is difficult to accurately calibrate the relative translation relationship between the vision and Lidar systems. The scene-based calibration method needs to pre-estimate the initial external parameter information when optimizing the relative rotation relationship and relative translation relationship, and is sensitive to delay information. Therefore, the current external parameter calibration of the vision system and the Lidar system has certain limitations. Summary of the invention

[0004] The purpose of this application is to provide a laser radar and camera calibration method, device and medium to address the practical needs of the existing technology that there are certain limitations in the external parameter calibration of the visual system and the lidar system.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a calibration method for a laser radar and a camera, the method comprising:

[0007] Acquire camera projection parameters of a camera and an image sequence captured by the camera, and acquire a radar point cloud frame sequence captured by a laser radar and a position and posture of the laser radar, wherein the camera is fixedly connected to the laser radar;

[0008] According to the image sequence and the position and posture of the laser radar, calibrate the delay of the laser radar relative to the camera, calibrate the first relative rotation amount of the laser radar relative to the camera in the external parameters, and determine the initial first relative translation amount of the external parameters;

[0009] Determine the position and posture of the camera according to the delay of the laser radar relative to the camera and the first relative rotation amount, and determine the visual point cloud corresponding to the camera according to the position and posture of the camera;

[0010] According to the visual point cloud, the camera projection parameters, the camera pose, the extrinsic parameters, the delay, the radar point cloud frame sequence and the lidar pose, the visual point cloud, the camera pose and the extrinsic parameters are optimized to obtain a target visual point cloud, a target pose of the camera and target extrinsic parameters.

[0011] As an optional implementation, the visual point cloud, the camera projection parameters, the camera pose, the extrinsic parameters, the delay, the radar point cloud frame sequence, and the laser radar pose are optimized to obtain a target visual point cloud, a target pose of the camera, and a target extrinsic parameter, including:

[0012] According to the visual point cloud, the camera projection parameters, the camera posture, the first relative rotation amount in the external parameters and the delay, a motion correlation minimum objective function is constructed, and the optimized visual point cloud, the optimized camera posture and the optimized external parameters are calibrated;

[0013] According to the radar point cloud frame sequence and the posture of the laser radar, a voxel map is constructed, the surface features of the radar point cloud frame sequence are extracted, the surface features are constrained, the surface feature constraints are determined, and according to the motion association minimum objective function and the surface feature constraints, the target visual point cloud, the target posture of the camera and the target extrinsic parameters are calibrated.

[0014] As an optional implementation manner, calibrating the delay of the laser radar relative to the camera according to the image sequence and the posture of the laser radar includes:

[0015] According to the image sequence, determining the relative rotation amount between each frame image and the first frame image in the image sequence;

[0016] According to the relative rotation amount between each frame image and the first frame image, a visual motion curve is constructed, and according to the position and posture of the laser radar, a radar motion curve is constructed;

[0017] The visual motion curve and the radar motion curve are sampled respectively at a preset sampling time interval to determine a visual angular velocity and a radar angular velocity;

[0018] Determining a delay parameter according to the visual angular rate, the radar angular rate, and the sampling time interval;

[0019] The delay of the laser radar relative to the camera is determined according to the delay parameter and the sampling time interval.

[0020] As an optional implementation, determining, according to the image sequence, the relative rotation amount between each frame image and the first frame image in the image sequence includes:

[0021] Extracting feature points of each frame image according to the image sequence, and performing image registration on each frame image according to the feature points of each frame image to obtain each pair of registered feature points of the successfully matched image pairs;

[0022] Determining, based on each pair of registration feature points of the image pair, a relative rotation amount of the camera from a first moment in the image pair to a second moment in the image pair;

[0023] According to the relative rotation amount of the camera from the first moment in the image pair to the second moment in the image pair, a visual pose graph is constructed to determine the relative rotation amount of each frame image in the image sequence with respect to the first frame image.

[0024] As an optional implementation manner, the calibration obtains a first relative rotation amount in the external parameters of the laser radar relative to the camera, including:

[0025] Determining a relative rotation amount of the laser radar from a first moment in the image pair to a second moment in the image pair based on the radar motion curve and a delay of the laser radar relative to the camera;

[0026] According to the relative rotation amounts of the laser radar from the first moment in each image pair to the second moment in each image pair and the relative rotation amounts of the camera from the first moment in each image pair to the second moment in each image pair, the first relative rotation amount in the external parameters of the laser radar relative to the camera is obtained by calibrating with a hand-eye calibration algorithm.

[0027] As an optional implementation, determining the position and posture of the camera according to the delay of the laser radar relative to the camera and the first relative rotation amount, and determining the visual point cloud corresponding to the camera according to the position and posture of the camera includes:

[0028] According to the registration results of each frame image in the image sequence, the same-name points of each registration feature point pair are obtained, and the same-name points of each registration feature point pair are triangulated to obtain the same-name points of each registration feature point pair of the camera in the spherical coordinate system;

[0029] Determine the position and posture of the camera according to the delay of the laser radar relative to the camera, the first relative rotation amount, and the radar motion curve;

[0030] Determine the same-name points and depth points of each pair of registration feature points of the camera in the world coordinate system according to the same-name points of each pair of registration feature points of the camera in the spherical coordinate system and the position and posture of the camera;

[0031] The visual point cloud corresponding to the camera is determined according to the depth points and the same-name points of each registration feature point pair of the camera in the world coordinate system.

[0032] As an optional implementation, constructing a voxel map according to the radar point cloud frame sequence and the position and posture of the laser radar, and extracting surface features of the radar point cloud frame sequence includes:

[0033] According to the position and posture of the laser radar, the radar point cloud frame sequence is converted from the radar coordinate system to the world coordinate system to obtain the radar point cloud frame sequence in the world coordinate system;

[0034] Constructing a voxel map according to the radar point cloud frame sequence in the world coordinate system and preset bounding box parameters;

[0035] Determining the radar point cloud mean and covariance of each voxel according to the voxel map;

[0036] According to each eigenvalue of the covariance, the obtained face feature, the target minimum eigenvalue and the eigenvector corresponding to the target minimum eigenvalue are marked.

[0037] As an optional implementation, the determining of the surface feature constraint and calibrating the target visual point cloud, the target pose of the camera and the target extrinsic parameters according to the motion association minimum objective function and the surface feature constraint include:

[0038] Determining a surface feature constraint according to the optimized visual point cloud, the mean value of the radar point cloud of the voxel, and a eigenvector corresponding to a minimum eigenvalue of the target;

[0039] The surface feature constraint is added to the motion association minimum objective function to calibrate the target visual point cloud, the target pose of the camera and the target extrinsic parameters.

[0040] In a second aspect, an embodiment of the present application provides a calibration device for a laser radar and a camera, the device comprising:

[0041] An acquisition module, used to acquire camera projection parameters of a camera and an image sequence acquired by the camera, and to acquire a radar point cloud frame sequence acquired by a laser radar and a position and posture of the laser radar, wherein the camera is fixedly connected to the laser radar;

[0042] A calibration module, used for calibrating the delay of the laser radar relative to the camera according to the image sequence and the posture of the laser radar, calibrating the first relative rotation amount of the laser radar relative to the camera in the external parameters, and determining the initial first relative translation amount of the external parameters;

[0043] A determination module, configured to determine a position and posture of the camera according to a delay of the laser radar relative to the camera and the first relative rotation amount, and determine a visual point cloud corresponding to the camera according to the position and posture of the camera;

[0044] The optimization processing module is used to optimize the visual point cloud, the camera pose and the external parameters according to the visual point cloud, the camera projection parameters, the camera pose, the external parameters, the delay, the radar point cloud frame sequence and the lidar pose to obtain the target visual point cloud, the camera's target pose and the target external parameters.

[0045] As an optional implementation, the optimization processing module is specifically used for:

[0046] According to the visual point cloud, the camera projection parameters, the camera posture, the first relative rotation amount in the external parameters and the delay, a motion correlation minimum objective function is constructed, and the optimized visual point cloud, the optimized camera posture and the optimized external parameters are calibrated;

[0047] According to the radar point cloud frame sequence and the posture of the laser radar, a voxel map is constructed, the surface features of the radar point cloud frame sequence are extracted, the surface features are constrained, the surface feature constraints are determined, and according to the motion association minimum objective function and the surface feature constraints, the target visual point cloud, the target posture of the camera and the target extrinsic parameters are calibrated.

[0048] As an optional implementation, the calibration module is specifically used for:

[0049] According to the image sequence, determining the relative rotation amount between each frame image and the first frame image in the image sequence;

[0050] According to the relative rotation amount between each frame image and the first frame image, a visual motion curve is constructed, and according to the position and posture of the laser radar, a radar motion curve is constructed;

[0051] The visual motion curve and the radar motion curve are sampled at preset sampling time intervals to determine a visual angular velocity and a radar angular velocity;

[0052] Determining a delay parameter according to the visual angular rate, the radar angular rate, and the sampling time interval;

[0053] The delay of the laser radar relative to the camera is determined according to the delay parameter and the sampling time interval.

[0054] As an optional implementation, the calibration module is specifically used for:

[0055] Extracting feature points of each frame image according to the image sequence, and performing image registration on each frame image according to the feature points of each frame image to obtain each pair of registered feature points of the successfully matched image pairs;

[0056] Determining, based on each pair of registration feature points of the image pair, a relative rotation amount of the camera from a first moment in the image pair to a second moment in the image pair;

[0057] According to the relative rotation amount of the camera from the first moment in the image pair to the second moment in the image pair, a visual pose graph is constructed to determine the relative rotation amount of each frame image in the image sequence with respect to the first frame image.

[0058] As an optional implementation, the calibration module is specifically used for:

[0059] Determining a relative rotation amount of the laser radar from a first moment in the image pair to a second moment in the image pair based on the radar motion curve and a delay of the laser radar relative to the camera;

[0060] According to the relative rotation amounts of the laser radar from the first moment in each image pair to the second moment in each image pair and the relative rotation amounts of the camera from the first moment in each image pair to the second moment in each image pair, the first relative rotation amount in the external parameters of the laser radar relative to the camera is obtained by calibrating with a hand-eye calibration algorithm.

[0061] As an optional implementation manner, the determining module is specifically used for:

[0062] According to the registration results of each frame image in the image sequence, the same-name points of each registration feature point pair are obtained, and the same-name points of each registration feature point pair are triangulated to obtain the same-name points of each registration feature point pair of the camera in the spherical coordinate system;

[0063] Determine the position and posture of the camera according to the delay of the laser radar relative to the camera, the first relative rotation amount, and the radar motion curve;

[0064] Determine the same-name points and depth points of each pair of registration feature points of the camera in the world coordinate system according to the same-name points of each pair of registration feature points of the camera in the spherical coordinate system and the position and posture of the camera;

[0065] The visual point cloud corresponding to the camera is determined according to the depth points and the same-name points of each registration feature point pair of the camera in the world coordinate system.

[0066] As an optional implementation, the optimization processing module is specifically used for:

[0067] According to the position and posture of the laser radar, the radar point cloud frame sequence is converted from the radar coordinate system to the world coordinate system to obtain the radar point cloud frame sequence in the world coordinate system;

[0068] Constructing a voxel map according to the radar point cloud frame sequence in the world coordinate system and preset bounding box parameters;

[0069] Determining the radar point cloud mean and covariance of each voxel according to the voxel map;

[0070] According to each eigenvalue of the covariance, the obtained face feature, the target minimum eigenvalue and the eigenvector corresponding to the target minimum eigenvalue are marked.

[0071] As an optional implementation, the processing module is specifically used for:

[0072] Determining a surface feature constraint according to the optimized visual point cloud, the mean value of the radar point cloud of the voxel, and a eigenvector corresponding to a minimum eigenvalue of the target;

[0073] The surface feature constraint is added to the motion association minimum objective function to calibrate the target visual point cloud, the target pose of the camera and the target extrinsic parameters.

[0074] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform the steps of the laser radar and camera calibration method as described in the first aspect above.

[0075] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the laser radar and camera calibration method as described in the first aspect above are executed.

[0076] The beneficial effects of this application are:

[0077] The present application provides a method, device and medium for calibrating a laser radar and a camera, wherein the camera and the laser radar are fixedly connected to the same device, and the camera projection parameters of the camera, the image sequence collected by the camera, the radar point cloud frame sequence collected by the laser radar and the position and posture of the laser radar are obtained. In the rough calibration process, the delay of the laser radar relative to the camera is obtained by rough calibration according to the image sequence collected by the camera and the position and posture of the laser radar, and then the first relative rotation amount in the external parameters of the laser radar relative to the camera is obtained, and the first relative translation amount in the external parameters of the laser radar relative to the camera is initialized. In the fine calibration process, the position and posture of the camera are first determined according to the delay and the first relative rotation amount obtained by the rough calibration, and then the visual point cloud corresponding to the camera is determined according to the position and posture of the camera; then, according to the visual point cloud, the camera projection parameters, the position and posture of the camera, the external parameters, the delay, the radar point cloud frame sequence and the position and posture of the laser radar, the visual point cloud, the position and posture of the camera and the external parameters are optimized multiple times, and the target visual point cloud, the target position and posture of the camera and the target external parameters are obtained by fine calibration. Since the delay of the lidar relative to the camera is calibrated in the motion-based coarse calibration process, the delay obtained by the coarse calibration is combined with the fine calibration process based on motion and point cloud surface feature alignment. Through multiple optimizations, the target visual point cloud, the target pose of the camera and the target extrinsic parameters are obtained, which greatly improves the calibration accuracy and efficiency and improves the robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 A schematic diagram of a flow chart of a laser radar and camera calibration method provided in an embodiment of the present application;

[0080] Figure 2 A schematic diagram of a process for obtaining a target visual point cloud, a target pose of a camera, and target extrinsic parameters in a calibration method for a laser radar and a camera provided in an embodiment of the present application;

[0081] Figure 3 A schematic diagram of a flow chart of obtaining the delay of the laser radar relative to the camera by calibrating the laser radar and camera calibration method provided in an embodiment of the present application;

[0082] Figure 4 A schematic diagram of a flow chart of determining the relative rotation amount between each frame image and the first frame image in an image sequence in a calibration method of a laser radar and a camera provided in an embodiment of the present application;

[0083] Figure 5 A schematic diagram of a flow chart of obtaining a first relative rotation amount in an external parameter of a laser radar relative to a camera by calibrating the laser radar and camera calibration method provided in an embodiment of the present application;

[0084] Figure 6 A schematic diagram of the process of determining the camera's position and the visual point cloud corresponding to the camera in the laser radar and camera calibration method provided in an embodiment of the present application;

[0085] Figure 7 A schematic diagram of a flow chart of extracting surface features of a radar point cloud frame sequence in a laser radar and camera calibration method provided in an embodiment of the present application;

[0086] Figure 8 Another schematic diagram of a process for obtaining a target visual point cloud, a target pose of a camera, and target extrinsic parameters of a laser radar and camera calibration method provided in an embodiment of the present application;

[0087] Fig. 9 A module structure diagram of a laser radar and camera calibration device provided in an embodiment of the present application;

[0088] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0090] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0091] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0092] In the visual Lidar system, the fusion of the visual system and the Lidar system requires the calibration of the external parameter information of the camera and the Lidar. The motion-based calibration method is difficult to accurately calibrate the relative translation relationship between the visual and Lidar systems. The scene-based calibration method requires the estimation of the initial external parameter information in advance and is sensitive to delay information. Therefore, the external parameter calibration of the visual system and the Lidar system currently has certain limitations.

[0093] Based on the above problems, the embodiment of the present application proposes a calibration method for a laser radar and a camera, in which the delay of the laser radar relative to the camera and the first relative rotation amount in the external parameters are calibrated in a rough calibration process based on motion. In the fine calibration process based on motion and point cloud surface feature alignment, the visual point cloud corresponding to the camera is determined according to the delay of the laser radar relative to the camera and the first relative rotation amount in the external parameters obtained by the rough calibration, and the visual point cloud, the camera's posture and the external parameters are optimized multiple times to obtain a more accurate target visual point cloud, the camera's target posture and the target external parameters. Without the aid of a calibration object and without estimating the initial external parameters, the delay and external parameters of the laser radar relative to the camera are accurately calibrated, thereby improving the calibration efficiency and accuracy.

[0094] Figure 1 The flowchart of the laser radar and camera calibration method provided in the embodiment of the present application is shown in FIG. 1 , and the execution subject of the method can be any computer device with computing processing capabilities. Figure 1 As shown, the method includes:

[0095] S101, obtaining camera projection parameters of a camera and an image sequence captured by the camera, and obtaining a radar point cloud frame sequence captured by a laser radar and a position and posture of the laser radar, wherein the camera is fixedly connected to the laser radar.

[0096] Optionally, the camera and the laser radar are fixedly connected to the same movable device, and the movable device is moved during the calibration process so that the camera collects images while the laser radar synchronously collects radar point clouds, and the image sequence collected by the camera is obtained within a preset time period. t} and the radar point cloud frame sequence in the radar coordinate system collected by the lidar And get the camera projection parameters F of the camera θ And the position of the laser radar

[0097] Among them, {} represents the set of time series, I t represents the image captured by the camera at time t, represents the jth 3D scanning point in the radar coordinate system collected by the laser radar at time t, F θ The camera projection parameters for projecting 3D points in the image captured by the camera to 2D points, Indicates the relative pose from the world coordinate system to the radar coordinate system at time t.

[0098] S102. Calibrate the delay of the laser radar relative to the camera according to the image sequence and the position of the laser radar, calibrate the first relative rotation amount in the external parameters of the laser radar relative to the camera, and determine the first relative translation amount in the initial external parameters.

[0099] Optionally, the calibration process includes a rough calibration process based on motion and a fine calibration process based on alignment of surface features of motion and point cloud. In the rough calibration process based on motion, according to the image sequence {I t} and the position of the LiDAR The rough calibration obtains the delay δt of the lidar relative to the camera, and then obtains the external parameter ΔT of the lidar relative to the camera c_l The first relative rotation R l_c Among them, the external parameter ΔT of the laser radar relative to the camera is c_l Including the first relative rotation R of the laser radar relative to the camera l_c And the first relative translation t of the laser radar relative to the camera l_c During the rough calibration process, the first relative translation t l_c Initialize and preset the first relative translation t of the laser radar relative to the camera in the initial external parameters l_c is zero.

[0100] S103, determining the position and posture of the camera according to the delay of the laser radar relative to the camera and the first relative rotation amount, and determining the visual point cloud corresponding to the camera according to the position and posture of the camera.

[0101] Optionally, in the fine calibration process based on motion and point cloud surface feature alignment, the delay δt of the laser radar relative to the camera and the first relative rotation amount R obtained by the rough calibration are l_c , determine the camera’s position That is, the visual pose. Among them, Represents the relative pose from the world coordinate system to the camera coordinate system at time t. Determine the visual point cloud corresponding to the camera through triangulation, coordinate transformation and depth point calculation Among them, the visual point cloud corresponding to the camera Located in the world coordinate system, different from the radar point cloud frame sequence Point clouds of each frame in the image, visual point clouds corresponding to the camera is a globally merged point cloud.

[0102] S104, optimizing the visual point cloud, the camera's pose and the external parameters according to the visual point cloud, the camera's projection parameters, the camera's pose, the external parameters, the delay, the radar point cloud frame sequence and the lidar's pose, and obtaining the target visual point cloud, the camera's target pose and the target external parameters.

[0103] Optionally, in the process of fine calibration based on motion and point cloud surface feature alignment, according to the visual point cloud corresponding to the camera Camera projection parameter F θ , camera pose The external parameter ΔT of the LiDAR relative to the camera c_l , the delay δt of the lidar relative to the camera, the radar point cloud frame sequence And the position of the laser radar Visual point cloud Camera pose And the external parameter ΔT of the lidar relative to the camera c_l Perform multiple optimization processes and register visual point clouds simultaneously And radar point cloud frame sequence Accurately calibrate to obtain the target visual point cloud The target pose of the camera And the target external parameter ΔT″ of the laser radar relative to the camera c_l .

[0104] In this embodiment, the camera and the laser radar are fixedly connected to the same device, and the camera projection parameters of the camera, the image sequence collected by the camera, the radar point cloud frame sequence collected by the laser radar, and the position and posture of the laser radar are obtained. In the rough calibration process, the delay of the laser radar relative to the camera is roughly calibrated according to the image sequence collected by the camera and the position and posture of the laser radar, and then the first relative rotation amount in the external parameters of the laser radar relative to the camera is obtained, and the first relative translation amount in the external parameters of the laser radar relative to the camera is initialized. In the fine calibration process, the position and posture of the camera is first determined according to the delay and the first relative rotation amount obtained by the rough calibration, and then the visual point cloud corresponding to the camera is determined according to the position and posture of the camera; then according to the visual point cloud, camera projection parameters, camera position and posture, external parameters, delay, radar point cloud frame sequence and the position and posture of the laser radar, the visual point cloud, the position and posture of the camera and the external parameters are optimized multiple times, and the target visual point cloud, the target position and posture of the camera and the target external parameters are obtained by fine calibration. Since the delay of the lidar relative to the camera is calibrated in the motion-based coarse calibration process, the delay obtained by the coarse calibration is combined with the fine calibration process based on motion and point cloud surface feature alignment. Through multiple optimizations, the target visual point cloud, the target pose of the camera and the target extrinsic parameters are obtained, which greatly improves the calibration accuracy and efficiency and improves the robustness.

[0105] The following describes in detail the process of optimizing the visual point cloud, camera pose and extrinsic parameters based on the visual point cloud, camera projection parameters, camera pose, extrinsic parameters, delay, radar point cloud frame sequence and lidar pose to obtain the target visual point cloud, camera target pose and target extrinsic parameters.

[0106] Figure 2 A schematic diagram of the process of obtaining a target visual point cloud, a target pose of a camera, and target external parameters of a laser radar and camera calibration method provided in an embodiment of the present application, such as Figure 2 As shown, in the above step S104, the visual point cloud, the camera's pose and the external parameters are optimized according to the visual point cloud, the camera projection parameters, the camera's pose, the external parameters, the delay, the radar point cloud frame sequence and the laser radar's pose to obtain the target visual point cloud, the camera's target pose and the target external parameters, including:

[0107] S201, constructing a motion-related minimum objective function according to the visual point cloud, camera projection parameters, camera posture, the first relative rotation amount in the external parameters, and the delay, and calibrating to obtain the optimized visual point cloud, the optimized camera posture, and the optimized external parameters.

[0108] Optionally, in the fine calibration process based on motion and point cloud surface feature alignment, according to the visual point cloud Camera projection parameter F θ , camera pose The external parameter ΔT of the LiDAR relative to the camera c_l The first relative rotation R l_c And the delay δt of the lidar relative to the camera, construct the minimum objective function of motion association Visual point cloud Camera pose and external parameter ΔT c_l Perform the first optimization process to obtain the optimized visual point cloud Optimized camera pose And the optimized external parameter ΔT′ c_l . Optimized external parameter ΔT′ c_l Including the second relative rotation R′ of the optimized laser radar relative to the camera l_c And the second relative translation t′ of the optimized laser radar relative to the camera l_c .

[0109] Among them, the camera's external parameter ΔT in the motion association minimum objective function is c_l Including the first relative rotation R of the laser radar relative to the camera obtained by rough calibration l_c And the first relative translation t of the initialized lidar relative to the camera l_c, the first relative translation t l_c is zero. Indicates t i The camera's position at the moment, represents the visual 3D point in the camera's visual point cloud, Indicates t i Momentary visual 3D point The associated feature points, B l is the radar motion curve, and δt is the delay of the lidar relative to the camera.

[0110] S202. Construct a voxel map according to the radar point cloud frame sequence and the laser radar's posture, extract the surface features of the radar point cloud frame sequence, constrain the surface features, determine the surface feature constraints, and calibrate the target visual point cloud, the camera's target posture, and the target extrinsic parameters according to the motion association minimum objective function and the surface feature constraints.

[0111] Optionally, according to the radar point cloud frame sequence in the radar coordinate system And the position of the laser radar Construct a voxel map (VoxelMap) to extract radar point cloud frame sequence The surface features are constrained and the surface features are determined. According to the minimum objective function of motion association And surface feature constraints, the optimized visual point cloud Optimized camera pose And the optimized external parameter ΔT′ c_l Perform the second optimization process to obtain the target visual point cloud The target pose of the camera And the target external parameter ΔT″ of the laser radar relative to the camera c_l .

[0112] In the second optimization process, the visual point cloud in the motion association minimum objective function is pre-set. Camera pose And the external parameter ΔT c_l Replaced with optimized visual point clouds respectively Optimized camera pose And the optimized external parameter ΔT′ c_l , to optimize the visual point cloud Optimized camera pose And the optimized external parameter ΔT′ c_l Perform the second optimization process.

[0113] In this embodiment, a motion-related minimum objective function is constructed based on the visual point cloud, camera projection parameters, camera posture, first relative rotation amount and delay, and the optimized visual point cloud, the optimized camera posture and the optimized external parameters are obtained through the first optimization process. According to the radar point cloud frame sequence and the laser radar posture in the radar coordinate system, a voxel map is constructed, the surface features of the radar point cloud frame sequence are extracted, and the surface feature constraints are determined. According to the motion-related minimum objective function and the surface feature constraints, the optimized visual point cloud, the optimized camera posture and the optimized external parameters are optimized for the second time to obtain the target visual point cloud, the target posture of the camera and the target external parameters of the laser radar relative to the camera. In the precise calibration process based on the alignment of surface features of motion and point cloud, the visual point cloud, the camera posture and the external parameters are optimized twice, and the calibration obtains a more accurate target visual point cloud, the target posture of the camera and the target external parameters of the laser radar relative to the camera, thereby improving the calibration accuracy.

[0114] The following is a detailed description of the process of calibrating the delay of the LiDAR relative to the camera based on the image sequence and the LiDAR's posture.

[0115] Figure 3 A schematic diagram of a process of calibrating the laser radar and camera calibration method provided in an embodiment of the present application to obtain the delay of the laser radar relative to the camera, such as Figure 3 As shown, the step of calibrating the delay of the laser radar relative to the camera according to the image sequence and the position of the laser radar in the above step S102 includes:

[0116] S301: Determine, according to an image sequence, a relative rotation amount between each frame image in the image sequence and a first frame image.

[0117] Optionally, in the motion-based coarse calibration process, the image sequence {I t} Perform feature point extraction, image registration, and visual pose graph construction to determine the image sequence {I t The relative rotation between each frame image and the first frame image Among them, the first frame image in the image sequence is the image captured by the camera at time t0.

[0118] S302: construct a visual motion curve according to the relative rotation between each frame image and the first frame image, and construct a radar motion curve according to the position and posture of the laser radar.

[0119] Optionally, according to the relative rotation amount between each frame image and the first frame image Construct a visual motion curve, which is a visual motion B-spline (Bspline) SO3 curve B c (t). According to the position of the laser radar Construct the radar motion curve, which is the radar motion Bspline SE3 curve B l (t). The B-spline curve is a piecewise continuous polynomial curve.

[0120] S303: Sample the visual motion curve and the radar motion curve respectively at a preset sampling time interval to determine the visual angular velocity and the radar angular velocity.

[0121] Optionally, the visual motion Bspline SO3 curve B is sampled at a preset sampling time interval Δt. c (t) is sampled and the visual angular rate {ω t,c}. Accordingly, the radar motion Bspline SE3 curve B is sampled at a preset sampling time interval Δt. l (t) is sampled and the radar angular velocity {ω t,l}.

[0122] S304: Determine a delay parameter according to the visual angular rate, the radar angular rate, and the sampling time interval.

[0123] Optionally, according to the visual angular rate {ω t,c}、Radar angular rate {ω t,l} and the sampling time interval Δt, by maximizing the function argmax j ∑||ω t+j×Δt,c ||·||ω t,l ||, determine the delay parameter j. Among them, ω t+j×Δt,c Represents the visual angular rate at time t+j×Δt.

[0124] S305. Determine the delay of the laser radar relative to the camera according to the delay parameter and the sampling time interval.

[0125] Optionally, according to the delay parameter j and the sampling time interval Δt, the product j×Δt of the delay parameter j and the sampling time interval Δt is taken as the delay δt of the laser radar relative to the camera, that is, δt=j×Δt.

[0126] In this embodiment, in the rough calibration process based on motion, the relative rotation between each frame image and the first frame image in the image sequence is determined by extracting feature points, aligning images, and constructing a visual pose graph of the image sequence, thereby constructing a visual motion curve, and constructing a radar motion curve according to the pose of the laser radar. The visual motion curve and the radar motion curve are sampled at a preset sampling time interval to determine the visual angular velocity and the radar angular velocity, and the delay parameter is determined based on the visual angular velocity, the radar angular velocity, and the sampling time interval, and the product of the delay parameter and the sampling time interval is used as the delay of the laser radar relative to the camera. Through the rough calibration based on motion, the delay of the laser radar relative to the camera is accurately calibrated.

[0127] The following describes in detail the process of determining the relative rotation amount between each frame image and the first frame image in the image sequence according to the image sequence.

[0128] Figure 4 A schematic diagram of a flow chart of determining the relative rotation amount between each frame image and the first frame image in an image sequence in a calibration method of a laser radar and a camera provided in an embodiment of the present application, such as Figure 4 As shown, the step of determining the relative rotation amount between each frame image and the first frame image in the image sequence according to the image sequence in the above step S301 includes:

[0129] S401 . Extract feature points of each frame of image according to the image sequence, and perform image registration on each frame of image according to the feature points of each frame of image to obtain each pair of registered feature points of the successfully matched image pairs.

[0130] Optionally, in the motion-based rough calibration process, according to the image sequence {I t}, and extract the image sequence {I t}. The feature extraction algorithm may be a scale invariant feature transformation (SIFT) algorithm or a speeded up robust features (SURF) algorithm. According to the feature points of each frame of the image, each frame of the image is matched with each other by a brute force matching (BruteMatch) algorithm or a K-nearest neighbor matching (KnnMatch) algorithm. The two frames of the image that are successfully matched are an image pair, and each pair of registered feature points of the successfully matched image pair is obtained. The registered feature point pair may be Indicates t i The kth feature point of the image at time Indicates t j The Sth feature point of the image at time t i is the first moment of image centering, t jis the second moment of image centering.

[0131] S402: Determine, based on each pair of registration feature points of the image pair, a relative rotation amount of the camera from a first moment in the image pair to a second moment in the image pair.

[0132] Optionally, according to each pair of registered feature points of the image pair, each pair of registered feature points of the image pair in the camera coordinate system is converted into feature points in the spherical coordinate system by coordinate transformation, and the matching relationship of each pair of registered feature points in the spherical coordinate system is determined. Then the matching relationship of the registration feature points in the spherical coordinate system is represents t in spherical coordinate system i The kth feature point of the image at time represents t in spherical coordinate system j The Sth feature point of the image at time t is calculated according to the matching relationship of each registered feature point pair in the spherical coordinate system, and then the rotation amount is decomposed according to the intrinsic matrix or homography matrix to determine the camera from the image pair at the first moment t i To the second moment t of image alignment j The relative rotation

[0133] S403: construct a visual pose graph according to the relative rotation of the camera from the first moment in the image pair to the second moment in the image pair, and determine the relative rotation of each frame image in the image sequence with respect to the first frame image.

[0134] Optionally, according to the camera from the first moment t of the image pair i To the second moment t of image alignment j The relative rotation The pose graph optimization (PGO) algorithm based on least squares nonlinear optimization constructs a visual pose graph and minimizes the function The image sequence obtained by optimization is {I t The relative rotation between each frame image and the first frame image in, The first moment t when the camera is centered in the image pair i To the second moment t of image alignment j The relative rotation of The second moment t is when the camera is centered on the image pair j The relative rotation to the first frame image.

[0135] In this embodiment, feature points of each frame image in the image sequence are extracted according to the image sequence captured by the camera, and each frame image is matched pairwise according to the feature points of each frame image to obtain each pair of registered feature points of the successfully matched image pair. Each pair of registered feature points of the image pair in the camera coordinate system is converted into feature points in the spherical coordinate system by coordinate transformation, and the matching relationship of each pair of registered feature points in the spherical coordinate system is determined, and then the relative rotation amount of the camera from the first moment in the image pair to the second moment in the image pair is determined. The visual pose graph is constructed according to the relative rotation amount of the camera from the first moment in the image pair to the second moment in the image pair, and the relative rotation amount of each frame image in the image sequence and the first frame image is obtained by minimizing the function optimization, so as to accurately construct the visual motion curve.

[0136] The following is a detailed description of the process of calibrating the first relative rotation amount in the external parameters of the laser radar relative to the camera.

[0137] Figure 5 A schematic diagram of a flow chart of obtaining a first relative rotation amount in an external parameter of the laser radar relative to the camera by calibrating the laser radar and camera calibration method provided in an embodiment of the present application, such as Figure 5 As shown, the step of calibrating the first relative rotation amount in the external parameters of the laser radar relative to the camera in the above step S102 includes:

[0138] S501. Determine the relative rotation amount of the laser radar from the first moment of the image alignment to the second moment of the image alignment according to the radar motion curve and the delay of the laser radar relative to the camera.

[0139] Optionally, according to the radar motion Bspline SE3 curve B l (t) and the delay δt of the lidar relative to the camera, the relative rotation from the first moment in the image pair to the second moment in the image pair is determined based on the following formula The relative rotation of the associated lidar from the first moment in the image pair to the second moment in the image pair

[0140]

[0141] in, B is the relative rotation of the laser radar from the first moment of the image pair to the second moment of the image pair, l is the radar motion Bspline SE3 curve, t i is the first moment of image centering, t j is the second moment of image centering, and δt is the delay of the lidar relative to the camera.

[0142] S502. Based on the relative rotation amounts of the laser radar from the first moment in each image pair to the second moment in each image pair and the relative rotation amounts of the camera from the first moment in each image pair to the second moment in each image pair, the first relative rotation amount in the external parameters of the laser radar relative to the camera is calibrated by a hand-eye calibration algorithm.

[0143] Optionally, according to the relative rotation amount of the laser radar from the first moment in each image pair to the second moment in each image pair And the relative rotation of the camera from the first moment in each image pair to the second moment in each image pair The hand-eye calibration algorithm is used to calibrate the external parameter ΔT of the LiDAR relative to the camera based on the rotation and translation motion of the X, Y and Z axes. c_l The first relative rotation R l_c .

[0144] In this embodiment, the relative rotation amount of the laser radar from the first moment in the image pair to the second moment in the image pair associated with the relative rotation amount of the camera from the first moment in the image pair to the second moment in the image pair is determined according to the radar motion curve and the delay of the laser radar relative to the camera. According to each relative rotation amount of the laser radar from the first moment in each image pair to the second moment in each image pair and each relative rotation amount of the camera from the first moment in each image pair to the second moment in each image pair, the first relative rotation amount in the external parameters of the laser radar relative to the camera is calibrated based on the rotation and translation motion of the three coordinate axes through the hand-eye calibration algorithm. The calibration accuracy of the first relative rotation amount in the external parameters of the laser radar relative to the camera is improved.

[0145] The following describes in detail the process of determining the position and posture of the camera based on the delay of the laser radar relative to the camera and the first relative rotation amount, and determining the visual point cloud corresponding to the camera based on the position and posture of the camera.

[0146] Figure 6 A schematic diagram of the process of determining the camera's position and the visual point cloud corresponding to the camera in the laser radar and camera calibration method provided in the embodiment of the present application, such as Figure 6 As shown, the step of determining the position and posture of the camera according to the delay and the first relative rotation amount of the laser radar relative to the camera in the above step S103, and determining the visual point cloud corresponding to the camera according to the position and posture of the camera, includes:

[0147] S601. Obtain the same-name points of each pair of registered feature points according to the registration results of each frame image in the image sequence, triangulate the same-name points of each pair of registered feature points, and obtain the same-name points of each pair of registered feature points in the spherical coordinate system of the camera.

[0148] Optionally, according to the image sequence {I t}, and associate the registration feature point pairs of the image pairs that are successfully matched in the rough calibration process to obtain the same-name points of each registration feature point pair.

[0149] Specifically, Indicates the tth i Frame image and t j The registration feature point pairs of the frame image, For the tth j Frame image and t k The registration feature point pairs of the frame image, The same-name points associated with each pair of registered feature points Represents the visual 3D point in the camera's visual point cloud, and the extended feature point subscript is Indicates t i Time homonymous point The associated feature points, then the same-name points of each pair of registered feature points extracted are

[0150] For each pair of registered feature points, Triangulate and transform the points with the same name into The associated feature points Convert to the spherical coordinate system and get the same-name points of each registration feature point pair of the camera in the spherical coordinate system

[0151] S602: Determine the position and posture of the camera according to the delay of the laser radar relative to the camera, the first relative rotation amount, and the radar motion curve.

[0152] Optionally, since the camera is fixedly connected to the laser radar, the relative position of the camera and the laser radar is fixed. l_c And radar motion Bspline SE3 curve B l (t), establish the association between the camera and the lidar to determine the camera’s position and posture Among them, the camera posture Includes the relative rotation from the world coordinate system to the camera coordinate system And the relative translation Determine t based on the following formula i The relative pose from the world coordinate system to the camera coordinate system at that moment The relative rotation in And the relative translation

[0153]

[0154] in, Indicates ti The relative rotation amount in the relative pose from the world coordinate system to the camera coordinate system at the moment, Indicates t i The relative translation from the world coordinate system to the camera coordinate system at the moment, R l_c is the external parameter ΔT of the laser radar relative to the camera c_l The first relative rotation in , δt is the delay of the lidar relative to the camera.

[0155] S603 . Determine the same-name points and depth points of each pair of registration feature points of the camera in the world coordinate system according to the same-name points of each pair of registration feature points of the camera in the spherical coordinate system and the position and posture of the camera.

[0156] Optionally, according to the same-name points of each registration feature point pair of the camera in the spherical coordinate system Select the two feature points with the longest distance from each associated point of the same name, and The relative rotation from the world coordinate system to the camera coordinate system And the characteristic points of the same name in the spherical coordinate system Get the same-name points of each registration feature point pair of the camera in the world coordinate system The coordinates of the two selected feature points are converted to the world coordinate system based on the following formula:

[0157]

[0158] in, are the characteristic points of the same name in the spherical coordinate system, t i The relative rotation amount in the relative pose from the world coordinate system to the camera coordinate system at the moment, are the characteristic points of the camera with the same name in the world coordinate system.

[0159] According to the characteristic points of the same name in the world coordinate system And the camera's position The relative translation from the world coordinate system to the camera coordinate system Determine the depth of the two selected feature points based on the following formula:

[0160]

[0161] in, is the depth point of the first feature point selected, is the depth point of the second feature point selected, is the first characteristic point of the same name point in the world coordinate system, is the second characteristic point of the same name point in the world coordinate system, ti The relative translation from the world coordinate system to the camera coordinate system at the moment, t j The relative translation from the world coordinate system to the camera coordinate system at the moment.

[0162] S604: Determine a visual point cloud corresponding to the camera according to the depth points and the same-name points of each registration feature point pair of the camera in the world coordinate system.

[0163] Optionally, the depth point of the first feature point based on the same name point and the depth point of the second feature point And the first feature point of the camera's point of the same name in the world coordinate system The second feature point with the same name as the point in the world coordinate system Determine the same-name points associated with the camera in the world coordinate system based on the following formula:

[0164]

[0165] in, are the same-name points of each pair of registration feature points in the world coordinate system, is the depth point of the first feature point of the same name point, is the depth point of the second feature point of the same name point, is the first feature point of the camera with the same name in the world coordinate system, It is the second feature point of the camera's point of the same name in the world coordinate system.

[0166] The same-name points of each registration feature point pair in the world coordinate system As the visual point cloud corresponding to the camera.

[0167] In this embodiment, according to the registration results of each frame image in the image sequence, each pair of registered feature points of the successfully matched image pair is associated to obtain the same-name points of each pair of registered feature points. The same-name points of each pair of registered feature points are converted to the spherical coordinate system by coordinate conversion to obtain the same-name points of each pair of registered feature points of the camera in the spherical coordinate system. According to the delay of the laser radar relative to the camera, the first relative rotation amount and the radar motion curve, the association between the camera and the laser radar is established to determine the position and posture of the camera. According to the same-name points of each pair of registered feature points of the camera in the spherical coordinate system and the position and posture of the camera, the two feature points with the farthest distance among the same-name points are selected to determine the same-name points and depth of each pair of registered feature points of the camera in the world coordinate system, and then determine the same-name points of each pair of registered feature points in the world coordinate system, and use the same-name points of each pair of registered feature points in the world coordinate system as the visual point cloud corresponding to the camera. The visual point cloud corresponding to the camera is determined by triangulating the same-name points of each pair of registered feature points.

[0168] The following is a detailed description of the process of constructing a voxel map and extracting surface features of the radar point cloud frame sequence based on the radar point cloud frame sequence and the laser radar's posture.

[0169] Figure 7 A schematic diagram of a flow chart of extracting surface features of a radar point cloud frame sequence in a laser radar and camera calibration method provided in an embodiment of the present application, such as Figure 7 As shown, the step of constructing a voxel map and extracting surface features of the radar point cloud frame sequence in the above step S202 according to the radar point cloud frame sequence and the laser radar posture includes:

[0170] S701. According to the position and posture of the laser radar, convert the radar point cloud frame sequence from the radar coordinate system to the world coordinate system to obtain the radar point cloud frame sequence in the world coordinate system.

[0171] Optionally, based on the laser radar’s position The radar point cloud frame sequence in the radar coordinate system is transformed by coordinate conversion Convert to the world coordinate system to obtain the radar point cloud frame sequence in the converted world coordinate system in, Represents the jth 3D scanning point in the world coordinate system collected by the LiDAR at time t.

[0172] S702: construct a voxel map according to the radar point cloud frame sequence in the world coordinate system and preset bounding box parameters.

[0173] Optionally, the bounding box parameters include the X coordinate and Y coordinate of the center of the bounding box and the width and height of the bounding box. According to the preset bounding box parameters, the radar point cloud frame sequence in the world coordinate system is Construct a voxel map, wherein the voxel map includes a plurality of voxels. Exemplarily, the preset bounding box parameters include an X coordinate iΔd of the center of the bounding box, a Y coordinate jΔd of the center of the bounding box, a width Δd of the bounding box, and a height Δd of the bounding box, wherein i and j are subscripts of the voxels.

[0174] S703 . Determine the radar point cloud mean and covariance of each voxel according to the voxel map.

[0175] Optionally, based on the constructed voxel map, the mean value of the radar point cloud inside each voxel in the voxel map is calculated using a function in an open source library for processing three-dimensional data. and covariance {Q i,j}. Among them, the covariance {Q i,j The matrix of} includes three eigenvalues, which are

[0176] S704. According to each eigenvalue of the covariance, the obtained face feature, the target minimum eigenvalue, and the eigenvector corresponding to the target minimum eigenvalue are marked.

[0177] Optionally, the covariance {Q i,j} are sorted and the target minimum eigenvalue is marked, where the target minimum eigenvalue is much smaller than the remaining eigenvalues ​​except the target minimum eigenvalue. Specifically, if Then it is marked as a face feature, and the voxel is denoted as {V i,j}, is the target minimum eigenvalue, and determines the target minimum eigenvalue The corresponding eigenvector N i,j .

[0178] In this embodiment, according to the position and posture of the laser radar, the radar point cloud frame sequence in the radar coordinate system is converted to the world coordinate system through coordinate transformation to obtain the radar point cloud frame sequence in the world coordinate system. According to the preset bounding box parameters, a voxel map is constructed for the radar point cloud frame sequence in the world coordinate system, and the radar point cloud mean and covariance of each voxel in the voxel map are calculated. According to each eigenvalue sorted by the covariance, the surface feature, the minimum eigenvalue of the target, and the eigenvector corresponding to the minimum eigenvalue of the target are marked. By constructing a voxel map for the radar point cloud frame sequence in the world coordinate system, the surface features of the radar point cloud frame sequence are extracted.

[0179] The following is a detailed description of the process of determining the surface feature constraints and calibrating the target visual point cloud, the target pose of the camera, and the target extrinsic parameters based on the motion association minimum objective function and the surface feature constraints.

[0180] Figure 8 Another schematic diagram of the process of obtaining the target visual point cloud, the target pose of the camera, and the target extrinsic parameters of the laser radar and camera calibration method provided in the embodiment of the present application is as follows: Figure 8 As shown, the step of determining the surface feature constraint in the above step S202, and calibrating the target visual point cloud, the target pose of the camera and the target extrinsic parameters according to the motion association minimum objective function and the surface feature constraint, includes:

[0181] S801, determining a surface feature constraint according to the optimized visual point cloud, the mean value of the voxel radar point cloud, and the eigenvector corresponding to the minimum eigenvalue of the target.

[0182] Optionally, based on the optimized visual point cloud The mean value of the radar point cloud for each voxel And the eigenvector corresponding to the minimum eigenvalue of each target {N i,j}, determine the surface feature constraints of each voxel as

[0183] S802, adding surface feature constraints to the motion association minimum objective function, calibrating the target visual point cloud, the target pose of the camera, and the target extrinsic parameters.

[0184] Optionally, the optimized visual point cloud in each voxel is The visual point cloud frames in are optimized for the second time. Specifically, if the optimized visual point cloud The optimized visual 3D points in Located in voxel V i,j In the case of motion association, the minimum objective function is Add voxels V i,j Face feature constraints Based on the following expression, the optimized visual point cloud Optimized camera pose And the optimized external parameter ΔT′ c_l Perform the second optimization process and calibrate the target visual point cloud The target pose of the camera And the target external parameter ΔT″ of the laser radar relative to the camera c_l :

[0185]

[0186] Among them, F θ is the camera projection parameter, To optimize the camera’s pose, Visual point cloud of the optimized camera The visual 3D points in t i Visual 3D points after constant optimization Associated feature points, ΔT′ c_l For the optimized external parameter, B l is the radar motion curve, δt is the delay of the lidar relative to the camera, V i,j The mean value of radar point cloud, N i,j is the eigenvector corresponding to the target minimum eigenvalue.

[0187] In this embodiment, the surface feature constraint of each voxel is determined based on the optimized visual point cloud, the mean value of the radar point cloud of the voxel, and the eigenvector corresponding to the minimum eigenvalue of the target. A second optimization process is performed on each optimized visual 3D point in the optimized visual point cloud in each voxel, and the surface feature constraint of each voxel is added to the motion-related minimum objective function to calibrate the target visual point cloud, the target pose of the camera, and the target extrinsic parameters. The calibration accuracy and robustness are further improved by adding the second optimization process of the surface feature constraint to the motion-related minimum objective function.

[0188] Based on the same inventive concept, the embodiments of the present application also provide a laser radar and camera calibration device corresponding to the laser radar and camera calibration method. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the laser radar and camera calibration method in the embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0189] Fig. 9 The module structure diagram of the laser radar and camera calibration device provided in the embodiment of the present application is as follows: Fig. 9 As shown, the device comprises:

[0190] The acquisition module 901 is used to acquire the camera projection parameters of the camera and the image sequence captured by the camera, and to acquire the radar point cloud frame sequence captured by the laser radar and the position and posture of the laser radar, wherein the camera is fixedly connected to the laser radar.

[0191] The calibration module 902 is used to calibrate the delay of the laser radar relative to the camera according to the image sequence and the posture of the laser radar, calibrate the first relative rotation amount in the external parameters of the laser radar relative to the camera, and determine the first relative translation amount in the initial external parameters.

[0192] The determination module 903 is used to determine the position and posture of the camera according to the delay of the laser radar relative to the camera and the first relative rotation amount, and determine the visual point cloud corresponding to the camera according to the position and posture of the camera.

[0193] The optimization processing module 904 is used to optimize the visual point cloud, the camera's pose and external parameters according to the visual point cloud, camera projection parameters, the camera's pose, external parameters, delay, radar point cloud frame sequence and lidar's pose to obtain the target visual point cloud, the camera's target pose and target external parameters.

[0194] As an optional implementation, the optimization processing module 904 is specifically used for:

[0195] According to the visual point cloud, camera projection parameters, camera pose, the first relative rotation amount in the external parameters and the delay, the motion correlation minimum objective function is constructed, and the optimized visual point cloud, optimized camera pose and optimized external parameters are calibrated.

[0196] According to the radar point cloud frame sequence and the posture of the lidar, a voxel map is constructed, the surface features of the radar point cloud frame sequence are extracted, the surface features are constrained, the surface feature constraints are determined, and the target visual point cloud, the target posture of the camera and the target extrinsic parameters are calibrated according to the motion association minimum objective function and the surface feature constraints.

[0197] As an optional implementation, the calibration module 902 is specifically used for:

[0198] According to the image sequence, the relative rotation amount between each frame image in the image sequence and the first frame image is determined.

[0199] The visual motion curve is constructed according to the relative rotation between each frame image and the first frame image, and the radar motion curve is constructed according to the position and posture of the lidar.

[0200] The visual motion curve and the radar motion curve are sampled respectively at a preset sampling time interval to determine the visual angular velocity and the radar angular velocity.

[0201] Based on the delay parameter and the sampling time interval, the delay of the lidar relative to the camera is determined.

[0202] As an optional implementation, the calibration module 902 is specifically used for:

[0203] According to the image sequence, feature points of each frame image are extracted, and image registration is performed on each frame image according to the feature points of each frame image to obtain each pair of registered feature points of the successfully matched image pairs.

[0204] According to each pair of registered feature points of the image pair, a relative rotation amount of the camera from a first moment in the image pair to a second moment in the image pair is determined.

[0205] According to the relative rotation of the camera from the first moment in the image pair to the second moment in the image pair, a visual pose graph is constructed to determine the relative rotation of each frame image in the image sequence with respect to the first frame image.

[0206] As an optional implementation, the calibration module 902 is specifically used for:

[0207] Based on the radar motion curve and the delay of the lidar relative to the camera, the relative rotation amount of the lidar from the first moment in the image pair to the second moment in the image pair is determined.

[0208] According to the relative rotation amounts of the laser radar from the first moment to the second moment in each image pair and the relative rotation amounts of the camera from the first moment to the second moment in each image pair, the first relative rotation amount in the external parameters of the laser radar relative to the camera is obtained by calibrating with the hand-eye calibration algorithm.

[0209] As an optional implementation manner, the determination module 903 is specifically configured to:

[0210] According to the registration results of each frame image in the image sequence, the same-name points of each registration feature point pair are obtained, and the same-name points of each registration feature point pair are triangulated to obtain the same-name points of each registration feature point pair of the camera in the spherical coordinate system.

[0211] The position and posture of the camera are determined according to the delay of the laser radar relative to the camera, the first relative rotation amount, and the radar motion curve.

[0212] According to the same-name points of each pair of registration feature points of the camera in the spherical coordinate system and the position and posture of the camera, the same-name points and depth points of each pair of registration feature points of the camera in the world coordinate system are determined.

[0213] According to the depth points and the same-name points of each registration feature point pair of the camera in the world coordinate system, the visual point cloud corresponding to the camera is determined.

[0214] As an optional implementation, the optimization processing module 904 is specifically used for:

[0215] According to the position and posture of the laser radar, the radar point cloud frame sequence is converted from the radar coordinate system to the world coordinate system to obtain the radar point cloud frame sequence in the world coordinate system.

[0216] A voxel map is constructed based on the radar point cloud frame sequence in the world coordinate system and the preset bounding box parameters.

[0217] According to the voxel map, the mean and covariance of the radar point cloud of each voxel are determined.

[0218] According to each eigenvalue of the covariance, the surface feature, the target minimum eigenvalue, and the eigenvector corresponding to the target minimum eigenvalue are marked.

[0219] As an optional implementation, the processing module 904 is specifically used for:

[0220] The surface feature constraints are determined based on the optimized visual point cloud, the mean value of the voxel radar point cloud, and the eigenvector corresponding to the minimum eigenvalue of the target.

[0221] Add surface feature constraints to the motion association minimum objective function to calibrate the target visual point cloud, the target pose of the camera, and the target extrinsic parameters.

[0222] The present application also provides a computer device, such as Fig.10 1 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, including: a processor 101, a memory 102 and a bus 103. The memory 102 stores machine-readable instructions executable by the processor 101 (for example, Fig. 9 In the device, the execution instructions corresponding to the acquisition module 901, the calibration module 902, the determination module 903 and the optimization processing module 904 are obtained, etc. When the computer device is running, the processor 101 communicates with the memory 102 through the bus 103. When the machine-readable instructions are executed by the processor 101, the steps of the calibration method of the laser radar and the camera in the above-mentioned embodiment are executed.

[0223] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the laser radar and camera calibration method in the above embodiment are executed.

[0224] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0225] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0226] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A laser radar and camera calibration method, characterized in that: include: Acquire camera projection parameters of a camera and an image sequence captured by the camera, and acquire a radar point cloud frame sequence captured by a laser radar and a position and posture of the laser radar, wherein the camera is fixedly connected to the laser radar; According to the image sequence and the position and posture of the laser radar, calibrate the delay of the laser radar relative to the camera, calibrate the first relative rotation amount of the laser radar relative to the camera in the external parameters, and determine the initial first relative translation amount of the external parameters; Determine the position and posture of the camera according to the delay of the laser radar relative to the camera and the first relative rotation amount, and determine the visual point cloud corresponding to the camera according to the position and posture of the camera; According to the visual point cloud, the camera projection parameters, the camera pose, the extrinsic parameters, the delay, the radar point cloud frame sequence and the lidar pose, the visual point cloud, the camera pose and the extrinsic parameters are optimized to obtain a target visual point cloud, a target pose of the camera and target extrinsic parameters.

2. The method according to claim 1, characterized in that: The method comprises: optimizing the visual point cloud, the camera's position and posture, the external parameters, the delay, the radar point cloud frame sequence, and the laser radar's position and posture to obtain a target visual point cloud, a target position and posture of the camera, and a target external parameter, including: According to the visual point cloud, the camera projection parameters, the camera posture, the first relative rotation amount in the external parameters and the delay, a motion correlation minimum objective function is constructed, and the optimized visual point cloud, the optimized camera posture and the optimized external parameters are calibrated; According to the radar point cloud frame sequence and the posture of the laser radar, a voxel map is constructed, the surface features of the radar point cloud frame sequence are extracted, the surface features are constrained, the surface feature constraints are determined, and according to the motion association minimum objective function and the surface feature constraints, the target visual point cloud, the target posture of the camera and the target extrinsic parameters are calibrated.

3. The method according to claim 1, characterized in that: The step of calibrating the delay of the laser radar relative to the camera according to the image sequence and the position and posture of the laser radar includes: According to the image sequence, determining the relative rotation amount between each frame image and the first frame image in the image sequence; According to the relative rotation amount between each frame image and the first frame image, a visual motion curve is constructed, and according to the position and posture of the laser radar, a radar motion curve is constructed; The visual motion curve and the radar motion curve are sampled respectively at a preset sampling time interval to determine a visual angular velocity and a radar angular velocity; Determining a delay parameter according to the visual angular rate, the radar angular rate, and the sampling time interval; The delay of the laser radar relative to the camera is determined according to the delay parameter and the sampling time interval.

4. The method according to claim 3, characterized in that Determining the relative rotation amount between each frame image and the first frame image in the image sequence according to the image sequence includes: Extracting feature points of each frame image according to the image sequence, and performing image registration on each frame image according to the feature points of each frame image to obtain each pair of registered feature points of the successfully matched image pairs; Determining, based on each pair of registration feature points of the image pair, a relative rotation amount of the camera from a first moment in the image pair to a second moment in the image pair; According to the relative rotation amount of the camera from the first moment in the image pair to the second moment in the image pair, a visual pose graph is constructed to determine the relative rotation amount of each frame image in the image sequence with respect to the first frame image.

5. The method according to claim 4, characterized in that The calibration obtains a first relative rotation amount in the external parameters of the laser radar relative to the camera, including: Determining a relative rotation amount of the laser radar from a first moment in the image pair to a second moment in the image pair based on the radar motion curve and a delay of the laser radar relative to the camera; According to the relative rotation amounts of the laser radar from the first moment in each image pair to the second moment in each image pair and the relative rotation amounts of the camera from the first moment in each image pair to the second moment in each image pair, the first relative rotation amount in the external parameters of the laser radar relative to the camera is obtained by calibrating with a hand-eye calibration algorithm.

6. The method according to claim 1, characterized in that Determining the position and posture of the camera according to the delay of the laser radar relative to the camera and the first relative rotation amount, and determining the visual point cloud corresponding to the camera according to the position and posture of the camera, includes: According to the registration results of each frame image in the image sequence, the same-name points of each registration feature point pair are obtained, and the same-name points of each registration feature point pair are triangulated to obtain the same-name points of each registration feature point pair of the camera in the spherical coordinate system; Determine the position and posture of the camera according to the delay of the laser radar relative to the camera, the first relative rotation amount, and the radar motion curve; Determine the same-name points and depth points of each pair of registration feature points of the camera in the world coordinate system according to the same-name points of each pair of registration feature points of the camera in the spherical coordinate system and the position and posture of the camera; The visual point cloud corresponding to the camera is determined according to the depth points and the same-name points of each registration feature point pair of the camera in the world coordinate system.

7. The method according to claim 2, characterized in that: The step of constructing a voxel map according to the radar point cloud frame sequence and the position and posture of the laser radar and extracting surface features of the radar point cloud frame sequence includes: According to the position and posture of the laser radar, the radar point cloud frame sequence is converted from the radar coordinate system to the world coordinate system to obtain the radar point cloud frame sequence in the world coordinate system; Constructing a voxel map according to the radar point cloud frame sequence in the world coordinate system and preset bounding box parameters; Determining the radar point cloud mean and covariance of each voxel according to the voxel map; According to each eigenvalue of the covariance, the obtained face feature, the target minimum eigenvalue and the eigenvector corresponding to the target minimum eigenvalue are marked.

8. The method according to claim 2, characterized in that: The determining of the surface feature constraint and calibrating the target visual point cloud, the target pose of the camera and the target external parameters according to the motion association minimum objective function and the surface feature constraint include: Determining a surface feature constraint according to the optimized visual point cloud, the mean value of the radar point cloud of the voxel, and a eigenvector corresponding to a minimum eigenvalue of the target; The surface feature constraint is added to the motion association minimum objective function to calibrate the target visual point cloud, the target pose of the camera and the target extrinsic parameters.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the laser radar and camera calibration method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the laser radar and camera calibration method as described in any one of claims 1 to 8 are executed.

Images