A laser radar and camera extrinsic parameter calibration method and device based on deep learning

Abstract

A method, device, and computer equipment for calibrating external parameters of a laser radar and a camera based on deep learning provided in an embodiment of the present invention utilizes an hourglass network to use skip layers to retain spatial information of each resolution, effectively processes and integrates features of different scales, and can effectively utilize features at each level to discover new features again by stacking multiple hourglass networks. A core selection network is introduced to cascade the deep features extracted by the two on the channel, and then multi-scale information is extracted from the cascaded depth map and image features, and feature aggregation is performed adaptively. The aggregated features are regressed through two fully connected branches to obtain the translation component and rotation component of the external parameters between the laser radar and the camera, which improves the calibration speed, reduces labor costs and experimental costs, and overcomes the dependence on specific calibration experimental scenes and specific experimental goals. From the experimental results, the calibration function is well implemented and good experimental results are achieved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a laser radar and camera extrinsic parameter calibration method and device based on deep learning, a computer device, and a non-transient computer-readable storage medium storing computer instructions. Background Art

[0002] In the past period of time, autonomous driving has become a very popular development direction, and the perception system, as the eyes of the autonomous driving system, is of great importance. The real environment is complex and changeable, and it is difficult for a perception system based on a single sensor to cope with it, so multi-sensor fusion has become a solution for many perception systems, especially the data fusion between LiDAR that can provide sparse 3D information and cameras with dense 2D information. Calibration is a basic problem in the field of computer vision and robotics. For example, in mobile platforms such as unmanned vehicles that are equipped with both LiDAR and cameras, calibration between LiDAR and cameras is required. The basis for the fusion of radar point cloud data and camera image data is to obtain accurate external parameters between LiDAR and camera through calibration. Summary of the invention

[0003] In view of this, embodiments of the present invention provide a deep learning-based laser radar and camera extrinsic parameter calibration method, apparatus, computer equipment, and a non-transitory computer-readable storage medium storing computer instructions.

[0004] In a first aspect, an embodiment of the present invention provides a method for calibrating laser radar and camera extrinsic parameters based on deep learning, the method comprising:

[0005] Obtain image data collected by the camera and point cloud data collected by the lidar;

[0006] Projecting the point cloud data onto the image plane of the camera to form a depth map;

[0007] Extracting first deep features of the image data and second deep features of the depth map respectively using a stacked hourglass network;

[0008] Using a kernel selection network to perform channel cascading on the first deep feature and the second deep feature to obtain a cascade feature, performing feature extraction on the cascade feature through convolution kernels with different receptive fields, determining the feature correlation between the depth map and the image data under different receptive fields, and using an attention mechanism to adaptively select different feature aggregation channels to obtain an aggregated feature;

[0009] The aggregated features are regressed through a fully connected layer to obtain the translation component and rotation component of the external parameters between the laser radar and the camera to complete the calibration.

[0010] As an optional solution, projecting the point cloud data onto the image plane of the camera to form a depth map includes:

[0011] Given the initial external parameter matrix T between the laser radar and the camera init And the camera internal parameter matrix K, each point P in the point cloud data i (x i ,y i ,z i ) are projected onto the image plane to form a depth map D, and the pixel coordinates corresponding to the depth map D are p i (u i , v i ), the pixel value is Z i , the projection process is as follows:

[0012]

[0013]

[0014] in, is the pixel coordinate P i The homogeneous coordinates of .

[0015] As an optional solution, it also includes:

[0016] Each frame of point cloud data is passed through the true value external parameter T LC Convert to the camera space coordinate system, then multiply by the random perturbation matrix △T and the camera internal parameters to convert into a depth map. At this time, the initial external parameter matrix corresponding to each frame of point cloud data is T init =△T*T LC .

[0017] As an optional solution, regressing the aggregated features through a fully connected layer to obtain the translation component and the rotation component of the external parameters between the laser radar and the camera includes:

[0018] The aggregated features are passed through a fully connected layer, and the output is a 1×3 translation vector t pred and the 1×4 rotation quaternion q pred =(q0,q1,q2,q3), the conversion formula between quaternion and rotation matrix:

[0019]

[0020] Set two loss functions as regression loss L T And point cloud distance loss L p :

[0021] L=λ T L T +λ pL p (4)

[0022] Among them, the regression loss L T The translation component loss L t and the rotational component loss L R Two parts, L t t pred With t gt The smooth L1 loss between t gt and q gt Corresponding to the added random rotation error and translation error, L R Calculate q gt =(q0,q1,q2,q3) and q pred =(q'0,q1',q'2,q'3), where λ T ,λ p All are experience values;

[0023]

[0024] L T =λ t L t +λ q L R (6)

[0025] Point cloud P = {P1, ...P N}, P i ∈R 3 Point cloud distance loss L p The calculation formula is as follows:

[0026]

[0027] As an optional solution, it also includes:

[0028] Evaluate the extrinsic parameters between the calibrated lidar and camera;

[0029] The inverse of the matrix obtained by regression Multiply by △T to obtain a new matrix, extract the translation vector t1 of the new matrix, and add the translation vector t1 and the true value translation vector t of the new matrix gt =[0,0,0] The absolute value of the difference between each component is used as the evaluation index E of the translation component t_i , i∈(x,y,z):

[0030] E t_i =|t 1_i -t gt_i | (8)

[0031] Transform the rotation matrix into Euler angles and calculate the angle error E of each axis respectively R_j , j∈(α,β,γ):

[0032] E R_j =|R 1_j -R gt_j | (9).

[0033] In a second aspect, an embodiment of the present invention provides a laser radar and camera extrinsic parameter calibration device based on deep learning, including:

[0034] An acquisition module is used to acquire image data collected by the camera and point cloud data collected by the laser radar;

[0035] A preprocessing module, used for projecting the point cloud data onto the image plane of the camera to form a depth map;

[0036] A feature extraction module, configured to respectively extract first deep features of the image data and second deep features of the depth map using a stacked hourglass network;

[0037] A feature aggregation module, used to use a core selection network to perform channel cascade on the first deep feature and the second deep feature to obtain a cascade feature, perform feature extraction on the cascade feature through convolution kernels with different receptive fields, determine the feature correlation between the depth map and the image data under different receptive fields, and use an attention mechanism to adaptively select different feature aggregation channels to obtain an aggregated feature;

[0038] The regression module is used to regress the aggregated features through a fully connected layer to obtain the translation component and rotation component of the external parameters between the laser radar and the camera to complete the calibration.

[0039] In a third aspect, an embodiment of the present invention provides a computer device, including:

[0040] at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned deep learning-based lidar and camera extrinsic parameter calibration method.

[0043] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the above-mentioned deep learning-based laser radar and camera extrinsic parameter calibration method.

[0044] In a fifth aspect, a computer program product is provided in an embodiment of the present invention, including a computer program, which, when executed by a processor, implements the above-mentioned deep learning-based laser radar and camera extrinsic parameter calibration method.

[0045] A method, device, computer equipment and non-transient computer-readable storage medium storing computer instructions for calibrating external parameters of a laser radar and a camera based on deep learning provided in an embodiment of the present invention utilizes an hourglass network to use skip layers to retain spatial information of each resolution, effectively processes and integrates features of different scales, and can effectively utilize features of each level to discover new features again by stacking multiple hourglass networks. A core selection network is introduced to cascade the deep features extracted by the two on the channel, and then multi-scale information is extracted from the cascaded depth map and image features, and feature aggregation is performed adaptively. The aggregated features are regressed through two fully connected branches to obtain the translation component and rotation component of the external parameters between the laser radar and the camera, which improves the calibration speed, reduces labor costs and experimental costs, and overcomes the dependence on specific calibration experimental scenes and specific experimental goals; from the experimental results, the calibration function is well realized and good experimental results are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of a laser radar and camera extrinsic parameter calibration method based on deep learning in an embodiment of the present invention;

[0047] Figure 2 It is a structural schematic diagram of an hourglass network in a laser radar and camera extrinsic parameter calibration method based on deep learning in an embodiment of the present invention;

[0048] Figure 3A It is a structural schematic diagram of a stacked hourglass network in a laser radar and camera extrinsic parameter calibration method based on deep learning provided in an embodiment of the present invention;

[0049] Figure 3B It is a schematic diagram of a core selection network structure in a laser radar and camera extrinsic calibration method based on deep learning in an embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of a neural network structure in a laser radar and camera extrinsic parameter calibration method based on deep learning provided in an embodiment of the present invention;

[0051] Figure 5A It is a schematic diagram of the fusion effect of the depth map and the RGB image before calibration in a laser radar and camera extrinsic parameter calibration method based on deep learning provided in an embodiment of the present invention;

[0052] Figure 5BIt is a schematic diagram of the fusion effect of the true depth map and the RGB image in a laser radar and camera extrinsic calibration method based on deep learning in an embodiment of the present invention;

[0053] Figure 5C It is a schematic diagram of the fusion effect of the depth map and the RGB image after calibration in a laser radar and camera extrinsic calibration method based on deep learning in an embodiment of the present invention;

[0054] Figure 6 It is a structural block diagram of a laser radar and camera extrinsic parameter calibration device based on deep learning in an embodiment of the present invention;

[0055] Figure 7 It is a structural block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] The terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0058] Combination Figure 1 As shown, an embodiment of the present invention provides a laser radar and camera extrinsic parameter calibration method based on deep learning, and the method includes:

[0059] S101, obtaining image data collected by a camera and point cloud data collected by a laser radar.

[0060] Point cloud data is three-dimensional and describes the shape, size, depth and other features of the scanned object. The image data collected by the camera is an RGB image. In this embodiment, the KITTI dataset is used. The KITTI dataset was jointly created by the Karlsruhe Institute of Technology and Toyota Research Institute of America. It is currently the largest computer vision algorithm evaluation dataset in the world for autonomous driving scenarios. This dataset is used to evaluate the performance of computer vision technologies such as stereo, optical flow, visual odometry, 3D object detection and 3D tracking in vehicle environments. KITTI contains real image data collected in scenes such as urban areas, rural areas and highways. Each image contains up to 15 vehicles and 30 pedestrians, as well as various degrees of occlusion and truncation. The entire dataset consists of 389 pairs of stereo images and optical flow maps, 39.2km of visual odometry sequences and more than 200k images of 3D annotated objects [1], sampled and synchronized at a frequency of 10Hz. In general, the original dataset is classified into 'Road', 'City', 'Residential', 'Campus' and 'Person'. For 3D object detection, the labels are subdivided into car, van, truck, pedestrian, pedestrian (sitting), cyclist, tram and misc. The point cloud in this dataset only contains the 3D coordinates of each point without the corresponding color information.

[0061] S102: Project the point cloud data onto an image plane of the camera to form a depth map.

[0062] Because point cloud data and image data are data of different modalities, point cloud data is much sparser than images. In the Kitti dataset, the RGB image resolution is about 1280×384 and the number of pixels is about 490,000. The corresponding effective number of points in a point cloud data sample is about 100,000. Projecting it onto the image plane, we get a depth map. There will be 100,000 pixels corresponding to the point cloud points on the depth map, and the remaining 390,000 pixels are 0. The sparsity of the depth map can be seen. The key to the calibration algorithm is to obtain the correlation between the depth map and the RGB image features. Therefore, in order to improve the accuracy of the calibration algorithm, we should fully extract the sparse depth map features and the cascaded depth map and RGB image features to obtain a more accurate correlation between the depth map features and the RGB image features.

[0063] In this embodiment, each frame of point cloud data is passed through the true value external parameter T LCConvert to the camera space coordinate system, then multiply by the random perturbation matrix △T and the camera internal parameters to convert into a depth map. At this time, the initial external parameter matrix corresponding to each frame of point cloud data is T init =△T*T LC Given the initial external parameter matrix T between the laser radar and the camera init And the camera internal parameter matrix K, each point P in the point cloud data i (x i ,y i , z i ) are projected onto the image plane to form a depth map D, and the pixel coordinates corresponding to the depth map D are P i (u i , v i ), the pixel value is Z i , the projection process is as follows:

[0064]

[0065]

[0066] in, is the pixel coordinate P i The homogeneous coordinates of .

[0067] S103: Use a stacked hourglass network to respectively extract first deep features of the image data and second deep features of the depth map.

[0068] like Figure 2 As shown, the hourglass network can use skip layers to retain the spatial information of each resolution, effectively process and integrate features of different scales, and by stacking multiple hourglass networks, it can effectively utilize features at each level to discover new features again. In this embodiment, the hourglass network is composed of Conv3*3, stride=2 and Conv3*3, stride=1.

[0069] like Figure 3A As shown, the stacked hourglass network consists of two symmetrical branches, that is, two hourglass networks are symmetrically composed, and the stacked hourglass network is used to extract the first deep features of the image data and the second deep features of the depth map respectively.

[0070] S104. Use a kernel selection network to perform channel cascade on the first deep feature and the second deep feature to obtain a cascade feature, perform feature extraction on the cascade feature through convolution kernels with different receptive fields, determine the feature correlation between the depth map and the image data under different receptive fields, and use an attention mechanism to adaptively select different feature aggregation channels to obtain an aggregated feature.

[0071] like Figure 3BAs shown in the figure, the Selective Kernel Networks (SKNet) implements an attention mechanism on the convolution kernel, that is, the network selects the appropriate convolution kernel by itself. Because a too large receptive field will ignore small useful information, and a too small receptive field will obtain too much local information, resulting in the loss of global information, so in order to obtain a more accurate feature correlation between the depth map and the image, the Selective Kernel Network cascades the first deep features and the second deep features extracted by the two on the channel, and then extracts features from the cascaded features through convolution kernels with different receptive fields, calculates the feature correlation between the depth map and the image under different receptive fields, uses the attention mechanism to adaptively select different feature aggregation channels to obtain aggregated features, cascades the deep features extracted by the two on the channel, and then extracts multi-scale information from the cascaded depth map and image features, and adaptively performs feature aggregation.

[0072] S105, regressing the aggregated features through a fully connected layer to obtain the translation component and rotation component of the external parameters between the laser radar and the camera, and completing the calibration.

[0073] Combination Figure 4 As shown in FIG. 1 , the neural network structure of the present method is shown. The aggregated features are passed through a fully connected layer, and the output is a 1×3 translation vector t pred and the 1×4 rotation quaternion q pred =(q0,q1,q2,q3), the conversion formula between quaternion and rotation matrix:

[0074]

[0075] Set two loss functions as regression loss L T And point cloud distance loss L p :

[0076] L=λ T L T +λ p L p (4)

[0077] Among them, the regression loss L T The translation component loss L t and the rotational component loss L R Two parts, L t t pred With t gt The smooth L1 loss between t gt and q gt Corresponding to the added random rotation error and translation error, L R Calculate q gt =(q0,q1,q2,q3) and q pred=(q'0,q1',q'2,q'3), where λ T ,λ p All are experience values;

[0078]

[0079] L T =λ t L t +λ q L R (6)

[0080] Point cloud P = {P1, ...P N}, P i ∈R 3 Point cloud distance loss L p The calculation formula is as follows:

[0081]

[0082] The aggregated features are regressed through two fully connected branches to obtain the translation and rotation components of the external parameters between the lidar and the camera. Experimental results show that this method improves the calibration accuracy.

[0083] A deep learning-based laser radar and camera extrinsic parameter calibration method provided in an embodiment of the present invention utilizes an hourglass network to use skip layers to retain spatial information of each resolution, effectively processes and integrates features of different scales, and can effectively utilize features at all levels to discover new features again by stacking multiple hourglass networks. A core selection network is introduced to cascade the deep features extracted by the two on the channel, and then multi-scale information is extracted from the cascaded depth map and image features, and feature aggregation is performed adaptively. The aggregated features are regressed through two fully connected branches to obtain the translation component and rotation component of the extrinsic parameters between the laser radar and the camera, which improves the calibration speed, reduces labor costs and experimental costs, and overcomes the dependence on specific calibration experimental scenarios and specific experimental goals; from the experimental results, the calibration function is well implemented and good experimental results are achieved.

[0084] In order to further verify the laser radar and camera extrinsic calibration method based on deep learning provided in the embodiment of the present invention, evaluation indicators are used for evaluation, specifically including:

[0085] The inverse of the matrix obtained by regression Multiply by △T to obtain a new matrix, extract the translation vector t1 of the new matrix, and add the translation vector t1 and the true value translation vector t of the new matrix gt =[0,0,0] The absolute value of the difference between each component is used as the evaluation index E of the translation component t_i , i∈(x,y,z):

[0086] E t_i =|t 1_i -t gt_i | (8)

[0087] To evaluate the rotation component, we transform the rotation matrix into Euler angles and calculate the angular error E for each axis separately. R_j , j∈(α,β,γ):

[0088] E R_j =|R 1_j -R gt_j | (9).

[0089] Experimental Results

[0090] Set Kitti dataset sequence00 as the test set. In this paper, we train 5 pre-trained models by adding random errors of (±1.5m, ±20°), (±1.0m, ±10°), (±0.5m, ±5°), (±0.2m, ±2°), and (±0.1m, ±1°), and then optimize the calibration results through iteration.

[0091] First, the point cloud of each frame of sequence00 P={P1,...P N}, P i ∈R 3 Multiply by the true value external parameter T LC Transfer to the camera coordinate system, then add a random error between [-1.5, 1.5] meters to the translation component; add a random error △T between [-20°, 20°] to the rotation component, multiply the point cloud data by △T to get P'={P'1,...P' N}, P′ i ∈R 3 , multiplied by the camera intrinsic parameter K to form the miscalibrated depth map Depth1;

[0092] The miscalibrated depth map and the corresponding RGB image are input into the pre-trained model (±1.5m, ±20°) to obtain T0, and P'={P1′,...P′ N}, P i '∈R 3 Multiply by (T0) -1 Get P" = {P1", ...P" N}, P i ″∈R 3 , multiplied by the camera intrinsic parameter K to form the miscalibrated depth map Depth2; after 5 iterations, we get:

[0093]

[0094] Substituting the evaluation index, the true value matrix is ​​the unit matrix E, and the experimental results are shown in Table 1:

[0095] <![CDATA[E t_x (cm)]]> <![CDATA[E t_y (cm)]]> <![CDATA[E t_z (cm)]]> <![CDATA[E R_α (°)]]> <![CDATA[E R_β (°)]]> <![CDATA[E R_γ (°)]]> Before calibration 23.0694 40.6102 114.2441 7.6107 7.0900 11.0167 After calibration 12.7356 13.2116 11.6163 0.0815 0.0576 0.0831

[0096] Table 1 Experimental results

[0097] Combination Figure 5A , 5B As shown in 5C, Figure 5A It is the fusion effect of the depth map and RGB image before calibration. Figure 5B It is the fusion effect of the true depth map and the RGB image. Figure 5C It is the fusion effect of the depth map and RGB image after calibration. From the experimental results, we can see that the errors of each translation component before calibration are all above 23cm, and the largest deviation is above 1m, but after the calibration algorithm, the errors of each translation component are reduced to about 11cm; the errors of each rotation component before calibration are all above 7°, and the largest deviation is above 11°, but after the calibration of the algorithm, the errors of each rotation component are reduced to about 0.08°, which well realizes the function of external parameter calibration between lidar and camera in the kitti dataset. The advantages of this technology: it improves the calibration speed, reduces the labor cost and experimental cost, and overcomes the dependence on specific calibration experimental scenes and specific experimental targets; from the experimental results, the calibration function is well realized and good experimental results are achieved.

[0098] An embodiment of the present invention also provides a method for calibrating laser radar and camera extrinsic parameters based on deep learning, including:

[0099] S501. Point cloud data preprocessing

[0100] Given the initial external parameter matrix Tinit between the laser radar and the camera, and the camera internal parameter matrix K, each point P in the point cloud data i (x i ,y i , z i ) are projected onto the image plane to form a depth map D, and its corresponding pixel coordinates are P i (u i , v i ), the pixel value is Z i , the projection process is as follows:

[0101]

[0102]

[0103] Here is the pixel coordinate P i The homogeneous coordinates of .

[0104] S502 Feature Extraction

[0105] The feature extraction uses a stacked hourglass network, which uses skip layers to retain spatial information at each resolution, effectively processing and integrating features of different scales. By stacking multiple hourglass networks, it can effectively utilize features at each level to discover new features again. The stacked hourglass network consists of two symmetrical branches, which are used to extract features of the image and depth map respectively.

[0106] S503, feature aggregation

[0107] Because a too large receptive field will ignore small useful information, while a too small receptive field will obtain too much local information, resulting in the loss of global information, so in order to obtain a more accurate feature correlation between the depth map and the image, the Selective Kernel Networks (SKNet) cascades the deep features extracted by the two on the channel, and then extracts the features of the cascade through convolution kernels with different receptive fields, calculates the feature correlation between the depth map and the image under different receptive fields, and uses the attention mechanism to adaptively select different feature aggregation channels.

[0108] S504, external reference regression

[0109] Through a fully connected layer, the output is a 1×3 translation vector t pred and the 1×4 rotation quaternion q pred =(q0,q1,q2,q3);

[0110] The conversion formula between quaternion and rotation matrix:

[0111]

[0112] S505. Loss function

[0113] We set two loss functions: regression loss L T And point cloud distance loss L p

[0114] L=λ T L T +λ p L p (4)

[0115] Among them, the regression loss L T The translation component loss L t and the rotational component loss L R Two parts, L t That is t pred With t gt The smooth L1 loss between gt and qgt Corresponding to the added random rotation error and translation error, L R Calculate q gt =(q0,q1,q2,q3) and q pred =(q'0,q'1,q'2,q'3) As shown in formula (5), after experiments, the λ T ,λ p Both are set to 0.5.

[0116]

[0117] L T =λ t L t +λ q L R (6)

[0118] Point cloud P = {P1, ...P N}, P i ∈R 3 Point cloud distance loss L p The calculation formula is as shown in formula (7). After experiments, the λ t Set to 1, λ p Set to 2:

[0119]

[0120] S506. Evaluation indicators

[0121] The inverse of the matrix obtained by regression Multiply by △T to obtain a new matrix, extract the translation vector t1 of the new matrix, and add the translation vector t1 and the true value translation vector t of the new matrix gt =[0,0,0] The absolute value of the difference between each component is used as the evaluation index E of the translation component t_i , i∈(x,y,z):

[0122] E t_i =|t 1_i -t gt_i | (8)

[0123] To evaluate the rotation component, we transform the rotation matrix into Euler angles and calculate the angular error E for each axis separately. R_j , j∈(α,β,γ):

[0124] E R_j =|R 1_j -R gt_j | (9)

[0125] Experimental Results

[0126] Set Kitti dataset sequence00 as the test set. In this paper, we train 5 pre-trained models by adding random errors of (±1.5m, ±20°), (±1.0m, ±10°), (±0.5m, ±5°), (±0.2m, ±2°), and (±0.1m, ±1°), and then optimize the calibration results through iteration.

[0127] 1. First, the point cloud of each frame of sequence00 P={P1,...P N}, P i ∈R 3 Multiply by the true value external parameter T LC Transfer to the camera coordinate system, then add a random error between [-1.5, 1.5] meters to the translation component; add a random error △T between [-20°, 20°] to the rotation component, multiply the point cloud data by △T to get, P = {P1, ...P N}, P i ∈R 3 Multiply it by the camera intrinsic parameter K to form the miscalibrated depth map Depth1;

[0128] 2. Input the miscalibrated depth map and the corresponding RGB image (±1.5m, ±20°) into the pre-trained model to obtain T0, and set P'={P1', ...P' N}, P i '∈R 3 Multiply by (T0) -1 Get P" = {P1", ...P" N}, P i ”∈R 3 , multiplied by the camera intrinsic parameter K to form the miscalibrated depth map Depth2; after 5 iterations, we get:

[0129]

[0130] Substituting the evaluation indicators, the true value matrix is ​​the unit matrix E, and the experimental results are obtained. Experiments have proved that this method improves the calibration speed, reduces labor costs and experimental costs, and overcomes the dependence on specific calibration experimental scenarios and specific experimental goals; from the experimental results, the calibration function is well realized and good experimental results are achieved.

[0131] Combination Figure 6 As shown, an embodiment of the present invention provides a laser radar and camera extrinsic parameter calibration device 600 based on deep learning, including:

[0132] An acquisition module 601 is used to acquire image data collected by a camera and point cloud data collected by a laser radar;

[0133] A pre-processing module 602, configured to project the point cloud data onto an image plane of the camera to form a depth map;

[0134] A feature extraction module 603, configured to respectively extract first deep features of the image data and second deep features of the depth map using a stacked hourglass network;

[0135] A feature aggregation module 604 is used to use a core selection network to perform channel cascade on the first deep feature and the second deep feature to obtain a cascade feature, perform feature extraction on the cascade feature through convolution kernels with different receptive fields, determine the feature correlation between the depth map and the image data under different receptive fields, and use an attention mechanism to adaptively select different feature aggregation channels to obtain an aggregated feature;

[0136] The regression module 605 is used to regress the aggregated features through a fully connected layer to obtain the translation component and rotation component of the external parameters between the laser radar and the camera to complete the calibration.

[0137] A laser radar and camera extrinsic parameter calibration device based on deep learning provided in an embodiment of the present invention utilizes an hourglass network to use skip layers to retain spatial information of each resolution, effectively processes and integrates features of different scales, and can effectively utilize features at each level to discover new features again by stacking multiple hourglass networks. A core selection network is introduced to cascade the deep features extracted by the two on the channel, and then multi-scale information is extracted from the cascaded depth map and image features, and feature aggregation is adaptively performed. The aggregated features are regressed through two fully connected branches to obtain the translation component and rotation component of the extrinsic parameters between the laser radar and the camera, which improves the calibration speed, reduces labor costs and experimental costs, and overcomes the dependence on specific calibration experimental scenes and specific experimental goals; from the experimental results, the calibration function is well realized and good experimental results are achieved.

[0138] Accordingly, according to an embodiment of the present invention, the present invention also provides a computer device, a readable storage medium and a computer program product.

[0139] Figure 7 FIG. 1 is a schematic diagram of the structure of a computer device 12 provided in an embodiment of the present invention. Figure 7 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 7 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0140] like Figure 7As shown, computer device 12 is in the form of a general-purpose computing device. Computer device 12 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Computer devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0141] Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16 , a system memory 28 , and a bus 18 that connects various system components including system memory 28 and processing unit 16 .

[0142] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0143] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0144] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0145] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0146] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the computer device 12, and / or may communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0147] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, for example, implementing the laser radar and camera extrinsic parameter calibration method based on deep learning provided in the embodiment of the present invention:

[0148] An embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein when the program is executed by a processor, the deep learning-based laser radar and camera extrinsic parameter calibration method provided in all the inventive embodiments of the present application.

[0149] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0150] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0151] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0152] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0153] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned deep learning-based laser radar and camera extrinsic parameter calibration method.

[0154] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0155] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A laser radar and camera extrinsic calibration method based on deep learning, characterized in that: The method comprises: Obtain image data collected by the camera and point cloud data collected by the lidar; Projecting the point cloud data onto the image plane of the camera to form a depth map; Extracting first deep features of the image data and second deep features of the depth map respectively using a stacked hourglass network; Using a kernel selection network to perform channel cascading on the first deep feature and the second deep feature to obtain a cascade feature, performing feature extraction on the cascade feature through convolution kernels with different receptive fields, determining the feature correlation between the depth map and the image data under different receptive fields, and using an attention mechanism to adaptively select different feature aggregation channels to obtain an aggregated feature; The aggregated features are regressed through a fully connected layer to obtain the translation component and rotation component of the external parameters between the laser radar and the camera to complete the calibration.

2. The laser radar and camera extrinsic calibration method based on deep learning according to claim 1, characterized in that: The step of projecting the point cloud data onto the image plane of the camera to form a depth map comprises: Given the initial external parameter matrix T between the laser radar and the camera init And the camera internal parameter matrix K, each point P in the point cloud data i (x i ,y i , z i ) are projected onto the image plane to form a depth map D, and the pixel coordinates corresponding to the depth map D are P i (u i , v i ), the pixel value is Z i , the projection process is as follows: in, is the pixel coordinate P i The homogeneous coordinates of .

3. The laser radar and camera extrinsic calibration method based on deep learning according to claim 1 or 2, characterized in that: Also includes: Each frame of point cloud data is passed through the true value external parameter T LC Convert to the camera space coordinate system, then multiply by the random perturbation matrix △T and the camera internal parameters to convert into a depth map. At this time, the initial external parameter matrix corresponding to each frame of point cloud data is T init =△T*T LC .

4. The laser radar and camera extrinsic calibration method based on deep learning according to claim 1, characterized in that: The step of regressing the aggregated features through a fully connected layer to obtain a translation component and a rotation component of an external parameter between the laser radar and the camera includes: The aggregated features are passed through a fully connected layer, and the output is a 1×3 translation vector t pred and the 1×4 rotation quaternion q pred =(q0,q1,q2,q3), the conversion formula between quaternion and rotation matrix: Set two loss functions as regression loss L T And point cloud distance loss L p : L=λ T L T +λ p L p (4) Among them, the regression loss L T The translation component loss L t and the rotational component loss L R Two parts, L t t pred With t gt The smooth L1 loss between t gt and q gt Corresponding to the added random rotation error and translation error, L R Calculate q gt =(q0,q1,q2,q3) and q pred =(q'0,q'1,q'2,q'3), where λ T ,λ p All are experience values; L T =λ t L t +λ q L R (6) Point cloud P = {P1, ...P N }, P i ∈R 3 Point cloud distance loss L p The calculation formula is as follows:

5. The laser radar and camera extrinsic calibration method based on deep learning according to claim 4, characterized in that: Also includes: Evaluate the extrinsic parameters between the calibrated lidar and camera; The inverse of the matrix obtained by regression Multiply by △T to obtain a new matrix, extract the translation vector t1 of the new matrix, and add the translation vector t1 and the true value translation vector t of the new matrix gt =[0,0,0] The absolute value of the difference between each component is used as the evaluation index E of the translation component t_i , i∈(x,y,z): E t_i =|t 1_i -t gt_i | (8) When rotating the components, transform the rotation matrix into Euler angles and calculate the angular error E of each axis separately. R_j , j∈(α,β,γ): E R_j =|R 1_j -R gt_j | (9)。 6. A laser radar and camera extrinsic parameter calibration device based on deep learning, characterized in that: include: An acquisition module is used to acquire image data collected by the camera and point cloud data collected by the laser radar; A preprocessing module, used for projecting the point cloud data onto the image plane of the camera to form a depth map; A feature extraction module, configured to respectively extract first deep features of the image data and second deep features of the depth map using a stacked hourglass network; A feature aggregation module, used to use a core selection network to perform channel cascade on the first deep feature and the second deep feature to obtain a cascade feature, perform feature extraction on the cascade feature through convolution kernels with different receptive fields, determine the feature correlation between the depth map and the image data under different receptive fields, and use an attention mechanism to adaptively select different feature aggregation channels to obtain an aggregated feature; The regression module is used to regress the aggregated features through a fully connected layer to obtain the translation component and rotation component of the external parameters between the laser radar and the camera to complete the calibration.

7. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the deep learning-based lidar and camera extrinsic parameter calibration method described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the deep learning-based laser radar and camera extrinsic parameter calibration method described in any one of claims 1-5.

Images