External parameter calibration method and system for mining laser radar and camera

Abstract

The invention discloses an external parameter calibration method and system for a mining laser radar and a camera, and belongs to the technical field of computer vision and robotics.The method comprises the steps that an original image and an original point cloud in a to-be-calibrated target scene are obtained; obtaining a preliminary alignment result according to the original point cloud data; optimizing the preliminary alignment result to obtain a laser radar fusion frame point cloud; performing panoramic segmentation on the original image, and constructing a 2D-3D effective target centroid corresponding relation; and based on a laser radar fusion frame point cloud and a 2D-3D effective target mass center corresponding relation, through panoramic segmentation, an effective target is extracted, an effective target panoramic information matching degree target function is constructed, the target function is optimized, and an external parameter optimal solution of the laser radar and a camera is obtained. According to the method, the automatic calibration precision of the external parameters of the mining laser radar and the camera can be effectively improved, and a research basis is provided for subsequent mining laser radar data and image fusion.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and robotics, and more specifically to an external parameter calibration method and system for a mining laser radar and a camera. Background Art

[0002] In recent years, my country's coal mines have begun to accelerate the construction of intelligent coal mines, among which "unmanned driving auxiliary transportation vehicles for mines" have become an important part of the intelligent construction of coal mines and an important standard for the acceptance of intelligent coal mines. It can be seen that the development of mining vehicles towards intelligence and even unmanned is an inevitable trend in the future. However, in order to achieve unmanned driving, mining vehicles must rely on accurate environmental perception, that is, using mining intrinsically safe vehicle-mounted sensors for perception.

[0003] As the most common vehicle-mounted sensors, LiDAR and cameras have different advantages and disadvantages. Cameras can provide color and texture information, making it easy to identify and segment targets, but they cannot directly provide distance information for sensing targets. LiDAR can directly obtain high-precision distance information, but it cannot perceive the texture and color of the target surface. Therefore, the combination of LiDAR and cameras can make up for each other's shortcomings and obtain richer and more accurate environmental perception information, which is crucial for three-dimensional target detection and SLAM mapping of unmanned mining vehicles.

[0004] In order to ensure the effective fusion between the LiDAR and the camera, external parameter calibration is required. External parameters refer to parameters that describe the relative position and posture between sensors, including translation vectors and rotation matrices. By accurately estimating the external parameters between the LiDAR and the camera, their data can be aligned so that they are consistent in the same coordinate system. In general, the external calibration methods of LiDAR and cameras can be divided into two categories: target-based and targetless. The targetless method does not require the production of specific calibration targets, but directly obtains appearance or motion information from the environment to automatically complete the estimation of external parameters. The traditional target-based external calibration method requires a lot of manpower and is not conducive to real-time calibration.

[0005] Although the targetless calibration method in the existing technology can save human intervention, it needs to provide relatively accurate initial calibration parameters, or the required laser point cloud is relatively dense, and the requirements for point cloud data are relatively high. However, the current mine-used intrinsically safe vehicle-mounted laser radar has explosion-proof power requirements, and the commonly used ones are mostly 16-line laser radars, which produce relatively sparse point clouds, affecting the accuracy of the mine-used intrinsically safe laser radar and camera external parameter calibration in the realization of mine-used unmanned driving. Summary of the invention

[0006] In view of the problems existing in the above-mentioned fields, the present invention proposes an external parameter calibration method and system for mining laser radar and camera, which can solve the technical problem that the current mining intrinsically safe vehicle-mounted laser radar has explosion-proof power requirements, and most commonly used are 16-line laser radars, and the point clouds generated are relatively sparse, which affects the accuracy of the external parameter calibration of mining intrinsically safe laser radar and camera in mining unmanned driving.

[0007] In order to solve the above technical problems, the present invention discloses an external parameter calibration method for a mining laser radar and a camera, comprising the following steps:

[0008] Obtain the original image and original point cloud in the target scene to be calibrated;

[0009] According to the original point cloud data, the historical frame point cloud data is preliminarily aligned with the current frame point cloud data through the RANSAC-based method to obtain the preliminary alignment result; the preliminary alignment result is optimized based on the ICP fine registration method to obtain the LiDAR fusion frame point cloud;

[0010] Perform panoptic segmentation on the original image to extract vehicles and traffic signs in the target scene as valid targets, and construct a 2D-3D valid target centroid correspondence;

[0011] Based on the obtained LiDAR fusion frame point cloud and the constructed 2D-3D effective target centroid correspondence, the effective target is extracted through panoramic segmentation, and the effective target is projected on the point cloud. The effective target point cloud is projected on the segmentation mask after the inverse distance transformation through the coarsely calibrated external parameters; by assigning different weight parameters to the number of points of the effective target projected on the matching map, the objective function of the effective target panoramic information matching degree is constructed;

[0012] According to the constructed effective target panoramic information matching objective function, the optimal solution of the external parameters of the lidar and camera is obtained by optimizing the objective function.

[0013] Preferably, obtaining the laser radar fusion frame point cloud comprises the following steps:

[0014] Get historical frame point cloud data and current frame point cloud data;

[0015] Based on the coarse registration of random sampling consistency, the historical frame point cloud data and the current frame point cloud data are downsampled and the FPFH features are extracted to obtain the feature point set in the historical frame point cloud and the feature point set in the current frame point cloud:

[0016]

[0017] In the formula, p si is the feature point in the historical frame point cloud, p ti is the feature point in the point cloud of the current frame;

[0018] Randomly select multiple feature point pairs from the feature point set {(p s1 ,p t1 ),(p s2 ,p t2 ),(p s3 ,p t3 ),...}; Use the selected feature point pairs to calculate the rigid body transformation matrix T through singular value decomposition; Use the transformation matrix T to transform all feature points p in the historical frame point cloud si Transform and get the transformed point p' si :

[0019] p' si =Tp si

[0020] Calculate the transformed point p' si The feature point p corresponding to the point cloud of the current frame ti When the distance error of the transformed point is less than the set distance threshold, the transformed point is called an inlier point, where the number of inliers is calculated as:

[0021]

[0022] In the formula, δ(·) is the indicator function, when ||p ti -p' si When ||<∈, the value is 1, otherwise, the value is 0; ∈ is the distance threshold;

[0023] Repeat the above steps for multiple iterations, and select the transformation matrix with the largest number of inliers as the rough registration result based on random sampling consistency;

[0024] Based on the iterative closest point method, the transformation matrix of the coarse registration based on random sampling consistency is optimized to obtain the historical frame point cloud position of the initialization transformation matrix T0 of the coarse registration based on random sampling consistency. si , through KD-Tree to accelerate the search for the nearest neighbor point q of the current frame point cloud tj :

[0025]

[0026] Among them, T k is the current transformation matrix, and the nearest neighbor point pair is By minimizing the error function, update the transformation matrix T k+1 :

[0027]

[0028] The error function is solved by singular value decomposition. After several iterations, the transformation matrix T converges to a stable value and the lidar fusion frame point cloud is obtained.

[0029] Preferably, the constructing of the 2D-3D effective target centroid correspondence comprises the following steps:

[0030] Through panoramic segmentation, each pixel and point cloud is assigned a category label and instance ID, and vehicles and traffic signs that are easy to segment and have obvious features are selected as valid targets, respectively, to obtain the instance centroid of the vehicle and the semantic centroid of the traffic sign;

[0031] The calculation formula for defining the two-dimensional centroid of the effective target m is:

[0032]

[0033] In the formula, is a set of pixels with valid target m, is the number of pixels of the valid target m;

[0035] The calculation formula for the three-dimensional centroid of the effective target m in the laser radar coordinate system is defined as:

[0036]

[0037] In the formula, represents the point cloud set of valid target m, Represents the number of point clouds of valid targets m;

[0039] According to the 2D centroid of the effective target m and the 3D centroid in the laser radar coordinate system, the 2D-3D centroid pair of the effective target is obtained.

[0040] Preferably, the method further includes obtaining external parameters for rough calibration of the laser radar and the camera according to the constructed 2D-3D effective target centroid correspondence, which specifically includes the following steps:

[0041] The initial extrinsic parameters of the lidar and camera are estimated by the EPnP algorithm in RANSAC to obtain a set of initial calibration extrinsic parameters for the rough calibration of the lidar and camera.

[0042] Get N valid targets and the 3D centroid of the valid targets Where i = 1, 2, ..., N, four control points are constructed through PCA, and the control points are used to represent each 3D centroid point

[0043]

[0044] In the formula, γ ij is the weight, satisfying are the corresponding control point coordinates;

[0045] Corresponding to the camera coordinate system It can be expressed as a linear combination of control points:

[0046]

[0047] In the formula, is the coordinate of the corresponding control point in the camera coordinate system;

[0048] General becomes the relationship between pixel coordinates and camera coordinates, get:

[0049]

[0050] In the formula, w i is the projection depth of the coordinate point, K is the camera internal parameter matrix, f u , f v ,u c , v c These are the internal parameters of the camera;

[0051] According to the distance-preserving property of Euclidean transformation, the 3D to 2D PNP problem is converted into a 3D to 3D rigid body transformation problem by solving the coordinates of the four control points in the camera coordinate system. The pose transformation between the lidar and the camera is solved through ICP iteration.

[0052] Preferably, the constructing of the effective target panoramic information matching objective function comprises the following steps:

[0053] Through the panoramic segmentation of the image, generate pixel-by-pixel effective target / background segmentation and build the target matching map

[0054]

[0055] In the formula, h i Pixel The matching value of

[0056] For LiDAR data, only the point clouds of valid targets are Projected in pixel coordinate system, point The matching value of is obtained through the target matching graph:

[0057]

[0058] The pixels of the effective target and other pixels Set to 1 and 0 respectively:

[0059]

[0060] Mask Perform an inverse distance transform to reward valid target point clouds projected near the edge of the mask:

[0061]

[0062] Among them, α and β are the weight parameters assigned to the target matching graph;

[0063] Matching graph Redefine as

[0064]

[0065] The matching degree of the effective target m is defined as:

[0066]

[0067] In the formula, is the number of LiDAR points of the valid target m; when the matching degree is larger, the external parameters are more accurate;

[0068] Different valid target point clouds have different numbers, and the importance of each valid target in the objective function is also different; according to the different weight parameters assigned to the number of valid target points projected on the matching graph, the objective function of the matching degree of the effective target panoramic information is defined as:

[0069]

[0070] Preferably, obtaining the optimal solution of the external parameters of the laser radar and the camera specifically includes the following steps:

[0071] The rough calibration is refined by optimizing the objective function C to obtain the optimization result of the external parameters; the larger the objective function value, the more accurate the external parameters;

[0072] By maximizing the objective function C, we can obtain the optimal external parameters for:

[0073]

[0074] The particle swarm optimization algorithm is used to optimize the objective function C. The specific particle swarm optimization process includes:

[0075] Initialize the particle swarm:

[0076] The number of particles N is 50, the dimension d of each particle is 6, corresponding to r = (r x ,r y,r z ) T and t=(t x ,t y ,t z ) T , where the position of the i-th particle is at Y i =(r xi ,r yi ,r zi ,t xi ,t yi ,t zi ), the speed is V i =(v xi ,v yi ,v zi ,u xi ,u yi ,u zi ), and i=0,1,…,N-1;

[0077] Initialize multiple particles using coarse calibration parameters (r0, t0):

[0078] Y0=(r 0, t0)

[0079] The remaining particles are initialized by adding random small perturbations d based on the initial solution:

[0080] Y i =(r 0, t0)+d,i=0,1,…,N-1

[0081] Calculate the fitness value of each particle:

[0082] Substitute the i-th particle in the k-th iteration into the objective function C to obtain the fitness of the particle; take the maximum value of the objective function as the goal, and obtain the optimal position of the individual obtained by the particle swarm algorithm up to the k-th generation and the global optimal position

[0083] Update particle velocity and position, where:

[0084] The particle velocity update formula is:

[0085]

[0086] Where w is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers in the interval [0,1]; the inertia weight indicates the degree to which the particle retains its current speed;

[0087] At the beginning of the calculation, the maximum value of w is selected. With continuous iterations, the search ability of the particles is dynamically adjusted according to the linearly decreasing value of w:

[0088]

[0089] In the formula, w max and w min are the maximum and minimum values ​​of the inertia weight, respectively, k max is the maximum number of iterations;

[0090] Update the particle position according to the velocity:

[0091]

[0092] When the maximum number of iterations or accuracy conditions are reached, the optimal solution for the external parameters of the lidar and camera is output.

[0093] Preferably, the obtaining of the original image and the original point cloud in the target scene to be calibrated specifically includes:

[0094] The original point cloud data of the acquisition data set comes from the 16-line GUJ50 mining intrinsically safe laser radar. The target scenes of the acquisition data set include indoor garages and outdoor environments in mining areas. The data set is synchronized and corrected at a rate of 10 Hz and contains 1120 frames of RGB images and fused frame point cloud data corresponding to the timestamp.

[0095] Preferably, it also includes an external parameter calibration system for a mining laser radar and a camera, including:

[0096] A data acquisition module is used to acquire the original image and original point cloud in the target scene to be calibrated;

[0097] The original point cloud fusion module is used to perform preliminary alignment between the historical frame point cloud data and the current frame point cloud data through a RANSAC-based method based on the original point cloud data to obtain a preliminary alignment result; the preliminary alignment result is optimized based on the ICP fine registration method to obtain the LiDAR fusion frame point cloud;

[0098] The original image target extraction module is used to perform panoramic segmentation on the original image to extract vehicles and traffic signs in the target scene as valid targets, and to construct a 2D-3D valid target centroid correspondence;

[0099] The module for constructing the objective function of the panoramic information matching degree of effective targets is used to extract effective targets through panoramic segmentation based on the obtained laser radar fusion frame point cloud and the constructed 2D-3D effective target centroid correspondence, and to perform point cloud projection on the effective targets, and to project the effective target point cloud onto the segmentation mask after the inverse distance transformation through the coarsely calibrated external parameters; by assigning different weight parameters to the number of points of the effective targets projected on the matching map, the objective function of the panoramic information matching degree of effective targets is constructed;

[0100] The module for obtaining the optimal solution of the external parameters is used to obtain the optimal solution of the external parameters of the lidar and the camera by optimizing the objective function according to the constructed effective target panoramic information matching degree objective function.

[0101] Compared with the prior art, the present invention has the following beneficial effects:

[0102] The extrinsic parameter calibration method of the mining laser radar and camera proposed in the present invention obtains a fused frame point cloud by fusing multiple frame point clouds to increase the point cloud density and enrich the point cloud information; the vehicles and traffic signs in the extracted target scene are taken as effective targets, and a 2D-3D target centroid correspondence is constructed to achieve a rough calibration of the mining laser radar and camera extrinsic parameters to obtain the initial extrinsic parameters; the effective target point cloud is projected onto the segmentation mask through the initial extrinsic parameters to construct a panoramic information matching objective function of the effective target image and the point cloud, and the greater the matching degree, the more accurate the extrinsic parameters; the objective function is further solved, and the greater the objective function value, the more accurate the external parameters, and the optimal solution for the extrinsic parameter calibration of the mining laser radar and camera is obtained by optimizing the objective function. This method effectively improves the automatic calibration accuracy of the mining laser radar and camera extrinsic parameters, and provides a research basis for subsequent mining laser radar data and image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 This is a framework diagram of the external parameter calibration method for mining laser radar and camera proposed by the present invention;

[0104] Figure 2 A flow chart of a precise calibration method according to an embodiment of the present invention;

[0105] Figure 3 This is the multi-frame point cloud fusion method process of the precise calibration method proposed in the embodiment of the present invention;

[0106] Figure 4 This is a comparison diagram of point cloud fusion before and after the precise calibration method proposed in an embodiment of the present invention;

[0107] Figure 5 A 2D-3D target centroid matching pair of the fine calibration method proposed in an embodiment of the present invention;

[0108] Figure 6 A flow chart of a particle swarm algorithm used in the precise calibration method proposed in an embodiment of the present invention;

[0109] Figure 7 This is a diagram of the experimental data acquisition instrument used in the embodiment of the present invention;

[0110] Figure 8 The result analysis of the rough calibration proposed in the embodiment of the present invention;

[0111] Fig. 9The MAE of disturbance fine calibration is added to the fine calibration method proposed in the embodiment of the present invention;

[0112] Fig.10 The distribution of translation and rotation errors of different point cloud inputs and method combinations on the data set for the fine calibration method proposed in the embodiment of the present invention;

[0113] Fig.11 This is a comparison diagram of the projection results of the precise calibration method proposed in an embodiment of the present invention and the projection results of the true value of the external parameter. DETAILED DESCRIPTION

[0114] The following will be combined with the attached embodiment of the present invention Figure 1-Figure 11 , the technical solutions in the embodiments of the present invention are clearly and completely described. It should be understood that the terms described in the present invention are only used to describe specific implementation methods and are not used to limit the present invention.

[0115] like Figure 1 As shown, the present invention proposes an external parameter calibration method for a mining laser radar and a camera, comprising the following steps:

[0116] S1: Obtain the original image and original point cloud of the target scene to be calibrated;

[0117] S2: Based on the original point cloud data, the historical frame point cloud data is preliminarily aligned with the current frame point cloud data through a RANSAC-based method to obtain a preliminary alignment result; the preliminary alignment result is optimized based on the ICP fine registration method to obtain the LiDAR fusion frame point cloud;

[0118] S3: Perform panoramic segmentation on the original image, extract vehicles and traffic signs in the target scene as valid targets, and construct 2D-3D valid target centroid correspondence;

[0119] S4: Based on the obtained LiDAR fusion frame point cloud and the constructed 2D-3D effective target centroid correspondence, the effective target is extracted through panoramic segmentation, and the effective target is projected on the point cloud. The effective target point cloud is projected on the segmentation mask after the inverse distance transformation through the coarsely calibrated external parameters; by assigning different weight parameters to the number of points of the effective target projected on the matching map, the objective function of the effective target panoramic information matching degree is constructed;

[0120] S5: Based on the constructed effective target panoramic information matching objective function, the optimal solution of the external parameters of the lidar and the camera is obtained by optimizing the objective function.

[0121] In step S1, the collected raw point cloud data comes from the 16-line GUJ50 mining intrinsically safe laser radar. The target scenes for collecting the raw point cloud data include indoor garages and outdoor environments in mining areas. The data set is synchronized and corrected at a rate of 10 Hz and contains 1120 frames of RGB images and fused frame point cloud data corresponding to the timestamp.

[0122] In step S2, obtaining the laser radar fusion frame point cloud includes the following steps:

[0123] The goal of the present invention is to align the point cloud and the image in the scene. Therefore, it is necessary to establish a mapping relationship between the lidar and the camera, project the point cloud of the lidar on the image plane, and obtain the geometric relationship and conversion parameters between the lidar and the camera so as to project them into the same coordinate system.

[0124] The point cloud is represented as n represents the total number of point clouds, and the superscript L represents the number of points in the laser radar coordinate system. Convert to a point in the camera coordinate system for:

[0125]

[0126] In the formula, the superscript C indicates that in the camera coordinate system, a pair (r, t) constitutes the extrinsic parameters between the lidar and the camera, t = (t x ,t y ,t z ) T represents the translation vector, R(r) represents the rotation vector r=(r x ,r y ,r z ) T Define the rotation matrix.

[0127] Where R(r) is obtained by angular axis representation, given the angular axis vector r, where the rotation axis is the unit vector The rotation angle is ||r||2, and the rotation matrix R(r) is expressed as:

[0128] R(r)=I+sin(θ)H+(1-cos(θ))H 2

[0129] Where θ = ||r||2, I is the unit matrix, H is the antisymmetric matrix of the rotation axis, and H is:

[0130]

[0131] Through the projection function The three-dimensional point in the camera coordinate system can be Projection to pixel coordinates In the figure, the superscript I indicates the pixel coordinate system.

[0132] The projection function K uses the intrinsic parameters of the camera such as focal length or distortion to calculate the pixel coordinates. The specific projection process is expressed as:

[0133]

[0134] The present invention assumes that the intrinsic parameters of the camera have been calibrated, and the focus is on exploring the accurate extrinsic parameters (r, t).

[0135] like Figure 2 As shown, it is a flow chart of the method proposed by the present invention. The external parameter calibration method of the mining laser radar and camera of the present invention includes coarse calibration and fine calibration.

[0136] In the coarse calibration part, the multi-frame point cloud fusion method is used to obtain the fused frame point cloud, so as to increase the point cloud density and enrich the point cloud information; the vehicles and traffic signs in the scene are extracted as valid targets through the panoramic segmentation method, and the coarse calibration of the mining lidar and camera extrinsic parameters is achieved by constructing the 2D-3D effective target center of mass correspondence.

[0137] In the fine calibration part, the effective target point cloud is projected onto the segmentation mask after inverse distance transformation through the coarsely calibrated extrinsic parameters, and the objective function of the effective target panoramic information matching is constructed. Finally, the particle swarm algorithm is used to maximize the objective function to obtain the optimal extrinsic parameters.

[0138] As an important step of the method proposed in the present invention, multi-frame point cloud fusion affects the subsequent point cloud segmentation task and the calibration accuracy of external parameters. The present invention selects three frames of point cloud data, takes the first two frames as historical frame point clouds, and aligns the historical frame point cloud data to the current frame point cloud through a multi-frame point cloud fusion algorithm, in order to enrich the current frame point cloud information and obtain a fused frame point cloud.

[0139] The multi-frame point cloud fusion algorithm process of the present invention is as follows: Figure 3 As shown in the figure, the historical frame point cloud data is first preliminarily aligned with the current frame point cloud data through a coarse registration method based on random sampling consistency RANSAC to obtain a preliminary alignment result; the fine registration method based on iterative closest point ICP is used to further optimize the preliminary alignment result to obtain a more accurate fused frame point cloud. The specific implementation process is as follows:

[0140] Coarse registration based on random sampling consistency

[0141] The historical frame point cloud and the current frame point cloud are preliminarily aligned through the random sampling consistency RANSAC algorithm. The historical frame point cloud data and the current frame point cloud data are downsampled and the FPFH features are extracted to obtain the feature point set in the historical frame point cloud and the feature point set in the current frame point cloud:

[0142]

[0143] In the formula, p si is the feature point in the historical frame point cloud, p ti is the feature point in the point cloud of the current frame.

[0144] Randomly select 4 feature point pairs from the feature point set {(p s1 ,p t1 ),(p s2 ,p t2 ),(p s3 ,p t3 ),(p s4 ,p t4 )}; Use the selected feature point pairs to calculate the rigid body transformation matrix T through singular value decomposition; Use the transformation matrix T to transform all feature points p in the historical frame point cloud si Transform and get the transformed point p' si :

[0145] p' si =Tp si

[0146] Calculate the transformed point p' si The feature point p corresponding to the point cloud of the current frame ti When the distance error of the transformed point is less than the set distance threshold, the transformed point is called an inlier point, where the number of inliers is calculated as:

[0147]

[0148] In the formula, δ(·) is the indicator function, when ‖p ti -p' si When ‖<∈, the value is 1, otherwise, the value is 0; ∈ is the distance threshold;

[0149] Repeat the above steps, and after multiple iterations, select the transformation matrix with the largest number of inliers as the coarse registration result based on random sampling consistency.

[0150] Precise registration based on iterative closest point ICP

[0151] After RANSAC rough registration, the transformation matrix of the rough registration based on random sampling consistency is optimized based on the iterative closest point ICP method to obtain the historical frame point cloud position of the initialization transformation matrix T0 of the rough registration based on random sampling consistency. si , through KD-Tree to accelerate the search for the nearest neighbor point q of the current frame point cloud tj :

[0152]

[0153] Among them, T k is the current transformation matrix, and the nearest neighbor point pair is By minimizing the error function, update the transformation matrix T k+1 :

[0154]

[0155] The error function is solved by singular value decomposition. After several iterations, the transformation matrix T converges to a stable value, and the fused frame point cloud of the lidar and camera is obtained.

[0156] Through the above-mentioned coarse registration based on random sampling consistency and fine registration based on iterative closest point ICP, the historical frame point cloud data can be effectively registered to the current frame point cloud, such as Figure 4 As shown in the figure, (a) is the previous frame point cloud, and (b) is the fused frame point cloud. Through comparative analysis, it is found that the fused frame point cloud has a higher point cloud density than the current frame point cloud, and the target contour is clearer and more complete.

[0157] The targetless calibration method has high requirements for good initial external parameters, because good initialization can speed up the convergence of the optimization process and avoid divergence or falling into local minima. However, the initial parameters of most current methods are measured manually or calibrated by hand and eye. However, manual measurement is time-consuming and labor-intensive, and hand-eye calibration relies on the motion information of the sensor. Therefore, for a targetless and fully autonomous calibration method, in order to achieve more accurate external parameter calibration, it is necessary to initialize and calibrate the external parameters.

[0158] In step S3, constructing a 2D-3D valid target centroid correspondence includes the following steps:

[0159] The present invention assigns a category label and instance ID to each pixel and point cloud through panoramic segmentation, and selects vehicles and traffic signs that are easy to segment and have obvious features as effective targets, and then obtains the instance centroid of the vehicle and the semantic centroid of the traffic sign, so as to increase the 2D-3D correspondence, such as Figure 5 As shown, it is a 2D-3D target centroid matching pair. Specifically, (a) is the 2D effective target centroid, and (b) is the 3D effective target centroid.

[0160] The two-dimensional centroid of the valid target m is defined as the sum of the coordinates of the pixels of the same instance or semantic category, and then divided by the number of pixels. The calculation formula is:

[0161]

[0162] In the formula, is a set of pixels with valid target m, is the number of pixels of the valid target m.

[0164] The three-dimensional centroid of the effective target m in the laser radar coordinate system is defined as the sum of the coordinates of all points of the same instance, divided by the number of points. The calculation formula is:

[0165]

[0166] In the formula, represents the point cloud set of valid target m, Represents the number of point clouds of valid targets m.

[0168] According to the 2D centroid of the effective target m and the 3D centroid in the laser radar coordinate system, the 2D-3D centroid pair of the effective target is obtained. Since some PnP linear methods (such as P3P) are difficult to utilize more point pair matching information, the present invention selects the EPnP algorithm in RANSAC to estimate the initial extrinsic parameters of the laser radar and camera, and obtains a set of good initial calibration parameters for fine calibration.

[0169] The EPnP algorithm in RANSAC constructs four control points through PCA, so that the three-dimensional centroid coordinates of the effective target can be represented by the linear combination of these four control points. Assuming that N effective targets are taken, the three-dimensional centroid of the effective target Where i = 1, 2, ..., N, four control points are constructed through PCA, and the control points are used to represent each 3D centroid point

[0170]

[0171] In the formula, γ ij is the weight, satisfying are the corresponding control point coordinates.

[0172] Similarly, we get It can be expressed as linear combination of control points:

[0173]

[0174] In the formula, is the coordinate of the corresponding control point in the camera coordinate system;

[0175] General becomes the relationship between pixel coordinates and camera coordinates, get:

[0176]

[0177] In the formula, w iis the projection depth of the coordinate point, K is the camera internal parameter matrix, f u , f v ,u c , v c These are camera internal parameters.

[0178] Will Further expansion, we get the expanded form:

[0179]

[0180] It can be seen that in the expansion, only the coordinates of the four control points in the camera coordinate system are unknown. There are 12 unknowns in total. When there are N matching pairs in the rough calibration, the expansion can be transformed into a linear equation system:

[0181] MX=0

[0182] Among them, M is the coefficient matrix of the linear equation, the size of which is 2N×12, and X is the coordinates of the four control points in the camera coordinate system to be solved.

[0183] According to the distance-preserving property of Euclidean transformation, the three-dimensional to two-dimensional PNP problem can be converted into a three-dimensional to three-dimensional rigid body transformation problem by calculating the coordinates of the four control points in the camera coordinate system, and the ICP iteration can be used to solve the pose transformation between the lidar and the camera.

[0184] In step S4, the specific fine calibration process, constructing an effective target panoramic information matching objective function, includes the following steps:

[0185] Through the panoramic segmentation of the image, generate pixel-by-pixel effective target / background segmentation and build the target matching map

[0186]

[0187] In the formula, h i Pixel The matching value of .

[0188] For LiDAR data, only the point clouds of valid targets are Projected in pixel coordinate system, point The matching value of is obtained through the target matching graph:

[0189]

[0190] The pixels of the effective target and other pixels Set to 1 and 0 respectively:

[0191]

[0192] Mask An inverse distance transform (IDT) is performed to prevent the binary image from having a zero space problem in the direction of the camera's observation axis and to avoid local maxima in the subsequent optimization process. By implementing the distance transform to reward valid target point clouds projected near the edge of the mask, the matching value can gradually decay from the boundary of the mask to its interior, thereby smoothing the objective function, which can be written as:

[0193]

[0194] Among them, α and β are weight parameters assigned to the target matching graph, and the corresponding values ​​are α=0.8 and β=0.6.

[0195] Matching graph Redefine as

[0196]

[0197] In order to express the consistency between the projection pixels of the valid target point cloud and its corresponding mask in the image, the matching degree of the valid target m is defined as:

[0198]

[0199] In the formula, is the number of LiDAR points of the valid target m; when the matching degree is larger, the external parameters are more accurate.

[0200] Different valid target point clouds have different numbers, and the importance of each valid target in the objective function is also different; according to the different weight parameters assigned to the number of valid target points projected on the matching graph, the objective function of the matching degree of the effective target panoramic information is defined as:

[0201]

[0202] In step S5, the optimal solution of the external parameters of the laser radar and the camera is obtained, including the following steps:

[0203] The rough calibration is refined by optimizing the objective function C to obtain the optimization result of the external parameters; the larger the objective function value, the more accurate the external parameters.

[0204] By maximizing the objective function C, we can obtain the optimal external parameters for:

[0205]

[0206] Since the objective function of the present invention is a non-convex function, some classic optimization methods such as gradient descent and Newton's method are not applicable. Therefore, the present invention uses the particle swarm optimization method to optimize the objective function. The particle swarm optimization algorithm simulates the movement behavior of a flock of birds or a school of fish and searches for the optimal solution through information sharing between individuals. Each particle represents a candidate solution and searches by updating its speed and position. The specific particle swarm algorithm process is as follows:

[0207] 1) Initialize the particle swarm:

[0208] The number of particles N is 50, the dimension d of each particle is 6, corresponding to r = (r x ,r y ,r z ) T and t=(t x ,t y ,t z ) T , where the position of the i-th particle is at Y i =(r xi ,r yi ,r zi ,t xi ,t yi ,t zi ), the speed is V i =(v xi ,v yi ,v zi ,u xi ,u yi ,u zi ), and i=0,1,…,N-1;

[0209] The present invention is based on coarse calibration parameters (r0, t0), which can be used to initialize multiple particles:

[0210] Y0=(r0,t0)

[0211] The remaining particles are initialized by adding random small perturbations d based on the initial solution:

[0212] Y i =(r 0, t0)+d,i=0,1,…,N-1

[0213] 2) Calculate the fitness value of each particle:

[0214] Substitute the i-th particle in the k-th iteration into the objective function C to obtain the fitness of the particle; take the maximum value of the objective function as the goal, and obtain the optimal position of the individual obtained by the particle swarm algorithm up to the k-th generation and the global optimal position

[0215] Update particle velocity and position, where:

[0216] 3) The particle velocity update formula is:

[0217]

[0218] Where w is the inertia weight, c1 and c2 are learning factors, and r1 and r2 are random numbers in the interval [0,1].

[0219] The inertia weight indicates the degree to which the particle retains its current velocity. Specifically, a larger inertia weight can improve the global convergence ability of the algorithm, and a smaller inertia weight can improve the local convergence ability of the algorithm.

[0220] At the beginning of the calculation, the maximum value of w is selected. With continuous iterations, the search ability of the particles is dynamically adjusted according to the linearly decreasing value of w:

[0221]

[0222] In the formula, w max and w min are the maximum and minimum values ​​of the inertia weight, respectively, k max is the maximum number of iterations.

[0223] Update the particle position according to the velocity:

[0224]

[0225] 4) Check termination conditions:

[0226] If the maximum number of iterations or accuracy conditions are reached, the optimal solution of the external parameters of the lidar and camera is output; otherwise, return to step 2) to recalculate the fitness value of each particle.

[0227] The flowchart of the particle swarm algorithm used in the present invention is as follows Figure 6 As shown, the algorithm can be used to solve the objective function and achieve the purpose of optimizing the external parameters of the laser radar and camera.

[0228] The present invention also proposes an external parameter calibration system for a mining laser radar and a camera, comprising:

[0229] A data acquisition module is used to acquire the original image and original point cloud in the target scene to be calibrated;

[0230] The original point cloud fusion module is used to perform preliminary alignment between the historical frame point cloud data and the current frame point cloud data through a RANSAC-based method based on the original point cloud data to obtain a preliminary alignment result; the preliminary alignment result is optimized based on the ICP fine registration method to obtain the LiDAR fusion frame point cloud;

[0231] The original image target extraction module is used to perform panoramic segmentation on the original image to extract vehicles and traffic signs in the target scene as valid targets, and to construct a 2D-3D valid target centroid correspondence;

[0232] The module for constructing the objective function of the panoramic information matching degree of effective targets is used to extract effective targets through panoramic segmentation based on the obtained laser radar fusion frame point cloud and the constructed 2D-3D effective target centroid correspondence, and to perform point cloud projection on the effective targets, and to project the effective target point cloud onto the segmentation mask after the inverse distance transformation through the coarsely calibrated external parameters; by assigning different weight parameters to the number of points of the effective targets projected on the matching map, the objective function of the panoramic information matching degree of effective targets is constructed;

[0233] The module for obtaining the optimal solution of the external parameters is used to obtain the optimal solution of the external parameters of the lidar and the camera by optimizing the objective function according to the constructed effective target panoramic information matching degree objective function.

[0234] The external parameter calibration method proposed in this application helps to achieve more convenient and accurate external parameter calibration of intrinsically safe laser radars and cameras for mine unmanned driving.

[0235] Example

[0236] In order to verify the accuracy and robustness of the proposed external parameter calibration method for mining laser radar and camera, a large number of experiments were conducted on a self-made dataset. Figure 7 The experimental acquisition instrument shown is installed on a trackless rubber-wheeled vehicle for acquisition. The image comes from the KBA12 (B) mining intrinsically safe camera with a resolution of 1920×1080, and the original point cloud data comes from the 16-line GUJ50 mining intrinsically safe laser radar. The acquisition scenes include indoor garages and outdoor environments in mining areas. The data set is synchronized and corrected at a rate of 10Hz. It contains 1120 frames of RGB images and fused frame point cloud data corresponding to the timestamp, and the external parameters between sensors are obtained by manual measurement as the experimental true value.

[0237] The present invention adopts the OneFormer model and the P3Former model with better segmentation performance to perform panoramic segmentation on the current frame image and the fused frame point cloud respectively.

[0238] The original images and fused frame point clouds in the homemade dataset are made into Cityscape and Semantic-KITTI formats, and the ratio of training set to validation set is 9:1. In order to speed up the convergence of the two models on the homemade dataset, the present invention uses the pre-trained weight files of the two models on the Cityscape dataset and the Semantic-KITTI dataset, respectively, and re-trains the two segmentation models, and uses the trained models for segmentation prediction.

[0239] In order to evaluate the performance of the proposed method relative to the reference calibration, the calibration results are analyzed based on the predicted translation and rotation errors of the extrinsic parameters, which are calculated by the Euclidean distance:

[0240]

[0241] Where t and r are the true values ​​of the rotation and translation vectors.

[0242] In addition, the calculation process follows the ZYX representation by calculating the mean absolute error (MAE) of the three components of the translation (X, Y, Z) and the MAE of the three Euler angles (roll, pitch, yaw).

[0243] The paper is implemented in C++ on a desktop computer using an Intel Core i7-8700 CPU and an Nvidia 3090 GPU.

[0244] Quantitative analysis of experimental results

[0245] The coarse calibration is performed in the self-made data set. The whole process is automatic. The present invention uses 50 frames of images and fused point clouds to perform initialization experiments based on the effective target centroid, and then no coarse calibration is performed. As shown in Table 1, the mean absolute error of the coarse calibration is shown. Figure 8 As shown in the figure, the result analysis of the coarse calibration proposed in the embodiment of the present invention shows the error distribution of the six parameters. Figure (a) shows that the translation errors of the three components of translation X, Y, and Z are mostly less than 0.3m, and Figure (b) shows that the rotation errors of the three Euler angles roll, pitch, and yaw are mostly less than 2. It can be seen that the coarse calibration can give a more reasonable initial value.

[0246] Table 1 Comparison of correction results of different methods under different point cloud inputs

[0247]

[0248] After the coarse calibration, 100 frames of images and fused point clouds were selected from the two scenes in the dataset for fine calibration experiments. The same initial external parameters obtained by coarse calibration were used for all experiments for fine calibration, and the results after fine calibration were compared with the reference calibration. As shown in Table 1, the mean absolute error (MAE) of fine calibration is listed. By comparison, it can be found that fine calibration has greatly improved over coarse calibration.

[0249] In order to verify the robustness of the fine calibration, the present invention perturbs each calibration parameter to varying degrees. Among them, when one parameter changes, the other five parameters maintain their true values. The perturbation range of rotation is set to [0,6°], with a total of 60 groups; the perturbation range of translation is set to [0,1m], with a total of 50 groups. The perturbation degree of each group of experiments is gradually increased, with the translation perturbation increasing by 0.02m each time and the rotation perturbation increasing by 0.1° each time. A single perturbation experiment was conducted on 100 frames of images and fused point clouds, and the average of the results of the 100 frames of fine calibration was taken as the final result.

[0250] like Fig. 9 As shown in the figure, Figure (a) shows the MAE of different degrees of disturbance added to each parameter of the three translation components X, Y, and Z, and Figure (b) shows the MAE of different degrees of disturbance added to each parameter of the three Euler angles roll, pitch, and yaw of rotation. Although the initial disturbance increases the difficulty of fine calibration, the MAE of translation disturbance and rotation disturbance only increases slightly. This shows that the degree of disturbance has no significant effect on fine calibration, indicating that the fine calibration method proposed in the present invention is robust and stable.

[0251] Table 2 Overall results of calibration parameter perturbation experiment

[0252]

[0253]

[0254] It can be seen from Table 2 that the calibration parameters after disturbance can control the error within the range less than the rough calibration error after fine calibration. The average errors of translation parameters ΔX, ΔY and ΔZ are 0.0223m, 0.0238m and 0.0210m, respectively, the standard deviations are 0.0062, 0.0094 and 0.0062, respectively, the maximum values ​​are 0.0339m, 0.0409m and 0.0326m, respectively, and the minimum values ​​are 0.0097m, 0.0073m and 0.0099m, respectively. The average errors of the rotation parameters Δroll, Δpitch and Δyaw are 0.1689°, 0.1434° and 0.1371°, respectively, the standard deviations are 0.0477, 0.0372 and 0.0389, the maximum values ​​are 0.2490°, 0.2132° and 0.2035°, and the minimum values ​​are 0.0691°, 0.0578° and 0.0512°, respectively. The above data analysis shows that the perturbation of the translation parameters and rotation parameters has little effect on the fine calibration, and the fine calibration algorithm can maintain a small error in a large range.

[0255] The present invention also compares and analyzes the method proposed by the present invention and the two methods based on semantic information using the same initial external parameters, and experiments are conducted on self-made data sets. The experimental settings include two inputs: single-frame point cloud and image, and fused frame point cloud and image. The semantic method and the method proposed by the present invention (precise calibration) are used for each input.

[0256] A total of 100 frames were used in the experiment. As shown in Table 1, the mean absolute error (MAE) of the calibration results is listed. Fig.10 The distribution diagram of the calibration error of the fine calibration method of the present invention and the calibration error based on semantic information under different inputs is shown. Figure (a) is the translation error and rotation error distribution of single-frame point cloud + semantic information, Figure (b) is the translation error and rotation error distribution of single-frame point cloud + fine calibration method of the present invention, Figure (c) is the translation error and rotation error distribution of fused frame point cloud + semantic information, and Figure (d) is the translation error and rotation error distribution of fused frame point cloud + fine calibration method of the present invention. By comparison, it can be clearly seen that using fused frame point cloud as input, the translation error of the fine calibration method of the present invention is 0.055m and the rotation error is 0.394°. Compared with single-frame point cloud as input, the translation error is reduced by 50.89% and the rotation error is reduced by 53.76%. Compared with the method based on semantic features, the translation error is reduced by 43.88% and the rotation error is reduced by 48.63%.

[0257] In summary, the advantages of the precision calibration method of the present invention can be attributed to the fact that the panoramic information used by the present invention can provide more accurate target position and boundary information, which helps to reduce mutual interference between targets. On the other hand, the present invention increases the density of the effective target point cloud in the current frame by fusing the historical frame point cloud with the current frame point cloud. This method not only enriches the point cloud data, but also improves the matching information of the point cloud, thereby reducing the calibration error.

[0258] Qualitative analysis of experimental results

[0259] In order to display the calibration results more intuitively, the effective target point cloud is projected onto the image plane by using the external parameters in Table 1. Tests are conducted on a self-made data set. The projection results of the precise calibration method of the present invention and the projection results of the true value of the external parameters are shown in Figure 2. Fig.11 As shown, Figure (a) is the external parameter projection result predicted by the precise calibration method of the present invention in the garage scene, Figure (b) is the external parameter true value projection result in the garage scene, Figure (c) is the external parameter projection result predicted by the precise calibration method of the present invention in the mining scene, and Figure (d) is the external parameter true value projection result in the mining scene. By comparison, it is found that the external parameter projection result predicted by the precise calibration method of the present invention is very similar to the true value projection result.

[0260] In order to evaluate the impact of the weight coefficient in the objective function on the calibration results, an ablation experiment was performed on the weight coefficient.

[0261] Different valid target point clouds have different numbers. Different weight parameters are assigned according to the number of valid target points projected on the matching graph. They are run under the same test conditions and datasets, as shown in Table 3. One considers the weight coefficient and the other does not.

[0262] Table 3 Ablation experimental data analysis of weight coefficients

[0263] Experimental setup Translation error Δt(m) Rotation error Δθ(°) Regardless of weight 0.086 0.634 Consider weight 0.055 0.394

[0264] By analyzing the data in Table 3, it can be seen that after considering the weight coefficient, the translation error is reduced by 36.05%, and the rotation error is reduced by 37.87%. Therefore, considering the weight coefficient can effectively improve the calibration performance.

[0265] The method proposed in the present invention can effectively improve the automatic calibration accuracy of mining laser radar and camera extrinsic parameters, and provide a research basis for the subsequent fusion of mining laser radar data and images.

[0266] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0267] In addition, unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as commonly understood by those skilled in the art to which the present invention belongs. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of any conflict with any incorporated document, the content of this specification shall prevail.

Claims

1. A method for calibrating external parameters of a mining laser radar and camera, characterized in that: The following steps are involved: Obtain the original image and original point cloud in the target scene to be calibrated; According to the original point cloud data, the historical frame point cloud data is preliminarily aligned with the current frame point cloud data through the RANSAC-based method to obtain the preliminary alignment result; the preliminary alignment result is optimized based on the ICP fine registration method to obtain the LiDAR fusion frame point cloud; Perform panorama segmentation on the original image, extract vehicles and traffic signs in the target scene as valid targets, and construct 2D-3D valid target centroid correspondence; Based on the obtained LiDAR fusion frame point cloud and the constructed 2D-3D effective target centroid correspondence, the effective target is extracted through panoramic segmentation, and the effective target is projected on the point cloud. The effective target point cloud is projected on the segmentation mask after the inverse distance transformation through the coarsely calibrated external parameters; by assigning different weight parameters to the number of points of the effective target projected on the matching map, the objective function of the effective target panoramic information matching degree is constructed; According to the constructed effective target panoramic information matching objective function, the optimal solution of the external parameters of the lidar and camera is obtained by optimizing the objective function.

2. The external parameter calibration method of the mining laser radar and camera according to claim 1 is characterized in that: The step of obtaining the laser radar fusion frame point cloud comprises the following steps: Get historical frame point cloud data and current frame point cloud data; Based on the coarse registration of random sampling consistency, the historical frame point cloud data and the current frame point cloud data are downsampled and the FPFH features are extracted to obtain the feature point set in the historical frame point cloud and the feature point set in the current frame point cloud: In the formula, p si is the feature point in the historical frame point cloud, p ti is the feature point in the point cloud of the current frame; Randomly select multiple feature point pairs from the feature point set {(p s1 , p t1 ), (p s2 , p t2 ), (p s3 , p t3 ), ...}; Use the selected feature point pairs to calculate the rigid body transformation matrix T through singular value decomposition; Use the transformation matrix T to transform all feature points p in the historical frame point cloud si Transform and get the transformed point p′ si : p′ si =Tp si Calculate the transformed point p′ si The feature point p corresponding to the point cloud of the current frame ti When the distance error of the transformed point is less than the set distance threshold, the transformed point is called an inlier point, where the number of inliers is calculated as: In the formula, δ(·) is the indicator function, when ||p ti -p′ si When ||<∈, the value is 1, otherwise, the value is 0; ∈ is the distance threshold; Repeat the above steps for multiple iterations, and select the transformation matrix with the largest number of inliers as the rough registration result based on random sampling consistency; Based on the iterative closest point method, the transformation matrix of the coarse registration based on random sampling consistency is optimized to obtain the historical frame point cloud position of the initialization transformation matrix T0 of the coarse registration based on random sampling consistency. si , through KD-Tree to accelerate the search for the nearest neighbor point q of the current frame point cloud tj : Among them, T k is the current transformation matrix, and the nearest neighbor point pair is By minimizing the error function, update the transformation matrix T k+1 : The error function is solved by singular value decomposition. After several iterations, the transformation matrix T converges to a stable value and the lidar fusion frame point cloud is obtained.

3. The external parameter calibration method of the mining laser radar and camera according to claim 2 is characterized in that: The construction of the 2D-3D effective target centroid correspondence relationship comprises the following steps: Through panoramic segmentation, each pixel and point cloud is assigned a category label and instance ID, and vehicles and traffic signs that are easy to segment and have obvious features are selected as valid targets, respectively, to obtain the instance centroid of the vehicle and the semantic centroid of the traffic sign; The calculation formula for defining the two-dimensional centroid of the effective target m is: In the formula, is a set of pixels with valid target m, is the number of pixels of the valid target m; The three-dimensional centroid calculation formula of the effective target m in the laser radar coordinate system is defined as: In the formula, represents the point cloud set of valid target m, Represents the number of point clouds of valid targets m; According to the 2D centroid of the effective target m and the 3D centroid in the laser radar coordinate system, the 2D-3D centroid pair of the effective target is obtained.

4. The external parameter calibration method of the mining laser radar and camera according to claim 3 is characterized in that: It also includes obtaining the external parameters of the rough calibration of the laser radar and the camera according to the constructed 2D-3D effective target centroid correspondence, which specifically includes the following steps: The initial extrinsic parameters of the lidar and camera are estimated by the EPnP algorithm in RANSAC to obtain a set of initial calibration extrinsic parameters for the rough calibration of the lidar and camera. Get N valid targets and the 3D centroid of the valid targets Where i = 1, 2, ..., N, four control points are constructed through PCA, and the control points are used to represent each 3D centroid point In the formula, γ ij is the weight, satisfying are the corresponding control point coordinates; Corresponding to the camera coordinate system It can be expressed as linear combination of control points: In the formula, is the corresponding control point coordinate in the camera coordinate system; General becomes the relationship between pixel coordinates and camera coordinates, get: In the formula, w i is the projection depth of the coordinate point, K is the camera internal parameter matrix, f u , f v ,u c , v c These are the internal parameters of the camera; According to the distance-preserving property of Euclidean transformation, the 3D to 2D PNP problem is converted into a 3D to 3D rigid body transformation problem by solving the coordinates of the four control points in the camera coordinate system. The pose transformation between the lidar and the camera is solved through ICP iteration.

5. The external parameter calibration method of the mining laser radar and camera according to claim 4 is characterized in that: The method of constructing the effective target panoramic information matching degree objective function comprises the following steps: Through the panoramic segmentation of the image, generate pixel-by-pixel effective target / background segmentation and build the target matching map In the formula, h i Pixel The matching value of For LiDAR data, only the point clouds of valid targets are Projected in pixel coordinate system, point The matching value of is obtained through the target matching graph: The pixels of the effective target and other pixels Set to 1 and 0 respectively: Mask Perform an inverse distance transform to reward valid target point clouds projected near the edge of the mask: Among them, α and β are the weight parameters assigned to the target matching graph; Matching graph Redefine as The matching degree of the effective target m is defined as: In the formula, is the number of LiDAR points of the valid target m; when the matching degree is larger, the external parameters are more accurate; Different valid target point clouds have different numbers, and the importance of each valid target in the objective function is also different; according to the different weight parameters assigned to the number of valid target points projected on the matching graph, the objective function of the matching degree of the effective target panoramic information is defined as:

6. The external parameter calibration method of the mining laser radar and camera according to claim 5 is characterized in that: The method of obtaining the optimal solution of the external parameters of the laser radar and the camera specifically includes the following steps: The rough calibration is refined by optimizing the objective function C to obtain the optimization result of the external parameters; the larger the objective function value, the more accurate the external parameters; By maximizing the objective function C, we can obtain the optimal external parameters for: The particle swarm optimization algorithm is used to optimize the objective function C. The specific particle swarm optimization process includes: Initialize the particle swarm: The number of particles N is 50, the dimension d of each particle is 6, corresponding to r = (r x , r y , r z ) T and t=(t x , t y , t z ) T , where the position of the i-th particle is at Y i =(r xi , r yi , r zi , t xi , t yi , t zi ), the speed is V i =(v xi , v yi , v zi ,u xi ,u yi ,u zi ), and i=0, 1,..., N-1; Initialize multiple particles using coarse calibration parameters (r0, t0): AND 0＝ (r0,t0) The remaining particles are initialized by adding random small perturbations d based on the initial solution: Y i =(r0,t0)+d,i=0,1,...,N-1 Calculate the fitness value of each particle: Substitute the i-th particle in the k-th iteration into the objective function C to obtain the fitness of the particle; take the maximum value of the objective function as the goal, and obtain the optimal position of the individual obtained by the particle swarm algorithm up to the k-th generation and the global optimal position Update particle velocity and position, where: The particle velocity update formula is: Where w is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers in the interval [0, 1]; the inertia weight indicates the degree to which the particle retains its current speed; At the beginning of the calculation, the maximum value of w is selected. With continuous iterations, the search ability of the particles is dynamically adjusted according to the linearly decreasing value of w: In the formula, w max and w min are the maximum and minimum values ​​of the inertia weight, respectively, k max is the maximum number of iterations; Update the particle position according to the velocity: When the maximum number of iterations or accuracy conditions are reached, the optimal solution for the external parameters of the lidar and camera is output.

7. The external parameter calibration method of mining laser radar and camera according to claim 1, characterized in that: The obtaining of the original image and the original point cloud in the target scene to be calibrated specifically includes: The original point cloud data of the acquisition data set comes from the 16-line GUJ50 mining intrinsically safe laser radar. The target scenes of the acquisition data set include indoor garages and outdoor environments in mining areas. The data set is synchronized and corrected at a rate of 10 Hz and contains 1120 frames of RGB images and fused frame point cloud data corresponding to the timestamp.

8. An external parameter calibration system for mining laser radar and camera, characterized in that: include: A data acquisition module is used to acquire the original image and original point cloud in the target scene to be calibrated; The original point cloud fusion module is used to perform preliminary alignment between the historical frame point cloud data and the current frame point cloud data through a RANSAC-based method based on the original point cloud data to obtain a preliminary alignment result; the preliminary alignment result is optimized based on the ICP fine registration method to obtain the LiDAR fusion frame point cloud; The original image target extraction module is used to perform panoramic segmentation on the original image to extract vehicles and traffic signs in the target scene as valid targets, and to construct a 2D-3D valid target centroid correspondence; The module for constructing the objective function of the panoramic information matching degree of effective targets is used to extract effective targets through panoramic segmentation based on the obtained laser radar fusion frame point cloud and the constructed 2D-3D effective target centroid correspondence, and to perform point cloud projection on the effective targets, and to project the effective target point cloud onto the segmentation mask after the inverse distance transformation through the coarsely calibrated external parameters; by assigning different weight parameters to the number of points of the effective targets projected on the matching map, the objective function of the panoramic information matching degree of effective targets is constructed; The module for obtaining the optimal solution of the external parameters is used to obtain the optimal solution of the external parameters of the lidar and the camera by optimizing the objective function according to the constructed effective target panoramic information matching degree objective function.

Images