An online calibration method and system for mining laser radar and camera

Abstract

The present invention discloses an online external parameter calibration method for a mining laser radar and camera, belonging to the field of mine detection. The calibration process relies on a laser radar, a camera, an information processing module, an illumination compensation module, and an external parameter calibration module. The laser radar and camera collect information; the information processing module is used to process laser point clouds and pixel images, the illumination compensation module provides an additional light source, and the external parameter calibration module performs external parameter calculations. The laser point cloud and image information of the pipeline area in the tunnel are collected, and then Kalman filtering is used for denoising. The centerline features of all pipelines in the point cloud and image are extracted, and the initial external parameters are obtained by hand-eye calibration. The final external parameters are obtained based on a matching algorithm. This method can automatically complete the external parameter calibration between the laser radar and the camera in the mine, laying the foundation for the efficient operation of the mine trackless rubber vehicle.

Description

Technical field

[0001] The present invention relates to an online calibration method and system for a mine-used laser radar and a camera, which is applicable to a mine environment and belongs to the field of mine detection. [Background Technology]

[0002] High-precision underground mine environmental mapping is crucial for ensuring effective operation. Mine surveying is essential for mine planning and design, exploration and construction, production and operations management, and even mine decommissioning. Roadways are crucial for ensuring production safety, and constructing high-precision environmental maps is essential for achieving efficient, safe, and productive mines. However, the complex road conditions within roadways and the instability of manual driving lead to frequent safety accidents underground. Therefore, the use of trackless rubber-coated mining vehicles in roadway environments can both ensure safety and improve production efficiency.

[0003] Autonomous driving for trackless rubber-coated mining vehicles primarily involves perception, planning, and decision-making. Perception is the prerequisite for all these processes. Relying solely on a single sensor to acquire information about the underground tunnel environment inevitably introduces significant cumulative errors. Furthermore, the harsh underground mine environment and severe interference make a single sensor inadequate for perception in the tunnel. The fusion of LiDAR and cameras can overcome the limitations of a single sensor in terms of information volume, stability, and accuracy, and is crucial for ensuring the safe operation of trackless mining vehicles.

[0004] Accurate extrinsic calibration of LiDAR and cameras is essential for achieving data fusion and precise sensing for trackless rubber-coated vehicles in mining operations. However, in actual mine tunnels, due to complex road conditions and harsh environments, calibrated LiDAR and cameras can experience severe extrinsic drift, necessitating recalibration of the sensors. Traditional checkerboard calibration methods suffer from low light conditions and inconvenience in mining environments. Therefore, it is necessary to propose an online calibration method that can utilize pipeline feature information in mines and perform calibration even in low-light environments, providing technical support for autonomous transportation of trackless rubber-coated vehicles in mining operations. [Summary of the invention]

[0005] To address the shortcomings of the above technologies, this paper provides an online calibration method for mining laser radar and cameras. This method can quickly and conveniently complete the automatic online calibration of mining trackless rubber-coated vehicles in the tunnel, laying the foundation for the efficient operation of mining trackless rubber-coated vehicles.

[0006] To achieve the above technical objectives, the present invention adopts the following technical solution: an online external parameter calibration method for a mining laser radar and camera, which is used for a mining trackless rubber vehicle equipped with a laser radar, a camera, an information acquisition module, an illumination compensation module, and an external parameter calibration module, and the steps are as follows:

[0007] Step 1: The mining trackless rubber vehicle uses a lidar and camera in the tunnel to simultaneously collect point clouds and images of a certain area in the tunnel. The camera takes pictures with the assistance of the illumination compensation module and sends the acquired point clouds and tunnel images to the information processing module.

[0008] Step 2: The information processing module uses the Kalman filter to perform noise reduction on the received point cloud and lane image data, removing outliers and discrete points in the point cloud and lane image data, and improving the quality of the image and point cloud;

[0009] Step 3: Perform image enhancement on the denoised lane image to improve the clarity of edge features and avoid possible local distortion of the image.

[0010] Step 4: Process the denoised point cloud, perform cylindrical surface fitting on all cylindrical pipes in the point cloud, and extract the central axis features of all fitted cylindrical pipes. Perform edge straight line fitting on the enhanced image, and extract the central axis of all cylindrical pipes in the image through symmetry detection.

[0011] Step 5: The central axis of the cylindrical pipe extracted from the point cloud and the central axis of the cylindrical pipe extracted from the image are transmitted to the external parameter calibration module (6). The external parameter calibration module (6) uses the input point cloud and the central axis of the cylindrical pipe extracted from the image to perform feature matching, and then establishes a hand-eye calibration equation. The hand-eye calibration algorithm is used to achieve a preliminary solution to the external parameters between the laser radar (2) and the camera (3). The point cloud coordinate system obtained by the laser radar is converted to the image coordinate system obtained by the camera, and the solution result is recorded in the external parameter calibration module (6);

[0012] Step 6: Project the point cloud representing the central axis feature in the point cloud using the external parameters solved in step 5 onto the pixel coordinate system where the central axis feature of the image pipeline is located, calculate the distance between each point cloud projection point and the nearest image pipeline central axis pixel point, and use the ICP algorithm to iterate the nearest point between each point cloud projection point constituting the central axis and the image pixel point. Through continuous iteration, reduce the distance error between the projection point and the pixel point, and finally complete the external parameter calibration between the lidar (2) and the camera (3).

[0013] Furthermore, the laser radar and camera are fixed in front of the mining trackless rubber-wheeled vehicle to ensure that there are no obstacles blocking the field of view in front of the laser radar and camera;

[0014] The information processing module connects the LiDAR and camera and is placed on the mining trackless rubber vehicle inside the dust cover. It is used to reduce noise, enhance features, and extract data collected by the LiDAR and camera.

[0015] The light compensation module is fixed on both sides of the mining trackless rubber vehicle to provide light source and compensate for the ambient light;

[0016] The external parameter calibration module is fixed on the mining trackless rubber vehicle and connected to the information processing module. The external parameter calibration module is performed through the characteristic information input by the information processing module.

[0017] Furthermore, the specific process of step 2 is as follows:

[0018] Define the state model and observation target of the point cloud and image. For the point cloud, select the spatial position of each point in the radar coordinate system and the reflection intensity of the point as the state model and observation target; for the image, select the gray value of each image pixel as the state model and observation target; initialize the built-in coefficients of the Kalman filter, and give the preset process noise covariance and observation noise covariance. By updating the state model of the point cloud and image, complete the denoising of the point cloud and image, and obtain the denoised radar point cloud P i With image pixel p i .

[0019] Furthermore, the specific process of step 3 is as follows:

[0020] The camera transmits the captured image information to the information processing module, which first processes the image through histogram equalization to ensure that the contrast of the entire image area is improved. The module then uses the multi-scale Retinex algorithm to further enhance the image contrast, brightness and color consistency, so that the details of both dim and bright areas in the image are preserved and the edge features in the image are highlighted.

[0021] Furthermore, the specific process of step 4 is as follows:

[0022] Preprocess the denoised point cloud and downsample it using voxelized grid filtering to prevent the number of point clouds from being excessive. Calculate the curvature of each downsampled point cloud, and calculate the minimum principal curvature k1 and maximum principal curvature k2 of each point.

[0023] The ratio of the minimum principal curvature k1 to the maximum principal curvature k2 of each point cloud is used for screening. If P i If k2 is greater than 100 times k1 at a point, the point is considered to belong to the point cloud of various pipelines laid in the tunnel, and the point is recorded as P i ′, stored in the information processing module for the next step; principal component analysis is used to analyze P i The point cloud in ′ is used to estimate the normal vector and construct the potential point semi-positive definite matrix as follows:

[0024]

[0025] Where n iRepresents the normal vector of the point cloud at point i, and T represents the transposed matrix;

[0026] The eigenvector of the potential point semi-positive definite matrix corresponding to the minimum eigenvalue The axial direction of the pipe where the potential point is located is considered;

[0027] The axis The point cloud on is projected onto a feature vector On the two-dimensional plane with the normal line passing through the origin O of the laser radar coordinate system L, each projection point is recorded as The distribution shape of all projected points is a circle. The standard equation of the circle is established and converted into a form suitable for least squares processing:

[0028]

[0029] The radius r and the center of the projected circle are fitted by algebraic fitting. To confirm, minimize the algebraic distance in the following formula:

[0030]

[0031] Where, the A matrix contains the transformed x and y coordinates of each projection point, each row represents a data, and the b vector contains the x and y coordinates of each point. 2 +y 2 , represents the constant term in the circle equation;

[0032] The center of the circle By transforming back to the three-dimensional coordinates, we can get a point on the central axis of the pipeline laid in the roadway point cloud, and record this point on the central axis as P * ;

[0033] Apply the above process iteratively to the potential point P i ′, and in each iteration, the point cloud is updated using the normal and distance standards, and all the axis vectors finally obtained are Perform point cloud interpolation to construct a feature set of the central axis of the pipeline laid in the tunnel

[0034] For the denoised lane image, it is converted into a grayscale image, and the gradient is calculated using the Sobel operator to obtain the gradient matrix G of the pixel image. It is determined whether the pixel gradient of any point is a maximum value in the neighborhood, and non-maximum pixel gradients are suppressed to eliminate spurious responses.

[0035] Set up adaptive threshold d C , the size varies with the extracted gradient value, and the threshold d is set C Determine whether the image pixel meets the edge feature requirements. If the gradient of the point is lower than d C, it is suppressed and not considered as an edge point, and is higher than d C Then the point is retained, and all the feature pixels that meet the requirements constitute the feature pixel set p i , and p i Identify pipeline contours in tunnels;

[0036] The skeletonization algorithm is applied to the identified pipeline contour features to simplify the contour into a one-dimensional central axis and the pixels representing the central axis are saved in the feature set. In the example, the vector of the central axis is denoted as

[0037] Furthermore, the specific process of step 5 is as follows:

[0038] Assume that the external parameter transformation matrix of the laser radar coordinate system to the camera coordinate system is: Then the time transformation from the lidar at time i to the camera at time j is: The relative position between the lidar and the camera remains unchanged. From time k-1 to time k, the relationship between the position changes of the lidar and the camera and their relative position is described as follows:

[0039]

[0040] Solve the above formula to obtain the initial external parameter matrix:

[0041]

[0042] Where, represents the matrix required for camera (3) to change its posture from time k-1 to time k; represents the matrix required by the laser radar (2) to change its posture from time k-1 to time k; and Represents the rotation matrix and translation vector in the extrinsic matrix.

[0043] Furthermore, the specific process of step 6 is as follows:

[0044] The initial external parameter matrix obtained in step 5 As the starting point of the ICP algorithm iteration, the feature set Convert to pixel coordinate system, Convert to pixel coordinate system Later, Perform neighborhood search within and establish feature set At The corresponding relationship between them is established, and the following error function is established:

[0045]

[0046] Then use the vector direction between line features to establish the angle error function:

[0047]

[0048] Where, e point (R,t) is the error function between the point cloud and the pixel point in the central axis, e arc (R) is the error function of the central axis angle; R and t represent the rotation matrix and translation vector respectively; c1, c2 are weight coefficients, and Represent the central axis vectors of each pipeline in the point cloud and image respectively;

[0049] The total error function is established as follows, and the current total error function e is calculated all The specific value of (R,t):

[0050] e all (R,t)=e point (R,t)+e arc (R)

[0051] Search by KD-tree Each point cloud in The nearest neighbor value l in the pixel set neighbor ;

[0052] Calculate the projection point set With pixel set The centroid coordinates of , and using singular value decomposition, the calculation of the extrinsic parameter matrix is ​​selected according to the following optimization problem to solve the rotation matrix with the minimum error:

[0053]

[0054] Where q i Represents the projected point set The center of mass, Represents a set of pixels The center of mass is obtained by taking the average value of the coordinates of all points in the point set; according to the obtained R * , calculate t * , get the external parameter matrix R * With t * , and use the solved R * With t * Restart (R * t * )·P i conversion;

[0055] Set the iteration direction threshold d e, calculate the external parameter matrix (R * t * ), the error function e all (R * ,t * ) and e all (R * ,t * ) and e all (R, t) for comparison, if e all (R * ,t * ) value increases, and the increase value exceeds the threshold d e , then the iteration direction is considered to be wrong, and the KD-tree is used to search again Every point in In addition to l neighbor The nearest neighbor value outside Change the iteration direction;

[0056] Set up the iteration stop threshold e, and judge whether the current external parameter (R * t * ), the error function e all Is (R,t) less than or equal to e? If so, then (R * t * ) as the final external parameter, otherwise repeat step 6.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. This invention establishes an iterative error threshold judgment scheme, which can change the iteration direction when the error exceeds the threshold. This reduces the local convergence phenomenon of traditional ICP algorithms and improves the stability of lidar and camera calibration in mine environments.

[0059] 2. The present invention uses various pipes laid in underground mine tunnels as calibration features. Compared with the complex tunnel surface under the mine, cylindrical feature fitting and center axis extraction are performed on the various cylindrical pipes laid. The stable pipe structure is easy to detect automatically, which can reduce external interference in the external parameter calibration process and improve the feature utilization and calibration accuracy under the mine.

Brief Description of the Drawings

[0060] Figure 1 Schematic diagram of the process of online calibration method of mining laser radar and camera in an embodiment of the present invention;

[0061] Figure 2 Schematic diagram of the process of centerline feature matching in an embodiment of the present invention;

[0062] Figure 3Schematic diagram of the structure of the online calibration system of mining laser radar and camera in an embodiment of the present invention.

[0063] In the figure, 1-mining trackless rubber vehicle, 2-laser radar, 3-camera, 4-information acquisition module, 5-illumination compensation module, 6-external parameter calibration module, 7-dust cover.

Specific implementation method

[0064] The following further describes the embodiments of the present invention in conjunction with the accompanying drawings:

[0065] like Figure 1 and Figure 2 As shown, the present invention discloses an online calibration method for mining laser radar and camera, which collects laser point cloud and image information of the pipeline area in the tunnel, then uses Kalman filtering to remove noise, extracts the centerline features of all pipelines in the point cloud and image, uses hand-eye calibration to obtain initial extrinsic parameters, and obtains final extrinsic parameters based on a matching algorithm. The specific steps are as follows:

[0066] Step 1: The mining trackless rubber vehicle 1 uses the laser radar 2 and the camera 3 in the tunnel to simultaneously collect the point cloud and image of a certain area in the tunnel. The camera 3 takes pictures with the assistance of the illumination compensation module 5 and sends the acquired point cloud and tunnel image to the information processing module 4;

[0067] Step 2: The information processing module 4 uses a Kalman filter to perform noise reduction on the received point cloud and lane image data, removes outliers and discrete points in the point cloud and lane image data, and improves the quality of the image and point cloud;

[0068] Define the state model and observation target of the point cloud and image. For the point cloud, select the spatial position of each point in the radar coordinate system and the reflection intensity of the point as the state model and observation target; for the image, select the gray value of each image pixel as the state model and observation target; initialize the built-in coefficients of the Kalman filter, and give the preset process noise covariance and observation noise covariance. By updating the state model of the point cloud and image, complete the denoising of the point cloud and image, and obtain the denoised radar point cloud P i With image pixel p i ;

[0069] Step 3: Perform image enhancement on the denoised lane image to improve the clarity of edge features and avoid possible local distortion of the image.

[0070] Camera 3 transmits the captured image information to information processing module 4. Information processing module 4 first processes the image through histogram equalization to ensure that the contrast of the entire image area is improved. Then, the multi-scale Retinex algorithm is used to further enhance the image contrast, brightness, and color consistency. This ensures that details in both dark and bright areas of the image are preserved and edge features in the image are highlighted.

[0071] Step 4: Process the denoised point cloud, perform cylindrical surface fitting on all cylindrical pipes in the point cloud, and extract the central axis features of all fitted cylindrical pipes. Perform edge straight line fitting on the enhanced image, and extract the central axis of all cylindrical pipes in the image through symmetry detection.

[0072] Preprocess the denoised point cloud and downsample it using voxelized grid filtering to prevent the number of point clouds from being excessive. Calculate the curvature of each downsampled point cloud, and calculate the minimum principal curvature k1 and maximum principal curvature k2 of each point.

[0073] The ratio of the minimum principal curvature k1 to the maximum principal curvature k2 of each point cloud is used for screening. If P i If k2 is greater than 100 times k1 at a point, the point is considered to belong to the point cloud of various pipelines laid in the tunnel, and the point is recorded as P i ′, stored in the information processing module for the next step; principal component analysis is used to analyze P i The point cloud in ′ is used to estimate the normal vector and construct the potential point semi-positive definite matrix as follows:

[0074]

[0075] Where n i Represents the normal vector of the point cloud at point i, and T represents the transposed matrix;

[0076] The eigenvector of the potential point semi-positive definite matrix corresponding to the minimum eigenvalue The axial direction of the pipe where the potential point is located is considered;

[0077] The axis The point cloud on is projected onto a feature vector On the two-dimensional plane with the normal line passing through the origin O of the laser radar coordinate system L, each projection point is recorded as The distribution shape of all projected points is a circle. The standard equation of the circle is established and converted into a form suitable for least squares processing:

[0078]

[0079] The radius r and the center of the projected circle are fitted by algebraic fitting. To confirm, minimize the algebraic distance in the following formula:

[0080]

[0081] Wherein, the A matrix contains the transformed x and y coordinates of each projection point, and each row represents a data. b The vector contains the x of each point 2 +y 2 , represents the constant term in the circle equation;

[0082] The center of the circle By transforming back to the three-dimensional coordinates, we can get a point on the central axis of the pipeline laid in the roadway point cloud, and record this point on the central axis as P * ;

[0083] Apply the above process iteratively to the potential point P i ′, and in each iteration, the point cloud is updated using the normal and distance standards, and all the axis vectors finally obtained are Perform point cloud interpolation to construct a feature set of the central axis of the pipeline laid in the tunnel

[0084] For the denoised lane image, it is converted into a grayscale image, and the gradient is calculated using the Sobel operator to obtain the gradient matrix G of the pixel image. It is determined whether the pixel gradient of any point is a maximum value in the neighborhood, and non-maximum pixel gradients are suppressed to eliminate spurious responses.

[0085] Set up adaptive threshold d C , the size varies with the extracted gradient value, and the threshold d is set C Determine whether the image pixel meets the edge feature requirements. If the gradient of the point is lower than d C , it is suppressed and not considered as an edge point, and is higher than d C Then the point is retained, and all the feature pixels that meet the requirements constitute the feature pixel set p i , and p i Identify pipeline contours in tunnels;

[0086] The skeletonization algorithm is applied to the identified pipeline contour features to simplify the contour into a one-dimensional central axis and the pixels representing the central axis are saved in the feature set. In the example, the vector of the central axis is denoted as

[0087] Step 5: The central axis of the cylindrical pipe extracted from the point cloud and the central axis of the cylindrical pipe extracted from the image are transmitted to the external parameter calibration module 6. The external parameter calibration module 6 first uses the input point cloud and the pipeline central axis features of the image to perform feature matching, and then establishes a hand-eye calibration equation. The hand-eye calibration algorithm is used to achieve a preliminary solution to the external parameters between the laser radar (2) and the camera (3). The point cloud coordinate system obtained by the laser radar 2 is converted to the image coordinate system obtained by the camera 3, and the solution result is recorded in the external parameter calibration module 6.

[0088] Assume that the external parameter transformation matrix of the laser radar coordinate system of the laser radar (2) is converted to the camera (3) coordinate system: Then the time transformation from the laser radar (2) at time i to the camera (3) at time j is: The relative position between the laser radar (2) and the camera (3) remains unchanged. During the period from time k-1 to time k, the relationship between the position changes and relative position of the laser radar (2) and the camera (3) can be described as follows:

[0089]

[0090] Solve the above formula to obtain the initial external parameter matrix:

[0091]

[0092] Where, represents the matrix required for camera (3) to change its posture from time k-1 to time k; represents the matrix required by the laser radar (2) to change its posture from time k-1 to time k; and Represents the rotation matrix and translation vector in the external parameter matrix;

[0093] Step 6: Project the point cloud representing the central axis feature in the point cloud onto the pixel coordinate system where the central axis feature of the image pipeline is located using the external parameters solved in step 5, calculate the distance between each point cloud projection point and the nearest pixel point of the central axis of the image pipeline, and use the ICP algorithm to perform the closest point iterative operation between the point cloud projection point and the image pixel point. Through continuous iteration, reduce the distance error between the projection point and the pixel point, and finally complete the external parameter calibration between the lidar (2) and the camera (3);

[0094] The initial external parameter matrix obtained in step 5 As the starting point of the ICP algorithm iteration, the feature set Convert to pixel coordinate system, Convert to pixel coordinate system Later, Perform neighborhood search within and establish feature set At The corresponding relationship between them is established, and the following error function is established:

[0095]

[0096] Then use the vector direction between line features to establish the angle error function:

[0097]

[0098] Where, e point (R,t) is the error function between the point cloud and the pixel point in the central axis, e arc (R) is the error function of the central axis angle; R and t represent the rotation matrix and translation vector respectively; c1, c2 are weight coefficients, and Represent the central axis vectors of each pipeline in the point cloud and image respectively;

[0099] The total error function is established as follows, and the current total error function e is calculated all The specific value of (R,t):

[0100] e all (R,t)=e point (R,t)+e arc (R)

[0101] Search by KD-tree Each point cloud in The nearest neighbor value l in the pixel set neighbor ;

[0102] Calculate the projection point set With pixel set The centroid coordinates of , and using singular value decomposition, the calculation of the extrinsic parameter matrix is ​​selected according to the following optimization problem to solve the rotation matrix with the minimum error:

[0103]

[0104] Where q i Represents the projected point set The center of mass, Represents a set of pixels The center of mass is obtained by taking the average value of the coordinates of all points in the point set; according to the obtained R * , calculate t * , get the external parameter matrix R * With t * , and use the solved R * With t * Restart (R *t * )·P i conversion;

[0105] Set the iteration direction threshold d e , calculate the external parameter matrix (R * t * ), the error function e all (R * ,t * ) and e all (R * ,t * ) and e all (R, t) for comparison, if e all (R * ,t * ) value increases, and the increase value exceeds the threshold d e , then the iteration direction is considered to be wrong, and the KD-tree is used to search again Every point in In addition to l neighbor The nearest neighbor value outside Change the iteration direction;

[0106] Set up the iteration stop threshold e, and judge whether the current external parameter (R * t * ), the error function e all Is (R,t) less than or equal to e? If so, then (R * t * ) as the final external parameter, otherwise repeat step 6;

[0107] like Figure 3 As shown, an online calibration system for mining laser radar and camera includes:

[0108] The laser radar 2 and the camera 3 are fixed in front of the mining trackless rubber-wheeled vehicle 1, inside the dust cover 7, to ensure that there are no obstacles blocking the field of view in front of the laser radar 2 and the camera 3;

[0109] The information processing module 4 is connected to the laser radar 2 and the camera 3 and is placed on the mining trackless rubber vehicle 1 to reduce noise, enhance features and extract data collected by the laser radar 2 and the camera 3;

[0110] The light compensation module 5 is fixed on both sides of the mining trackless rubber vehicle 1 to provide light source and perform light compensation for the environment;

[0111] The external parameter calibration module 6 is fixed on the mining trackless rubber vehicle 1 and is connected to the information processing module 4 , and calibrates the external parameters through the characteristic information input by the information processing module 4 .

Claims

1. A method for online external parameter calibration of mining laser radar and camera, characterized in that: The online calibration system of the mining laser radar and camera includes a mining trackless rubber vehicle (1) provided with a laser radar (2), a camera (3), an information acquisition module (4), an illumination compensation module (5), and an external parameter calibration module (6), and the steps are as follows: Step 1: The mining trackless rubber vehicle (1) uses a laser radar (2) and a camera (3) in a tunnel to simultaneously collect a point cloud and an image of a certain area in the tunnel. The camera (3) takes pictures with the assistance of a light compensation module (5) and sends the acquired point cloud and tunnel image to an information processing module (4); Step 2: The information processing module (4) uses the Kalman filter to perform noise reduction on the received point cloud and lane image data, removes outliers and discrete points in the point cloud and lane image data, and improves the quality of the image and point cloud; Step 3: Perform image enhancement on the denoised lane image to improve the clarity of edge features and avoid possible local distortion of the image. Step 4: Process the denoised point cloud, perform cylindrical surface fitting on all cylindrical pipes in the point cloud, and extract the central axis features of all fitted cylindrical pipes. Perform edge straight line fitting on the enhanced image, and extract the central axis of all cylindrical pipes in the image through symmetry detection. Step 5: The central axis of the cylindrical pipe extracted from the point cloud and the central axis of the cylindrical pipe extracted from the image are transmitted to the external parameter calibration module (6). The external parameter calibration module (6) uses the input point cloud and the central axis of the cylindrical pipe extracted from the image to perform feature matching, and then establishes a hand-eye calibration equation. The hand-eye calibration algorithm is used to achieve a preliminary solution to the external parameters between the laser radar (2) and the camera (3). The point cloud coordinate system obtained by the laser radar (2) is converted to the image coordinate system obtained by the camera (3), and the solution result is recorded in the external parameter calibration module (6); Step 6: Project the point cloud representing the central axis feature in the point cloud using the external parameters solved in step 5 onto the pixel coordinate system where the central axis feature of the image pipeline is located, calculate the distance between each point cloud projection point and the nearest pixel point of the central axis of the image pipeline, and use the ICP algorithm to iterate the nearest point between each point cloud projection point constituting the central axis and the image pixel point. Through continuous iteration, reduce the distance error between the projection point and the pixel point, and finally complete the external parameter calibration between the lidar (2) and the camera (3); With the initial external parameter matrix As the starting point of the ICP algorithm iteration, the feature set Convert to pixel coordinate system, After conversion to pixel coordinate system, Perform neighborhood search within and establish feature set and The corresponding relationship between them is established, and the following error function is established: ； Then use the vector direction between line features to establish the angle error function: ； Where, is the error function between the point cloud and the pixel point in the central axis, is the error function of the central axis angle; and Represent the rotation matrix and translation vector respectively; is the weight coefficient, and Represent the central axis vectors of each pipeline in the point cloud and image respectively; The total error function is established as follows, and the current total error function is calculated The specific numerical size of: ； Search by KD-tree Each point cloud in The nearest neighbor value in the pixel set ; Calculate the projection point set With pixel set The centroid coordinates of , and using singular value decomposition, the calculation of the extrinsic parameter matrix is ​​selected according to the following optimization problem to solve the rotation matrix with the minimum error: ； Where, Represents the projected point set The center of mass, Represents a set of pixels The center of mass is obtained by taking the average value of the coordinates of all points in the point set; according to the obtained , calculate , and get the external parameter matrix and , and use the solved and Re-do conversion; Setting the iteration direction threshold , calculate the external parameter matrix of the current external parameter iteration step Next, the error function size, and will and For comparison, if The value of increases and exceeds the threshold , then the iteration direction is considered to be wrong, and the KD-tree is used to search again Every point in In addition The nearest neighbor value outside , change the iteration direction; Setting the iteration stopping threshold , judge in the current external reference Next, the error function Is it less than or equal to If so, then As the final external parameter, otherwise repeat step 6.

2. The online extrinsic parameter calibration method for a mining laser radar and camera according to claim 1, characterized in that: The laser radar (2) and the camera (3) are fixed in front of the mining trackless rubber-wheeled vehicle (1), ensuring that there are no obstacles blocking the field of view in front of the laser radar (2) and the camera (3); The information processing module (4) is connected to the laser radar (2) and the camera (3), and is placed on the mining trackless rubber vehicle (1) and inside the dust cover (7), and is used for performing noise reduction, feature enhancement and extraction on the data collected by the laser radar (2) and the camera (3); The light compensation module (5) is fixed on both sides of the mining trackless rubber vehicle (1) and is used to provide a light source to compensate for the light of the environment; The external parameter calibration module (6) is fixed on the mining trackless rubber vehicle (1) and is connected to the information processing module, and calibrates the external parameters through the characteristic information input by the information processing module (5).

3. The online external parameter calibration method for a mining laser radar and camera according to claim 1 is characterized in that: The specific process of step 2 is: Define the state model and observation target of the point cloud and image. For the point cloud, select the spatial position of each point in the radar coordinate system and the reflection intensity of the point as the state model and observation target; for the image, select the grayscale value of each image pixel as the state model and observation target; initialize the built-in coefficients of the Kalman filter, and give the preset process noise covariance and observation noise covariance. By updating the state model of the point cloud and image, complete the denoising of the point cloud and image, and obtain the denoised radar point cloud With image pixels .

4. The online external parameter calibration method for a mining laser radar and camera according to claim 1, characterized in that: The specific process of step 3 is: The camera (3) transmits the captured image information to the information processing module (4). The information processing module (4) first processes the image through histogram equalization to ensure that the contrast of the entire image area is improved, and then uses the multi-scale Retinex algorithm to further enhance the image contrast, brightness and color consistency, so that the details of the dark and bright areas in the image are retained and the edge features in the image are highlighted.

5. The online external parameter calibration method for a mining laser radar and camera according to claim 1 is characterized in that: The specific process of step 4 is: Preprocess the denoised point cloud and downsample it by voxelized grid filtering to prevent the point cloud from being too numerous. Calculate the curvature of each downsampled point cloud and calculate the minimum principal curvature of each point. With the maximum principal curvature ; The minimum principal curvature of each point cloud With the maximum principal curvature The ratio of is screened, if Point More than 100 times , then the point is considered to belong to the point cloud of various pipelines laid in the tunnel, and the point is recorded as , stored in the information processing module for the next step; principal component analysis is used to The point cloud in is used to estimate the normal vector and construct the potential point semi-positive definite matrix as follows: ； Where, Representative The normal vector of the point cloud at point, represents the transposed matrix; The eigenvector of the potential point semi-positive definite matrix corresponding to the minimum eigenvalue The axial direction of the pipe where the potential point is located is considered; The axis The point cloud on is projected onto a feature vector is a normal and passes through the lidar coordinate system origin On the two-dimensional plane, each projection point is recorded as , all the projection points are distributed in a circle. The standard equation of the circle is established and converted into a form suitable for least squares processing: ； The radius of the projected circle is calculated by algebraic fitting. With the center To confirm, minimize the algebraic distance in the following formula: ； Where, The matrix contains the transformed and Coordinates, each row represents a data, The vector contains the , represents the constant term in the circle equation; The center of the circle Transform back to the three-dimensional coordinates, that is, get a point on the central axis of the pipeline laid in the roadway point cloud, and record the point on the central axis as ; Apply the above process iteratively to potential points , and in each iteration, the point cloud is updated using the normal and distance standards, and all the axis vectors finally obtained are Perform point cloud interpolation to construct a feature set of the central axis of the pipeline laid in the tunnel ; For the denoised lane image, convert it into a grayscale image and calculate the gradient according to the Sobel operator to obtain the gradient matrix of the pixel image ; Determine whether the pixel gradient of any point is a maximum value in the neighborhood, and suppress the non-maximum pixel gradient to eliminate spurious responses; Setting Adaptive Thresholds , the size varies with the extracted gradient value, and the threshold is set Determine whether the image pixel meets the edge feature requirements. If the gradient of the point is lower than , it is suppressed and not used as an edge point, and is higher than Then the point is retained, and all the feature pixels that meet the requirements constitute the feature pixel set , and Identify pipeline contours in tunnels; The skeletonization algorithm is applied to the identified pipeline contour features to simplify the contour into a one-dimensional central axis and the pixels representing the central axis are saved in the feature set. In the example, the vector of the central axis is denoted as .

6. The online external parameter calibration method for a mining laser radar and camera according to claim 1 is characterized in that: The specific process of step 5 is: Assume that the external parameter transformation matrix of the laser radar coordinate system to the camera coordinate system is: , then the laser radar at time to the camera in time The time transformation is: , the relative pose between the lidar and the camera remains unchanged. Time's up Over time, the relationship between the position changes and relative positions of the lidar and camera can be described as follows: ； Solve the above formula to obtain the initial external parameter matrix: ； Where, Indicates that the camera (3) is from Time has come The matrix required to change its posture at all times; Indicates that the laser radar (2) Time has come The matrix required to change its posture at all times; and Represents the rotation matrix and translation vector in the extrinsic matrix.

Images