Camera and laser radar calibration device and method for tunnel boring machine

Abstract

The present invention discloses a camera and laser radar calibration device and method for a tunneling machine, which belongs to the field of mine tunneling. It includes a calibration plate rotation module that is arranged on the top of the tunneling machine and can be moved according to the position of the camera radar fusion module arranged on the tunneling machine and used as a calibration. The calibration plate rotation module includes a calibration plate, and a universal mechanical arm that can move freely and take into account all the laser radars and industrial cameras arranged on the tunneling machine is arranged below the calibration plate; a chessboard is arranged in the calibration plate, and four hollow circles are arranged on the calibration plate in the four directions of the upper, lower, left and right of the chessboard, respectively, wherein four ArUco codes are arranged on the calibration plate on the outside of the four corners of the chessboard between every two hollow circles. It can better adapt to the harsh environment under the mine, and provide support for the positioning, navigation, and automatic control of the tunneling machine without blocking the field of view of the sensor, which not only ensures that mining activities can be carried out efficiently and safely, but also increases the visual range of the camera and radar.

Description

[Technical field]

[0001] The invention relates to a camera and laser radar calibration device and method for a tunneling machine, belonging to the technical field of mine tunneling. [Background technology]

[0002] Mine excavation technology occupies an important position in the development of the mining industry. However, due to the complex mining environment, low visibility, and reliance on manual operation by workers, excavation errors are large, and over-excavation and under-excavation are easy to occur. Therefore, the automation and digitization of mine excavation are crucial to improving the efficiency and sustainability of mining. The three-dimensional reconstruction of underground mine tunnels can effectively realize the visualization of unknown environments, provide solutions for real-time display of the posture of the tunneling machine relative to the tunnel and realize directional excavation, especially the use of cameras and laser radars for data collection, which can make up for the lack of information, accuracy, and stability of a single sensor, and is of great significance to the realization of intelligent and automated operation of coal mine equipment and the improvement of coal mine excavation efficiency.

[0003] The calibration of camera and LiDAR is the basis for the fusion of the two data, and is a key step to ensure data consistency and accuracy, and to achieve accurate three-dimensional reconstruction and positioning of the tunnel boring machine. However, in the actual mining working environment, calibration often takes a lot of time, complex processes and high costs, which is not conducive to the realization of intelligent and automated mining excavation. At the same time, in actual mining excavation, due to the vibration of the tunnel boring machine equipment and the regular maintenance and replacement of underground equipment, visual parameter drift is prone to occur between the camera and LiDAR. This requires the equipment to be able to quickly and conveniently complete a new round of visual calibration, but the existing visual calibration technology has great limitations in mines, and the calibration process is not convenient enough, and it is impossible to fully and quickly calibrate the sensors at multiple positions on the tunnel boring machine. Therefore, it is necessary to design a more flexible device and method suitable for the calibration of multiple visual sensors on the tunnel boring machine to provide technical support for the realization of intelligent mining excavation. [Summary of the invention]

[0004] In view of the shortcomings of the above-mentioned technology, a camera and laser radar calibration device and method for a roadheader are provided. Through the device and method, visual calibration can be completed quickly and conveniently without affecting the normal operation of the roadheader's visual module.

[0005] To achieve the above-mentioned purpose, the present invention discloses a camera and laser radar calibration device for a roadheader, comprising a calibration plate rotation module which is arranged on the top of the roadheader and can be moved according to the position of a camera radar fusion module arranged on the roadheader and used as a calibration plate, the calibration plate rotation module comprises a calibration plate, the camera radar fusion module comprises a laser radar and an industrial camera, a universal mechanical arm which can move freely and take into account all the laser radars and industrial cameras arranged on the roadheader is arranged below the calibration plate; the calibration plate is a rectangular plate structure, a checkerboard for calibration is arranged in the center of the calibration plate, and four directions of the checkerboard, namely, up, down, left, and right, are arranged on the calibration plate There are four hollow circles, among which four ArUco codes are respectively arranged on the calibration board outside the four corners of the chessboard between every two hollow circles. The four ArUco codes are tilted according to the diagonal of the calibration board. The IDs corresponding to the four ArUco codes are 0, 1, 2, and 3, respectively, and are arranged in a clockwise order; the laser radar and the industrial camera use the depth difference of the hollow circles for edge extraction; the industrial camera uses the chessboard to obtain the initial calibration parameters of the intrinsic calibration and the extrinsic calibration respectively, and the laser radar uses the chessboard to obtain the initial calibration parameters of the extrinsic calibration; the industrial camera uses the ArUco code for rapid positioning and error correction.

[0006] Furthermore, the camera radar fusion module includes a sensor connection block, on which a laser radar and an industrial camera are respectively provided on the left and right sides, wherein side support blocks are respectively provided on the outer sides of the laser radar and the industrial camera; the camera radar fusion module is fixed above the tunnel boring machine body through a pitch rotating platform, and the pitch rotating platform includes a rotating base fixed on the tunnel boring machine body, and a pitch bracket is provided on the rotating base, and a driving motor is fixed on the pitch bracket by bolts, and the driving motor is connected to the side support blocks on both sides of the camera radar fusion module and provides driving force; the pitch rotating platform realizes an angle adjustment of the pitch bracket within a range of 360° in the horizontal plane through the rotating base, and the driving motor is fixed to the pitch bracket by bolts, driving the camera radar fusion module to realize 180° adjustment in the vertical direction.

[0007] Furthermore, the bottom of the universal robotic arm is arranged on the tunnel boring machine through a truncated cone rotating platform, the truncated cone rotating platform includes a supporting base fixed on the tunnel boring machine body, and a rotatable truncated cone is provided on the supporting base; the universal robotic arm includes a robotic arm swing mechanism, which includes a lower rocker arm and an upper rocker arm connected in sequence, the lower rocker arm is connected to the rotatable truncated cone through a cylindrical pin, the end of the lower rocker arm is connected to the upper rocker arm through a cylindrical pin, and a hydraulic telescopic rod is connected to the end of the upper rocker arm; the truncated cone rotating platform can rotate 360° in a horizontal plane through the rotatable rotating truncated cone and the supporting base, thereby driving the universal robotic arm swing mechanism and the calibration plate rotation module to adjust the angle in the horizontal direction.

[0008] Furthermore, the calibration plate rotation module includes a rotating disc base connected to the hydraulic telescopic rod, and the forward and backward movement of the calibration plate rotation module is controlled by the hydraulic telescopic rod. A rotatable rotating disc is installed on the rotating disc base, and a connecting base is provided on the rotating disc. A driving motor and a rotating bracket are installed on the connecting base, and a calibration plate is installed on the rotating bracket for providing a chessboard required for calibration, including corner point information of the chessboard, center point information of the hollow circle and ID information of four ArUco codes; the calibration plate rotation module is connected to the hydraulic telescopic rod through a rotating disc base, and a rotating disc is installed on the rotating disc base. The connecting base and the rotating disc are fixedly connected together, and a driving motor, a rotating bracket and a calibration plate are installed thereon. The calibration plate can be adjusted to different angles through the rotating disc and the driving motor, so that the calibration plate is in an observable posture of the camera and lidar fusion module.

[0009] Furthermore, the robot arm swing mechanism is located between the truncated table rotating platform and the calibration plate rotation module. The lower rocker arm of the robot arm swing mechanism is connected to the rotating truncated table and the upper rocker arm through cylindrical pins. The hydraulic telescopic rod is arranged at the top of the upper rocker arm. The lower rocker arm and the upper rocker arm can both rotate 180° in the vertical plane, and they are in a driven relationship. When the lower rocker arm rotates in the vertical plane, the upper rocker arm and the hydraulic telescopic rod need to be controlled to move at the same time, so as to adjust the spatial position of the calibration plate rotation module in the vertical and horizontal directions.

[0010] A working method for a camera and laser radar calibration device for a tunnel boring machine, the steps of which are as follows:

[0011] Remotely control the rotation base and the pitch bracket of the pitch rotation platform to rotate, and adjust the horizontal and vertical angles of the camera radar fusion module in real time to ensure that the observation direction of the camera radar fusion module is facing the calibration board during calibration;

[0012] Remotely control the rotating table of the rotating table platform to rotate, and adjust the horizontal angle of the calibration plate in real time to ensure that the angle deviation between the front of the calibration plate and the observation direction of the camera radar fusion module in the axial direction around the rotating table is controlled within 30° during calibration;

[0013] Remotely control the rotation angle of the upper and lower rocker arms of the universal manipulator, the telescopic distance of the hydraulic telescopic rod, and adjust the horizontal and vertical positions of the calibration board in real time to ensure that the camera radar fusion module can observe the calibration board at multiple positions, and that all the corner points of the chessboard on the calibration board should be within the field of view of the camera radar fusion module, and the corner points of the chessboard should not be missing or blurred in the field of view of the camera radar fusion module;

[0014] Remotely control the rotation disk and drive motor of the calibration plate rotation module to rotate, so as to adjust the angle of the calibration plate in the radial and axial directions of the upper rocker arm, ensure that the angle deviation between the front of the calibration plate and the observation direction of the camera radar fusion module in the axial direction of the drive motor is controlled within 30°, and can rotate 360° around the axial direction of the upper rocker arm, so as to facilitate the stowage after calibration;

[0015] Through the joint adjustment of the pitch rotating platform, the truncated rotating platform, the mechanical arm swing mechanism, and the calibration plate rotation module, ensure that before and during the calibration, the camera radar fusion module needs to capture multiple different relative viewing angles with the calibration plate to collect point cloud data and image data, and the industrial camera is collecting the calibration plate image data and the laser radar is collecting the point cloud data. It is necessary to ensure that all the corners of the calibration plate are within the field of view of the camera radar fusion module, and there is no missing corner point or blurred corner point of the calibration plate due to angle and position reasons. After the calibration data is collected, the control calibration plate is attached to the main body of the tunnel boring machine and is ready to be unfolded when it needs to be calibrated again.

[0016] During calibration, attention should be paid to the noise of the laser radar and industrial camera at the current position. If the noise suddenly increases due to excessive dust in a certain area due to dusty environment factors at the current position, the camera radar fusion module is rotated and pitched by the pitch rotation mechanism to change the viewing angle of the camera radar module until the noise in the observation angle of the laser radar and industrial camera is reduced. The round table rotating platform, the mechanical arm swing mechanism, and the calibration plate rotation module are controlled by the remote control system to make the calibration plate within the observation range of the current camera radar fusion module, and there is no missing corner point or blurred corner point phenomenon. All mechanisms are kept stationary in this state, and the industrial camera is used to collect image information of the calibration plate, and the laser radar is used to collect radar point cloud data for more than 10 seconds.

[0017] The position of the camera radar module is kept unchanged, and the relative posture of the calibration plate and the camera radar module is changed by adjusting the position of the calibration plate, so that the acquired calibration plate image information is tilted by 10° in the four directions of up, down, left, and right respectively: the spatial position of the calibration plate is adjusted by the movement of the truncated table rotating platform, the mechanical arm swing mechanism, and the calibration plate rotating module, so as to ensure that the camera radar fusion module can collect the camera image and radar point cloud of the calibration plate (22) in the above-mentioned tilted direction at different distances from the calibration plate (22);

[0018] After the camera radar module has collected the image data and point cloud data of the calibration plate (22), the calibration plate (22) is folded and retracted under the joint adjustment of the truncated table rotating platform, the mechanical arm swing mechanism, and the calibration plate rotating module, so that it can be attached to the main body of the tunnel boring machine after the calibration is completed without blocking the sensor's field of view.

[0019] If a laser radar and camera for obstacle detection and autonomous navigation are installed in front of or on the side of the tunnel boring machine body, when the laser radar and camera at that position need to be recalibrated, the calibration plate is adjusted to that position, and multiple sets of data are collected by continuously changing the relative position of the calibration plate relative to the laser radar and camera at that position to complete the sensor fusion calibration of the laser radar and camera at that position.

[0020] Furthermore, the camera radar fusion module will upload the collected point cloud and image data wirelessly. The internal reference data of the industrial camera will first be calibrated through MATLAB, and the image information of the checkerboard in the calibration plate at different angles and distances will be collected to ensure that the number of images is not less than 20. The camera calibration toolbox of MATLAB is used to extract the characteristic corner points on the checkerboard to calculate the internal reference of the industrial camera. If there is an image with a calibration error greater than 0.5 pixels, the corresponding image will be deleted and recalibrated; the final internal reference calibration result will be saved for the subsequent accuracy evaluation and error correction;

[0021] The ArUco code is detected on the calibration board image captured by the industrial camera. The ArUco module of Opencv is used to detect the ArUco code on the calibration board, identify the ID information of each ArUco code, and obtain the pixel coordinates of the center point of the ArUco code (25) in the calibration board pixel coordinate system; the calibrated camera intrinsic parameters are used to calculate the reprojection error of the ArUco code:

[0022]

[0023] Where N is the number of ArUco codes, (u i ,v i ) is the pixel coordinate of the ArUco code in the actual image, (u′ i ,v′ i ) is the reprojected coordinate obtained by back-projecting the internal reference calibration result; R e Indicates the reprojection error of the camera intrinsic parameter used for the ArUco code;

[0024] A threshold is set to determine whether the reprojection error is within the threshold d. If it exceeds the threshold, the error is compensated by the pixel coordinates of the center point of the ArUco code, and the reprojection error is reduced by the least squares method to improve the accuracy and stability of the internal parameters.

[0025] Further, the process of external parameter calibration is:

[0026] First, according to the relative position relationship between the industrial camera and the laser radar, the preset initial external parameters are input into the external parameter calibration program. The industrial camera uses the Harris algorithm to detect the corner points of the checkerboard in the uploaded image collected on the calibration plate. The laser radar uses the different reflectivity of the laser on different colors to detect the corner points of the uploaded point cloud. Then, the ICP feature point matching algorithm is used to batch process the extracted calibration plate corner points. In view of the low light and high noise characteristics of the underground environment, the matching is inevitably partial or incomplete. The convergence result is used as the first-stage external parameter value and stored in the external parameter calibration program.

[0027] Use the laser radar to obtain the depth change of the hollow circle of the calibration plate, and extract the edge features of the hollow circle in the point cloud obtained by the laser radar: set the depth threshold d L , let the depth of any point be P i d , the depth values ​​of adjacent points on the same scan line in the point cloud are The depth difference of each point in the point cloud is calculated using the following formula:

[0028]

[0029] According to the depth threshold d L Determine whether the depth difference of the point cloud meets the edge feature requirements. Greater than the depth threshold d L , it is regarded as an edge point cloud, and all feature point clouds that meet the requirements constitute the edge point cloud set P 0～n ; Construct the circular edge through the edge point cloud set, and solve the position of each circle center in the circular point cloud through the known hollow circle radius of the calibration plate;

[0030] The edge pixel set of the circular point cloud is extracted by using the gradient intensity change of the image taken by the industrial camera at the hollow circle. The center position of each hollow circle in the image of the calibration plate taken by the industrial camera is solved by using the known hollow circle radius of the calibration plate.

[0031] The external parameter values ​​of the first stage are used as the starting point of the iteration of the second stage. The coordinates of the center of the hollow circle in the radar coordinate system and the coordinates of the center of the hollow circle in the image pixel coordinate system are used to establish external parameter constraints. The calculation formula is as follows:

[0032]

[0033] In the formula, O i Indicates the coordinates of the center of the hollow point cloud, o i represents the pixel center coordinates, and Represents the final solved external parameter values, representing the rotation matrix part and the translation matrix part respectively; the final and Write the program to complete the final visual calibration.

[0034] Beneficial effects: This device integrates the camera and the laser radar, and can calibrate the camera and the laser radar at the same time, providing more convenient technical support for the calibration of the laser radar and the camera in the mine. At the same time, in order to better adapt to the harsh environment of the mine, a correction mechanism is introduced to ensure the stability and accuracy of the calibration results. Without blocking the field of view of the sensor, the automation level of the tunnel boring machine is enhanced, providing more accurate support for the positioning, navigation, and automatic control of the tunnel boring machine, ensuring that mining activities can be carried out efficiently and safely, and increasing the visual range of the camera and radar, providing more reliable guarantee for the realization of intelligent mine excavation.

Brief Description of the Drawings

[0035] Figure 1 It is a schematic diagram of the overall structure of the camera and laser radar calibration device for a roadheader of the present invention;

[0036] Figure 2 It is a schematic diagram of the pitch rotation platform structure of the camera and laser radar calibration device for a roadheader of the present invention;

[0037] Figure 3 It is a schematic diagram of the structure of a camera radar fusion module of a camera and laser radar calibration device for a roadheader according to the present invention;

[0038] Figure 4 It is a schematic diagram of the structure of the truncated table rotating platform of the camera and laser radar calibration device for a roadheader of the present invention;

[0039] Figure 5 A schematic diagram of the mechanical arm swing of the camera and laser radar calibration device for a tunnel boring machine according to the present invention;

[0040] Figure 6 It is a schematic diagram of a calibration plate rotation module of a camera and laser radar calibration device for a roadheader according to the present invention;

[0041] Figure 7 It is a schematic diagram of the structure of a calibration plate of a camera and laser radar calibration device for a roadheader according to the present invention;

[0042] Figure 8 This is a working flow chart of the camera and laser radar calibration device for a roadheader of the present invention;

[0043] In the figure, 1- tunnel boring machine body, 2- rotating base, 3- pitch bracket, 4- driving motor, 5- bolt, 6- side support block, 7- laser radar, 8- sensor connection block, 9- industrial camera, 10- support base, 11- rotating round table, 12- cylindrical pin, 13- lower rocker arm, 14- cylindrical pin, 15- upper rocker arm, 16- hydraulic telescopic rod, 17- rotating disc base, 18- rotating disc, 19- connecting base, 20- driving motor, 21- rotating bracket, 22- calibration plate, 23- hollow circle, 24- chessboard, 25- ArUco code. [Specific implementation method]

[0044] The present invention will be further described below in conjunction with the accompanying drawings:

[0045] like Figure 1 As shown, the present invention discloses a camera and laser radar calibration device for a tunnel boring machine, comprising a pitch rotating platform and a truncated rotating platform fixed above a tunnel boring machine body 1, a camera radar fusion module is arranged on the pitch rotating platform, a mechanical arm swing mechanism is arranged on the truncated rotating platform, and a calibration plate rotation module is connected to the mechanical arm swing mechanism; a control system for remotely controlling the pitch rotating platform, the truncated rotating platform, the mechanical arm swing mechanism, and the calibration plate rotation module is also arranged on the tunnel boring machine body 1, and a data system for transmitting data collected by the camera radar fusion module is also arranged;

[0046] like Figure 2 As shown, the pitch rotating platform includes a rotating base 2 fixed on the main body of the tunnel boring machine, a pitch bracket 3 is arranged above the rotating base 2, and a driving motor 4 is connected to the pitch bracket 3 through a bolt 5;

[0047] like Figure 3 As shown, the camera radar fusion module includes a side support block 6 connected to the drive motor 4, a laser radar 7 and an industrial camera 9 fixed on the side support block 6, and a sensor connection block 8 connecting the laser radar 7 and the industrial camera 9. The laser radar 7 is used to obtain point cloud information, and the industrial camera 9 is used to obtain image information;

[0048] like Figure 5 As shown, the truncated table rotating platform comprises a support base 10 fixed on the tunnel boring machine body 1, and a rotatable truncated table 11 is provided on the support base 10;

[0049] The mechanical arm swing mechanism includes a lower rocker arm 13 and an upper rocker arm 15. The lower rocker arm 13 is connected to the rotatable round table 11 through a cylindrical pin 12. The end of the lower rocker arm 13 is connected to the upper rocker arm 15 through a cylindrical pin 14. A hydraulic telescopic rod 16 is connected to the end of the upper rocker arm.

[0050] like Figure 6As shown, the calibration plate rotation module includes a rotating disc base 17 connected to a hydraulic telescopic rod 16, and the calibration plate rotation module is pushed forward and backward by the hydraulic telescopic rod 16. A rotatable rotating disc 18 is installed on the rotating disc base 17, and a connecting base 19 is provided on the rotating disc 18. A driving motor 20 and a rotating bracket 21 are installed on the connecting base 19. A calibration plate 22 is installed on the rotating bracket 21 to provide the characteristic points required for calibration;

[0051] The industrial camera 9 extracts the corner points of the calibration plate 22 through the Harris detection algorithm, and the laser radar 7 extracts the corner points of the calibration plate 22 through the different laser reflectivities on different colors. The extracted feature corner point coordinates will be matched through the ICP algorithm.

[0052] The main body of the tunnel boring machine is provided with an electric control system for driving the pitch rotating platform, the truncated rotating platform, the mechanical arm swing mechanism, and the calibration plate rotating module to adjust the position, and is also provided with a transmission system for providing data transmission for the camera radar fusion module.

[0053] The pitch rotating platform is located on the main body 1 of the tunnel boring machine. The pitch bracket 3 is adjusted within a range of 360° in the horizontal plane through the rotating base 2. The driving motor 4 is fixed to the pitch bracket 3 through bolts 5, driving the camera radar fusion module to achieve 180° adjustment in the vertical direction.

[0054] The truncated rotating platform is located on the main body 1 of the tunnel boring machine, and is composed of a rotatable rotating truncated platform 11 and a supporting base 10. It can rotate 360° in the horizontal plane, thereby driving the mechanical arm swing mechanism and the calibration plate rotating module to adjust the angle in the horizontal direction;

[0055] The mechanical arm swing mechanism is located between the truncated table rotating platform and the calibration plate rotation module. The lower rocker arm 13 of the mechanical arm swing mechanism is connected to the rotating truncated table 11 and the upper rocker arm 15 through a cylindrical pin 12. A hydraulic telescopic rod 16 is provided on the top of the upper rocker arm 15. The lower rocker arm 13 and the upper rocker arm 15 can both rotate 180° in the vertical plane, and they are in a driven relationship. The hydraulic telescopic rod 16 can be telescoped along the axial direction of the upper rocker arm 15, and it is in a driven relationship with the upper rocker arm 15. When the lower rocker arm 13 rotates in the vertical plane, it is necessary to control the upper rocker arm 15 and the hydraulic telescopic rod 16 to move at the same time, so as to adjust the spatial position of the calibration plate rotation module in the vertical and horizontal directions.

[0056] The calibration plate rotation module is located on the robotic arm swing mechanism, and is connected to the hydraulic telescopic rod 16 through a rotating disc base 17. A rotating disc 18 is installed on the rotating disc base 17, and a rotating disc 18 is installed on a connecting base 19. The connecting base 19 and the rotating disc 18 are fixedly connected together, and a driving motor 20, a rotating bracket 21, and a calibration plate 22 are installed thereon. The calibration plate 22 can be adjusted to different angles through the rotating disc 18 and the driving motor 20, so that the calibration plate is in an observable posture of the camera and lidar fusion module.

[0057] like Figure 7 As shown, the specially designed new calibration plate consists of a hollow circle 23, a chessboard 24, and an ArUco code 25. The laser radar 7 and the industrial camera 9 use the depth difference of the hollow circle 23 to extract the edge; the chessboard 24 is used for the internal reference calibration of the industrial camera 9 and the initial calibration parameter calculation of the external reference calibration of the laser radar 7 and the industrial camera 9; the ArUco code is used for the rapid positioning and error correction of the industrial camera 9 during the calibration process.

[0058] like Figure 8 As shown, a working method of a camera and laser radar calibration device for a tunnel boring machine, the steps are as follows:

[0059] The rotating base 2 and the pitch bracket 3 of the pitch rotating platform are remotely adjusted by the remote control system to rotate, thereby adjusting the angles of the camera radar fusion module in the horizontal and vertical directions in real time to ensure that the observation direction of the camera radar fusion module is toward the calibration board during calibration;

[0060] The rotating truncated table 11 of the truncated table rotating platform is remotely adjusted by the remote control system to rotate, thereby adjusting the angle of the calibration plate 22 in the horizontal direction in real time to ensure that the angle deviation between the front of the calibration plate and the observation direction of the camera radar fusion module in the axial direction around the rotating truncated table 11 is controlled within 30° during calibration;

[0061] The rotation angle of the upper rocker arm 15 and the lower rocker arm 13 of the robotic arm swing mechanism and the telescopic distance of the hydraulic telescopic rod 16 are remotely adjusted through the remote control system, and the position of the calibration plate 22 in the horizontal and vertical directions is adjusted in real time to ensure that the camera radar fusion module can observe the calibration plate at multiple positions, and all the corner points on the calibration plate should be within the field of view of the camera radar fusion module, and there should be no missing corner points or blurred corner points.

[0062] The rotation of the rotating disk 18 and the driving motor 20 of the calibration plate rotation module is remotely adjusted by the remote control system, thereby adjusting the angle of the calibration plate in the radial and axial directions of the upper rocker arm 15, ensuring that the angle deviation between the front of the calibration plate and the observation direction of the camera radar fusion module in the axial direction around the driving motor 20 is controlled within 30°, and can be rotated 360° around the axial direction of the upper rocker arm 15, so as to facilitate the stowage after calibration;

[0063] Through the joint adjustment of the pitch rotating platform, the truncated rotating platform, the mechanical arm swing mechanism, and the calibration plate rotation module, it is ensured that before and during the calibration, the camera radar fusion module can collect data in multiple different relative positions with the calibration plate, and during the data collection process, all the corners of the calibration plate are within the field of view of the camera radar fusion module, and there is no missing corner point or corner point blur caused by angle and position reasons. After the calibration data is collected, the control calibration plate is attached to the tunnel boring machine body 1 and is unfolded when it needs to be calibrated again.

[0064] The remote control system transmits information with the camera radar fusion module wirelessly. During calibration, attention should be paid to the noise of the laser radar 7 and the industrial camera 9 at the current position. If the noise suddenly increases due to environmental factors such as dust at the current position, the camera radar fusion module is rotated and pitched through the pitch rotation mechanism until the noise in the observation angle of the laser radar 7 and the industrial camera 9 is reduced. The remote control system controls the truncated table rotation platform, the mechanical arm swing mechanism, and the calibration plate rotation module to make the calibration plate 22 within the observation range of the current camera radar fusion module, and there is no missing corner point or blurred corner point. Keep all mechanisms stationary in this state, take pictures, and collect radar point cloud data for more than 10 seconds; keep the position of the camera radar module unchanged, and change the relative posture of the calibration plate 22 and it. The spatial position of the calibration plate 22 is adjusted by the movement of the truncated table rotating platform, the mechanical arm swing mechanism, and the calibration plate rotating module to ensure that the camera radar fusion module can collect data in multiple different relative postures; after the data collection is completed, the posture of the calibration plate 22 is changed by the joint adjustment of the truncated table rotating platform, the mechanical arm swing mechanism, and the calibration plate rotating module, so that it can be attached to the tunnel boring machine body 1 after the calibration is completed, without blocking the sensor field of view. If a laser radar and camera for obstacle detection and autonomous navigation are provided in front of or on the side of the tunnel boring machine body 1, when the laser radar and camera at this position need to be recalibrated, the calibration plate 22 is adjusted to this position, and multiple sets of data are collected by continuously changing the relative posture of the calibration plate 22 relative to the laser radar and camera at this position, so as to complete the sensor fusion calibration of the laser radar and camera at this position.

[0065] The camera radar fusion module uploads the collected point cloud and image data wirelessly. The internal parameters of the industrial camera 9 will first be calibrated through matlab, and image information at different angles and distances will be collected to ensure that the number of images is not less than 20. The characteristic corner points on the chessboard 24 are extracted through the matlab camera calibration toolbox to calculate the internal parameters of the industrial camera 9. If there is an image with a calibration error greater than 0.5 pixels, the corresponding image will be deleted and recalibrated. The final internal parameter calibration result will be saved for the subsequent accuracy evaluation and error correction.

[0066] Perform ArUco code detection on the images captured by the industrial camera, use the ArUco module of Opencv to detect the ArUco code 25 on the calibration board, identify the ID information of each ArUco code 25, and obtain its pixel coordinates on the image. The calibrated camera intrinsic parameters are used to calculate the reprojection error of the ArUco code 25:

[0067]

[0068] Where N is the number of ArUco codes 25, (u i ,v i ) is the 25-pixel coordinate of the ArUco code in the actual image, (u′ i ,v′ i ) is the reprojected coordinate obtained by back-projecting the intrinsic calibration result.

[0069] A threshold is set to determine whether the reprojection error is within the threshold d. If it exceeds the threshold range, secondary calibration is performed using the pixel coordinates of the ArUco code 25.

[0070] The process of external parameter calibration is as follows: first, the preset external parameters are written into the control program according to the relative position relationship between the industrial camera 9 and the laser radar 7. The industrial camera 9 uses the Harris algorithm to detect the corner points of the checkerboard 24 in the uploaded image. The laser radar 7 uses the different reflectivity of the laser on different colors to detect the corner points of the checkerboard 24 on the uploaded point cloud. Then, the ICP feature point matching algorithm is used to batch process the extracted feature points. In view of the low light and high noise characteristics of the underground environment, the matching will inevitably have local convergence or incomplete convergence. The convergence result is used as the first stage external parameter value and stored in the program.

[0071] Furthermore, the depth change of the laser radar at the hollow circle 23 is used to extract the edge features of the hollow circle 23 in the point cloud: the depth threshold d is set L , let the depth of any point be P i d , the depth values ​​of adjacent points on the same scan line are Calculate the depth difference of each point in the point cloud

[0072]

[0073] According to the depth threshold d L Determine whether the point cloud meets the edge feature requirements. Greater than the depth threshold d L , it is regarded as an edge point cloud, and all feature point clouds that meet the requirements constitute the edge point cloud set P 0～n . The circular edge is constructed by edge point cloud set, and the position of each circle center in the point cloud is solved by the known hollow circle 23 radius of the calibration plate;

[0074] The circular edge pixel set is extracted by using the gradient intensity change of the image at the hollow circle 23, and the position of each circle center in the image is solved by the known radius of the hollow circle 23 of the calibration plate;

[0075] The external parameter values ​​of the first stage are used as the starting point of the second stage iteration, and the external parameter constraints of all the hollow point cloud centers and pixel centers in the collected data are established using the Kabsch registration method:

[0076]

[0077] In the formula, O i Indicates the coordinates of the center of the hollow point cloud, o i represents the pixel center coordinates, and Indicates the final solved external parameter values, representing the rotation matrix part and the translation matrix part respectively. and Write the program to complete the final visual calibration.

Claims

1. A camera and lidar calibration device for a tunnel boring machine, Features: The invention comprises a calibration plate rotation module which is arranged on the top of a roadheader and moves according to the position of a camera radar fusion module arranged on the roadheader and serves as a calibration plate. The calibration plate rotation module comprises a calibration plate (22). The camera radar fusion module comprises a laser radar (7) and an industrial camera (9). A universal mechanical arm which can move freely and take into account all the laser radars (7) and industrial cameras (9) arranged on the roadheader is arranged below the calibration plate (22). The calibration plate (22) is a rectangular plate structure. A checkerboard (24) for calibration is arranged at the center of the calibration plate (22). Four hollow circles (23) are arranged on the calibration plate (22) in the four directions of the upper, lower, left and right of the checkerboard (24). The checkerboard is located between every two hollow circles (23). Four ArUco codes (25) are respectively arranged on the calibration plate (22) at the outer sides of the four corners of (24), and the four ArUco codes (25) are arranged according to the diagonal tilt of the calibration plate (22). The IDs corresponding to the four ArUco codes (25) are 0, 1, 2, and 3, respectively, and are arranged in a clockwise order; the laser radar (7) and the industrial camera (9) use the depth difference of the hollow circle (23) to extract the edge; the industrial camera (9) uses the chessboard (24) to obtain the initial calibration parameters of the internal reference calibration and the external reference calibration, respectively, and the laser radar (7) uses the chessboard (24) to obtain the initial calibration parameters of the external reference calibration; the industrial camera (9) uses the ArUco code (25) to perform rapid positioning and error correction; The camera radar fusion module uploads the collected point cloud and image data wirelessly. The internal reference data of the industrial camera (9) will first be calibrated through MATLAB. Image information of the chessboard (24) in the calibration plate at different angles and distances is collected to ensure that the number of images is not less than 20. The camera calibration toolbox of MATLAB is used to extract the characteristic corner points on the chessboard (23) to calculate the internal reference of the industrial camera (9). If there is an image whose calibration error is greater than 0.5 pixels, the corresponding image is deleted and recalibrated. The final internal reference calibration result is saved to facilitate the subsequent accuracy evaluation and error correction. The ArUco code (25) is detected on the calibration board image captured by the industrial camera (9), and the ArUco module of Opencv is used to detect the ArUco code (25) on the calibration board, and the ID information of each ArUco code (25) is identified, and the pixel coordinates of the center point of the ArUco code (25) in the calibration board pixel coordinate system are obtained; the calibrated camera intrinsic parameters are used for the reprojection error calculation of the ArUco code (25): Where N is the number of ArUco codes (25), (u i ,v i ) is the pixel coordinate of the ArUco code (25) in the actual image, (u i ′,v i ′) is the reprojected coordinate obtained by back-projecting the internal reference calibration result; R e Indicates the reprojection error of the camera intrinsic parameter used for the ArUco code; A threshold is set to determine whether the reprojection error is within the threshold d. If it exceeds the threshold, the error is compensated by the pixel coordinates of the center point of the ArUco code (25). The reprojection error is reduced by the least squares method to improve the accuracy and stability of the internal parameters. The process of external parameter calibration is as follows: first, according to the relative position relationship between the industrial camera (9) and the laser radar (7), the preset initial external parameters are input into the external parameter calibration program; the industrial camera (9) uses the Harris algorithm to detect the corner points of the chessboard (24) in the uploaded image collected on the calibration plate (22); the laser radar (7) uses the different reflectivity of lasers on different colors to detect the corner points of the chessboard (24) on the uploaded point cloud; then, the ICP feature point matching algorithm is used to batch process the extracted corner points of the calibration plate (22); in view of the low light and high noise characteristics of the underground environment, the matching is inevitably subject to local convergence or incomplete convergence; the convergence result is used as the first stage external parameter value and stored in the external parameter calibration program; Furthermore, the depth change at the hollow circle (23) of the calibration plate (22) is obtained by using the laser radar (7), and the edge features of the hollow circle (23) in the point cloud obtained by the laser radar (7) are extracted: a depth threshold d is set L , let the depth of any point be The depth values ​​of adjacent points on the same scan line in the point cloud are The depth difference of each point in the point cloud is calculated using the following formula: According to the depth threshold d L Determine whether the depth difference of the point cloud meets the edge feature requirements. Greater than the depth threshold d L , it is regarded as an edge point cloud, and all feature point clouds that meet the requirements constitute the edge point cloud set P 0～n ; Construct a circular edge through the edge point cloud set, and solve the position of each center of the circular point cloud through the radius of the hollow circle (23) known by the calibration plate (22); By using the gradient intensity change of the image captured by the industrial camera (9) at the hollow circle (23), the edge pixel set of the circular point cloud is extracted, and the position of the center of each hollow circle (23) in the image of the calibration plate (22) captured by the industrial camera (9) is solved by using the known radius of the hollow circle (23) of the calibration plate; The external parameter value of the first stage is used as the starting point of the iteration of the second stage. The coordinates of the center of the hollow circle (23) in the radar coordinate system and the coordinates of the center of the hollow circle (23) in the image pixel coordinate system are used to establish external parameter constraints. The calculation formula is as follows: In the formula, O i Indicates the coordinates of the center of the hollow point cloud, o i represents the pixel center coordinates, and Represents the final solved external parameter values, representing the rotation matrix part and the translation matrix part respectively; the final and Write the program to complete the final visual calibration.

2. A camera and laser radar calibration device for a tunnel boring machine according to claim 1, Features: The camera radar fusion module comprises a sensor connection block (8), and a laser radar (7) and an industrial camera (9) are respectively arranged on the left and right sides of the sensor connection block (8), wherein the outer sides of the laser radar (7) and the industrial camera (9) are respectively provided with side support blocks (6); the camera radar fusion module is fixed on the top of the tunnel boring machine body (1) through a pitch rotation platform, and the pitch rotation platform comprises a rotating base (2) fixed on the tunnel boring machine body (1), and a pitch bracket (3) is arranged on the rotating base (2), and a driving motor a (4) is fixed on the pitch bracket (3) through bolts (5), and the driving motor a (4) is connected to the side support blocks (6) on both sides of the camera radar fusion module and provides driving force; the pitch rotation platform realizes the pitch bracket (3) to adjust the angle within a range of 360 degrees in the horizontal plane through the rotating base (2), and the driving motor a (4) is fixed to the pitch bracket (3) through bolts (5), so as to drive the camera radar fusion module to realize 180 degrees of adjustment in the vertical direction.

3. A camera and laser radar calibration device for a tunnel boring machine according to claim 1, Features: The bottom of the universal mechanical arm is arranged on the tunnel boring machine through a truncated rotating platform, the truncated rotating platform comprises a support base (10) fixed on the tunnel boring machine body (1), and a rotating truncated platform (11) is arranged on the support base (10); the universal mechanical arm comprises a mechanical arm swing mechanism, which comprises a lower rocker arm (13) and an upper rocker arm (15) connected in sequence, the lower rocker arm (13) is connected to the rotating truncated platform (11) through a cylindrical pin a (12), the end of the lower rocker arm (13) is connected to the upper rocker arm (15) through a cylindrical pin b (14), and a hydraulic telescopic rod (16) is connected to the end of the upper rocker arm (15); the truncated rotating platform rotates 360 degrees in a horizontal plane through the rotating truncated platform (11) and the support base (10), thereby driving the universal mechanical arm swing mechanism and the calibration plate rotation module to adjust the angle in the horizontal direction.

4. A camera and laser radar calibration device for a tunnel boring machine according to claim 1, Features: The calibration plate rotation module comprises a rotating disc base (17) connected to a hydraulic telescopic rod (16), and the forward and backward movement of the calibration plate rotation module is controlled by the hydraulic telescopic rod (16). A rotating disc (18) capable of rotation is installed on the rotating disc base (17), and a connecting base (19) is provided on the rotating disc (18). A driving motor b (20) and a rotating bracket (21) are installed on the connecting base (19). A calibration plate (22) is installed on the rotating bracket (21) for providing a chessboard (24) required for calibration, including corner point information of the chessboard (24), a hollow circle (22) and a rotation bracket (21). 3) center information and ID information of four ArUco codes; the calibration plate rotation module is connected to the hydraulic telescopic rod (16) through a rotating disk base (17), a rotating disk (18) is installed on the rotating disk base (17), a connecting base (19) and the rotating disk (18) are fixedly connected together, and a driving motor b (20), a rotating bracket (21), and a calibration plate (22) are installed thereon, and the calibration plate (22) is adjusted to different angles through the rotating disk (18) and the driving motor b (20), so that the calibration plate is in an observable posture of the camera and laser radar fusion module.

5. A camera and laser radar calibration device for a tunnel boring machine according to claim 3, Features: The mechanical arm swing mechanism is located between the truncated table rotating platform and the calibration plate rotating module. The lower rocker arm (13) of the mechanical arm swing mechanism is connected to the rotating truncated table (11) and the upper rocker arm (15) through a cylindrical pin a (12). The hydraulic telescopic rod is arranged at the top of the upper rocker arm (15). The lower rocker arm (13) and the upper rocker arm (15) can both rotate 180 degrees in a vertical plane and are in a driven relationship. When the lower rocker arm (13) rotates in the vertical plane, the upper rocker arm (15) and the hydraulic telescopic rod (16) need to be controlled to move at the same time, so as to adjust the spatial position of the calibration plate rotating module in the vertical and horizontal directions.

6. A working method using the camera and laser radar calibration device for a roadheader according to any one of claims 1 to 5, Features Here are the steps: The rotating base (2) and the pitch bracket (3) of the pitch rotating platform are remotely controlled to rotate, and the angles of the camera radar fusion module in the horizontal direction and the vertical direction are adjusted in real time to ensure that the observation direction of the camera radar fusion module is toward the calibration plate during calibration; The rotating truncated table (11) of the truncated table rotating platform is remotely controlled to rotate, and the angle of the calibration plate (22) in the horizontal direction is adjusted in real time to ensure that the angle deviation between the front of the calibration plate and the observation direction of the camera radar fusion module in the axial direction around the rotating truncated table (11) is controlled within 30 degrees during calibration; Remotely control the rotation angle of the upper rocker arm (15) and the lower rocker arm (13) of the universal mechanical arm and the telescopic distance of the hydraulic telescopic rod (16), and adjust the position of the calibration plate (22) in the horizontal direction and in the vertical direction in real time to ensure that the camera radar fusion module can observe the calibration plate (22) at multiple positions, and that the corner points of the chessboard (24) on the calibration plate (22) should all be within the field of view of the camera radar fusion module, and the corner points of the chessboard (24) should not be missing or blurred in the field of view of the camera radar fusion module; The rotating disk (18) and the driving motor b (20) of the calibration plate rotation module are remotely controlled to rotate, thereby adjusting the angle of the calibration plate in the radial and axial directions of the upper rocker arm (15), ensuring that the angle deviation between the front face of the calibration plate (22) and the observation direction of the camera radar fusion module in the axial direction around the driving motor b (20) is controlled within 30°, and can rotate 360° around the axial direction of the upper rocker arm (15), so as to facilitate the folding after calibration; Under the joint adjustment of the pitch rotating platform, the truncated rotating platform, the mechanical arm swing mechanism, and the calibration plate rotating module, it is ensured that before starting the calibration and during the calibration phase, the camera radar fusion module needs to capture multiple points of view with different relative viewing angles to the calibration plate (22) to collect point cloud data and image data, and that when the industrial camera (9) is collecting image data of the calibration plate (22) and the laser radar (7) is collecting point cloud data, it is necessary to ensure that all corner points of the calibration plate (22) are within the field of view of the camera radar fusion module, and that there is no missing corner point or blurred corner point of the calibration plate (22) due to angle and position reasons; After the calibration data is collected, the control calibration plate is retracted and attached to the tunnel boring machine body (1), waiting to be unfolded when calibration is required again; During calibration, attention should be paid to the noise of the laser radar (7) and the industrial camera (9) at the current position. If the dust environment causes excessive dust in a certain area at the current position, resulting in a sudden increase in noise, the camera radar fusion module is rotated and pitched by the pitch rotation mechanism to change the viewing angle of the camera radar module until the noise in the observation angle of the laser radar (7) and the industrial camera (9) is reduced. The truncated table rotation platform, the mechanical arm swing mechanism, and the calibration plate rotation module are controlled by the remote control system so that the calibration plate (22) is within the observation range of the current camera radar fusion module, and there is no corner point missing or corner point blur. All mechanisms are kept stationary in this state, and the industrial camera (9) is used to collect image information of the calibration plate (22), and the laser radar (7) is used to collect radar point cloud data for more than 10 seconds; The position of the camera radar module is kept unchanged, and the relative position of the calibration plate (22) and the camera radar module is changed by adjusting the position of the calibration plate (22), so that the acquired image information of the calibration plate (22) is tilted by 10° in the four directions of up, down, left and right respectively: the spatial position of the calibration plate (22) is adjusted by the movement of the truncated table rotating platform, the mechanical arm swing mechanism, and the calibration plate rotating module, so as to ensure that the camera radar fusion module can collect the camera image and radar point cloud of the calibration plate (22) in the above-mentioned tilted direction at different distances from the calibration plate (22); After the camera radar module has collected the image data and point cloud data of the calibration plate (22), the calibration plate (22) is folded and retracted under the joint adjustment of the truncated table rotating platform, the mechanical arm swing mechanism, and the calibration plate rotating module, so that it can be attached to the tunnel boring machine body (1) after the calibration is completed, without blocking the sensor field of view; If a laser radar and a camera for obstacle detection and autonomous navigation are provided in front of or on the side of the tunnel boring machine body (1), when the laser radar and the camera at that position need to be recalibrated, the calibration plate (22) is adjusted to that position, and multiple sets of data are collected by continuously changing the relative posture of the calibration plate (22) relative to the laser radar and the camera at that position, thereby completing the sensor fusion calibration of the laser radar and the camera at that position.

Images