Online joint calibration method for guniting robot laser radar and monocular camera

Abstract

The application provides a shotcrete robot laser radar and monocular camera online joint calibration method, relates to the technical field of sensor calibration, is used for online calibration of a shotcrete robot laser radar and a monocular camera, and comprises the following steps: monitoring vibration in real time through an IMU and actively triggering calibration, controlling a mechanical arm to make a I-shaped high-reflective light identification enter a common field of view of the sensors; performing gamma correction and image enhancement on an image, and performing statistical outlier removal and reflection intensity threshold filtering on point clouds; using a known size identification model to extract three-dimensional edge features through TF transformation, and then optimizing external parameters through a re-projection error and an LM algorithm. The application can complete real-time, high-precision and strong-robust online fine-tuning of external parameters under the working conditions of high dust, strong vibration and low texture of a mine shotcrete, and ensures the geometric consistency of point cloud and image fusion modeling.

Description

Technical Field

[0001] This invention relates to the field of sensor calibration technology, and more particularly to the field of joint calibration technology of lidar and monocular camera, specifically to an online joint calibration method for lidar and monocular camera of a shotcrete robot. Background Technology

[0002] In mine roadway shotcreting operations, the joint calibration of LiDAR and monocular camera is fundamental to fusing geometric and textural information, and its accuracy directly affects the quality of 3D modeling. Existing calibration methods are mainly divided into four categories: First, offline calibration based on artificial targets, which has high accuracy but depends on the controlled environment and cannot cope with millimeter-level extrinsic parameter drift caused by vibration during operation; second, targetless self-calibration based on environmental features, which matches natural edges or planes extracted from the scene; however, the roadway wall texture is sparse and has extremely low contrast, and dust and water mist severely degrade the features, making it difficult to extract stable constraints; third, SLAM calibration based on motion trajectory alignment, which uses hand-eye models to calculate relative poses, but underground monocular vision suffers from severe scale drift, leading to inaccurate translation vector calculations; and fourth, end-to-end regression methods based on deep learning, which are limited by the scarcity of underground data and insufficient edge computing power, making real-time deployment difficult. To address the aforementioned issues, existing solutions propose fitting the roadway equipment plane using RANSAC and extracting the intersection line as a feature, followed by two-stage optimization to solve the extrinsic parameters. However, environmental features are easily lost under dust cover, RANSAC plane fitting computation is expensive, it cannot track high-frequency vibration offset in real time, and the roadway wall changes continuously with the accumulation of slurry during the shotcreting process, causing the calibration benchmark under the static assumption to shift, ultimately leading to parameter correction errors.

[0003] Therefore, there is an urgent need to develop an online joint calibration method for lidar and monocular camera for shotcrete robots operating under harsh conditions of high dust, strong vibration, and weak texture. Summary of the Invention

[0004] The purpose of this invention is to provide an online joint calibration method for the lidar and monocular camera of a shotcrete robot, so as to solve the problems of low external parameter calibration accuracy, poor real-time performance, and easy divergence in the harsh environment of shotcrete in the prior art.

[0005] To address the above objectives, this invention provides an online joint calibration method for a shotcrete robot's lidar and monocular camera, comprising: S1. Connect the lidar and monocular camera, and obtain the initial extrinsic parameter matrix through offline calibration; S2. The vibration is monitored in real time by the inertial measurement unit to obtain vibration information. When the vibration exceeds the preset threshold, an online calibration command is triggered. S3. Responding to the online calibration command, a highly reflective I-shaped calibration board with a known topology template is introduced, and the monocular camera image data and lidar point cloud data of the calibration board are collected simultaneously. S4. Filter the high reflectivity point cloud corresponding to the calibration board in the lidar point cloud data, and process the monocular camera data according to the high reflectivity point cloud to generate a binary edge map. On the image side, extract the two-dimensional edge sampling points of the binary edge map and generate the image edge information field. On the point cloud, fit the calibration board spatial plane based on the high reflectivity point cloud and combine it with the topological template of the calibration board to obtain three-dimensional edge sampling points with edge category labels and construct a point cloud feature point set. S5. Based on the point cloud feature point set, combined with the known topological template of the high-reflectivity I-shaped calibration plate and the image edge information field, calculate the forward residual, reverse residual, direction consistency residual, corner topological residual and prior residual, and construct a composite objective function. S6. Construct an extrinsic perturbation model. Based on the composite objective function, iteratively optimize the extrinsic matrix using the initial extrinsic matrix as the initial value, and output the optimized extrinsic matrix for online joint calibration of the laser radar and monocular camera of the shotcrete robot.

[0006] In S1, the lidar and monocular camera are fixed together by a rigid bracket, and the fixed lidar and monocular camera are installed as a whole on the base of the shotcrete robot's robotic arm. The initial extrinsic parameter matrix is ​​obtained by offline calibration based on minimizing the corner reprojection error of the checkerboard calibration board. The initial extrinsic parameter matrix is: ; In the formula, Represents the initial extrinsic parameter matrix. This represents the initial rotation matrix of the lidar to the monocular camera. This is the initial translation vector from the lidar to the monocular camera.

[0007] In S2, the inertial measurement unit is installed on the robotic arm of the shotcrete robot. The vibration information includes the cumulative amount of high-frequency vibration, the instantaneous acceleration change, and the angular velocity change. When any vibration information exceeds the corresponding preset threshold, an online calibration command is triggered. In S3, the topological template of the high-reflectivity I-shaped calibration plate includes multiple edge segments and corner points with labels. The edge segments include at least the outer edge segment of the upper crossbeam, the outer edge segment of the lower crossbeam, the side edge segment of the middle vertical beam, and the concave edge segment. The corner points include at least the convex corner points and the concave corner points. Each edge segment includes the segment endpoints, direction vector, segment length, category label, and adjacency relationship; Category labels include horizontal outer edge, vertical outer edge, concave edge, and corner category.

[0008] In S4, point cloud data points with echo intensity greater than a preset echo intensity threshold are selected from the lidar point cloud data to construct a high reflectivity point cloud. Using the initial extrinsic parameter matrix, the high reflectivity point cloud is projected onto the planar image of the monocular camera image data. The point cloud projection area is obtained on the planar image. Image processing, including image enhancement, high reflectivity area segmentation, and edge detection, is performed on the image in the point cloud projection area to generate a two-dimensional edge map. The image edge information field includes the image edge distance field, the image edge orientation field, and the image edge confidence field; The image edge distance field is obtained by performing a distance transformation on the two-dimensional edge map, including the horizontal edge distance field and the vertical edge distance field. The image edge direction field is obtained by performing image gradient direction detection and line detection on the two-dimensional edge map. The image edge confidence field is jointly determined by the image gradient magnitude, the brightness response of highly reflective areas, and edge connectivity.

[0009] In S5, outlier removal is performed on the highly reflective point cloud. After removing outliers, the highly reflective point cloud is fitted to the calibration board space plane using RANSAC plane fitting. The highly reflective point cloud is then projected onto the fitted calibration board space plane to construct a local coordinate system for the calibration board. In the local coordinate system of the calibration board, the highly reflective point cloud is matched with the topological template of the calibration board. Based on the pose of the calibration board in the lidar coordinate system, samples are taken at equal intervals on each edge segment to obtain several 3D edge sampling points with corresponding category labels for the edge segments, thus constructing a point cloud feature point set. The feature point set of the point cloud is: ; In the formula, Represents the feature point set of a point cloud. This represents the total number of 3D edge sampling points in the point cloud feature point set. This represents the index of the number of 3D edge sampling points in the point cloud feature point set. This represents the total number of edge segments corresponding to the sampled points in the point cloud feature point set. This represents the index of the number of edge segments corresponding to the sampling points in the point cloud feature point set. Indicates the first The three-dimensional coordinates of each three-dimensional edge sampling point Indicates the first The edge segment corresponding to each 3D edge sampling point Category tags, Indicates the first The edge segment corresponding to each 3D edge sampling point directional vector, Indicates the first The confidence level of the point cloud of a three-dimensional edge sampling point.

[0010] In S6, the 3D edge sampling points are transformed to the camera coordinate system according to the current extrinsic matrix, and then projected onto the image plane of the binary edge map through the monocular camera intrinsic matrix. The distance value is read in the image edge distance field, the image edge unit direction vector is read in the image edge direction field, and the image edge confidence is read in the image edge confidence field. The positive residual and the direction consistency residual are calculated. The positive residual is: ; In the formula, Indicates the first The positive residual of each three-dimensional edge sampling point Indicates the first Pixel coordinates of three-dimensional edge sampling points Indicates the first Image edge confidence of 3D edge sampling points This is a function for calculating the confidence score of image edges. Indicated in pixels The distance value is obtained from the image edge distance field; The directional consistency residual is: ; ; ; In the formula, Indicates the first The directional consistency residual of each three-dimensional edge sampling point Represents absolute value. Indicated in pixels The image edge unit direction vector is obtained from the image edge direction field. Indicates the first Normalized unit projection tangential direction of each 3D edge sampling point Represents the transpose of a matrix or vector. Represents perspective projection function exist Jacobian matrix at the location, for Three-dimensional coordinates in the camera coordinate system Indicates the first The tangential direction vector of each 3D edge sampling point in the camera coordinate system This represents the rotation matrix of the lidar to the monocular camera. This represents the Euclidean norm of a vector.

[0011] First, the intrinsic parameter matrix of the monocular camera is used to back-project the two-dimensional edge sampling points into the direction of the normalized ray emitted from the optical center to obtain the normalized ray. Then, the normalized ray is transformed to the lidar coordinate system according to the current extrinsic parameter matrix. The back-projection points of the two-dimensional edge sampling points in the local coordinate system of the calibration board are obtained by fitting the calibration board spatial plane. Finally, the back-projection points are transformed to the local coordinate system of the calibration board according to the pose transformation matrix from the lidar coordinate system to the local coordinate system of the calibration board. Read the distance value in the image edge distance field, read the image edge unit direction vector in the image edge direction field, read the image edge confidence in the image edge confidence field, determine the set of candidate edge segments that match the image edge pixels, and calculate the inverse residual; The reverse residual is: ; ; In the formula, Indicates the index of the number of two-dimensional edge sampling points. Indicates the first The inverse residual of a two-dimensional edge sampling point Indicates the first The pixel coordinates of the back projection points of each two-dimensional edge sampling point in the local coordinate system of the calibration board. Indicates the first The pixel coordinates of a two-dimensional edge sampling point Indicates the first Image edge confidence of two-dimensional edge sampling points This represents the function for calculating the minimum value. Indicates the back projection point to the edge segment Euclidean distance, This represents the pose transformation matrix from the lidar coordinate system to the local coordinate system of the calibration board. This represents the set of candidate edge segments.

[0012] Extract the three-dimensional corner points corresponding to the corner points in the lidar coordinate system, transform the three-dimensional corner points to the camera coordinate system according to the current extrinsic matrix, and then obtain the projection coordinates of the three-dimensional corner points on the image plane according to the monocular camera intrinsic matrix to obtain the two-dimensional corner points and calculate the corner point topological residuals. The corner topological residual is: ; In the formula, Indicates the index of the number of corner points. This represents the intrinsic parameter matrix of a monocular camera. Indicates the first Confidence of a three-dimensional corner point Indicates the first The confidence level of the two-dimensional corner points corresponding to the three-dimensional corner points Indicates the first The coordinates of a 3D corner point in the lidar coordinate system Represents the first image plane pixel coordinates of a two-dimensional corner point Represents the translation vector. Represents the perspective projection function; The relative transformation of the current extrinsic matrix with respect to the initial extrinsic matrix is ​​mapped to the Lie algebra space to obtain a six-dimensional extrinsic offset vector. The offset vector is then normalized according to the prior covariance matrix of the extrinsic parameters to construct the prior residuals of the extrinsic parameters. ; In the formula, Indicates the prior residuals of external parameters. This represents the square root information matrix of the prior covariance matrix of the extrinsic parameters. Represents the prior covariance matrix of the extrinsic parameters. Indicates from Li Qun To Lie algebra logarithmic mapping, This represents the operator that transforms a Lie algebra matrix into a six-dimensional vector. This represents the current extrinsic parameter matrix.

[0013] The composite objective function is: ; ; ; ; ; ; In the formula, This represents the external parameter perturbation. This represents a composite objective function. For positive edge distance, For edge direction consistency terms, For the reverse template distance item, For corner topology terms, As an external reference prior term, , , , , They represent , , , , The weighting coefficients, , , , They represent , , , Robust kernel function, The information matrix representing the topological residuals at corner points.

[0014] The adaptively damped robust LM algorithm iteratively optimizes the extrinsic parameters. The adaptively damped robust LM algorithm updates the extrinsic parameters using Lie algebra perturbations. ; In the formula, For the first The adaptive damping factor for the next iteration. Indicates by A diagonal matrix composed of the diagonal elements. This is the prior information matrix of the extrinsic parameters. Indicates the first The weighted approximation Hessian matrix of the next iteration. Indicates by A diagonal matrix composed of the diagonal elements. Indicates the first The weighted gradient vector of the next iteration Indicates the first The increment of the extrinsic parameters in the next iteration Indicates the first The extrinsic perturbation quantity of the next iteration.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] This invention achieves high-precision online refinement of lidar and camera extrinsic parameters by constructing an I-shaped topological constraint and a bidirectional directional range field. It fundamentally avoids feature mismatch and corner confusion by using forced matching of convex / concave corner point categories and edge direction consistency constraints. Simultaneously, through deep fusion of point cloud reflection intensity and image gradient multi-source confidence weighting, combined with a robust kernel function and IMU vibration-adaptive extrinsic parameter priors, it significantly resists interference from harsh working conditions such as dust, overexposure, and mechanical vibration. Furthermore, real-time degradation detection based on Hessian matrix condition number and an active attitude switching mechanism solve the optimization null space drift problem at the hardware action level, ensuring global observability in ill-conditioned scenarios. Finally, combined with robust LM iterative solution of adaptive damping factor, the algorithm achieves extremely high calibration accuracy, strong anti-interference capability, and stable numerical convergence capability in complex shotcrete operations, realizing fully automatic and highly reliable online joint calibration. Attached Figure Description

[0017] Figure 1This is a schematic diagram of the reprojection error curves before and after online calibration in a fixed external parameter bias experiment;

[0018] Figure 2 This is a schematic diagram of the online correction curve of the rotating external parameter in a fixed external parameter bias experiment;

[0019] Figure 3 This is a schematic diagram of the online correction curve of the translational external parameter in a fixed external parameter bias experiment;

[0020] Figure 4 This is a schematic diagram of the reprojection error reduction rate curve for a fixed external parameter bias experiment.

[0021] Figure 5 This is a schematic diagram of the online correction curve of the rotating external parameter in a continuous vibration bias experiment;

[0022] Figure 6 This is a schematic diagram of the online correction curve of the translational external parameters in a continuous vibration bias experiment;

[0023] Figure 7 This is a schematic diagram of the reprojection error reduction rate curve of a continuous vibration bias experiment.

[0024] Figure 8 This is a schematic diagram of the reprojection error curves before and after online calibration in a continuous vibration bias experiment.

[0025] Figure 9 It is a superimposed projection image after online calibration of the continuous vibration bias experiment. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] An online joint calibration method for the lidar and monocular camera of a shotcrete robot includes: S1. Connect the lidar and monocular camera, and obtain the initial extrinsic parameter matrix through offline calibration; S2. The vibration is monitored in real time by the inertial measurement unit to obtain vibration information. When the vibration exceeds the preset threshold, an online calibration command is triggered. S3. Responding to the online calibration command, a highly reflective I-shaped calibration board with a known topology template is introduced, and the monocular camera image data and lidar point cloud data of the calibration board are collected simultaneously. S4. Filter the high reflectivity point cloud corresponding to the calibration board in the lidar point cloud data, and process the monocular camera data according to the high reflectivity point cloud to generate a binary edge map. On the image side, extract the two-dimensional edge sampling points of the binary edge map and generate the image edge information field. On the point cloud, fit the calibration board spatial plane based on the high reflectivity point cloud and combine it with the topological template of the calibration board to obtain three-dimensional edge sampling points with edge category labels and construct a point cloud feature point set. S5. Based on the point cloud feature point set, combined with the known topological template of the high-reflectivity I-shaped calibration plate and the image edge information field, calculate the forward residual, reverse residual, direction consistency residual, corner topological residual and prior residual, and construct a composite objective function. S6. Construct an extrinsic perturbation model. Based on the composite objective function, iteratively optimize the extrinsic matrix using the initial extrinsic matrix as the initial value, and output the optimized extrinsic matrix for online joint calibration of the laser radar and monocular camera of the shotcrete robot.

[0028] In S1, the lidar and monocular camera are fixed together by a rigid bracket, and the fixed lidar and monocular camera are installed as a whole on the base of the shotcrete robot's robotic arm. The initial extrinsic parameter matrix is ​​obtained by offline calibration based on minimizing the corner reprojection error of the checkerboard calibration board. The initial extrinsic parameter matrix is: ; In the formula, Represents the initial extrinsic parameter matrix. This represents the initial rotation matrix of the lidar to the monocular camera. This is the initial translation vector from the lidar to the monocular camera.

[0029] In S2, the inertial measurement unit is installed on the robotic arm of the shotcrete robot. The vibration information includes the cumulative amount of high-frequency vibration, the instantaneous acceleration change, and the angular velocity change. When any vibration information exceeds the corresponding preset threshold, an online calibration command is triggered. In S3, the topological template of the high-reflectivity I-shaped calibration plate includes multiple edge segments and corner points with labels. The edge segments include at least the outer edge segment of the upper crossbeam, the outer edge segment of the lower crossbeam, the side edge segment of the middle vertical beam, and the concave edge segment. The corner points include at least the convex corner points and the concave corner points. Each edge segment includes the segment endpoints, direction vector, segment length, category label, and adjacency relationship; Category labels include horizontal outer edge, vertical outer edge, concave edge, and corner category.

[0030] In S4, point cloud data points with echo intensity greater than a preset echo intensity threshold are selected from the lidar point cloud data to construct a high reflectivity point cloud. Using the initial extrinsic parameter matrix, the high reflectivity point cloud is projected onto the planar image of the monocular camera image data. The point cloud projection area is obtained on the planar image. Image processing, including image enhancement, high reflectivity area segmentation, and edge detection, is performed on the image in the point cloud projection area to generate a two-dimensional edge map. The image edge information field includes the image edge distance field, the image edge orientation field, and the image edge confidence field; The image edge distance field is obtained by performing a distance transformation on the two-dimensional edge map, including the horizontal edge distance field and the vertical edge distance field. The image edge direction field is obtained by performing image gradient direction detection and line detection on the two-dimensional edge map. The image edge confidence field is jointly determined by the image gradient magnitude, the brightness response of highly reflective areas, and edge connectivity.

[0031] In S5, outlier removal is performed on the highly reflective point cloud. After removing outliers, the highly reflective point cloud is fitted to the calibration board space plane using RANSAC plane fitting. The highly reflective point cloud is then projected onto the fitted calibration board space plane to construct a local coordinate system for the calibration board. In the local coordinate system of the calibration board, the highly reflective point cloud is matched with the topological template of the calibration board. Based on the pose of the calibration board in the lidar coordinate system, samples are taken at equal intervals on each edge segment to obtain several 3D edge sampling points with corresponding category labels for the edge segments, thus constructing a point cloud feature point set. The feature point set of the point cloud is: ; In the formula, Represents the feature point set of a point cloud. This represents the total number of 3D edge sampling points in the point cloud feature point set. This represents the index of the number of 3D edge sampling points in the point cloud feature point set. This represents the total number of edge segments corresponding to the sampled points in the point cloud feature point set. This represents the index of the number of edge segments corresponding to the sampling points in the point cloud feature point set. Indicates the first The three-dimensional coordinates of each three-dimensional edge sampling point Indicates the first The edge segment corresponding to each 3D edge sampling point Category tags, Indicates the first The edge segment corresponding to each 3D edge sampling point directional vector, Indicates the first The confidence level of the point cloud of a three-dimensional edge sampling point.

[0032] In S6, the 3D edge sampling points are transformed to the camera coordinate system according to the current extrinsic matrix, and then projected onto the image plane of the binary edge map through the monocular camera intrinsic matrix. The distance value is read in the image edge distance field, the image edge unit direction vector is read in the image edge direction field, and the image edge confidence is read in the image edge confidence field. The positive residual and the direction consistency residual are calculated. The positive residual is: ; In the formula, Indicates the first The positive residual of each three-dimensional edge sampling point Indicates the first Pixel coordinates of three-dimensional edge sampling points Indicates the first Image edge confidence of 3D edge sampling points This is a function for calculating the confidence score of image edges. Indicated in pixels The distance value is obtained from the image edge distance field; The directional consistency residual is: ; ; ; In the formula, Indicates the first The directional consistency residual of each three-dimensional edge sampling point Represents absolute value. Indicated in pixels The image edge unit direction vector is obtained from the image edge direction field. Indicates the first Normalized unit projection tangential direction of each 3D edge sampling point Represents the transpose of a matrix or vector. Represents perspective projection function exist Jacobian matrix at the location, for Three-dimensional coordinates in the camera coordinate system Indicates the first The tangential direction vector of each 3D edge sampling point in the camera coordinate system This represents the rotation matrix of the lidar to the monocular camera. This represents the Euclidean norm of a vector.

[0033] First, the intrinsic parameter matrix of the monocular camera is used to back-project the two-dimensional edge sampling points into the direction of the normalized ray emitted from the optical center to obtain the normalized ray. Then, the normalized ray is transformed to the lidar coordinate system according to the current extrinsic parameter matrix. The back-projection points of the two-dimensional edge sampling points in the local coordinate system of the calibration board are obtained by fitting the calibration board spatial plane. Finally, the back-projection points are transformed to the local coordinate system of the calibration board according to the pose transformation matrix from the lidar coordinate system to the local coordinate system of the calibration board. Read the distance value in the image edge distance field, read the image edge unit direction vector in the image edge direction field, read the image edge confidence in the image edge confidence field, determine the set of candidate edge segments that match the image edge pixels, and calculate the inverse residual; The reverse residual is: ; ; In the formula, Indicates the index of the number of two-dimensional edge sampling points. Indicates the first The inverse residual of a two-dimensional edge sampling point Indicates the first The pixel coordinates of the back projection points of each two-dimensional edge sampling point in the local coordinate system of the calibration board. Indicates the first The pixel coordinates of a two-dimensional edge sampling point Indicates the first Image edge confidence of two-dimensional edge sampling points This represents the function for calculating the minimum value. Indicates the back projection point to the edge segment Euclidean distance, This represents the pose transformation matrix from the lidar coordinate system to the local coordinate system of the calibration board. This represents the set of candidate edge segments.

[0034] Extract the three-dimensional corner points corresponding to the corner points in the lidar coordinate system, transform the three-dimensional corner points to the camera coordinate system according to the current extrinsic matrix, and then obtain the projection coordinates of the three-dimensional corner points on the image plane according to the monocular camera intrinsic matrix to obtain the two-dimensional corner points and calculate the corner point topological residuals. The corner topological residual is: Li Qun In the formula, Indicates the index of the number of corner points. This represents the intrinsic parameter matrix of a monocular camera. Indicates the first Confidence of a three-dimensional corner point Indicates the first The confidence level of the two-dimensional corner points corresponding to the three-dimensional corner points Indicates the first The coordinates of a 3D corner point in the lidar coordinate system Represents the first image plane pixel coordinates of a two-dimensional corner point Represents the translation vector. Represents the perspective projection function; The relative transformation of the current extrinsic matrix with respect to the initial extrinsic matrix is ​​mapped to the Lie algebra space to obtain a six-dimensional extrinsic offset vector. The offset vector is then normalized according to the prior covariance matrix of the extrinsic parameters to construct the prior residuals of the extrinsic parameters. ; In the formula, Indicates the prior residuals of external parameters. This represents the square root information matrix of the prior covariance matrix of the extrinsic parameters. Represents the prior covariance matrix of the extrinsic parameters. Indicates from Li Qun To Lie algebra logarithmic mapping, This represents the operator that transforms a Lie algebra matrix into a six-dimensional vector. This represents the current extrinsic parameter matrix.

[0035] The composite objective function is: ; ; ; ; ; ; In the formula, This represents the external parameter perturbation. This represents a composite objective function. For positive edge distance, For edge direction consistency terms, For the reverse template distance item, For corner topology terms, As an external reference prior term, , , , , They represent , , , , The weighting coefficients, , , , They represent , , , Robust kernel function, The information matrix representing the topological residuals at corner points.

[0036] The adaptively damped robust LM algorithm iteratively optimizes the extrinsic parameters. The adaptively damped robust LM algorithm updates the extrinsic parameters using Lie algebra perturbations. ; In the formula, For the first The adaptive damping factor for the next iteration. Indicates by A diagonal matrix composed of the diagonal elements. This is the prior information matrix of the extrinsic parameters. Indicates the first The weighted approximation Hessian matrix of the next iteration. Indicates by A diagonal matrix composed of the diagonal elements. Indicates the first The weighted gradient vector of the next iteration Indicates the first The increment of the extrinsic parameters in the next iteration Indicates the first The extrinsic perturbation quantity of the next iteration.

[0037] Adaptive damping factor The system automatically adjusts its parameters based on the decrease in the objective function before and after each iteration. When the current iteration decreases the objective function and improves the overlap between the radar edge projection and the image edge, the system accepts this extrinsic parameter update and reduces... This makes the algorithm closer to Gaussian-Newton updates to improve convergence speed; when the objective function increases or projection diverges in the current iteration, the system rejects the current extrinsic parameter update and increases the... This makes the algorithm closer to gradient descent updates, thus improving stability.

[0038] The forward residual, directional consistency residual, reverse residual, corner topological residual, and prior residual of the composite objective function are all functions of the current extrinsic parameter matrix, and their values ​​change with the change of the extrinsic parameter perturbation. The positive residual in the composite objective function is: ; The directional consistency residual is: ; ; ; ; The reverse residual is: ; ; ; ; The corner topological residual is: ; The prior residuals of the external parameters are: ; In the formula, The representation is defined as follows: , , , , All are items affected by external parameters. This indicates that based on the current external parameters, the first... The coordinates obtained by back-projecting the two-dimensional edge sampling points onto the local coordinate system of the calibration plate , These are the rotation and translation parts of the current extrinsic parameter matrix, respectively. This is the current extrinsic parameter matrix. This is the pose transformation matrix from the local coordinate system of the calibration board to the coordinate system of the lidar under the current extrinsic parameters. The pose transformation matrix is ​​calculated from the current extrinsic parameter matrix and the known pose of the calibration board. The perturbation of the current extrinsic parameters relative to the initial extrinsic parameters is represented by a six-dimensional Lie algebra vector. The extrinsic parameters are iteratively optimized using an adaptively damped robust LM algorithm. The weighted approximation Hessian matrix is ​​constructed by weighting the Jacobian matrices of each residual term. The Jacobian matrix is ​​a 2x3 matrix, representing the linear change in the image plane projection coordinates caused by a small change in the coordinates of a point in three-dimensional space. The prior information matrix of the extrinsic parameters is the inverse of the prior covariance matrix of the extrinsic parameters. The coefficients of each residual term are used to balance the contribution of each residual in the composite objective function. They are adjusted in real time according to the working conditions during the shotcreting operation (dust concentration, vibration amplitude, lighting conditions). When the overall confidence level of the image edges is low (due to severe dust), the coefficients are reduced. and ,improve When point cloud noise is high, reduce the reliance on point cloud edge sampling. and ,improve When the vibration is severe, reduce This allows for greater adjustment of the extrinsic parameters. The initial values ​​of the coefficients of each residual term are determined based on the magnitude of each residual term under the initial extrinsic parameters.

[0039] Li Qun It is a continuous group consisting of all rigid body motions (rotation and translation) in three-dimensional space, using the Lie group. Corresponding Lie algebra Transforming complex rigid body pose estimation problems (such as SLAM and robot hand-eye calibration) into unconstrained vector optimization problems avoids numerical instability caused by orthogonal constraints on rotation matrices and allows for efficient local differentiation and updates using exponential / logarithmic mappings on manifolds. Lie algebras It is Li Qun The tangent space at the unit element is essentially a local linearization of the rigid body motion manifold, usually a six-dimensional vector, or equivalently a 4×4 matrix.

[0040] In S1, during the offline calibration stage, the control robot arm drives the checkerboard calibration board to change no less than 10 different postures within the shared field of view of the lidar and the monocular camera. The reprojection error equation from the 3D laser corner point to the 2D image corner point is constructed using the camera intrinsic parameter matrix, and nonlinear optimization is performed using the Levenberg-Marquardt (LM) algorithm. The extrinsic parameter matrix that minimizes the reprojection error is iteratively solved as the initial extrinsic parameter matrix.

[0041] The initial extrinsic parameter matrix is ​​a 4×4 homogeneous transformation matrix, specifically in the form of: ; In the formula, Represents the initial extrinsic parameter matrix. For the initial 3×3 rotation matrix of the lidar to the monocular camera, The initial translation vector from the lidar to the monocular camera is 3×1. The initial extrinsic parameter matrix is ​​obtained by minimizing the corner reprojection error of the checkerboard calibration board, and the initial prior covariance matrix corresponding to the initial extrinsic parameter matrix is ​​determined based on the reprojection residual covariance during offline calibration. The initial prior covariance matrix is ​​used for the weighted calculation of the prior residuals. It is a zero matrix with 1 row and 3 columns.

[0042] The initial prior covariance matrix is ​​set as a diagonal matrix: ; In the formula, It is a diagonal matrix. Let represent the squares of the permissible prior standard deviations for the three rotation directions, respectively. These represent the squares of the permissible prior standard deviations for the three translation directions.

[0043] The topology template of the high-reflectivity I-beam calibration plate is a pre-stored digital geometric model in the system. Based on the known physical dimensions of the calibration plate (the length, width, and thickness of the upper, lower, and middle vertical beams are all known parameters), it precisely records the endpoint coordinates, direction vectors, and lengths of all edge segments of the upper, lower, and middle vertical beams in the local coordinate system of the calibration plate, and strictly defines the three-dimensional positions of convex and concave corner points. Simultaneously, the topology template assigns clear category labels (such as horizontal outer edge, vertical side edge, convex corner point, and concave corner point) and their adjacency topology relationships to each edge and corner point. The topology template is a structured dataset that precisely records the position, size, type labels, and interconnection topology relationships of all key geometric features (lines and points) on the calibration plate. This topological template acts as an absolute geometric reference throughout the calibration process. It not only guides edge segment matching and corner point localization during point cloud segmentation, but also prevents mismatches by forcing consistency between the category labels (convexity / concavity attributes) of 3D and 2D corner points. Furthermore, it serves as the target contour for backprojection constraints, ensuring that image edge points, after being backprojected onto the calibration board plane, must fall near the corresponding line segments of the template, thus effectively suppressing interference from background noise and local occlusion. Each edge segment has endpoint coordinates, direction vector, actual length, category label (e.g., horizontal outer edge, vertical outer edge), and adjacency relationships. Each corner point has 3D coordinates, a corner point category label (convex or concave), and adjacent edge numbers.

[0044] The system first coarsely projects the highly reflective point cloud onto the image based on the previous extrinsic parameters, and generates a region of interest centered on the projected area. Subsequently, image enhancement, highly reflective region segmentation, and edge detection are performed within the region of interest to obtain a binary edge map.

[0045] A distance transformation is performed on the binary edge map to obtain the image edge distance field. The edge direction field is calculated based on the image gradient direction or line detection results, and the edge confidence field is calculated based on the image gradient magnitude, brightness response of highly reflective areas, and edge connectivity. To distinguish different edge directions of the I-shaped calibration plate, the system establishes horizontal and vertical edge distance fields separately. When the template edge to which the 3D edge point belongs is a horizontal edge, the distance residual is preferentially read from the horizontal edge distance field; when the template edge to which the 3D edge point belongs is a vertical edge, the distance residual is preferentially read from the vertical edge distance field.

[0046] For any m-th 3D edge sampling point The system is based on the current external parameters Transform it to the camera coordinate system, and then use the camera intrinsic parameters. Projected onto the image plane: ; The system reads distance values ​​from the image edge distance field and constructs positive edge distance residuals: ; in, The confidence level of the point cloud at the m-th 3D edge sampling point is determined by the point cloud reflection intensity, the plane fitting residual, and the neighborhood density. Image edge confidence, representing pixel points. The reliability of nearby image edges can be determined by the image gradient magnitude, the response of highly reflective areas, and edge connectivity. In actual calculations... and It can be obtained from the image edge distance field and the image edge confidence field through bilinear interpolation. Therefore, Indicates the first The weighted distance residual between the three-dimensional edge sampling points of the radar and the corresponding image edge after the projection points are projected onto the image. The closer the projected points are to the real I-shaped edges in the image, and the higher the confidence of the point cloud and the image of the observation, the stronger the constraint effect of the residual on the optimization of the extrinsic parameters. When the observation is affected by dust, weak edges, occlusion or mismatch, its confidence decreases, and its impact on the optimization results decreases accordingly.

[0047] In S5, edge segments whose direction consistency meets the preset conditions are selected based on the local edge direction of the two-dimensional edge sampling points and the direction vector of each topological template edge segment, and a candidate edge segment set is constructed.

[0048] The system projects the tangential direction of the 3D edge sampling points onto the image plane based on the current extrinsic parameters, thus obtaining its 2D projection tangential direction. Let the first... The tangential direction of each point in the lidar coordinate system is: Its tangential direction in the camera coordinate system is: ; in, Let be the rotation matrix of the lidar to the monocular camera. Let the coordinates of the edge sampling point in the camera coordinate system be: ; Perspective projection function exist The Jacobian matrix at point is Then the projection direction of the tangential direction of the three-dimensional edge onto the image plane is: ; Normalize it to obtain the unit projection tangential direction: ; Simultaneously, the system reads projected pixels from the image edge direction field. Image edge unit direction vector at the location .in, It can be determined by the image gradient direction, LSD line detection results, or Hough line detection results, and obtained through bilinear interpolation. The system constructs edge direction consistency residuals: ; in, For the first The confidence level of the point cloud of a 3D edge sampling point For projected pixels The image edge confidence score is calculated. Since edge direction itself does not distinguish between positive and negative directions, the following method is used: and The absolute value of the inner product Indicates the degree of consistency between the two edge directions. This represents the absolute value. When the 3D edge projection direction is the same as or opposite to the image edge direction, this value is close to 1, and the direction consistency residual is close to 0; when the two are nearly perpendicular, this value is close to 0, and the direction consistency residual increases. The vector inner product is a scalar obtained by multiplying the corresponding components of two vectors and then summing them. It is used to describe the relationship and projection between vectors.

[0049] Therefore, the edge direction consistency residual is used to constrain the direction of the radar three-dimensional edge projection to be consistent with the corresponding edge direction in the image, avoiding mismatch between the three-dimensional horizontal edge and the vertical edge of the image, or mismatch between the three-dimensional vertical edge and the horizontal edge of the image, thereby improving the accuracy of edge matching of the high reflectivity I-shaped calibration plate and the stability of extrinsic parameter optimization.

[0050] The Adaptive Robust Levenberg-Marquardt (LM) algorithm is an improved numerical optimization method specifically designed for nonlinear least squares problems, incorporating dynamic weighting and prior regularization strategies into the traditional LM algorithm. In each iteration, the algorithm constructs a weighted approximate Hessian matrix and a weighted gradient vector based on the current residual vector and Jacobian matrix. The weight matrix is ​​composed of robust kernel weights for each observation, image edge confidence, and point cloud reflectance confidence, dynamically suppressing the adverse effects of outliers such as dust noise, overexposed edges, occlusion, and mismatches on the optimization direction. In the calibration task of a shotcrete robot, this algorithm can efficiently converge to the global optimum within a finite number of iterations, exhibiting high accuracy, strong robustness, and real-time online computation capabilities.

[0051] When calibrating using an online joint calibration method combining a laser radar and a monocular camera on a shotcrete robot, the initial offline extrinsic parameters are first acquired, and an error compensation model for an I-shaped calibration plate is built. Then, the IMU triggers an online calibration command in real time, controlling the robotic arm to bring the reflective I-shaped calibration plate into the shared field of view of the radar and camera. Next, gamma correction and image enhancement processing are used to improve the distance, direction, and position accuracy of the image edges. After filtering out the reflective point cloud, the plane fitting of the calibration plate and the template matching of the I-shaped calibration plate are completed. The forward edge distance error, reverse edge distance error, direction consistency error, corner topology error, and extrinsic parameter mismatch error are extracted. Then, the adaptive damping factor LM algorithm is used to optimize the current extrinsic parameters. Finally, it is determined whether the system has degraded: if degradation is determined, the robotic arm is controlled to perform the operation of clamping the watch core; if no degradation is determined, the extrinsic parameter matrix is ​​directly updated.

[0052] Figure 1 This demonstrates the change in average reprojection error before and after online calibration in an experiment with fixed extrinsic parameter bias. The blue curve represents the average reprojection error after introducing fixed extrinsic parameter bias without online correction, while the orange curve represents the average reprojection error after online optimization of the extrinsic parameters using the method of this invention. Figure 1 It is evident that the reprojection error before optimization is significantly higher than that after optimization, and exhibits substantial fluctuations when extrinsic parameters are biased. After online joint calibration, the reprojection error is rapidly reduced and maintained within a low range. This result demonstrates that the method of this invention can effectively correct radar-camera extrinsic parameters by using constraints from a highly reflective I-shaped calibration plate, image edge distance field constraints, and point cloud edge constraints, thereby improving cross-modal projection alignment accuracy, even with fixed extrinsic parameter deviations.

[0053] Figure 2 The figure shows the online correction curves for the rotational extrinsic parameters in a fixed extrinsic parameter bias experiment. In the figure, Δrx, Δry, and Δrz represent the rotational corrections around the x-axis, y-axis, and z-axis, respectively, in mrad. The curves reflect the estimation and compensation process for rotational errors in each direction when there is a rotational deviation in the extrinsic parameters. The rotational correction changes with the optimization process, indicating that the algorithm does not only reduce projection errors at the image level but can also solve for the rotational compensation of the extrinsic parameters in the corresponding coordinate axis directions.

[0054] Figure 3 The figure shows the online correction curves for the translational extrinsic parameters in a fixed extrinsic parameter bias experiment. In the figure, Δtx, Δty, and Δtz represent the translational correction along the x-axis, y-axis, and z-axis, respectively, in mm. The curves reflect the estimation and compensation process for the relative position offset between the radar and camera using the method of this invention. Combined with... Figure 1The results of the reprojection error reduction show that the method of the present invention can not only reduce the pixel error after the point cloud is projected onto the image, but also output physically meaningful rotation and translation extrinsic parameter corrections, thereby providing accurate extrinsic parameters for subsequent point cloud-image fusion modeling.

[0055] Figure 4 The curve showing the percentage decrease in reprojection error during experiments with fixed extrinsic parameter bias is presented. This percentage decrease is calculated from the average reprojection error before and after online calibration and is used to quantitatively evaluate the suppression effect of the method of this invention on the projection error caused by extrinsic parameter bias. A higher percentage decrease indicates a more significant reduction in error after online optimization. Figure 4 It can be seen that, under the condition of fixed extrinsic parameter bias, the method of the present invention can continuously reduce the reprojection error after the radar point cloud is projected onto the image plane, indicating that the method has good extrinsic parameter error correction capability and cross-modal alignment accuracy improvement effect.

[0056] Figures 5 to 8 The study demonstrates the correction amount of extrinsic parameters, the reduction rate of reprojection error, and the change of reprojection error before and after online calibration under continuous vibration conditions. Under continuous vibration conditions, the error fluctuates significantly with mechanical disturbances before optimization, while the error remains within a low range after optimization, indicating that this method can provide online response and compensation for extrinsic parameter drift caused by dynamic disturbances.

[0057] Figure 5 This is a graph showing the online correction amount of the rotational extrinsic parameters under continuous vibration conditions. In the graph, Δrx, Δry, and Δrz represent the rotational correction amounts around the x-axis, y-axis, and z-axis, respectively. Continuous vibration causes dynamic disturbances in the relative attitude between the radar and the camera. The continuous change in the rotational correction amount curve demonstrates that the method of this invention can sense and compensate for the rotational drift of the extrinsic parameters caused by mechanical vibration in real time.

[0058] Figure 6 The figure shows the online correction curves for translational extrinsic parameters under continuous vibration conditions. In the figure, Δtx, Δty, and Δtz represent the translational correction along the x-axis, y-axis, and z-axis, respectively. The curves demonstrate that when the robotic arm or sensor mounting structure is continuously disturbed, the method of this invention can output translational compensation in the corresponding directions, thereby maintaining the spatial alignment between the radar point cloud and the camera image.

[0059] Figure 7 This is the reprojection error reduction rate curve under continuous vibration conditions. This curve is used to evaluate the degree to which online calibration suppresses dynamic disturbance errors. Figure 7 It is evident that the method of the present invention can still maintain a certain error reduction effect under continuous vibration conditions, indicating that the method has good dynamic disturbance adaptability and robustness.

[0060] Figure 8The figure shows the change in average reprojection error before and after online calibration under continuous vibration conditions. Before optimization, the error fluctuated significantly due to mechanical vibration, while after optimization, the error remained within a low range. This indicates that the method of this invention can provide online response and compensation for external parameter drift under non-static conditions, thereby improving the stability of radar and camera fusion during the operation of the shotcrete robot.

[0061] Figure 9 This is a projection overlay image after online calibration for a continuous vibration bias experiment, reflecting the projection overlay effect after online calibration under continuous vibration conditions. In the image, the red box represents the result of projecting the I-shaped topology template or radar edge points onto the image plane based on the online optimized radar-camera extrinsic parameters; the highlighted areas in the image represent the highly reflective calibration plate areas, and the light-colored edges represent the actual edges detected by the image end. Figure 9 As can be seen, the red projection outline basically coincides with the edge of the I-shaped calibration plate in the image, indicating that the online optimized extrinsic parameters can achieve a good radar-camera cross-modal alignment effect.

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An online joint calibration method for a shotcrete robot's lidar and monocular camera, characterized in that, include: S1. Connect the lidar and monocular camera, and obtain the initial extrinsic parameter matrix through offline calibration; S2. The vibration is monitored in real time by the inertial measurement unit to obtain vibration information. When the vibration exceeds the preset threshold, an online calibration command is triggered. S3. Responding to the online calibration command, a highly reflective I-shaped calibration board with a known topology template is introduced, and the monocular camera image data and lidar point cloud data of the calibration board are collected simultaneously. S4. Filter the high reflectivity point cloud corresponding to the calibration board in the lidar point cloud data, and process the monocular camera data according to the high reflectivity point cloud to generate a binary edge map. On the image side, extract the two-dimensional edge sampling points of the binary edge map and generate the image edge information field. On the point cloud, fit the calibration board spatial plane based on the high reflectivity point cloud and combine it with the topological template of the calibration board to obtain three-dimensional edge sampling points with edge category labels and construct a point cloud feature point set. S5. Based on the point cloud feature point set, combined with the known topological template of the high-reflectivity I-shaped calibration plate and the image edge information field, calculate the forward residual, reverse residual, direction consistency residual, corner topological residual and prior residual, and construct a composite objective function. S6. Construct an external parameter perturbation model. Based on the composite objective function, iteratively optimize the external parameter matrix using the initial external parameter matrix as the initial value, and output the optimized external parameter matrix for online joint calibration of the laser radar and monocular camera of the shotcrete robot. The composite objective function is: ； ； ； ； ； ； In the formula, This represents the external parameter perturbation. This represents a composite objective function. For positive edge distance, Indicates the first The positive residual of each three-dimensional edge sampling point For edge direction consistency terms, Indicates the first The directional consistency residual of each three-dimensional edge sampling point For the reverse template distance item, Indicates the first The inverse residual of each two-dimensional edge sampling point For corner topology terms, Indicates the first Corner topological residuals of two-dimensional corner points As an external reference prior term, Indicates the prior residuals of external parameters. , , , , They represent , , , , The weighting coefficients, , , , They represent , , , Robust kernel function, The information matrix representing the topological residuals at corner points; The adaptively damped robust LM algorithm iteratively optimizes the extrinsic parameters. The adaptively damped robust LM algorithm updates the extrinsic parameters using Lie algebra perturbations. ； In the formula, For the first The adaptive damping factor for the next iteration. Indicates by A diagonal matrix composed of the diagonal elements. This is the extrinsic prior information matrix. Indicates the first The weighted approximation Hessian matrix of the next iteration. Indicates by A diagonal matrix composed of the diagonal elements. Indicates the first The weighted gradient vector of the next iteration Indicates the first The increment of the extrinsic parameters in the next iteration Indicates the first The extrinsic perturbation quantity of the next iteration.

2. The online joint calibration method for the laser radar and monocular camera of the shotcrete robot according to claim 1, characterized in that, In S1, the lidar and monocular camera are fixed together by a rigid bracket, and the fixed lidar and monocular camera are installed as a whole on the base of the shotcrete robot's robotic arm. The initial extrinsic parameter matrix is ​​obtained by offline calibration based on minimizing the corner reprojection error of the checkerboard calibration board. The initial extrinsic parameter matrix is: ； In the formula, Represents the initial extrinsic parameter matrix. This represents the initial rotation matrix of the lidar to the monocular camera. This is the initial translation vector from the lidar to the monocular camera.

3. The online joint calibration method for the laser radar and monocular camera of the shotcrete robot according to claim 2, characterized in that, In S2, the inertial measurement unit is installed on the robotic arm of the shotcrete robot. The vibration information includes the cumulative amount of high-frequency vibration, the instantaneous acceleration change, and the angular velocity change. When any vibration information exceeds the corresponding preset threshold, an online calibration command is triggered. In S3, the topological template of the high-reflectivity I-shaped calibration plate includes multiple edge segments and corner points with labels. The edge segments include at least the outer edge segment of the upper crossbeam, the outer edge segment of the lower crossbeam, the side edge segment of the middle vertical beam, and the concave edge segment. The corner points include at least the convex corner points and the concave corner points. Each edge segment includes the segment endpoints, direction vector, segment length, category label, and adjacency relationship; Category labels include horizontal outer edge, vertical outer edge, concave edge, and corner category.

4. The online joint calibration method for the laser radar and monocular camera of the shotcrete robot according to claim 3, characterized in that, In S4, point cloud data points with echo intensity greater than a preset echo intensity threshold are selected from the lidar point cloud data to construct a high reflectivity point cloud. Using the initial extrinsic parameter matrix, the high reflectivity point cloud is projected onto the planar image of the monocular camera image data. The point cloud projection area is obtained on the planar image. Image processing, including image enhancement, high reflectivity area segmentation, and edge detection, is performed on the image in the point cloud projection area to generate a two-dimensional edge map. The image edge information field includes the image edge distance field, the image edge orientation field, and the image edge confidence field; The image edge distance field is obtained by performing a distance transformation on the two-dimensional edge map, including the horizontal edge distance field and the vertical edge distance field. The image edge direction field is obtained by performing image gradient direction detection and line detection on the two-dimensional edge map. The image edge confidence field is jointly determined by the image gradient magnitude, the brightness response of highly reflective areas, and edge connectivity.

5. The online joint calibration method for the laser radar and monocular camera of the shotcrete robot according to claim 4, characterized in that, In S5, outlier removal is performed on the highly reflective point cloud. After removing outliers, the highly reflective point cloud is fitted to the calibration board space plane using RANSAC plane fitting. The highly reflective point cloud is then projected onto the fitted calibration board space plane to construct a local coordinate system for the calibration board. In the local coordinate system of the calibration board, the highly reflective point cloud is matched with the topological template of the calibration board. Based on the pose of the calibration board in the lidar coordinate system, samples are taken at equal intervals on each edge segment to obtain several 3D edge sampling points with corresponding category labels for the edge segments, thus constructing a point cloud feature point set. The feature point set of the point cloud is: ； In the formula, Represents the feature point set of a point cloud. This represents the total number of 3D edge sampling points in the point cloud feature point set. This represents the index of the number of 3D edge sampling points in the point cloud feature point set. This represents the total number of edge segments corresponding to the sampled points in the point cloud feature point set. This represents the index of the number of edge segments corresponding to the sampling points in the point cloud feature point set. Indicates the first The three-dimensional coordinates of each three-dimensional edge sampling point Indicates the first The edge segment corresponding to each 3D edge sampling point Category tags, Indicates the first The edge segment corresponding to each 3D edge sampling point directional vector, Indicates the first The confidence level of the point cloud of a three-dimensional edge sampling point.

6. The online joint calibration method for the laser radar and monocular camera of the shotcrete robot according to claim 5, characterized in that, In S6, the 3D edge sampling points are transformed to the camera coordinate system according to the current extrinsic matrix, and then projected onto the image plane of the binary edge map through the monocular camera intrinsic matrix. The distance value is read in the image edge distance field, the image edge unit direction vector is read in the image edge direction field, and the image edge confidence is read in the image edge confidence field. The positive residual and the direction consistency residual are calculated. The positive residual is: ； In the formula, Indicates the first The positive residual of each three-dimensional edge sampling point Indicates the first Pixel coordinates of three-dimensional edge sampling points Indicates the first Image edge confidence of 3D edge sampling points This is a function for calculating the confidence score of image edges. Indicated in pixels The distance value is obtained from the image edge distance field; The directional consistency residual is: ； ； ； In the formula, Indicates the first The directional consistency residual of each three-dimensional edge sampling point Represents absolute value. Indicated in pixels The image edge unit direction vector is obtained from the image edge direction field. Indicates the first Normalized unit projection tangential direction of each 3D edge sampling point Represents the transpose of a matrix or vector. Represents perspective projection function exist Jacobian matrix at the location, for Three-dimensional coordinates in the camera coordinate system Indicates the first The tangential direction vector of each 3D edge sampling point in the camera coordinate system This represents the rotation matrix of the lidar to the monocular camera. This represents the Euclidean norm of a vector.

7. The online joint calibration method for the laser radar and monocular camera of the shotcrete robot according to claim 6, characterized in that, First, the intrinsic parameter matrix of the monocular camera is used to back-project the two-dimensional edge sampling points into the direction of the normalized ray emitted from the optical center to obtain the normalized ray. Then, the normalized ray is transformed to the lidar coordinate system according to the current extrinsic parameter matrix. The back-projection points of the two-dimensional edge sampling points in the local coordinate system of the calibration board are obtained by fitting the calibration board spatial plane. Finally, the back-projection points are transformed to the local coordinate system of the calibration board according to the pose transformation matrix from the lidar coordinate system to the local coordinate system of the calibration board. Read the distance value in the image edge distance field, read the image edge unit direction vector in the image edge direction field, read the image edge confidence in the image edge confidence field, determine the set of candidate edge segments that match the image edge pixels, and calculate the inverse residual; The reverse residual is: ； ； In the formula, Indicates the index of the number of two-dimensional edge sampling points. Indicates the first The inverse residual of each two-dimensional edge sampling point Indicates the first The pixel coordinates of the back projection points of each two-dimensional edge sampling point in the local coordinate system of the calibration board. Indicates the first The pixel coordinates of a two-dimensional edge sampling point Indicates the first Image edge confidence of two-dimensional edge sampling points This represents the function for calculating the minimum value. Indicates the back projection point to the edge segment Euclidean distance, This represents the pose transformation matrix from the lidar coordinate system to the local coordinate system of the calibration board. This represents the set of candidate edge segments.

8. The online joint calibration method for the laser radar and monocular camera of the shotcrete robot according to claim 7, characterized in that, Extract the three-dimensional corner points corresponding to the corner points in the lidar coordinate system, transform the three-dimensional corner points to the camera coordinate system according to the current extrinsic matrix, and then obtain the projection coordinates of the three-dimensional corner points on the image plane according to the monocular camera intrinsic matrix to obtain the two-dimensional corner points and calculate the corner point topological residuals. The corner topological residual is: ； In the formula, Indicates the index of the number of corner points. This represents the intrinsic parameter matrix of a monocular camera. Indicates the first Confidence of a three-dimensional corner point Indicates the first The confidence level of the two-dimensional corner points corresponding to the three-dimensional corner points Indicates the first The coordinates of a 3D corner point in the lidar coordinate system Represents the first image plane pixel coordinates of a two-dimensional corner point Represents the translation vector. Represents the perspective projection function; The relative transformation of the current extrinsic matrix with respect to the initial extrinsic matrix is ​​mapped to the Lie algebra space to obtain a six-dimensional extrinsic offset vector. The offset vector is then normalized according to the prior covariance matrix of the extrinsic parameters to construct the prior residuals of the extrinsic parameters. ； In the formula, Indicates the prior residuals of external parameters. This represents the square root information matrix of the prior covariance matrix of the extrinsic parameters. Represents the prior covariance matrix of the extrinsic parameters. Indicates from Li Qun To Lie algebra logarithmic mapping, This represents the operator that transforms a Lie algebra matrix into a six-dimensional vector. This represents the current extrinsic parameter matrix.

Images