An underground online joint calibration method

Abstract

The present invention relates to an underground online joint calibration method, belonging to the field of underground detection technology. The method comprises obtaining anchor net image data collected by an industrial camera and anchor net point cloud data collected by a laser radar; extracting the first edge contour of the anchor net from the anchor net image data using an image processing method; extracting the second edge contour of the anchor net from the anchor net point cloud data using a point cloud processing method; performing a coarse matching of the first edge contour and the second edge contour based on a particle swarm search method of longhorn beetle whiskers; and performing a precise matching of the first edge contour and the second edge contour, specifically: taking measurement errors into account, constructing a line-to-line ICP problem, and performing nonlinear least squares optimization to solve the optimal external parameters and obtain a calibration result. During the matching process, it is easy to fall into a local optimal solution. The particle swarm search method of longhorn beetle whiskers can prevent the local optimal solution, thereby reducing the number of searches and ensuring that the local optimal solution will not be fallen into.

Description

Technical Field

[0001] The present invention relates to the field of downhole detection technology, and in particular to an online joint calibration method for downhole laser radar and industrial cameras. Background Art

[0002] With the increasing application of LiDAR and industrial cameras in unmanned systems, the use of multi-sensor fusion to enhance their perception capabilities is gaining increasing attention. The fusion of LiDAR and industrial cameras holds significant significance in underground rock bolting. LiDAR can quickly and accurately measure the shape and size of surfaces, while industrial cameras can capture a wide range of surface details and provide higher-precision measurements. Combining these two approaches enables more accurate 3D model reconstruction, leading to a better understanding of underground structures, more accurate assessment of their mechanical properties, and effective stability prediction. Calibration can help us better integrate LiDAR and industrial cameras, enabling tasks such as anchor hole detection, sign reading, and worker and obstacle identification. Given that jitter or positional offset between the LiDAR and industrial cameras makes the initial calibration extrinsic parameters ineffective, online correction is necessary, requiring joint calibration of the LiDAR and industrial cameras.

[0003] Existing calibration methods generally use specific calibration reference objects, such as chessboard calibration plates or 3D-shaped reference objects, to assist the algorithm in finding relevant features. These methods often require manual intervention and often require human intervention. However, calibrating these reference objects before each mission is impractical. To reduce reliance on scene features, current automatic calibration methods still face two challenges: First, while algorithms for specific calibration objects do not require manual intervention, they require high reference object requirements and must be placed in advance. Second, methods that do not rely on reference objects require initial parameters and rely on extrinsic parameters instead of feature matching, using egomotion estimation. Motion estimation-based methods require accurate and sufficient egomotion state estimation and the presence of 3D geometric features and trackable visual features in the scene. The lack of these features can lead to positioning slippage in LiDAR ranging estimation, thus affecting the accuracy of calibration results. Alternatively, there are methods that apply deep learning to extrinsic parameter estimation for industrial cameras and LiDARs. However, these methods require the input of ground-truth depth data as supervision data. However, obtaining accurate ground-truth depth values is often difficult in real-world scenarios, and deep learning methods suffer from poor interpretability and generalization.

[0004] In summary, although the above methods can complete online calibration, they are not robust enough to the environment and have low accuracy. The method of extracting information such as the depth edge of the point cloud is only applicable in long-distance and abundant reference objects. However, due to the poor lighting conditions and few reference objects underground, the above methods cannot be applied to online calibration tasks underground.

[0005] Due to the low visibility underground, the clarity of images collected by general cameras is significantly reduced, and the texture information underground cannot be recognized. The present invention can use underground objects (such as anchor nets) for online calibration without the need for human intervention or calibration plates with special features. Summary of the Invention

[0006] To solve the above technical problems, the present invention provides an online joint calibration method for underground laser radar and industrial cameras. The technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention provides an underground online joint calibration method, comprising:

[0008] S1, moving the laser radar and the industrial camera so that the anchor net is placed in the field of view of the laser radar and the industrial camera, and obtaining the anchor net image data collected by the industrial camera and the anchor net point cloud data collected by the laser radar;

[0009] S2, extracting a first edge contour of the anchor net from the anchor net image data using an image processing method;

[0010] S3, extracting a second edge contour of the anchor net from the anchor net point cloud data using a point cloud processing method;

[0011] S4, performing a rough matching of the first edge contour and the second edge contour based on a particle swarm search method of longicorn beetle whiskers;

[0012] S5, performing precise matching on the first edge profile and the second edge profile, specifically: taking measurement errors into account, constructing a line-to-line ICP problem, and performing nonlinear least squares optimization to solve the optimal extrinsic parameters to obtain a calibration result.

[0013] Furthermore, before extracting the first edge contour of the anchor net from the anchor net image data using an image processing method, the method further includes:

[0014] The Zhang Zhengyou calibration method is used to calibrate the internal parameters of the industrial camera.

[0015] Furthermore, when extracting the first edge contour of the anchor net from the anchor net image data using an image processing method according to the characteristics of the anchor net, S2 includes:

[0016] S21, extracting the initial edge contour of the anchor network using the Canny algorithm according to the characteristics of the anchor network;

[0017] S22, fitting the initial edge contour using a Hough transform algorithm to obtain a contour line of the anchor network;

[0018] S23, remove the contour lines whose length is less than a preset threshold from the contour lines of the anchor network, and use the remaining contour lines as the first edge contour of the anchor network. The pixel coordinates of the remaining contour lines are put into the kd tree tr1, and the image edge, i.e., the remaining anchor network contour lines, are recorded as

[0019] Furthermore, when extracting the second edge contour of the anchor net from the anchor net point cloud data using a point cloud processing method, S3 includes:

[0020] S31, setting the side length of the voxel grid, and dividing the anchor point cloud data into M, N, and L parts along the X, Y, and Z coordinate axes according to the side length of the voxel grid. After dividing the anchor point cloud data into M*N*L voxel grids, plane fitting is performed on the anchor point cloud data in each voxel grid to obtain multiple fitting planes, the normal vector of each fitting plane is calculated, and after eliminating the intersection lines in the same fitting plane, dense interpolation is performed on the remaining intersection lines to obtain a depth-continuous edge point cloud;

[0021] S32, projecting the reflection intensity of the anchor network point cloud data in each fitting plane along its normal vector direction to obtain a 2D intensity image corresponding to each fitting plane, extracting the reflection intensity mutation edge point cloud in the 2D intensity image using the Canny algorithm, and then obtaining the reflection intensity mutation edge point cloud in the 3D intensity image using inverse projection;

[0022] S33: Determine the depth-continuous edge point cloud and the reflection intensity mutation edge point cloud as a second edge contour of the anchor network.

[0023] Furthermore, when performing a rough match between the first edge contour and the second edge contour using a particle swarm search method based on longicorn beetle whiskers, S4 includes:

[0024] S41, pre-set the population size to P, that is, there are P longhorn beetles, and the maximum number of iterations is N max , the learning factors are c1 and c2, the inertia weight is w, and the maximum and minimum speeds are V max 、V min Finally, a search range is preset based on the positional relationship between the industrial camera and the lidar. The update speed v of each beetle in the population is randomly initialized within the preset search range. A population is randomly generated within the preset search range and the randomly generated population is initialized using the Tent chaotic sequence so that the parameters in R and t are distributed within the preset search range.

[0025] The expression of the Tent chaotic sequence is recorded as Expression 1, and the specific expression 1 is as follows:

[0026]

[0027] Among them, Rand represents a random number, R and t generated by chaos initialization are x i =(T1, T2, T3, T4, T5, T6, T7), which means generating a particle;

[0028] S42, after projecting the three-dimensional coordinates of the second edge contour onto the pixel coordinate system, remove the projection points that exceed the image size range of the industrial camera from the projected pixel coordinates, and store the remaining projection points in another KD tree tr2; use the projection points in tr2 as query points, perform a nearest neighbor query in the image edge contour KD tree tr1, filter out the projection points whose distances from the nearest neighbor points to the remaining projection points are within the threshold, and then establish a nearest neighbor matching relationship. Traverse all points in tr2 and establish a nearest neighbor matching relationship for all points that meet the requirements; wherein, set expression 2, and calculate the pixel coordinates of the three-dimensional coordinates of the point cloud edge projected onto the pixel coordinate system according to expression 2, and the specific expression 2 is as follows:

[0029]

[0030] Among them, u represents the x-axis coordinate in the pixel coordinate system, v represents the y-axis coordinate in the pixel coordinate system, and z c Indicates the actual depth information of the point in the pixel coordinate system, f x Represents the transverse focal length component parameter, f y represents the longitudinal focal length component parameter, (u0, v0) represents the actual position of the principal point, R represents the rotation matrix between the laser radar and the camera, t represents the translation matrix between the laser radar and the camera, (X L ,Y L ,Z L ) represents the three-dimensional coordinates of the laser radar, and π is the distortion model of the industrial camera;

[0031] S43, setting expression three, and using expression three to calculate the coordinates of the left and right whiskers of the longicorn beetle, the expression three is specifically as follows:

[0032]

[0033] Where n represents the nth iteration, dir represents a random unit vector, represents the position coordinates of the left and right whiskers of the nth iteration, d0 represents the distance between the two whiskers of the longicorn, and x n Indicates the coordinates of the nth iteration of the longicorn;

[0034] S44, setting an expression 4, and using the expression 4 to calculate the fitness value of the position coordinates of the left and right whiskers of the longhorn beetle. The expression 4 is specifically as follows:

[0035]

[0036] Where N represents the number of lines, The direction vector representing the edge contour of the anchor network in the pixel plane, Represents the direction vector of the anchor network’s point cloud edge contour projected onto the pixel plane, (x·y) represents the inner product of x and y. When both are unit vectors, (x·y) is the cosine value of vector x and vector y, are the linear coefficients of the edge contour of the anchor hole in the pixel coordinate system, The projection point of the three-dimensional data onto the pixel plane;

[0037] S45, respectively obtain the extreme values of the population and the individual. The extreme value of the individual is expressed as Expression 5, and the extreme value of the population is expressed as Expression 6. The specific expression 5 is as follows:

[0038] i is the i-th longicorn;

[0039] The expression six is specifically as follows:

[0040] P is the number of populations;

[0041] S46, determining whether the extreme value of the individual and the extreme value of the population are optimal values, if so, the program terminates; if not, proceeding to the next step;

[0042] S47, setting Expression 7 and Expression 8 to update the speed and position of the group, and using Expression 7 and Expression 8 to make the leading particle jump out of the local optimal solution and approach the global optimal solution;

[0043] The expression seven is specifically as follows:

[0044]

[0045] Among them, r1 and r2 are random numbers, x i (n) is the position of the i-th longicorn, and n is the number of iterations;

[0046] The expression eight is specifically as follows:

[0047] x i (n+1)=x i (n)+v i (n+1)

[0048] When the update speed v i (n+1) exceeds V max , then take V max , similarly when the update speed v i (n+1) is less than V min , then take V min ;

[0049] S48, judging whether the maximum number of iterations has been reached, if not, then repeating S42-S47; if so, then exiting the program loop and outputting the optimal solution.

[0050] Furthermore, S42 includes:

[0051] For a single projection point p i , use it as the query point to perform k-nearest neighbor search in tr1, and the obtained k-nearest neighbor point set is The formula is used to centralize the k nearest neighbor points by subtracting the mean. Finally, according to the principle that the eigenvector corresponding to the minimum eigenvalue after the singular value decomposition of the covariance matrix is the normal vector of the fitting line, the direction of the image edge contour is calculated; in the same way, the projection point in tr2 is used as the query point, and the nearest neighbor query is performed in the image edge contour KD tree tr2 and the nearest neighbor matching relationship is established. All points in tr2 are traversed, and the nearest neighbor matching relationship is established for those that meet the requirements; a single projection point is used as the query point to perform a k-nearest neighbor search in tr2, and the k nearest neighbor point sets obtained are solved using the above method to obtain the direction of the point cloud edge contour; the matches between the point cloud edge contour and the image edge contour with an angle between [-30°, 30°] or [150°, 270°] are retained and counted, and the statistical number is recorded as N.

[0052] Furthermore, the S5 includes:

[0053] S51, considering the measurement noise, project the edge points of the point cloud to the image plane, and the edge points of the image {q i , n i} Expression 9, the expression 9 is as follows:

[0054]

[0055] in, L w i represents the measurement noise of the lidar, I w i represents the measurement noise of the camera;

[0056] The expression nine is a constraint formula;

[0057] Wherein, the expression 9 is nonlinear. The expression 9 is linearized and the expression 10 is obtained by the first-order Taylor expansion. The specific expression 10 is as follows:

[0058]

[0059] Among them, r i is the initial value of the external parameter, is the corresponding formula approximation, is the derivative of the external parameter, is the derivative with respect to the noise;

[0060] S52: After optimizing the expression 10, the expression 11 is obtained. The specific expression 11 is as follows:

[0061]

[0062] S53. Update the current iteration value to obtain Expression 12, which is as follows:

[0063]

[0064] The expression twelve is iterated until convergence, and the result after convergence is the optimal external parameter.

[0065] In a second aspect, the present invention provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method according to the first aspect.

[0066] All the above optional technical solutions can be combined arbitrarily, and the present invention does not provide detailed descriptions of the structures after each combination.

[0067] By means of the above solution, the beneficial effects of the present invention are as follows:

[0068] Due to the poor lighting conditions and low visibility underground, industrial cameras equipped with light sources can illuminate the underground area over a large area, which is more conducive to the recognition of industrial cameras; using the underground anchor net for online calibration avoids the use of calibration objects that require human intervention or have special characteristics, which is conducive to improving the accuracy of online calibration; the anchor net itself is characterized by being composed of multiple horizontal lines and multiple vertical lines that are parallel and intersecting with each other. During the matching process, it is easy to fall into the local optimal solution. The particle swarm search method of longhorn beetle whiskers can prevent local optimal solutions, which can not only reduce the number of searches but also ensure that it will not fall into the local optimal solution; in the precise matching process, the optimization direction is clarified, which can reduce the number of optimization times while improving the calibration accuracy.

[0069] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is an overall flow chart of an underground online combined calibration method provided by the present invention.

[0071] Figure 2 This is a schematic diagram of the optimization of the rotation matrix in the downhole online joint calibration method provided by the present invention.

[0072] Figure 3 This is a linear method in an underground online joint calibration method provided by the present invention. To the straight line distance.

[0073] Figure 4 This is an overall program flowchart of an underground online joint calibration method provided by the present invention. DETAILED DESCRIPTION

[0074] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0075] like Figure 1 and Figure 4 As shown in FIG, an online combined calibration method for underground provided by the present invention, the execution subject of the method can be a computing device, or a device, program, chip, etc. in the computing device. The present invention is described by taking the computing device as an example. Figure 1 and Figure 4 As shown, the method may include:

[0076] S0, use Zhang Zhengyou calibration method to calibrate the internal parameters of industrial cameras.

[0077] Specifically, a calibration board with black and white checkerboard is prepared. The corner size of the squares is 9 rows × 7 columns, and the size of each square is 10cm × 10cm. A camera is used to capture 40 images at different angles and distances. Then, the Matlab calibration toolbox is used to complete the intrinsic parameter calibration process. According to the work requirements, the calibration reprojection error is within one pixel, which meets the requirements.

[0078] S1, move the laser radar and industrial camera to place the anchor net in the field of view of the laser radar and industrial camera, and obtain the anchor net image data collected by the industrial camera and the anchor net point cloud data collected by the laser radar.

[0079] Specifically, the anchor trolley and its robotic arm are moved to allow the industrial camera to be positioned directly below the anchor net to collect anchor net image data. The anchor trolley controls the drill arm to move directly below the anchor net, and the LiDAR operates to fully scan and sample the anchor net, collecting anchor net point cloud data.

[0080] S2, extracting the first edge contour of the anchor net from the anchor net image data using an image processing method.

[0081] Specifically, when S2 extracts the first edge contour of the anchor net from the anchor net image data using an image processing method according to the characteristics of the anchor net, it includes: S21, extracting the initial edge contour of the anchor net using the Canny algorithm according to the characteristics of the anchor net. S22, fitting the initial edge contour using the Hough transform algorithm to obtain the contour line of the anchor net. S23, removing the contour lines whose length is less than a preset threshold from the contour lines of the anchor net, and taking the remaining contour lines as the first edge contour of the anchor net, and putting their pixel coordinates into the kd tree tr1, and recording the image edge, i.e., the remaining anchor net contour line, as

[0082] S3, extracting the second edge contour of the anchor network from the anchor network point cloud data using a point cloud processing method.

[0083] Specifically, when S3 uses the point cloud processing method to extract the second edge contour of the anchor network from the anchor network point cloud data, it includes: S31, setting the voxel grid side length, and dividing the anchor network point cloud data into M, N, and L parts along the three coordinate axes of X, Y, and Z according to the voxel grid side length. After dividing the anchor network point cloud data into M*N*L voxel grids, plane fitting is performed on the anchor network point cloud data in each voxel grid to obtain multiple fitting planes, the normal vector of each fitting plane is calculated, and after eliminating the intersection lines in the same fitting plane, dense interpolation is performed on the remaining intersection lines to obtain a deep continuous edge point cloud. For example, the voxel grid side length cell is set, and the three coordinate axes X, Y, and Z of the sampling point cloud are equally divided into M, N, and L parts. The point cloud data is then divided into M*N*L voxel grids. The Ransac algorithm is then used to fit the plane to the point cloud in each voxel grid. The intersection lines of the normal vectors of adjacent planes within the preset angle range [30°, 150°] are retained, and the intersection lines that are actually in the same plane are eliminated. Dense interpolation is performed on the remaining intersection lines to obtain depth-continuous edges.

[0084] S32: Project the reflection intensity of the anchor network point cloud data in each fitting plane along its normal vector direction to obtain a 2D intensity image corresponding to each fitting plane, then use the Canny algorithm to extract the reflection intensity mutation edge point cloud in the 2D intensity image, and then use inverse projection to obtain the reflection intensity mutation edge point cloud in the 3D intensity image. For example, the point cloud data collected by the lidar includes X, Y, Z, and reflection intensity. The reflection intensity of the point cloud data in each plane is projected along its normal vector direction to obtain a 2D point cloud intensity image, and then use the Canny operator to extract the reflection intensity mutation edge in the 2D intensity image, and then use inverse projection to obtain a 3D reflection intensity mutation edge point cloud.

[0085] S33, the depth continuous edge point cloud and the reflection intensity mutation edge point cloud are determined as the second edge contour of the anchor network. For example, the extracted point cloud edge is recorded as Includes depth continuous edge point cloud and reflection intensity mutation edge point cloud.

[0086] S4, performing a rough match between the first edge contour and the second edge contour based on a particle swarm search method of longicorn beetle whiskers.

[0087] That is to say, the edge contours of the anchor network in the image and the edge contours of the anchor network in the point cloud are matched, which is the coarse matching process.

[0088] Among them, such as Figure 2 As shown in the figure, coarse matching is to adjust the rotation matrix and translation matrix so that the distance between the 2D point projected from the point cloud to the pixel coordinate system is closest to the corresponding real 2D point. The specific method is as follows:

[0089] S41, pre-set the population size to P (ie, there are P longhorn beetles), the maximum number of iterations is N max , the learning factors are c1 and c2, the inertia weight is w, and the maximum and minimum speeds are V max 、V min Finally, a search range is preset according to the positional relationship between the industrial camera and the lidar. The update speed v of each longhorn beetle in the population is randomly initialized within the preset search range, and a population is randomly generated within the preset search range (the position xi of each longhorn beetle is determined). The randomly generated population is initialized using the Tent chaotic sequence, so that R (quaternion) and the seven parameters in t (the unit quaternion q is used to represent the rotation matrix R) are distributed throughout the preset search range, ensuring that the initialized population is random, ergodic, and regular.

[0090] The expression of the Tent chaotic sequence is recorded as Expression 1, which is as follows:

[0091]

[0092] Among them, Rand represents a random number, R and t generated by chaos initialization are x i =(T1, T2, T3, T4, T5, T6, T7), that is, a particle is generated, and then the above operation is repeated according to the population size, and the chaos initialization is completed.

[0093] S42, after projecting the three-dimensional coordinates of the second edge contour onto the pixel coordinate system, remove the projection points that exceed the image size range of the industrial camera (e.g., the circular field of view of Mid-70 can reach 70.4 degrees in both horizontal and vertical directions, and the resolution of the Hikvision camera is 2592×2048) from the projected pixel coordinates, and store the remaining projection points in another KD tree tr2; use the projection points in tr2 as query points, perform a nearest neighbor query in the image edge contour KD tree tr1, filter out the projection points whose distances from the nearest neighbor points to the remaining projection points are within a threshold (d=3), and then establish a nearest neighbor matching relationship. Traverse all points in tr2 and establish a nearest neighbor matching relationship for all points that meet the requirements; wherein, set expression 2, and calculate the pixel coordinates of the three-dimensional coordinates of the point cloud edge projected onto the pixel coordinate system according to expression 2, and expression 2 is specifically as follows:

[0094]

[0095] Among them, u represents the x-axis coordinate in the pixel coordinate system, v represents the y-axis coordinate in the pixel coordinate system, and z c Indicates the actual depth information of the point in the pixel coordinate system, f x Represents the transverse focal length component parameter, f y represents the longitudinal focal length component parameter, (u0, v0) represents the actual position of the principal point, R represents the rotation matrix between the laser radar and the camera, t represents the translation matrix between the laser radar and the camera, (X L ,Y L ,Z L ) represents the three-dimensional coordinates of the laser radar, and π is the distortion model of the industrial camera.

[0096] Furthermore, for a single projection point p i , use it as the query point to perform k-nearest neighbor search in tr1, and the obtained k-nearest neighbor point set is The formula is used to centralize the k nearest neighbor points by subtracting the mean. Finally, according to the principle that the eigenvector corresponding to the minimum eigenvalue after the singular value decomposition of the covariance matrix is the normal vector of the fitting line, the direction of the image edge contour is calculated; in the same way, the projection point in tr2 is used as the query point, and the nearest neighbor query is performed in the image edge contour KD tree tr2 and the nearest neighbor matching relationship is established. All points in tr2 are traversed, and the nearest neighbor matching relationship is established for those that meet the requirements; a single projection point is used as the query point to perform a k-nearest neighbor search in tr2, and the k nearest neighbor point sets obtained are solved using the above method to obtain the direction of the point cloud edge contour; the matches between the point cloud edge contour and the image edge contour with an angle between [-30°, 30°] or [150°, 270°] are retained and counted, and the statistical number is recorded as N.

[0097] S43, setting expression three, and using expression three to calculate the coordinates of the left and right whiskers of the longicorn beetle. The specific expression three is as follows:

[0098]

[0099] Where n represents the nth iteration, dir represents a random unit vector, represents the position coordinates of the left and right whiskers of the nth iteration, d0 represents the distance between the two whiskers of the longicorn, and x n Indicates the coordinates of the longhorn beetle in the nth iteration (the coordinates of the longhorn beetle will be initialized when the population is randomly created).

[0100] Further, such as Figure 2 and Figure 3 As shown, S42 may include:

[0101] S44, setting expression 4, and using expression 4 to calculate the fitness value of the position coordinates of the left and right whiskers of the longhorn beetle. The specific expression 4 is as follows:

[0102]

[0103] Where N represents the number of lines, The direction vector representing the edge contour of the anchor network in the pixel plane, Represents the direction vector of the anchor network’s point cloud edge contour projected onto the pixel plane, (x·y) represents the inner product of x and y. When both are unit vectors, (x·y) is the cosine value of vector x and vector y, are the linear coefficients of the edge contour of the anchor hole in the pixel coordinate system, The projection point of the 3D data onto the pixel plane.

[0104] S45, respectively obtain the extreme values of the population and the individual. The extreme value of the individual is Expression 5, and the extreme value of the population is Expression 6. The specific expression of Expression 5 is as follows:

[0105] i is the i-th longicorn;

[0106] Expression six is as follows:

[0107] P is the size of the population.

[0108] The optimal value of fitness is the optimal value of the population extreme value, that is, n is the number of iterations.

[0109] S46, judging whether the extreme value of the individual and the extreme value of the population are optimal values, if so, the program terminates; if not, proceeding to the next step;

[0110] S47, setting expressions 7 and 8 to update the speed and position of the group, using expressions 7 and 8 to make the leading particle jump out of the local optimal solution and approach the global optimal solution;

[0111] Expression seven is as follows:

[0112]

[0113] Among them, r1 and r2 are random numbers, x i (n) is the position of the i-th longicorn, and n is the number of iterations;

[0114] Expression eight is as follows:

[0115] x i (n+1)=x i (n)+v i (n+1)

[0116] When the update speed v i (n+1) exceeds V max , then take V max , similarly when the update speed v i (n+1) is less than V min , then take V min .

[0117] Furthermore, considering that using a large step size throughout the optimization process will make it difficult to find the most accurate global optimal solution, the present invention adopts a variable step size method, setting the initial value η to 1.0, if the current optimal fitness value transformation matrix g best Less than the optimal transformation matrix F best Parameter, indicating that the optimal step length is reasonable, otherwise, reduce the search step length, the new step length solution is: w = δw, the step length factor δ is Where m represents the number of times the global optimal transformation matrix is updated, and n represents the number of iterations under the current global optimal transformation matrix m;

[0118] S48, judging whether the maximum number of iterations has been reached, if not, then repeating S42-S47; if so, then exiting the program loop and outputting the optimal solution.

[0119] S5, accurately matching the first edge profile and the second edge profile, specifically: taking the measurement error into account, constructing a line-to-line ICP problem, and performing nonlinear least squares optimization to solve the optimal extrinsic parameters to obtain a calibration result.

[0120] That is, taking the sensor measurement error into account, the formula is the update iteration strategy, where SE(3) represents the special Euclidean group, se(3) represents the tangent space of the Euclidean space, that is, the Lie algebra, Represents the unit vector exist After adding δT to the tangent plane se(3) at , it is transformed back to SE(3) through the exponential mapping. The antisymmetric matrix representing the cross product of the mapping. The exact extrinsic parameter matrices R and T can be solved by updating and iterating through the nonlinear least squares method.

[0121] Specifically, S5 may include:

[0122] S51. Considering the measurement noise, the observation point of the lidar can be expressed as ( L P i + L w i ), the camera’s observation point can be expressed as (q i + I w i ), the edge points of the point cloud ( L P i + L w i ) is projected onto the image plane, and the image edge {q i , n i}Expression nine, expression nine is as follows:

[0123]

[0124] in, L w i represents the measurement noise of the lidar, I w i represents the measurement noise of the camera;

[0125] Expression nine is the constraint formula.

[0126] Among them, expression 9 is nonlinear. Expression 9 is linearized and expression 10 is obtained through first-order Taylor expansion. The specific expression 10 is as follows:

[0127]

[0128] Among them, r i is the initial value of the external parameter, is the corresponding formula approximation, is the derivative of the external parameter, is the derivative with respect to the noise.

[0129] S52. After optimization, expression 10 is obtained to obtain expression 11.

[0130] in,

[0131]

[0132] Substitute the matched corresponding points into expression 10, and the result is as follows:

[0133]

[0134] in, Σ=diag(Σ1,…,Σ N )

[0135] The error function obtained from formula (1) is:

[0136]

[0137] From the perspective of maximum likelihood estimation, the objective function is as follows:

[0138]

[0139] Formula (3) can be used to obtain the optimization direction δT by taking the partial derivative of δT and setting it to 0. * , that is, Expression 11: Expression 11 is as follows:

[0140]

[0141] S53. Update the current iteration value to obtain Expression 12. The specific expression 12 is as follows:

[0142]

[0143] Iterate expression 12 until convergence, and the result after convergence is the optimal external parameter.

[0144] Due to the poor lighting conditions and low visibility underground, large-scale illumination of the underground by an industrial camera equipped with a light source is more conducive to the recognition of the industrial camera; online calibration is carried out using the anchor net underground, avoiding the use of calibration objects that require human intervention or have special features, which is conducive to improving the accuracy of online calibration; the edge contour of the anchor net can be more effectively extracted by using the depth continuity feature and the reflection intensity mutation feature; the anchor net itself is characterized by being composed of multiple horizontal lines and multiple vertical lines that are parallel and intersecting with each other. During the matching process, it is easy to fall into the local optimal solution. The particle swarm search method of the longicorn beetle whiskers can prevent the local optimal solution, which can not only reduce the number of searches, but also ensure that it will not fall into the local optimal solution; in the precise matching process, the optimization direction is clarified, which can reduce the number of optimization times while improving the calibration accuracy.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An underground online joint calibration method, characterized in that: include: S1, moving the laser radar and the industrial camera so that the anchor net is placed in the field of view of the laser radar and the industrial camera, and obtaining the anchor net image data collected by the industrial camera and the anchor net point cloud data collected by the laser radar; S2, extracting a first edge contour of the anchor net from the anchor net image data using an image processing method; S3, extracting a second edge contour of the anchor net from the anchor net point cloud data using a point cloud processing method; S4, performing a rough matching of the first edge contour and the second edge contour based on a particle swarm search method of longicorn beetle whiskers; S5, performing precise matching on the first edge profile and the second edge profile, specifically: taking measurement errors into account, constructing a line-to-line ICP problem, and performing nonlinear least squares optimization to solve the optimal extrinsic parameters to obtain a calibration result.

2. The underground online joint calibration method according to claim 1, characterized in that: Before extracting the first edge contour of the anchor net from the anchor net image data using an image processing method, the method further includes: The Zhang Zhengyou calibration method is used to calibrate the internal parameters of the industrial camera.

3. The downhole online joint calibration method according to claim 1 or 2, characterized in that: When extracting the first edge contour of the anchor net from the anchor net image data using an image processing method according to the characteristics of the anchor net, S2 includes: S21, extracting the initial edge contour of the anchor network using the Canny algorithm according to the characteristics of the anchor network; S22, fitting the initial edge contour using a Hough transform algorithm to obtain a contour line of the anchor network; S23, remove the contour lines whose length is less than a preset threshold from the contour lines of the anchor network, and use the remaining contour lines as the first edge contour of the anchor network. The pixel coordinates of the remaining contour lines are put into the kd tree tr1, and the image edge, i.e., the remaining anchor network contour lines, are recorded as 4. The underground online joint calibration method according to claim 3, characterized in that: The step S3, when extracting the second edge contour of the anchor net from the anchor net point cloud data using a point cloud processing method, includes: S31, setting the side length of the voxel grid, and dividing the anchor point cloud data into M, N, and L parts along the X, Y, and Z coordinate axes according to the side length of the voxel grid. After dividing the anchor point cloud data into M*N*L voxel grids, plane fitting is performed on the anchor point cloud data in each voxel grid to obtain multiple fitting planes, the normal vector of each fitting plane is calculated, and after eliminating the intersection lines in the same fitting plane, dense interpolation is performed on the remaining intersection lines to obtain a depth-continuous edge point cloud; S32, projecting the reflection intensity of the anchor network point cloud data in each fitting plane along its normal vector direction to obtain a 2D intensity image corresponding to each fitting plane, extracting the reflection intensity mutation edge point cloud in the 2D intensity image using the Canny algorithm, and then obtaining the reflection intensity mutation edge point cloud in the 3D intensity image using inverse projection; S33: Determine the depth-continuous edge point cloud and the reflection intensity mutation edge point cloud as a second edge contour of the anchor network.

5. The downhole online joint calibration method according to claim 4, characterized in that: When performing a rough match between the first edge contour and the second edge contour in a particle swarm search method based on longicorn beetle whiskers, S4 includes: S41, pre-set the population size to P, that is, there are P longhorn beetles, and the maximum number of iterations is N max , the learning factors are c1 and c2, the inertia weight is w, and the maximum and minimum speeds are V max 、V min Finally, a search range is preset based on the positional relationship between the industrial camera and the lidar. The update speed v of each beetle in the population is randomly initialized within the preset search range. A population is randomly generated within the preset search range and the randomly generated population is initialized using the Tent chaotic sequence so that the parameters in R and t are distributed within the preset search range. The expression of the Tent chaotic sequence is recorded as Expression 1, and the specific expression 1 is as follows: Among them, Rand represents a random number, R and t generated by chaos initialization are x i =(T1, T2, T3, T4, T5, T6, T7), which means generating a particle; S42, after projecting the three-dimensional coordinates of the second edge contour onto the pixel coordinate system, remove the projection points that exceed the image size range of the industrial camera from the projected pixel coordinates, and store the remaining projection points in another KD tree tr2; use the projection points in tr2 as query points, perform a nearest neighbor query in the image edge contour KD tree tr1, filter out the projection points whose distances from the nearest neighbor points to the remaining projection points are within the threshold, and then establish a nearest neighbor matching relationship. Traverse all points in tr2 and establish a nearest neighbor matching relationship for all points that meet the requirements; wherein, set expression 2, and calculate the pixel coordinates of the three-dimensional coordinates of the point cloud edge projected onto the pixel coordinate system according to expression 2, and the specific expression 2 is as follows: Among them, u represents the x-axis coordinate in the pixel coordinate system, v represents the y-axis coordinate in the pixel coordinate system, and z c Indicates the actual depth information of the point in the pixel coordinate system, f x Represents the transverse focal length component parameter, f y represents the longitudinal focal length component parameter, (u0, v0) represents the actual position of the principal point, R represents the rotation matrix between the laser radar and the camera, t represents the translation matrix between the laser radar and the camera, (X L ,Y L ,Z L ) represents the three-dimensional coordinates of the laser radar, and π is the distortion model of the industrial camera; S43, setting expression three, and using expression three to calculate the coordinates of the left and right whiskers of the longicorn beetle, the expression three is specifically as follows: Where n represents the nth iteration, dir represents a random unit vector, represents the position coordinates of the left and right whiskers of the nth iteration, d0 represents the distance between the two whiskers of the longicorn, and x n Indicates the coordinates of the nth iteration of the longicorn; S44, setting an expression 4, and using the expression 4 to calculate the fitness value of the position coordinates of the left and right whiskers of the longhorn beetle. The expression 4 is specifically as follows: Where N represents the number of lines, The direction vector representing the edge contour of the anchor network in the pixel plane, Represents the direction vector of the anchor network’s point cloud edge contour projected onto the pixel plane, (x·y) represents the inner product of x and y. When both are unit vectors, (x·y) is the cosine value of vector x and vector y, are the linear coefficients of the edge contour of the anchor hole in the pixel coordinate system, The projection point of the three-dimensional data onto the pixel plane; S45, respectively obtain the extreme values of the population and the individual. The extreme value of the individual is expressed as Expression 5, and the extreme value of the population is expressed as Expression 6. The specific expression 5 is as follows: i is the i-th longicorn; The expression six is specifically as follows: P is the number of populations; S46, determining whether the extreme value of the individual and the extreme value of the population are optimal values, if so, the program terminates; if not, proceeding to the next step; S47, setting Expression 7 and Expression 8 to update the speed and position of the group, and using Expression 7 and Expression 8 to make the leading particle jump out of the local optimal solution and approach the global optimal solution; The expression seven is specifically as follows: Among them, r1 and r2 are random numbers, x i (n) is the position of the i-th longicorn, and n is the number of iterations; The expression eight is specifically as follows: x i (n+1)=x i (n)+v i (n+1) When the update speed v i (n+1) exceeds V max , then take V max , similarly when the update speed v i (n+1) is less than V min , then take V min ; S48, judging whether the maximum number of iterations has been reached, if not, then repeating S42-S47; if so, then exiting the program loop and outputting the optimal solution.

6. The underground online joint calibration method according to claim 5, characterized in that: S42 includes: For a single projection point p i , use it as the query point to perform k-nearest neighbor search in tr1, and the obtained k-nearest neighbor point set is The formula is used to centralize the k nearest neighbor points by subtracting the mean. Finally, according to the principle that the eigenvector corresponding to the minimum eigenvalue after the singular value decomposition of the covariance matrix is the normal vector of the fitting line, the direction of the image edge contour is calculated; in the same way, the projection point in tr2 is used as the query point, and the nearest neighbor query is performed in the image edge contour KD tree tr2 and the nearest neighbor matching relationship is established. All points in tr2 are traversed, and the nearest neighbor matching relationship is established for those that meet the requirements; a single projection point is used as the query point to perform a k-nearest neighbor search in tr2, and the k nearest neighbor point sets obtained are solved using the above method to obtain the direction of the point cloud edge contour; the matches between the point cloud edge contour and the image edge contour with an angle between [-30°, 30°] or [150°, 270°] are retained and counted, and the statistical number is recorded as N.

7. The downhole online joint calibration method according to claim 5, characterized in that: The S5 includes: S51, considering the measurement noise, project the edge points of the point cloud to the image plane, and the edge points of the image {q i , n i Expression 9, the expression 9 is specifically as follows: in, L w i represents the measurement noise of the lidar, I w i represents the measurement noise of the camera; The expression nine is a constraint formula; Wherein, the expression 9 is nonlinear. The expression 9 is linearized and the expression 10 is obtained by the first-order Taylor expansion. The specific expression 10 is as follows: Among them, r i is the initial value of the external parameter, is the corresponding formula approximation, is the derivative of the external parameter, is the derivative with respect to the noise; S52: After optimizing the expression 10, the expression 11 is obtained. The specific expression 11 is as follows: S53. Update the current iteration value to obtain Expression 12, which is as follows: The expression twelve is iterated until convergence, and the result after convergence is the optimal external parameter.

8. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 7.

Images