A calibration method, system, and storage medium for sensors of wide-body mining trucks

Abstract

The present invention discloses a calibration method, system, and storage medium for sensors of wide-body mining trucks, including S1: obtaining original point cloud data and image data of the sensor, removing outliers in the original point cloud data to obtain first-processed point cloud data; S2: establishing a voxel space based on the first-processed point cloud data, projecting the first-processed point cloud data into the voxel space to form voxel points; then extracting edge information from the image data to obtain an edge straight-line segment point cluster, and outputting natural features in combination with the voxel points; S3: establishing a pass-through filter to filter the first-processed point cloud data to obtain second-processed point cloud data, then fitting the second-processed point cloud data to obtain a point cloud plane, and finally downsampling the point cloud plane to obtain third-processed point cloud data; S4: establishing an external parameter correspondence relationship model based on the third-processed point cloud data and natural features, and solving it in the form of a weighted loss function to obtain sensor external parameter calibration parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor calibration, and in particular, discloses a calibration method, system, and storage medium for sensors of wide-body mining trucks. Background Art

[0002] Most current autonomous driving is based on the driving scheme of passenger cars, where lasers and CCD cameras are installed around the vehicle, while most sensors of mining trucks (trucks used for mining, simply referred to as mining trucks) are installed on top of the vehicle cab.

[0003] The calibration scheme of sensors usually adopts two methods: with a standard calibration object or without a calibration object. In the method with a standard calibration object, lasers and CCD cameras are generally used to provide data with standard features; in the method without a calibration object, only natural features of the environment, such as edge and plane feature information, can be used. The main working processes of the two calibration methods are: the sensors collect data respectively, use detection algorithms for the data, extract algorithms to implement the detection and extraction of features, and then use matching algorithms to implement the matching of feature information in laser data and feature information in image data.

[0004] In the prior art, in Paper 1 (Automatic Calibration for Non-repetitive Scanning Solid-State lidar and Camera Systens), a standard calibration object is used. The vehicle is a VGA car with an overall height of about 1 meter, and an aluminum alloy profile is used to build a platform to install lidar and CCD camera sensors. In Paper 2 (Pixel-level Extrinsic Self Calibration of High Resolution Lidar and Camera in Targetless Environments), a camera bracket is used to fix the lidar and CCD camera, with a height of about 1.5 meters.

[0005] The change in the installation height of the sensor will cause changes in the blind area and vertical field of view. When the sensor is installed higher, the features detected by the algorithm will become fewer, and the accuracy of the calibration result will become lower. The basic dimensions of a wide-body mining truck are length * width * height: 9 m * 5 m * 4.5 m, and the dimensions of a general passenger car are length * width * height: approximately 4.5 m * 1.9 m * 1.6 m. It can be seen that the height of a wide-body mining truck is more than twice that of a passenger car. Therefore, for a wide-body mining truck, the accuracy of the calibration result becomes lower with the calibration methods in the prior art. Summary of the Invention

[0006] Aiming at the problem of low calibration accuracy of sensors in wide-body mining trucks in the prior art, the present invention proposes a calibration method, system, and storage medium for sensors in wide-body mining trucks. By jointly establishing a constraint function using the characteristics of standard calibration objects and the edge features in the natural environment background, the constraint conditions are increased, and the calibration accuracy is improved.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A calibration method for sensors in wide-body mining trucks specifically includes the following steps:

[0009] S1: Obtain the original point cloud data and image data of the sensor, and remove the outliers in the original point cloud data to obtain the first processed point cloud data;

[0010] S2: Establish a voxel space according to the first processed point cloud data, and project the first processed point cloud data into the voxel space to form voxel points; then extract the edge information from the image data to obtain an edge straight-line segment point cluster, and output natural features in combination with the voxel points;

[0011] S3: Establish a pass-through filter to filter the first processed point cloud data to obtain the second processed point cloud data, then fit the second processed point cloud data to obtain a point cloud plane, and finally downsample the point cloud plane to obtain the third processed point cloud data;

[0012] S4: Establish an external parameter correspondence model according to the third processed point cloud data and natural features, and solve it in the form of a weighted loss function to obtain the sensor external parameter calibration parameters.

[0013] Preferably, the sensor includes a lidar and a CCD camera.

[0014] Preferably, in S1, the method for removing outliers in the original point cloud data is:

[0015] Construct each frame of the obtained point cloud data into a KD-tree structure, calculate the K-nearest neighbors of any point cloud Pi, the average distance disKi between neighbors, the mean distance meanK, the standard deviation stdK, and the empirical value beta, so as to calculate the threshold T = meanK + (±beta) * stdK; if the average distance disKi of any point cloud Pi > T, it means that the point cloud is an outlier, and the outlier is removed.

[0016] Preferably, S2 includes:

[0017] S2-1: Construct the boundaries Lx = |X_max - X_min|, Ly = |Y_max - Y_min|, Lz = |Z_max - Z_min|; X_max represents the maximum value of the X-axis, X_min represents the minimum value of the X-axis, Y_max represents the maximum value of the Y-axis, Y_min represents the minimum value of the Y-axis, Z_max represents the maximum value of the Z-axis, and Z_min represents the minimum value of the Z-axis;

[0018] S2-2: With (X_min, Y_min, Z_min) as the origin and the voxel resolution Rvoxel as the side length, divide along the XYZ axes to obtain the voxel space;

[0019] S2-3: Project any point cloud P (P x , P y , P z ) in the first processed point cloud data into the voxel space to obtain voxel points, and the coordinates of the voxel points are (q x + 1, q y + 1, q z + 1); Taking the X-axis coordinate as an example, the calculation formula is:

[0020] S2-4: In the voxel space, use the RANSAC method to perform linear fitting and plane fitting on any voxel point; If the angle between the line or plane after fitting between two adjacent voxel points is within 30 degrees to 150 degrees, then retain the two adjacent voxel points and the line segment and plane after their fitting;

[0021] S2-5: Extract the edge information of the CCD image data through the traditional Canny operator to obtain the edge straight line segment point clusters represents the jth point in the ith point cluster; Then establish a KD tree for the extracted edge straight line segment point clusters to calculate the mean value and eigenvalue q i represents the mean value of the ith edge straight line segment point cluster, N represents the total number of points, and T represents the transpose matrix;

[0022] S2-6: Convert the voxel points, line segments, and planes retained in S2-4 into the CCD image data, verify the orthogonality ni of the point cloud line segments converted into the image data and the edge straight line segments in S2-5, and output the non-orthogonal natural features (n i , q i ).

[0023] Preferably, the said S3 includes:

[0024] S3-1: Construct a straight-through rate filter Bound = [X_min, X_max, Y_min, Y_max, Z_min, Z_max] for cutting the first processed point cloud data; X_max represents the maximum value of the X-axis of the calibration plate area in the first point cloud data, X_min represents the minimum value of the X-axis of the calibration plate area, Y_max represents the maximum value of the Y-axis of the calibration plate area, Ymin represents the minimum value of the Y-axis of the calibration plate area, Z_max represents the maximum value of the Z-axis of the calibration plate area, and Z_min represents the minimum value of the Z-axis of the calibration plate area;

[0025] S3-2: Select the maximum value in the Z-axis direction of the filtered data as the upper bound Z of the calibration plate top ; Then calculate the lower bound Z according to the side length of the checkerboard down , and the point cloud data within the upper and lower bounds [Z top , Z down is the second processed point cloud data:

[0026]

[0027] In formula (1), L 2 represents the function for calculating the 2-norm; mean represents the function for calculating the mean; P xy represents the x-axis and y-axis coordinate values of the point; represents the distribution of the width H along the z-axis direction; ratio represents the threshold parameter; N h represents the number of corner points of the checkerboard in the height angle; G s represents the side length of the checkerboard;

[0028] S3-3: Use the RANSAC method to fit a plane in the Y-axis direction for the second processed point cloud data. If the distance between any point cloud P i (X i , Y i , Z i ) in the second processed point cloud data and the plane fitted in the Y-axis direction is greater than the threshold deta, remove the point; otherwise, keep it, and obtain the fitted plane P':

[0029]

[0030] In formula (2), A, B, C, and D represent the parameters of the spatial plane equation, and deta represents the threshold;

[0031] S3-4: Map the second processed point cloud data to the fitted plane P' to obtain the point cloud plane P I ,

[0032]

[0033] In formula (3), Tr represents the mapping ratio parameter;

[0034] S3-5: Divide the point cloud plane P I into multiple grids according to a square with side length ρ. In each grid, retain a maximum of seta point clouds per square centimeter to obtain a new point cloud plane. Subsequently, use voxel filtering in the PCL library to downsample the new point cloud plane to obtain the third processed point cloud data.

[0035] Preferably, the said S4 includes:

[0036] S4-1: The external calibration parameters of the sensor are The superscript C represents image data, and the subscript L represents lidar data. represents the external calibration parameters for converting image data to point cloud data. represents the rotation matrix from image data to point cloud data. represents the translation parameter from image data to point cloud data.

[0037] Construct a loss function using the conversion relationship between point cloud data and image data:

[0038]

[0039] In formula (4), l 0 represents the loss function. represents the transpose of n i , F represents the distortion parameter, and π represents the pinhole model parameter, i.e., internal calibration. represents the external parameters of the lidar. represents the coordinate of the lidar point cloud data. represents the pixel coordinate of the CDD camera.

[0040] S4-2: According to the result of the third processed point cloud data, construct a reflection intensity model S C :

[0041] S C ={(x, y, z, I)|x ∈ (O, N w G s ), y ∈ (O, N h G s ), z = 0, I = δ(x, y, z)}; (5)

[0042] In formula (5), the calibration plate size is {N w , N h , G s}, N w represents the width corner point of the calibration plate, N h represents the height corner point of the calibration plate, G srepresents the side length of the checkerboard; I represents the mean obtained after classifying the third processed point cloud data into two categories using the Gaussian mixture model;

[0043] S4-3: Establish the weighted constraint function l sum , and use the L-M solution method in the Ceres Solver library to obtain the external parameter:

[0044] l sum = αl 0 + β(l 1 + l 2 ) (6)

[0045] In formula (6), l 0 , l 1 , l 2 represent the loss function, and α and β are coefficients.

[0046] Preferably, the expressions of the said l 1 , l 2 are:

[0047]

[0048]

[0049] In formula (7), the point cloud data P L is transformed into the image data coordinate I s represents the intensity information of the point in the S C model, I P represents the intensity information corresponding to the P C point, is the reflection intensity model, G i represents the corner point closest to the point P C ; δ in represents {(x,y)∈P C |X min ≤ x ≤ X max , Y min ≤ y ≤ Y max ,(X,Y)∈S C}}, l 1 , l 2 represents the loss function, and the minimum distance function d(P C ,G i ) is d(P C ,G i ) = argminL 2 (P C ,G i ).

[0050] Preferably, in S2-4, the cosine theorem in space is used to determine the angle between lines, and the angle between spatial planes is determined by using the angle between the normal vectors of planes.

[0051] The present invention also provides a calibration system for sensors of wide-body mining trucks, including:

[0052] A data acquisition module, configured to acquire point cloud data of a lidar and image data of a CCD camera;

[0053] A preprocessing module, configured to preprocess the point cloud data of the lidar, including removing outliers, so as to obtain first processed point cloud data;

[0054] A first feature extraction module, configured to extract first features of the first processed point cloud data and the image data of the CCD camera;

[0055] A second feature extraction module, configured to construct a pass-through filter to filter and fit the first processed point cloud data, and output second features;

[0056] A calibration module, configured to construct a loss function according to the first features and the second features, and calibrate the external parameters of the lidar and the CCD camera.

[0057] Preferably, steps of the calibration method are performed when the computer program is executed by a processor.

[0058] In summary, due to the adoption of the above technical solutions, compared with the prior art, the present invention has at least the following beneficial effects:

[0059] By jointly establishing a constraint function using the features of a standard calibration object and the edge features in the natural environment background, the present invention increases the constraint conditions, and can preferably solve the problem that due to the change in the installation height of the sensor, the calibration object (the standard calibration object and the natural environment background) in the field of view becomes smaller, the features are compressed (reduced), and the constraints become fewer, thereby improving the calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the detection of a calibration object by a prior art sensor.

[0061] Figure 2 It is a schematic flowchart of a calibration method for sensors of wide-body mining trucks according to an exemplary embodiment of the present invention.

[0062] Figure 3 It is a schematic diagram of a calibration system for sensors of wide-body mining trucks according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described in detail below in conjunction with embodiments and specific implementation manners. However, it should not be understood that the scope of the above-mentioned subject matter of the present invention is limited to the following embodiments. Any technology implemented based on the content of the present invention belongs to the scope of the present invention.

[0064] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention.

[0065] As Figure 1 shown, the sensor 2 is respectively installed at the positions of 400 cm and 150 cm high on the wall 1. The vertical field of view angle of the sensor 2 at the 150 cm high position is 25.6 degrees. A standard calibration object 3 with a height of 200 cm can be detected at a longitudinal distance of about 645.55 cm, and it occupies about 70% of the vertical field of view angle of the sensor in the vertical direction. For the sensor 2 at the 400 cm high position, the standard calibration object 3 with a height of 200 cm can be completely detected at a longitudinal distance of about 1773.71 cm, and it occupies about 20% - 30% of the vertical field of view angle of the sensor in the vertical direction.

[0066] It can be seen that the higher the sensor height, the more serious the vertical compression of the standard calibration object in the data, and the same effect is brought horizontally, which increases the difficulty of feature detection and feature extraction. For the same algorithm and the same calibration object, when the sensor is installed higher, the features detected by the algorithm will be fewer, and the accuracy of the calibration result will be lower.

[0067] The basic dimensions of a wide-body mining truck are length * width * height: 9 m * 5 m * 4.5 m, and the dimensions of a general passenger car are length * width * height: approximately 4.5 m * 1.9 m * 1.6 m. It can be seen that the height of the wide-body mining truck is more than twice that of the passenger car, so it is not suitable for general calibration methods.

[0068] As Figure 2 shown, the present invention provides a calibration method for sensors of wide-body mining trucks, which adopts a scheme of combining natural features and artificial features in the point cloud data of lidar and the image data of CCD cameras to realize the external parameter calibration of the lidar and CCD cameras of the autonomous driving system based on wide-body mining trucks, so that the lidar point cloud data and the CCD camera image data overlap at the pixel level and form a corresponding relationship, that is, the coordinate systems correspond.

[0069] Specifically, it includes the following steps:

[0070] S1: Obtain the original point cloud data of each frame of the sensor, remove the outliers in the original point cloud data, and obtain the first processed point cloud data.

[0071] In the present invention, sensors (lidar and CCD camera) are installed on an adjustable angle platform, and then the platform is installed on the cab of a wide-body mining truck.

[0072] In this embodiment, the working mode of the lidar sensor is non-repetitive scanning. It is necessary to accumulate multiple frames of point clouds to obtain a point cloud set with a very high density. However, due to reasons such as the system error of the sensor, the collected point cloud contains error terms, such as outliers.

[0073] The method for removing outliers in the original point cloud data is as follows:

[0074] Construct each frame of the obtained data into a common KD-tree structure, calculate the K-nearest neighbors of any point cloud Pi, the average distance disKi between neighbors, the mean distance meanK, the standard deviation stdK, and an empirical value beta (obtained from multiple debugging data), so as to calculate the threshold T = meanK + (±beta) * stdK; if the average distance disKi of any point cloud Pi > T, it means that the point cloud is an outlier, and the outlier is removed; after the calculation of multiple frames of point clouds is completed and the outliers are removed, they are accumulated to obtain the first processed point cloud data.

[0075] S2: Read the coordinate information from the first processed point cloud data, establish a voxel space, and project the first processed point cloud data into the voxel space to form voxel points; then extract the edge information of the image data and output natural features in combination with the voxel points.

[0076] In this embodiment, the point cloud data in the first processed data all contain coordinates (coordinate detection and recording are carried out by the sensor itself);

[0077] S2-1: Construct the boundaries Lx = |X_max - X_min|, Ly = |Y_max - Y_min|, Lz = |Z_max - Z_min|; X_max represents the maximum value of the X-axis, X_min represents the minimum value of the X-axis, Y_max represents the maximum value of the Y-axis, Y_min represents the minimum value of the Y-axis, Z_max represents the maximum value of the Z-axis, and Z_min represents the minimum value of the Z-axis.

[0078] S2-2: Taking (X_min, Y_min, Z_min) as the origin and the voxel resolution Rvoxel as the side length, while maintaining the same coordinate direction as the original coordinate system, divide on the XYZ three axes to obtain the voxel space, and the voxel space is a subspace of the three-dimensional space of the original point cloud data.

[0079] S2-3: For any point cloud P in the first processed point cloud data (Px , P y , P z ) Projected into the voxel space, the coordinates of the corresponding point cloud (which can be marked as voxel points) are (q x + 1, q y + 1, q z + 1); taking the X-axis coordinate as an example, the calculation formula is: Then the X-axis coordinate is q x + 1; the Y-axis and Z-axis coordinates are calculated in the same way.

[0080] S2-4: In the voxel space, use the original RANSAC method to perform multiple straight-line fittings and plane fittings on any voxel point; if the angle between the straight line or plane after fitting between two adjacent voxel points is within 30 degrees to 150 degrees, then retain the indexes of the two adjacent voxel points and the straight line and plane after their fittings; if the angle is not within 30 degrees to 150 degrees, then do not retain the indexes.

[0081] In this embodiment, the cosine theorem in space is used to determine the angle between space straight lines; the angle between space planes can be determined by using the angle between plane normal vectors, and the intersection line of space planes can be determined by a system of ternary linear equations composed of two planes.

[0082] S2-5: Extract the edge information of the CCD image data through the traditional Canny operator to obtain a cluster of edge straight-line segments represents the j-th point in the i-th point cluster; then establish a KD tree for the extracted cluster of edge straight-line segments to calculate the mean value of the K nearest neighbors and eigenvalues q i represents the mean value of the i-th cluster of edge straight-line segments, N represents the total number of points, and T represents the transpose matrix.

[0083] S2-6: According to formula (1), convert the voxel points retained in S2-4 into the image data, verify the orthogonality ni between the point cloud straight-line segments converted into the image data and the edge straight-line segments in S2-5, and eliminate the orthogonal line segments through orthogonality, and output the non-orthogonal (n i , q i ) to form natural features.

[0084] S3: Establish a pass-through filter to filter the first processed point cloud data to obtain the second processed point cloud data, then fit the second processed point cloud data to obtain a point cloud plane, and finally downsample the point cloud plane to obtain the third processed point cloud data.

[0085] S3-1: Construct a straight-through rate filter Bound = [X_min, X_max, Y_min, Y_max, Z_min, Z_max] for cutting the first point cloud data; X_max represents the maximum value of the X-axis of the calibration plate area in the first point cloud data, X_min represents the minimum value of the X-axis of the calibration plate area, Y_max represents the maximum value of the Y-axis of the calibration plate area, Ymin represents the minimum value of the Y-axis of the calibration plate area, Z_max represents the maximum value of the Z-axis of the calibration plate area, and Z_min represents the minimum value of the Z-axis of the calibration plate area.

[0086] S3-2: Select the maximum value in the Z-axis direction of the filtered data as the upper bound Z of the calibration plate top ; then calculate the lower boundary Z according to the side length of the checkerboard down , and the point cloud data within the upper and lower boundaries [Z top , Z down is the second processed point cloud data, and the point cloud data not within it is excluded.

[0087]

[0088] In formula (2), L 2 represents the function for calculating the 2-norm; mean represents the function for calculating the mean; P xy represents the x-axis and y-axis coordinate values of the point; represents the distribution of the width H along the z-axis direction; ratio represents the threshold parameter; N h represents the number of points in the height angle of the checkerboard; G s represents the side length of the checkerboard.

[0089] S3-3: Use the RANSAC method to fit the plane in the Y-axis direction for the second processed point cloud data. For any point cloud P i (X i , Y i , Z i ) in the second processed point cloud data, if the distance from the plane is greater than the threshold deta, the point is excluded; otherwise, it is retained, and the fitted plane P' that best fits the real plane is obtained.

[0090]

[0091] In formula (3), A, B, C, and D represent the parameters of the spatial plane equation, and deta represents the preset distance. S3-4: Map the second processed point cloud data to the fitted plane P' to obtain a smoother point cloud plane P I ,

[0092]

[0093] In formula (4), Tr represents the mapping ratio parameter;

[0094] S3-5: Divide the point cloud plane P I into multiple grids according to squares with a side length of ρ (unit: m). In each grid, retain a maximum of seta point clouds per square centimeter to obtain a new point cloud plane. Subsequently, use voxel filtering in the PCL library to downsample the new point cloud plane to obtain a sparse point cloud, which is marked as the third processed point cloud data.

[0095] S4: Establish an external parameter correspondence model between the point cloud data and the image data, that is, a loss function; here, a weighted form of multiple loss functions is used to solve for the external parameter calibration parameters.

[0096] S4-1: In this embodiment, the external parameter calibration parameters of the sensor are The superscript C represents the image data, and the subscript L represents the lidar data. represents the external parameter calibration parameter for converting the image data to the point cloud data. represents the rotation matrix from the image data to the point cloud data. represents the translation parameter from the image data to the point cloud data.

[0097] Construct a corresponding model to convert the point cloud data coordinates to the image data coordinate system, that is, use the conversion relationship between the point cloud data and the image data, and the loss function L 0 is:

[0098]

[0099] In formula (5), l 0 represents the loss function. represents the transpose of n i . F represents the distortion parameter, and π represents the pinhole model parameter, that is, the internal parameter calibration, which can be obtained from the CCD camera data by the Zhang Zhengyou method; represents the external parameter of the lidar. represents the point cloud data coordinates of the lidar. represents the pixel coordinates of the CDD camera (mainly using the edge line segment information in the image data).

[0100] S4-2: According to the results of the third processed point cloud data, construct a new loss function;

[0101] First, construct a reflection intensity model S C :

[0102] S C ={(x,y,z,I)|x∈(0,N w G S ),y∈(0,N h G s), z = 0, I = δ(x, y, z)}; (6)

[0103] In formula (6), the calibration board size is {N w , N h , G s}, N w represents the width corner points of the calibration board, N h represents the height corner points of the calibration board, G s represents the side length of the checkerboard. I represents the binary value corresponding to the binary function δ, and the binary threshold is the mean value obtained by performing 2-class classification on the intensity information of the fourth processed point cloud using the Gaussian mixture model.

[0104] Without considering size scaling, the obtained point cloud and the standard model can be aligned:

[0105]

[0106]

[0107] In formula (7), the point cloud data P L is transformed into the image data coordinates I s represents the intensity information of the points in the S C model, I P represents the intensity information corresponding to the P C points, is the reflection intensity model, G i represents the corner point closest to the point P C ; δ in represents {(x, y) ∈ P C | X min ≤ x ≤ X max , Y min ≤ y ≤ Y max , (X, Y) ∈ S C}, l 1 , l 2 represents the loss function, d(P C , G i ) is the minimum distance function d(P C , G i ) = argminL 2 (P C , G i )

[0108] S4-3: Establish the weighted constraint function L sum , and an optimized external parameter can be obtained using the L-M solution method in the Ceres Solver library.

[0109] l sum = αl0 +β(l 1 +l 2 ) (7)

[0110] In formula (7), l 0 、l 1 、l 2 represent calibration values, and α and β are set coefficients.

[0111] The weighted constraint function l sum is a non-linear function, where the unknowns are the external parameters An optimized external parameter can be obtained by using the L-M solution method in the Ceres Solver library.

[0112] Based on the above method, as Figure 3 shown, the present invention also provides a calibration system for a wide-body mining truck sensor, including:

[0113] A data acquisition module for acquiring point cloud data of a lidar and image data of a CCD camera;

[0114] A preprocessing module for preprocessing the point cloud data of the lidar, including removing outlier points, so as to obtain first processed point cloud data;

[0115] A first feature extraction module for extracting first features of the first processed point cloud data and the image data of the CCD camera;

[0116] A second feature extraction module for constructing a pass-through filter to filter and fit the first processed point cloud data and output second features;

[0117] A calibration module for constructing a loss function according to the first feature and the second feature and calibrating the external parameters of the lidar and the CCD camera.

[0118] The present invention also provides a computer-readable storage medium, on which a computer program is stored. The computer program, when executed by a processor, implements the steps of the calibration method described in the above embodiments.

[0119] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present invention, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present invention.

Claims

1. A calibration method for wide-body mining truck sensors, It is characterized in that The specific steps include: S1: Acquire original point cloud data and image data from the sensor, remove outliers from the original point cloud data, and obtain first processed point cloud data; S2: establishing a voxel space according to the first processed point cloud data, and projecting the first processed point cloud data into the voxel space to form voxel points; then extracting edge information from the image data to obtain edge straight line segment point clusters, and outputting natural features in combination with the voxel points; S3: Establishing a straight-through filter to filter the first processed point cloud data to obtain second processed point cloud data, then fitting the second processed point cloud data to obtain a point cloud plane, and finally downsampling the point cloud plane to obtain third processed point cloud data; S4: Establish an external parameter correspondence model based on the third processed point cloud data and natural features, and use the loss function weighted form to solve it to obtain the sensor external parameter calibration parameters.

2. A calibration method for a wide-body mining truck sensor as claimed in claim 1, It is characterized in that The sensors include laser radar and CCD camera.

3. A calibration method for a wide-body mining truck sensor as claimed in claim 1, It is characterized in that In S1, the method for removing outliers in the original point cloud data is: Construct each frame of point cloud data obtained into a KD tree structure, calculate the K nearest neighbors of any point cloud Pi, the average distance disKi between neighbors, the mean distance meanK, the standard deviation stdK and the empirical value beta, and then calculate the threshold T = meanK + (±beta) * stdK; If the average distance disKi of any point cloud Pi>T, it means that the point cloud is an outlier and the outlier is removed.

4. A calibration method for a wide-body mining truck sensor as claimed in claim 1, It is characterized in that The S2 includes: S2-1: Construct the boundary Lx=|X_max-X_min|, Ly=|Y_max-Y_min|, Lz=|Z_max-Z_min|; X_max represents the maximum value of the X axis, X_min represents the minimum value of the X axis, Y_max represents the maximum value of the Y axis, Y_min represents the minimum value of the Y axis, Z_max represents the maximum value of the Z axis, and Z_min represents the minimum value of the Z axis; S2-2: Take (X_min, Y_min, Z_min) as the origin and the voxel resolution Rvoxel as the side length, divide it on the XYZ axis to get the voxel space; S2-3: Project any point cloud P (P x , P y , P z ) in the first processed point cloud data into the voxel space to obtain a voxel point, and the coordinates of the voxel point are (q x + 1, q y + 1, q z + 1); taking the X-axis coordinate as an example, the calculation formula is: S2-4: In the voxel space, use the RANSAC method to perform straight line fitting and plane fitting on any voxel point; if the angle of the fitted straight line or plane between two adjacent voxel points is within 30 degrees to 150 degrees, then the two adjacent voxel points and their fitted straight line segments and planes are retained; S2-5: Extract the edge information from the CCD image data through the traditional Canny operator to obtain the edge straight-line segment point clusters Denote the j-th point in the i-th point cluster; then build a KD tree for the extracted edge straight-line segment point clusters to calculate the mean of the K nearest neighbors and eigenvalues q i Denote the mean of the i-th edge straight-line segment point cluster, N denote the total number of points, and T denote the transpose matrix; S2-6: Convert the voxel points, line segments, and planes retained in S2-4 into CCD image data, verify the orthogonality ni of the point cloud line segments converted into image data and the edge line segments in S2-5, and output the non-orthogonal natural features (n i ,q i ).

5. A calibration method for a wide-body mining truck sensor as claimed in claim 1, It is characterized in that The S3 includes: S3-1: Construct a through filter Bound = [X_min, X_max, Y_min, Y_max, Z_min, Z_max] for cutting the first processed point cloud data; X_max represents the maximum value of the X-axis of the calibration board area in the first point cloud data, X_min represents the minimum value of the X-axis of the calibration board area, Y_max represents the maximum value of the Y-axis of the calibration board area, Ymin represents the minimum value of the Y-axis of the calibration board area, Z_max represents the maximum value of the Z-axis of the calibration board area, and Z_min represents the minimum value of the Z-axis of the calibration board area; S3-2: Select the maximum value in the Z-axis direction of the filtered data as the upper bound Z of the calibration board top ; Then calculate the lower boundary Z according to the side length of the checkerboard down , and the point cloud data within the upper and lower boundaries [Z top , Z down is the second processed point cloud data: In formula (1), L 2 represents the function for calculating the 2-norm; mean represents the function for calculating the mean; P xy represents the x-axis and y-axis coordinate values of a point; represents the distribution of the width H along the z-axis direction; ratio represents the threshold parameter; N h represents the number of points of the height angle of the checkerboard; G s represents the side length of the checkerboard; S3-3: Apply the RANSAC method to the second processed point cloud data to fit a plane in the Y-axis direction. If the distance between any point cloud P i (X i , Y i , Z i ) in the second processed point cloud data and the plane fitted in the Y-axis direction is greater than the threshold deta, remove this point; otherwise, keep it, and obtain the fitted plane P': In formula (2), A, B, C, and D represent the parameters of the spatial plane equation, and deta represents the threshold; S3-4: Map the second processed point cloud data onto the fitted plane P' to obtain the point cloud plane P I , In formula (3), Tr represents the mapping ratio parameter; S3-5: Divide the point cloud plane P I into multiple grids according to squares with side length ρ, and retain a maximum of seta point clouds per square centimeter in each grid to obtain a new point cloud plane. Subsequently, use voxel filtering in the PCL library to downsample the new point cloud plane to obtain the third processed point cloud data.

6. A calibration method for a wide-body mining truck sensor according to claim 1, characterized in that, the S4 includes: S4-1: The extrinsic calibration parameters of the sensor are The superscript C represents image data, and the subscript L represents lidar data. represents the extrinsic calibration parameters for transforming image data to point cloud data. represents the rotation matrix from image data to point cloud data. represents the translation parameter from image data to point cloud data; Construct a loss function using the conversion relationship between point cloud data and image data: In formula (4), l 0 represents the loss function, represents the transpose of n i , F represents the distortion parameter, and π represents the pinhole model parameter, i.e., the internal parameter calibration; represents the external parameter of the lidar, represents the coordinate of the point cloud data of the lidar, represents the pixel coordinate of the CDD camera; S4-2: Construct the reflection intensity model S based on the result of the third processed point cloud data C : S C = {(x, y, z, I) | x ∈ (0, N w G s ), y ∈ (0, N h G s ), z = 0, I = δ(x, y, z)}; (5) In formula (5), the calibration board size is {N w , N h , G s}, where N w represents the corner points of the width of the calibration board, N h represents the corner points of the height of the calibration board, and G s represents the side length of the checkerboard; I represents the mean value obtained after classifying the third processed point cloud data into two categories using the Gaussian mixture model; S4-3: Establish the weighted constraint function l sum , and use the L-M solution method in the Ceres Solver library to obtain the external parameter: l sum ＝αl 0 +β(l 1 +l 2 ) (6) In formula (6), l 0 , l 1 , l 2 represent the loss function, and α and β are coefficients.

7. A calibration method for a wide-body mining truck sensor according to claim 6, characterized in that, The said l 1 , l 2 The expression of is: The point cloud data P in formula (7) L is transformed into the image data coordinates I s represents S C The intensity information of the points in the model, I P represents P C The intensity information corresponding to the point is the reflection intensity model, G i represents the corner point closest to the point P C ; δ in represents {(x,y)∈P C |X min ≤x≤X max , Y min ≤y≤Y max ,(X,Y)∈S C}, l 1 , l 2 represents the loss function, d(P C ,G i ) The minimum distance function d(P C ,G i ) = arg min L 2 (P C ,G i ).

8. A calibration method for a wide-body mining truck sensor according to claim 4, characterized in that, In the S2-4, the cosine theorem of space is used to determine the included angle of the straight line, and the included angle of the spatial plane is determined by using the included angle of the plane normal.

9. A calibration system for a wide-body mining truck sensor, characterized in that, including: A data acquisition module for acquiring the point cloud data of the lidar and the image data of the CCD camera; A preprocessing module for preprocessing the point cloud data of the lidar, including removing outliers to obtain the first processed point cloud data; A first feature extraction module for extracting the first features of the first processed point cloud data and the image data of the CCD camera; A second feature extraction module for constructing a through filter to filter and fit the first processed point cloud data and output the second features; A calibration module for constructing a loss function according to the first feature and the second feature to calibrate the external parameters of the lidar and the CCD camera.

10. A computer-readable storage medium with a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the calibration method according to any one of claims 1 to 8.

Images