A laser radar external parameter calibration method for a robot and a related device

Abstract

The application belongs to the field of image processing, and discloses a laser radar external parameter calibration method for a robot and a related device, which comprises collecting camera pictures under a target environment and point view point clouds based on a non-repetitive scanning laser radar, then extracting point view checkerboard point clouds by using a checkerboard calibration object segmentation scheme based on a graph view, then obtaining checkerboard three-dimensional corner points of a laser radar system according to a prior three-dimensional corner point of a checkerboard model system, a coordinate transformation matrix of the checkerboard model system and a checkerboard coordinate system, and a coordinate transformation matrix of the checkerboard coordinate system and the laser radar system, and finally obtaining external parameters of a camera and a laser radar according to checkerboard two-dimensional corner points of the camera pictures and the checkerboard three-dimensional corner points of the laser radar system. By using the checkerboard calibration object segmentation scheme based on the graph view, the dependence on the environment is reduced, the scheme has universality and high stability, and the scheme does not need to adjust parameters, and has strong user friendliness.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing, and relates to a laser radar external parameter calibration method for a robot and a related device. BACKGROUND

[0002] Accurate laser radar external parameters and camera external parameters are prerequisites for multi-sensor fusion algorithms, and therefore a stable and effective laser radar external parameter calibration scheme needs to be provided. However, most existing laser radar external parameter calibration schemes are for mechanical laser radars with repetitive scanning, and are not suitable for rotating mirror laser radars with non-repetitive scanning. For non-repetitive scanning laser radar external parameter calibration, a calibration scheme with a calibration object suitable for non-repetitive scanning is commonly used at present.

[0003] However, the calibration scheme with a calibration object is not suitable for the case where there are other planar objects around, and cannot adapt to point cloud density, which is related to integral time of the point cloud, distance and direction of the object from the laser radar, and type of the laser radar. Based on the above analysis, it can be seen that the calibration scheme with a calibration object has poor generalization and stability, and is prone to false detection in chessboard partitioning. SUMMARY

[0004] The application aims to overcome the shortcomings of the existing calibration scheme with a calibration object, that is, poor generalization and stability and inability to stably partition the chessboard calibration object, and provides a laser radar external parameter calibration method for a robot and a related device.

[0005] To achieve the above-mentioned purpose, the application adopts the following technical scheme:

[0006] In a first aspect, the application provides a laser radar external parameter calibration method for a robot, comprising:

[0007] obtaining a camera picture of a target environment and a point view point cloud based on a non-repetitive scanning laser radar;

[0008] projecting the point view point cloud to a graph view to obtain a laser point intensity graph and a laser point depth graph, and generating an index hash table by taking an image pixel index as a key and a laser point index as a value during the projection;

[0009] extracting an edge graph of the laser point depth graph, extracting a plurality of contour point sets of the edge graph, mapping the plurality of contour point sets to the laser point depth graph one by one, obtaining a closed region surrounded by the plurality of contour point sets in the laser point depth graph, and performing a region enhancement operation on the closed region to obtain a plurality of candidate graphs;

[0010] extracting the checkerboard two-dimensional corner points of each candidate image, and obtaining the checkerboard point cloud of the point view according to the checkerboard two-dimensional corner points of each candidate image and the index hash table; projecting the checkerboard point cloud to a plane to obtain a checkerboard plane point cloud, and obtaining a coordinate transformation matrix of the checkerboard coordinate system and the laser radar system according to the checkerboard plane point cloud;

[0011] constructing a checkerboard model, and obtaining a coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system;

[0012] obtaining the checkerboard three-dimensional corner points of the laser radar system according to the three-dimensional corner points under the prior checkerboard model system, the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system, and the coordinate transformation matrix of the checkerboard coordinate system and the laser radar system;

[0013] extracting the checkerboard two-dimensional corner points of the camera picture, and obtaining the extrinsic parameters of the camera and the laser radar according to the checkerboard two-dimensional corner points of the camera picture and the checkerboard three-dimensional corner points of the laser radar system.

[0014] Optionally, the projecting the point view point cloud to the image view to obtain the laser point intensity image and the laser point depth image comprises:

[0015] calculating the pixel position of each laser point by using the heading angle, the pitch angle, the angle random error of the laser radar, and the horizontal and vertical field angles of the laser radar;

[0016] obtaining the laser point intensity image and the laser point depth image by using the intensity value and the depth value of each laser point in the point view point cloud as pixel values, combining the pixel position of each laser point, and filling the pixels with pixel values of 0 in the laser point intensity image and the laser point depth image by using the closing operation;

[0017] the region enhancement operation on the closed region comprises: setting the pixel values outside the closed region in the intensity image to 0.

[0018] Optionally, the extracting the edge image of the laser point depth image comprises:

[0019] performing two-dimensional convolution operation on the depth image by using a gradient convolution kernel to obtain an edge image, and then performing threshold segmentation on the edge image to obtain the edge image of the laser point depth image.

[0020] Optionally, the extracting the checkerboard two-dimensional corner points of each candidate image, and obtaining the checkerboard point cloud of the point view according to the checkerboard two-dimensional corner points of each candidate image and the index hash table comprises:

[0021] judging whether each candidate image has a checkerboard by using a two-dimensional checkerboard detection algorithm of an OpenCV algorithm library; and extracting the checkerboard two-dimensional corner points of each candidate image having a checkerboard;

[0022] Taking the two-dimensional corner points of each candidate image with the chessboard as input, a convex hull fitting algorithm is used to generate a convex hull of the two-dimensional corner points of each candidate image with the chessboard;

[0023] Taking the convex hull points in the convex hull of the two-dimensional corner points of each candidate image with the chessboard as input, a minimum area rectangle fitting algorithm is used to generate a rectangular region on each candidate image with the chessboard, and the rectangular region is extended outward by a δ pixel boundary to obtain a chessboard mask of each candidate image with the chessboard; wherein δ represents the pixel distance between two two-dimensional corner points of adjacent chessboards of each candidate image with the chessboard;

[0024] According to each chessboard mask and the index hash table, the image pixels on each chessboard mask are taken as the key to index out the chessboard laser point index of each candidate image with the chessboard, and the chessboard point cloud of the point view is obtained according to the chessboard laser point index of each candidate image with the chessboard.

[0025] Optionally, the projecting the chessboard point cloud to a plane to obtain a chessboard plane point cloud, and obtaining a coordinate transformation matrix of the chessboard coordinate system and the laser radar system according to the chessboard plane point cloud comprises:

[0026] The point cloud voxel downsampling is used on the chessboard point cloud to obtain a down-sampled chessboard point cloud;

[0027] The random sample consensus algorithm is used to process the down-sampled chessboard point cloud to obtain a denoised chessboard point cloud;

[0028] The least square method is used to fit a plane, and the denoised chessboard point cloud is orthogonally projected to the plane to obtain a chessboard plane point cloud;

[0029] The principal component analysis is used on the chessboard plane point cloud to obtain a coordinate transformation matrix of the chessboard coordinate system and the laser radar system.

[0030] Optionally, the obtaining the coordinate transformation matrix of the chessboard model system and the chessboard coordinate system comprises:

[0031] Taking the laser points of the chessboard plane point cloud of the chessboard coordinate system falling on the chessboard model and falling on the correct position of the chessboard model as more, the cost of the cost function is lower as the optimization target, and a cost function taking the coordinate transformation matrix of the chessboard model system and the chessboard coordinate system as the to-be-optimized parameters is constructed;

[0032] The L-BFGS-B optimization method is used to solve the cost function to obtain the coordinate transformation matrix of the chessboard model system and the chessboard coordinate system;

[0033] The grid of the chessboard grid model falling in the correct position indicates that when the intensity of the laser point is greater than a given intensity threshold, the laser point is considered to fall on the grid of the white chessboard grid model, otherwise the laser point is considered to fall on the grid of the black chessboard grid model.

[0034] The intensity threshold is obtained by: estimating the probability distribution function of the laser point intensity by using a kernel density estimation function; using a peak detection algorithm of a Scipy algorithm library on the probability distribution function of the laser point intensity to obtain two maximum peaks and average the corresponding intensity values to obtain the intensity threshold.

[0035] Optionally, the extrinsic parameters of the camera and the laser radar are obtained according to the chessboard two-dimensional corner points of the camera picture and the chessboard three-dimensional corner points of the laser radar system, and include:

[0036] The extrinsic parameters of the camera and the laser radar are obtained by using an SQPnP solver according to the chessboard two-dimensional corner points of the camera picture and the chessboard three-dimensional corner points of the laser radar system.

[0037] In the second aspect of the present application, a laser radar extrinsic parameter calibration system for a robot is provided, comprising:

[0038] The data acquisition module is configured to acquire a camera picture of a target environment and a point view point cloud based on a non-repetitive scanning laser radar.

[0039] The projection module is configured to project the point view point cloud to a graph view to obtain a laser point intensity graph and a laser point depth graph, and generate an index hash table by taking an image pixel index as a key and taking a laser point index as a value during the projection.

[0040] The graph segmentation module is configured to extract an edge graph of the laser point depth graph, extract a plurality of contour point sets of the edge graph, map the plurality of contour point sets to the laser point depth graph one by one, acquire a plurality of closed regions surrounded by the plurality of contour point sets in the laser point depth graph, and perform a region enhancement operation on the closed regions to obtain a plurality of candidate graphs.

[0041] The first transformation matrix acquisition module is configured to extract chessboard two-dimensional corner points of the plurality of candidate graphs, obtain a chessboard point cloud of the point view according to the chessboard two-dimensional corner points of the plurality of candidate graphs and the index hash table, project the chessboard point cloud to a plane to obtain a chessboard plane point cloud, and obtain a coordinate transformation matrix of a chessboard coordinate system and a laser radar system according to the chessboard plane point cloud.

[0042] The second transformation matrix acquisition module is configured to construct a chessboard grid model and obtain a coordinate transformation matrix of the chessboard grid model system and the chessboard coordinate system.

[0043] The coordinate system conversion module is configured to obtain the three-dimensional corner points of the checkerboard in the laser radar system according to the three-dimensional corner points of the checkerboard in the prior checkerboard model system, the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system, and the coordinate transformation matrix of the checkerboard coordinate system and the laser radar system.

[0044] The calibration module is configured to extract the two-dimensional corner points of the checkerboard in the camera picture, and obtain the extrinsic parameters of the camera and the laser radar according to the two-dimensional corner points of the checkerboard in the camera picture and the three-dimensional corner points of the checkerboard in the laser radar system.

[0045] In a third aspect, the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned method for calibrating the extrinsic parameters of the laser radar of the robot when executing the computer program.

[0046] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the steps of the above-mentioned method for calibrating the extrinsic parameters of the laser radar of the robot are implemented when the processor executes the computer program.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] The method for calibrating the extrinsic parameters of the laser radar of the robot according to the present application projects the point view point cloud to the graph view, obtains the laser point intensity graph and the laser point depth graph, extracts the edge graph of the laser point depth graph and each contour point set of the edge graph, maps each contour point set to the laser point depth graph in sequence, obtains the closed region surrounded by each contour point set in the laser point depth graph, and performs a region enhancement operation on the closed region to obtain each candidate graph, extracts the two-dimensional corner points of the checkerboard in each candidate graph, and obtains the checkerboard point cloud of the point view according to the two-dimensional corner points of the checkerboard in each candidate graph and the index hash table, thereby realizing the checkerboard calibration object segmentation scheme based on the graph view, reducing the dependence on the environment, and having universality and high stability without the need for adjusting parameters, and having strong user friendliness in actual application. Then, the checkerboard point cloud is projected to a plane to obtain the checkerboard plane point cloud, the coordinate transformation matrix of the checkerboard coordinate system and the laser radar system is obtained according to the checkerboard plane point cloud, the checkerboard model is constructed, and the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system is obtained; the three-dimensional corner points of the checkerboard in the laser radar system are obtained according to the three-dimensional corner points of the checkerboard in the prior checkerboard model system, the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system, and the coordinate transformation matrix of the checkerboard coordinate system and the laser radar system; finally, the two-dimensional corner points of the checkerboard in the camera picture are extracted, and the extrinsic parameters of the camera and the laser radar are obtained according to the two-dimensional corner points of the checkerboard in the camera picture and the three-dimensional corner points of the checkerboard in the laser radar system, so that the overall calibration scheme has high stability and high precision. Attached Figure Description

[0049] Figure 1 This is a flowchart of a lidar extrinsic parameter calibration method for robots according to an embodiment of the present invention.

[0050] Figure 2 This is a block diagram of a lidar extrinsic parameter calibration system for robots according to an embodiment of the present invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0053] The present invention will now be described in further detail with reference to the accompanying drawings:

[0054] See Figure 1 In one embodiment of the present invention, in view of the problem that the existing extrinsic parameter calibration scheme based on calibration objects cannot stably segment the checkerboard calibration object, a method for extrinsic parameter calibration of LiDAR for robots is provided, which can effectively improve the versatility and stability of extrinsic parameter calibration of non-repetitive scanning LiDAR and cameras.

[0055] Specifically, the method for calibrating the extrinsic parameters of a LiDAR used in robots includes the following steps:

[0056] S1: Acquire camera images of the target environment and point cloud views based on non-repeating scanning LiDAR.

[0057] S2: Project the point view point cloud to the map view to obtain a laser point intensity map and a laser point depth map; and generate an index hash table with image pixel indexes as keys and laser point indexes as values during the projection.

[0058] S3: Extract an edge map of the laser point depth map, and extract each contour point set of the edge map; and map each contour point set to the laser point depth map in sequence, obtain a closed region surrounded by each contour point set in the laser point depth map, and perform a region enhancement operation on the closed region to obtain each candidate map.

[0059] S4: Extract a checkerboard two-dimensional corner point of each candidate map, and obtain a checkerboard point cloud of the point view according to the checkerboard two-dimensional corner point of each candidate map and the index hash table; project the checkerboard point cloud to a plane to obtain a checkerboard plane point cloud, and obtain a coordinate transformation matrix of the checkerboard coordinate system and the laser radar system according to the checkerboard plane point cloud.

[0060] S5: Construct a checkerboard model, and obtain a coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system.

[0061] S6: Obtain a checkerboard three-dimensional corner point of the laser radar system according to the prior three-dimensional corner point under the checkerboard model system, the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system, and the coordinate transformation matrix of the checkerboard coordinate system and the laser radar system.

[0062] S7: Extract a checkerboard two-dimensional corner point of the camera picture, and obtain an extrinsic parameter of the camera and the laser radar according to the checkerboard two-dimensional corner point of the camera picture and the checkerboard three-dimensional corner point of the laser radar system.

[0063] In summary, the laser radar external parameter calibration method for the robot of the present application projects the point view point cloud to the graph view, obtains the laser point intensity graph and the laser point depth graph, then extracts the edge graph of the laser point depth graph, and extracts each contour point set of the edge graph, then maps each contour point set to the laser point depth graph in turn, and obtains the closed region surrounded by each contour point set in the laser point depth graph, and performs a region enhancement operation on the closed region to obtain each candidate graph, then extracts the checkerboard two-dimensional corner points of each candidate graph, and obtains the checkerboard point cloud of the point view according to the checkerboard two-dimensional corner points of each candidate graph and the index hash table, realizes the checkerboard calibration object segmentation scheme based on the graph view, reduces the dependence on the environment, has universality while showing high stability, and does not need to adjust the parameters, and has strong user friendliness in actual application. Then project the checkerboard point cloud to the plane to obtain the checkerboard plane point cloud, and obtain the coordinate transformation matrix of the checkerboard coordinate system and the laser radar system according to the checkerboard plane point cloud, then construct the checkerboard model, and obtain the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system; obtain the checkerboard three-dimensional corner points of the laser radar system according to the three-dimensional corner points under the prior checkerboard model system, the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system, and the coordinate transformation matrix of the checkerboard coordinate system and the laser radar system; finally extract the checkerboard two-dimensional corner points of the camera picture, and obtain the external parameters of the camera and the laser radar according to the checkerboard two-dimensional corner points of the camera picture and the checkerboard three-dimensional corner points of the laser radar system, the overall calibration scheme has strong stability and high precision.

[0064] In a possible implementation, the projecting the point view point cloud to the graph view to obtain the laser point intensity graph and the laser point depth graph comprises: calculating the pixel position of each laser point by using the heading angle, the pitch angle, the angle random error of the laser radar, and the horizontal and vertical field of view angle of the laser radar; obtaining the laser point intensity graph and the laser point depth graph by taking the intensity value and the depth value of each laser point in the point view point cloud as the pixel value and combining the pixel position of each laser point; and filling the pixels with pixel value 0 in the laser point intensity graph and the laser point depth graph by using the closing operation.

[0065] In a possible implementation, the performing the region enhancement operation on the closed region comprises: setting the pixel value outside the closed region in the intensity graph to 0.

[0066] In a possible implementation, the extracting the edge graph of the laser point depth graph comprises: performing a two-dimensional convolution operation on the depth graph by using a gradient convolution kernel to obtain an edge image, and then performing threshold segmentation on the edge image to obtain the edge graph of the laser point depth graph.

[0067] In a possible implementation, the method further includes: determining whether each candidate image has a checkerboard by using a two-dimensional checkerboard detection algorithm of an OpenCV algorithm library; extracting two-dimensional corner points of the checkerboard of each candidate image that has the checkerboard; using a convex hull fitting algorithm to generate a two-dimensional corner point convex hull of each candidate image that has the checkerboard, by using the two-dimensional corner points of the checkerboard of each candidate image that has the checkerboard as input; using a minimum area rectangle fitting algorithm to generate a rectangular region on each candidate image that has the checkerboard, by using the convex hull points in the two-dimensional corner point convex hull of each candidate image that has the checkerboard as input, and extending the rectangular region outward by a δ-pixel boundary to obtain a checkerboard mask of each candidate image that has the checkerboard; wherein δ represents a pixel distance between two two-dimensional corner points of the checkerboard of each candidate image that has the checkerboard; and indexing checkerboard laser point indexes of each candidate image that has the checkerboard, by using image pixels on each checkerboard mask as a key and the index hash table as input, and obtaining the checkerboard point cloud of the point view, by using the checkerboard laser point indexes of each candidate image that has the checkerboard.

[0068] In a possible implementation, the method further includes: obtaining a checkerboard plane point cloud by projecting the checkerboard point cloud to a plane, and obtaining a coordinate transformation matrix of a checkerboard coordinate system and a LiDAR system, by using the checkerboard plane point cloud.

[0069] In a possible implementation, the acquiring the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system includes: taking the cost function as an optimization objective, the more the laser points of the checkerboard plane point cloud in the checkerboard coordinate system fall on the grid of the checkerboard model and the grid of the correct position of the checkerboard model, the lower the cost of the cost function; constructing the cost function with the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system as the to-be-optimized parameters; using an L-BFGS-B optimization method to solve the cost function to obtain the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system; wherein the grid of the correct position of the checkerboard model indicates that when the intensity of the laser point is greater than a given intensity threshold, it is considered that the laser point should fall on the grid of the white checkerboard model, otherwise the laser point should fall on the grid of the black checkerboard model; wherein the intensity threshold is obtained by: estimating the probability distribution function of the laser point intensity by using a kernel density estimation function; using a peak detection algorithm of a Scipy algorithm library for the probability distribution function of the laser point intensity to obtain two maximum peaks and their corresponding intensity values and average them to obtain the intensity threshold.

[0070] wherein the L-BFGS-B optimization method (Limited-memory Broyden-Fletcher-Goldfarb-Shanno) is a limited-memory quasi-Newton method, which is an improved method of the L-BFGS method, and is used to process constrained optimization problems based on the implementation of the Scipy library.

[0071] In a possible implementation, the obtaining the extrinsic parameters of the camera and the lidar according to the checkerboard two-dimensional corner points of the camera picture and the checkerboard three-dimensional corner points of the lidar system includes: using an SQPnP solver to solve the extrinsic parameters of the camera and the lidar according to the checkerboard two-dimensional corner points of the camera picture and the checkerboard three-dimensional corner points of the lidar system.

[0072] In a possible implementation, the method for calibrating the extrinsic parameters of the lidar of the robot can be divided into the following steps:

[0073] The first step is to collect the camera picture of the target environment and the point view point cloud of the non-repetitive scanning lidar.

[0074] Specifically, the picture of the target environment is collected by using a camera, and the point cloud of the target environment is collected by using a non-repetitive scanning lidar, and the point cloud is stacked based on time sequence to obtain the point cloud after integration and in the point view, that is, the point view point cloud of the non-repetitive scanning lidar.

[0075] The second step is to divide the checkerboard calibration object based on the point view. Specifically, the following steps are included:

[0076] Step 21, project the laser points in the point view point cloud to the image view, with the intensity value and the depth value of the laser points as the pixel value, and calculate the pixel position of the laser points using the heading angle, the pitch angle, the angle random error of the lidar, the horizontal and vertical field of view angle of the lidar, to obtain the laser point intensity image and the laser point depth image. In addition, project the point view point cloud to the image view at the same time, and generate an index hash table with the image pixel index as the key and the laser point index as the value.

[0077] Step 22, fill the pixels with value 0 in the image view (including the laser point intensity image and the laser point depth image) using the closing operation, and then perform two-dimensional convolution operation on the laser point depth image using the gradient convolution kernel to obtain the edge image, and then perform threshold segmentation on the edge image to obtain the edge image of the laser point depth image.

[0078] Step 23, use the contour extraction algorithm to obtain a series of contour point sets from the edge image of the laser point depth image. Each contour point set corresponds to a closed region surrounded by the contour point elements in the set. Map each contour point set to the laser point depth image in turn, and obtain the closed region surrounded by each contour point set in the laser point depth image, and then perform a region enhancement operation on each closed region (i.e. set the pixel value outside the closed region in the laser point intensity image to 0) to obtain a series of candidate images after region enhancement.

[0079] Step 24, use the two-dimensional checkerboard detection algorithm of the OpenCV algorithm library on each candidate image to determine whether there is a checkerboard, and obtain the two-dimensional corner points of the checkerboard in each candidate image where the checkerboard exists. Then use the convex hull fitting algorithm to generate a two-dimensional corner point convex hull using the two-dimensional corner points as input, and then use the minimum area rectangle fitting algorithm to generate a rectangular region on each candidate image where the checkerboard exists using the convex hull points in the two-dimensional corner point convex hull as input, and then extend the rectangular region outward by δ pixel boundary, where δ represents the pixel distance between two adjacent two-dimensional corner points, to obtain a new region after inflation, called checkerboard mask. At the same time, based on the index hash table constructed above, use the image pixels on the checkerboard mask as the key to index out the laser point index of the checkerboard, and use this laser point index to extract the checkerboard point cloud.

[0080] Step 3: Extract the three-dimensional corner points of the checkerboard under the point cloud. Specifically, the following steps are included:

[0081] Step 31, post-processing of the checkerboard point cloud. Use point cloud voxel downsampling on the checkerboard point cloud to obtain the downsampled checkerboard point cloud; use the random sample consensus algorithm to process the downsampled checkerboard point cloud to obtain the denoised checkerboard point cloud; use the least squares method to fit a plane model, and orthogonally project the denoised checkerboard point cloud to the plane to obtain the checkerboard plane point cloud.

[0082] Step 32, using principal component analysis on the chessboard plane point cloud to obtain the coordinate transformation matrix between the chessboard coordinate system and the laser radar system.

[0083] Step 33, constructing a cost function with the coordinate transformation matrix between the chessboard model system and the chessboard coordinate system as the optimization parameters. When the laser points of the chessboard coordinate system chessboard plane point cloud fall on the chessboard model as much as possible and fall on the correct position of the chessboard model grid as much as possible, the cost of the cost function is lower. Then use L-BFGS-B optimization method to solve the cost function to obtain the coordinate transformation matrix between the chessboard model system and the chessboard coordinate system.

[0084] Wherein, the meaning of falling on the correct position of the chessboard model grid is that when the intensity of the laser point is greater than a given intensity threshold, it is considered that the laser point should fall on the white chessboard model grid, otherwise the laser point should fall on the black chessboard model grid.

[0085] Optionally, the intensity threshold is obtained using the following scheme: estimate the probability distribution function of the laser point intensity by kernel density estimation function; use the peak detection algorithm of Scipy algorithm library on the probability distribution function to find two maximum peaks and their corresponding intensity values; then take the mean of the two intensity values to obtain the intensity threshold.

[0086] In a possible implementation, the cost function Specifically:

[0087]

[0088] Wherein:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] Wherein, represents the laser point under the chessboard model system The loss value caused by falling on the incorrect chessboard model grid; the second term represents The loss value of not falling on the chessboard model.

[0095] Wherein, is used to judge Whether it falls on the correct grid of the checkerboard model. For judging Whether it is located on the checkerboard model.c i Is The color of the grid of the checkerboard model theoretically located on the basis of the intensity value of the estimated. The color of the grid of the checkerboard model actually located. Corresponding to The color of the grid of the checkerboard model actually located. Is the distance weight. Where, Δx1, Δx2 respectively represent The distance to the left and right two vertical boundary lines of its adjacent grid of the checkerboard model, and Δy1, Δy2 to the upper and lower two horizontal boundary lines of its adjacent grid of the checkerboard model.

[0096] Step 34, based on the prior checkerboard model system under the three-dimensional corner point, combined with the coordinate transformation matrix of the checkerboard coordinate system and the laser radar system obtained in step 32, that is, the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system obtained in step 33, the three-dimensional corner point of the checkerboard of the laser radar system is obtained through the coordinate system transformation.

[0097] Fourth step: based on two-dimensional and three-dimensional corner point pair, construct perspective n-point problem and solve the external parameter.

[0098] The checkerboard two-dimensional corner point of the camera picture and the checkerboard three-dimensional corner point of the laser radar system are input, the perspective n-point problem is constructed, and the SQPnP solver is used to solve the external parameter of the laser radar and the camera.

[0099] The laser radar external parameter calibration method for the robot of the application, compared with the existing calibration object for non-repetitive scanning laser radar and camera external parameter calibration scheme, the advantages of the application mainly lie in that the checkerboard calibration object segmentation scheme based on graph view is adopted, which ensures the stability and universality of calibration. At the same time, due to less dependence on the environment, it is basically not necessary to set adaptive adjustment parameters, and the stability of the algorithm is higher; on the other hand, it is not necessary to adjust the parameters, and it has strong user friendliness in actual application.

[0100] The following is an apparatus embodiment of the application, which can be used to execute the method embodiment of the application. For details not disclosed in the apparatus embodiment, please refer to the method embodiment of the application.

[0101] Referring to Figure 2In still another embodiment of the present application, a laser radar extrinsic parameter calibration system for a robot is provided, which can be used to implement the above-mentioned laser radar extrinsic parameter calibration method for a robot. Specifically, the laser radar extrinsic parameter calibration system for a robot comprises a data acquisition module, a projection module, a graph segmentation module, a first transformation matrix acquisition module, a second transformation matrix acquisition module, a coordinate system conversion module, and a calibration module.

[0102] The data acquisition module is configured to acquire a camera picture of a target environment and a point view point cloud based on a non-repetitive scanning laser radar. The projection module is configured to project the point view point cloud to a graph view to obtain a laser point intensity graph and a laser point depth graph, and generate an index hash table by taking an image pixel index as a key and a laser point index as a value during the projection. The graph segmentation module is configured to extract an edge graph of the laser point depth graph, extract a plurality of contour point sets of the edge graph, sequentially map the plurality of contour point sets to the laser point depth graph, acquire a closed region surrounded by the plurality of contour point sets in the laser point depth graph, and perform a region enhancement operation on the closed region to obtain a plurality of candidate graphs. The first transformation matrix acquisition module is configured to extract a checkerboard two-dimensional corner point of each candidate graph, and obtain a checkerboard point cloud of the point view according to the checkerboard two-dimensional corner point of each candidate graph and the index hash table. The checkerboard point cloud is projected to a plane to obtain a checkerboard plane point cloud, and a coordinate transformation matrix between a checkerboard coordinate system and a laser radar system is obtained according to the checkerboard plane point cloud. The second transformation matrix acquisition module is configured to construct a checkerboard model, and acquire a coordinate transformation matrix between a checkerboard model system and the checkerboard coordinate system. The coordinate system conversion module is configured to obtain a checkerboard three-dimensional corner point of the laser radar system according to a three-dimensional corner point in the checkerboard model system, the coordinate transformation matrix between the checkerboard model system and the checkerboard coordinate system, and the coordinate transformation matrix between the checkerboard coordinate system and the laser radar system. The calibration module is configured to extract a checkerboard two-dimensional corner point of the camera picture, and obtain an extrinsic parameter between the camera and the laser radar according to the checkerboard two-dimensional corner point of the camera picture and the checkerboard three-dimensional corner point of the laser radar system.

[0103] The foregoing embodiments of the laser radar extrinsic parameter calibration method for a robot involve all related contents of the steps, which can be cited to the functional description of the corresponding functional modules of the laser radar extrinsic parameter calibration system for a robot in the embodiments of the present application, and will not be described here in detail.

[0104] The division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, another division mode can be used. In addition, each functional module in each embodiment of the present application can be integrated in one processor, or can be physically separated, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0105] In another embodiment of the present application, a computer device is provided, which comprises a processor and a memory, the memory is configured to store a computer program, the computer program comprises program instructions, and the processor is configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are particularly suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method process or a corresponding function; the processor in the embodiments of the present application can be used for the operation of the laser radar external parameter calibration method of the robot.

[0106] In another embodiment of the present application, the present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a computer device, and is configured to store programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the computer device, and of course can also include an expansion storage medium supported by the computer device. The computer readable storage medium provides a storage space, and the storage space stores an operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the laser radar external parameter calibration method for the robot in the above embodiments.

[0107] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0108] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0109] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0111] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing the technical solutions of the present application, but not for limiting it. Although the present application is described in detail with reference to the above embodiments, those skilled in the field should understand that the specific embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A method for calibrating extrinsic parameters of a laser radar for a robot, characterized in that, The method comprises the following steps: acquiring a camera picture of a target environment and a point-view point cloud based on a non-repetitive scanning laser radar; projecting the point-view point cloud to a graph view to obtain a laser point intensity graph and a laser point depth graph; generating an index hash table using the image pixel index as the key and the laser point index as the value during the projection; extracting an edge graph of the laser point depth graph and extracting a plurality of contour point sets of the edge graph; mapping the plurality of contour point sets to the laser point depth graph one by one, acquiring a closed region surrounded by the plurality of contour point sets in the laser point depth graph, and performing a region enhancement operation on the closed region to obtain a plurality of candidate graphs; extracting a checkerboard two-dimensional corner point of each candidate graph, and obtaining a checkerboard point cloud of the point view according to the checkerboard two-dimensional corner point of each candidate graph and the index hash table; projecting the checkerboard point cloud to a plane to obtain a checkerboard plane point cloud, and obtaining a coordinate transformation matrix of a checkerboard coordinate system and a laser radar system according to the checkerboard plane point cloud; constructing a checkerboard model and acquiring a coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system; obtaining a checkerboard three-dimensional corner point of the laser radar system according to the prior three-dimensional corner point under the checkerboard model system, the coordinate transformation matrix of the checkerboard model system and the checkerboard coordinate system, and the coordinate transformation matrix of the checkerboard coordinate system and the laser radar system; extracting a checkerboard two-dimensional corner point of the camera picture, and obtaining an extrinsic parameter of the camera and the laser radar according to the checkerboard two-dimensional corner point of the camera picture and the checkerboard three-dimensional corner point of the laser radar system. 2.The method for calibrating extrinsic parameters of lidar of a robot according to claim 1, characterized in that, The method of projecting the point-view point cloud to the graph view to obtain the laser point intensity graph and the laser point depth graph comprises the following steps: calculating the pixel position of each laser point by using the heading angle, the pitch angle, the angle random error of the laser radar, and the horizontal and vertical field angles of the laser radar; obtaining the laser point intensity graph and the laser point depth graph by using the intensity value and the depth value of each laser point in the point-view point cloud as the pixel value and combining the pixel position of each laser point; and filling the pixels with a pixel value of 0 in the laser point intensity graph and the laser point depth graph by using a closing operation; the region enhancement operation on the closed region comprises: setting the pixel value outside the closed region in the intensity graph to 0. 3.The method for calibrating extrinsic parameters of lidar for robots according to claim 1, characterized in that, The method of extracting the edge graph of the laser point depth graph comprises the following steps: performing a two-dimensional convolution operation on the depth graph by using a gradient convolution kernel to obtain an edge image, and then performing threshold segmentation on the edge image to obtain the edge graph of the laser point depth graph. 4.The method for calibrating extrinsic parameters of lidar for robots according to claim 1, wherein, The method of extracting the checkerboard two-dimensional corner point of each candidate graph and obtaining the checkerboard point cloud of the point view according to the checkerboard two-dimensional corner point of each candidate graph and the index hash table comprises the following steps: judging whether each candidate graph has a checkerboard by using a two-dimensional checkerboard detection algorithm of an OpenCV algorithm library; and extracting the checkerboard two-dimensional corner point of each candidate graph having a checkerboard; using a convex hull fitting algorithm to generate a two-dimensional corner point convex hull of each candidate graph having a checkerboard by taking the two-dimensional corner point of each candidate graph having a checkerboard as the input. The convex points in the two-dimensional corner convex hull of each candidate image with a chessboard are taken as input, a rectangular region is generated on each candidate image with a chessboard using a minimum area rectangle fitting algorithm, and the rectangular region is extended by δ pixel boundaries to obtain a chessboard mask of each candidate image with a chessboard; wherein δ represents the pixel distance between two two-dimensional corner points of adjacent chessboards of each candidate image with a chessboard; According to each chessboard mask and the index hash table, the image pixels on each chessboard mask are taken as keys to index out the chessboard laser point index of each candidate image with a chessboard, and the chessboard point cloud of the point view is obtained according to the chessboard laser point index of each candidate image with a chessboard. 5.The method for calibrating extrinsic parameters of lidar for robots according to claim 1, wherein, The chessboard point cloud is projected onto a plane to obtain a chessboard plane point cloud, and the coordinate transformation matrix of the chessboard coordinate system and the laser radar system is obtained according to the chessboard plane point cloud. Point cloud voxel downsampling is used on the chessboard point cloud to obtain a down-sampled chessboard point cloud; A random sample consensus algorithm is used to process the down-sampled chessboard point cloud to obtain a noise-reduced chessboard point cloud; A least squares method is used to fit a plane, and the noise-reduced chessboard point cloud is orthogonally projected onto the plane to obtain a chessboard plane point cloud; Principal component analysis is used on the chessboard plane point cloud to obtain the coordinate transformation matrix of the chessboard coordinate system and the laser radar system. 6.The method for calibrating extrinsic parameters of lidar for robots according to claim 1, wherein, The coordinate transformation matrix of the chessboard model system and the chessboard coordinate system includes: The more the laser points of the chessboard plane point cloud of the chessboard coordinate system fall on the grid of the chessboard model and the correct position of the chessboard model, the lower the cost of the cost function, which is the optimization goal, and a cost function with the coordinate transformation matrix of the chessboard model system and the chessboard coordinate system as the to-be-optimized parameters is constructed; The L-BFGS-B optimization method is used to solve the cost function to obtain the coordinate transformation matrix of the chessboard model system and the chessboard coordinate system; Wherein, the grid of the chessboard model in the correct position means that when the intensity of the laser point is greater than a given intensity threshold, it is considered that the laser point should fall on the grid of the white chessboard model, otherwise the laser point should fall on the grid of the black chessboard model; Wherein, the intensity threshold is obtained by: estimating the probability distribution function of the laser point intensity by a kernel density estimation function; using the peak detection algorithm of the Scipy algorithm library on the probability distribution function of the laser point intensity to obtain two maximum peaks and their corresponding intensity values and average them to obtain the intensity threshold. 7.The method for calibrating extrinsic parameters of lidar for robots according to claim 1, wherein, The extrinsic parameters of the camera and the laser radar are obtained according to the chessboard two-dimensional corner points of the camera picture and the chessboard three-dimensional corner points of the laser radar system, including: The extrinsic parameters of the camera and the laser radar are obtained by using the SQPnP solver according to the chessboard two-dimensional corner points of the camera picture and the chessboard three-dimensional corner points of the laser radar system. 8.A laser radar extrinsic parameter calibration system for a robot, characterized by, It includes: A data acquisition module for acquiring camera pictures of a target environment and point view point clouds based on a non-repetitive scanning laser radar; A projection module for projecting the point view point cloud onto the image view to obtain a laser point intensity map and a laser point depth map; And an index hash table is generated by taking image pixel indexes as keys and laser point indexes as values during projection; The image segmentation module is configured to extract an edge map of the laser point depth map, extract a plurality of contour point sets of the edge map, and sequentially map the contour point sets to the laser point depth map, obtain a closed region surrounded by the contour point sets in the laser point depth map, and perform a region enhancement operation on the closed region to obtain a plurality of candidate graphs. The first transformation matrix obtaining module is configured to extract a checkerboard two-dimensional corner point of each candidate graph, obtain a checkerboard point cloud of the point view according to the checkerboard two-dimensional corner point of each candidate graph and the index hash table, project the checkerboard point cloud to a plane to obtain a checkerboard plane point cloud, and obtain a coordinate transformation matrix between a checkerboard coordinate system and a laser radar system according to the checkerboard plane point cloud. The second transformation matrix obtaining module is configured to construct a checkerboard model and obtain a coordinate transformation matrix between the checkerboard model system and the checkerboard coordinate system. The coordinate system conversion module is configured to obtain a checkerboard three-dimensional corner point of the laser radar system according to a three-dimensional corner point in the prior checkerboard model system, the coordinate transformation matrix between the checkerboard model system and the checkerboard coordinate system, and the coordinate transformation matrix between the checkerboard coordinate system and the laser radar system. The calibration module is configured to extract a checkerboard two-dimensional corner point of a camera picture, and obtain an extrinsic parameter of the camera and the laser radar according to the checkerboard two-dimensional corner point of the camera picture and the checkerboard three-dimensional corner point of the laser radar system.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method for calibrating an extrinsic parameter of a laser radar of a robot according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the method for calibrating an extrinsic parameter of a laser radar of a robot according to any one of claims 1 to 7.

Images