Monocular laser radar and camera extrinsic parameter calibration method and storage medium

Abstract

The application relates to a single-line laser radar and camera external parameter calibration method and a storage medium, and belongs to the technical field of computers.The method comprises the following steps: pre-calibrating internal parameters of a camera; starting an encoder and a single-line laser radar, and controlling a motor to rotate; acquiring a rotation angle of a rotating body collected by the encoder and a plurality of frames of point cloud data of a calibration board collected by the single-line laser radar at each rotation angle; converting the point cloud data to a first coordinate system according to the rotation angle corresponding to each frame of point cloud data, so as to obtain dense point cloud data of the calibration board in the first coordinate system; controlling the camera to collect image data of the calibration board in a second coordinate system; matching the dense point cloud data and the image data, so as to obtain an external parameter calibration result of the first coordinate system and the second coordinate system; the method can solve the problem that a traditional calibration method cannot realize external parameter calibration between a single-line laser radar and a camera; and the external parameter calibration between the single-line laser radar and the camera is realized.

Description

Technical Field

[0001] This application relates to a method and storage medium for calibrating the extrinsic parameters of a single-line lidar and camera, belonging to the field of computer technology. Background Technology

[0002] With the development of artificial intelligence, multiple sensors are now commonly used to achieve recognition functions. For example, point cloud data collected by LiDAR and image data collected by a camera are combined to perform recognition tasks. However, data collected by different sensors are usually obtained in different coordinate systems. Therefore, when fusing data from different sensors, it is necessary to pre-calibrate the transformation relationship between the coordinate systems corresponding to the different sensors.

[0003] Traditional calibration methods between LiDAR and cameras include using multi-line LiDAR for extrinsic parameter calibration. For example, the invention patent application No. 202110607494.X, "A Calibration Method for LiDAR and Camera," discloses the following: The LiDAR acquires N frames of point cloud data including a calibration board, and records the coordinates of each point in each frame of point cloud data and its corresponding ray ID; from the point cloud data pc_final, the edge points of the calibration board are found; the found edge points of the calibration board are transformed into the pseudo-camera coordinate system and projected onto the imaging plane of the pseudo-camera to obtain a projected image; from the projected image, the 2D coordinates of the corner points of the calibration board in the pseudo-camera coordinate system are found; based on the multiple sets of 2D coordinates of the corner points of the calibration board in the pseudo-camera coordinate system and their corresponding 3D coordinates in the camera coordinate system, the transformation matrix between the camera coordinate system and the pseudo-camera coordinate system is obtained.

[0004] As described above, the LiDAR acquires point cloud data corresponding to multiple ray IDs, which is necessary for calibration between a multi-line LiDAR and the camera. However, since a single-line LiDAR only has one line per frame of point cloud data, it cannot find the edge points of the calibration board using the same method as a multi-line LiDAR. Therefore, the above method cannot be applied to calibration scenarios between a single-line LiDAR and the camera, and cannot achieve extrinsic parameter calibration between them. Summary of the Invention

[0005] This application provides a method and storage medium for extrinsic parameter calibration between a single-line lidar and a camera, which can realize extrinsic parameter calibration between the single-line lidar and the camera. This application provides the following technical solution:

[0006] On one hand, a method for calibrating the extrinsic parameters of a single-line lidar and a camera is provided. The calibration environment of the single-line lidar and the camera includes a calibration plate and a rotating structure. The rotating structure includes a fixed-end body, a motor mounted on the fixed-end body, an encoder connected to the drive shaft of the motor, and a rotating body. The single-line lidar is mounted on the rotating body. The encoder is used to detect the rotation angle of the drive shaft. The method includes:

[0007] The camera's intrinsic parameters are pre-calibrated;

[0008] The encoder and single-line lidar are activated to control the motor rotation;

[0009] The rotation angle of the rotating body collected by the encoder, and the multi-frame point cloud data of the calibration board collected by the single-line lidar at each rotation angle are obtained.

[0010] According to the rotation angle corresponding to each frame of point cloud data, the point cloud data is transformed to the first coordinate system to obtain the dense point cloud data in the first coordinate system corresponding to the calibration board; the first coordinate system is established based on the position of the single-line lidar when it is used.

[0011] The camera is controlled to acquire images of the calibration board, obtaining image data of the calibration board in a second coordinate system; the second coordinate system is established based on the position of the camera.

[0012] The dense point cloud data is matched with the image data to obtain the extrinsic parameter calibration results of the first coordinate system and the second coordinate system.

[0013] Optionally, the pre-calibration of the camera's intrinsic parameters includes:

[0014] The intrinsic parameters of the camera are calibrated using the calibration methods provided by OpenCV to obtain the model parameters of the camera.

[0015] Optionally, the position of the single-line lidar during use is the position when the rotation angle is 0, and the first coordinate system is established based on this position; the step of transforming the point cloud data to the first coordinate system according to the rotation angle corresponding to each frame of point cloud data to obtain the dense point cloud data in the first coordinate system corresponding to the calibration board includes:

[0016] For point cloud data in each frame where the rotation angle is not 0, the rotation angle of the single-line radar is determined by the data collected by the encoder.

[0017] The representation of the rotation vector in the first coordinate system is determined based on the installation position of the single-line lidar on the rotating structure;

[0018] Based on the rotation vector and the rotation angle, the transformation relationship between the rotating single-line lidar coordinate system and the first coordinate system is determined;

[0019] Based on the transformation relationship, each frame of point cloud data collected by the rotating single-line lidar is transformed to the first coordinate system and superimposed to obtain the dense point cloud of the calibration plate in the first coordinate system.

[0020] Optionally, the calibration board is rectangular, and the calibration board includes black cells and white cells spaced apart; the step of matching the dense point cloud data with the image data to obtain the extrinsic parameter calibration results of the first coordinate system and the second coordinate system includes:

[0021] Edge extraction is performed on the dense point cloud data to obtain the edge contours of the dense point cloud data;

[0022] The edge contour is estimated by line estimation to obtain the edge lines in the four first coordinate systems and the edge corner points in the four first coordinate systems formed by the intersection of each pair of adjacent edge lines.

[0023] The intensity information of the dense point cloud data is used to determine the corner points of the m*n cells in the first coordinate system of the black cells and the white cells;

[0024] The edges of the calibration plate in the image data are identified to obtain four edge lines in the second coordinate system and four edge corner points in the second coordinate system;

[0025] Identify the black and white cells in the image data to obtain m*n cell corner points in the second coordinate system;

[0026] Match the edge lines in the first coordinate system with the edge lines in the second coordinate system, match the edge corner points in the first coordinate system with the edge corner points in the second coordinate system, match the cell corner points in the first coordinate system with the cell corner points in the second coordinate system, and combine these with camera intrinsic parameters to form a PnP (Perspective-n-Point) problem.

[0027] Solve the PnP problem to obtain the extrinsic parameter calibration results.

[0028] Optionally, the step of performing edge extraction on the dense point cloud data to obtain the edge contours of the dense point cloud data includes:

[0029] The edge contours are extracted using the scatter contour alpha shapes algorithm.

[0030] Optionally, determining the corner points of the m*n cells in the first coordinate system using the intensity information of the dense point cloud data includes:

[0031] Perform planar estimation on the dense point cloud data;

[0032] Project all points in the dense point cloud data onto the estimated plane;

[0033] Retrieve intensity information of other points within a certain range around each point;

[0034] If the intensity information conforms to the preset rules, the point is determined to be a cell corner point.

[0035] Optionally, identifying the edges of the calibration plate in the image data to obtain four edge lines in the second coordinate system and four edge corner points in the second coordinate system includes:

[0036] The corner response algorithm is used to obtain all cell corners in the second coordinate system of the image data;

[0037] Based on all cell corner points in the second coordinate system, and according to the pre-measured width of the blank area at the edge of the calibration plate and the width of the square cell, the edge lines and edge corner points in the second coordinate system are calculated.

[0038] Optionally, the start encoder and single-line lidar include:

[0039] The single-line lidar is controlled to synchronously acquire data with the encoder to obtain point cloud data for each frame and the corresponding rotation angle for each frame of point cloud data.

[0040] Optionally, the motor rotates at a preset speed at a constant speed, and the method further includes:

[0041] For each frame of point cloud data, the timestamp of each rotation angle is compared with the timestamp of the point cloud data to obtain the difference in timestamps;

[0042] The angle difference is determined based on the minimum difference in timestamps and the preset speed;

[0043] The rotation angle corresponding to the point cloud data is determined based on the rotation angle corresponding to the minimum value of the timestamp difference and the angle difference.

[0044] Optionally, the motor rotates at a preset speed at a constant speed, and the method further includes:

[0045] For each frame of point cloud data, the timestamp of each rotation angle is compared with the timestamp of the point cloud data to obtain the difference in timestamps;

[0046] The angle difference is determined based on the minimum difference in timestamps and the preset speed;

[0047] The rotation angle corresponding to the point cloud data is determined based on the rotation angle corresponding to the minimum value of the timestamp difference and the angle difference.

[0048] On the other hand, a computer-readable storage medium is provided, wherein a program is stored in the storage medium, and the program, when executed by a processor, is used to implement the single-line lidar and camera extrinsic parameter calibration method described in the first aspect.

[0049] The beneficial effects of this application include at least the following: pre-calibrating the intrinsic parameters of the camera; starting the encoder and single-line LiDAR, controlling the motor to rotate; acquiring the rotation angle of the rotating body collected by the encoder, and multi-frame point cloud data of the calibration board collected by the single-line LiDAR at each rotation angle; converting the point cloud data to the first coordinate system according to the rotation angle corresponding to each frame of point cloud data, obtaining dense point cloud data in the first coordinate system corresponding to the calibration board; the first coordinate system is established based on the position of the single-line LiDAR during use; controlling the camera to acquire images of the calibration board, obtaining image data of the calibration board in the second coordinate system; the second coordinate system is established based on the position of the camera; matching the features of the dense point cloud data with the features of the image data, obtaining the extrinsic parameter calibration results of the first and second coordinate systems; solving the problem that traditional calibration methods cannot achieve extrinsic parameter calibration between the single-line LiDAR and the camera; since the single-line LiDAR is rotated by a motor to obtain dense point cloud data of the calibration board, and the dense point cloud data is used to perform extrinsic parameter calibration with the image data acquired by the camera, extrinsic parameter calibration between the single-line LiDAR and the camera can be achieved.

[0050] In addition, the accuracy of the calibration results can be improved by calibrating the extrinsic parameters by matching the edges of the rectangle, the corners of the edges, and the corners of the black and white cells.

[0051] In addition, using the pixel values ​​of each pixel in the image data to determine the edges and corners of the calibration board can reduce the difficulty of identifying edges and corners and save equipment resources.

[0052] In addition, by controlling the single-line lidar and encoder to collect data synchronously, the difficulty of obtaining the rotation angle corresponding to the point cloud data can be reduced, and the efficiency of obtaining the rotation angle can be improved.

[0053] In addition, rotating the single-line lidar point cloud data to the first coordinate system according to the rotation angle measured by the encoder and superimposing it into a dense point cloud increases the degree of freedom of single-line lidar data acquisition and improves the flexibility of single-line lidar data acquisition.

[0054] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the structure of a single-line lidar and camera extrinsic parameter calibration system provided in one embodiment of this application;

[0056] Figure 2 This is a schematic diagram of a frame of point cloud data acquired by a multi-line lidar according to an embodiment of this application;

[0057] Figure 3 This is a side view of a frame of point cloud data acquired by a single-line lidar according to an embodiment of this application;

[0058] Figure 4 This is a top view of a frame of point cloud data acquired by a single-line lidar according to an embodiment of this application;

[0059] Figure 5 This is a schematic diagram of a rotating structure provided in one embodiment of this application;

[0060] Figure 6 This is a schematic diagram of a multi-line lidar irradiating a calibration board according to an embodiment of this application;

[0061] Figure 7 This is a schematic diagram of dense point cloud data provided in one embodiment of this application;

[0062] Figure 8 This is a flowchart of a single-line lidar and camera extrinsic parameter calibration method provided in one embodiment of this application. Detailed Implementation

[0063] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0064] Figure 1 This is a schematic diagram of the structure of a single-line lidar and camera extrinsic parameter calibration system provided in one embodiment of this application. The system includes: a single-line lidar 110, a camera 120, a calibration plate 130, a rotating structure 140, and a calibration device 150.

[0065] The single-line lidar 110 and camera 120 are located in the same calibration environment. A single-line lidar 110 refers to a lidar whose laser source emits a single-line beam. The amount of point cloud data acquired by a single-line lidar 110 is small; using only one frame of point cloud data is insufficient to complete the extrinsic parameter calibration of the single-line lidar 110 and camera 120.

[0066] For example: a frame of point cloud data acquired by a multi-line lidar, such as... Figure 2 As shown, in the same scene, a side view of a frame of point cloud data acquired by a single-line lidar is as follows. Figure 3 As shown, the top view is as follows Figure 4 As shown, by Figure 3 and Figure 4 It can be seen that a single frame of point cloud data collected by a single-line lidar is far less than a single frame of point cloud data collected by a multi-line lidar.

[0067] To address the aforementioned issues, this application proposes a rotating shaft structure 140. A single-line lidar is mounted on the rotating shaft structure, causing the lidar to rotate so that its scanning range at least includes the spatial area where the calibration plate 130 is located.

[0068] refer to Figure 5 The rotating structure 140 includes a fixed-end body 141, a motor 142 mounted on the fixed-end body, an encoder 143 connected to the drive shaft of the motor 142, and a rotating body 144. A single-line lidar is mounted on the rotating body 144.

[0069] The fixed end body 141 is used to provide support for the rotating structure and ensure the stability of the rotating structure during the rotation of the motor.

[0070] The output end of the motor 142 has a drive shaft (or output shaft) to output the rotational force of the motor 142 through the drive shaft.

[0071] The encoder 143 and the rotating body 144 are mounted on the drive shaft and rotate under the drive of the drive shaft.

[0072] Encoder 143 is a device that encodes and converts signals (such as bitstreams) or data into a signal form that can be used for communication, transmission, and storage. In this embodiment, encoder 143 is used to measure the rotation angle of motor 142. Encoder 143 can be an angle encoder or a rotary encoder. This embodiment does not limit the implementation of encoder 143. Encoder 143 is a high-precision encoder.

[0073] In one example, the encoder includes a photoelectric code disk with a rotatable central shaft, multiple gratings on the code disk, and light-emitting diodes (LEDs) and phototubes located on either side of the gratings. Light emitted from the LEDs illuminates the gratings, and the phototubes detect the light projected after passing through the gratings. The phototubes output several pulse signals, based on which the rotation angle of the central shaft can be calculated. Based on this principle, connecting the central shaft to a drive shaft allows for the calculation of the drive shaft's rotation angle.

[0074] The rotating body includes a support plate for placing a single-line lidar. The single-line lidar is fixedly connected to the support plate, and the fixing method includes, but is not limited to, threaded connection, snap-fit ​​connection, or adhesive connection. This embodiment does not limit the fixing method.

[0075] The rotating body can be integrally connected to the fixed end body, or the rotating body can be detachably connected to the fixed end body. This embodiment does not limit the connection method between the rotating body and the fixed end body.

[0076] In this embodiment, the single-line lidar is installed behind the rotating structure. During the rotation of the rotating structure, the acquisition center of the single-line lidar coincides with the rotation center of the single-line lidar.

[0077] The camera 120 is used to acquire image data. The camera 120 can be an electronic device with image acquisition capabilities, such as a camera or video camera. This embodiment does not limit the type of camera.

[0078] The calibration board 130 is used to provide the same calibration features for the single-line lidar and the camera. In one example, the calibration board 130 is rectangular and includes spaced black and white cells; the dense point cloud data is matched with the image data to obtain the extrinsic parameter calibration results for the first and second coordinate systems.

[0079] refer to Figure 6 Traditionally, when using multi-line lidar and cameras for extrinsic parameter calibration, the multi-line lidar is used to illuminate a calibration board, and the endpoints of the multi-line lidar are connected to form the plane on which the calibration board is located. Then, the plane on which the calibration board is located is matched with the image of the calibration board captured by the camera to obtain the extrinsic parameter calibration result between the multi-line lidar and the camera.

[0080] However, the point cloud of a single-line lidar does not have multiple endpoints, therefore, it is impossible to use multiple endpoints to determine the plane on which the calibration board is located. To address this technical problem, this embodiment rotates each frame of the single-line lidar and superimposes them to form a complete point cloud model, thereby achieving extrinsic parameter calibration between the single-line lidar and the camera.

[0081] The result of flipping and overlaying the point cloud from a single-line lidar system is as follows: Figure 7 As shown.

[0082] Specifically, in this embodiment, the single-line lidar, encoder, motor, and camera are all connected to the calibration device via a wired connection.

[0083] The calibration equipment, which is a computer, is used to determine the extrinsic parameter calibration results between the single-line lidar and the camera.

[0084] The calibration equipment is used for: pre-calibrating the intrinsic parameters of the camera; starting the encoder and single-line LiDAR, and controlling the motor rotation; acquiring the rotation angle of the rotating body collected by the encoder, and multi-frame point cloud data of the calibration board collected by the single-line LiDAR at each rotation angle; transforming the point cloud data to the first coordinate system according to the rotation angle corresponding to each frame of point cloud data to obtain the dense point cloud data corresponding to the calibration board; the first coordinate system is established based on the position of the single-line LiDAR during use; controlling the camera to acquire images of the calibration board to obtain image data of the calibration board in the second coordinate system; the second coordinate system is established based on the position of the camera; and matching the dense point cloud data with the image data to obtain the extrinsic parameter calibration results of the first and second coordinate systems.

[0085] In this embodiment, by using a motor to drive the single-line lidar to rotate to obtain dense point cloud data of the calibration board, and using the dense point cloud data and the image data acquired by the camera to perform extrinsic parameter calibration, the extrinsic parameter calibration between the single-line lidar and the camera can be realized.

[0086] Optionally, this application uses the single-line lidar and camera extrinsic parameter calibration methods provided in various embodiments for... Figure 1 The following explanation uses the calibration equipment in the external parameter calibration system as an example.

[0087] Figure 8 This is a flowchart of a single-line lidar and camera extrinsic parameter calibration method according to an embodiment of this application. The method includes at least the following steps:

[0088] Step 801: Pre-calibrate the camera's intrinsic parameters.

[0089] The camera's intrinsic parameters are calibrated using the calibration methods provided by OpenCV to obtain the camera's model parameters.

[0090] Step 802: Start the encoder and single-line lidar to control the motor rotation.

[0091] In one example, in response to an external parameter calibration command, the calibration device controls the motor to rotate and activates the encoder and single-line lidar.

[0092] The extrinsic parameter calibration command is used to instruct the start of extrinsic parameter calibration between the single-line lidar and the camera. The extrinsic parameter calibration command may be generated when the calibration device receives a trigger operation applied to the extrinsic parameter calibration control, or when a preset calibration time is reached, or it may be sent by other devices. This embodiment does not limit the method of obtaining the extrinsic parameter calibration command.

[0093] The external parameter calibration control can be a physical button or a virtual control displayed on a touch screen. This embodiment does not limit the implementation method of the external parameter calibration control.

[0094] Step 803: Obtain the rotation angle of the rotating body collected by the encoder, and the multi-frame point cloud data of the calibration board collected by the single-line lidar at each rotation angle.

[0095] In one example, the calibration device controls a single-line lidar and encoder to synchronously acquire data to obtain each frame of point cloud data and the corresponding rotation angle. In this process, the single-line lidar and encoder are synchronized beforehand to ensure that they acquire data synchronously.

[0096] In another example, the data acquisition time of the single-line lidar and the encoder may be out of sync.

[0097] Step 804: According to the rotation angle corresponding to each frame of point cloud data, the point cloud data is transformed to the first coordinate system to obtain the dense point cloud data in the first coordinate system corresponding to the calibration board.

[0098] The first coordinate system is established based on the position of the single-line lidar during use. For example, the position of the single-line lidar during use is the position of the single-line lidar when the motor is not rotating. In this case, the first coordinate system is established with the data acquisition center of the single-line lidar as the origin, the plane containing the point cloud defined inside the single-line lidar as the xoy plane, and the axis perpendicular to the xoy plane as the z-axis.

[0099] Because the coordinate values ​​of point cloud data acquired by a single-line lidar are all within the coordinate system established at the current position of the lidar, and the lidar is rotating, meaning that point cloud data acquired at different positions belong to different coordinate systems, it is necessary to transform each frame of point cloud data to the same coordinate system for overlay and fusion. In this embodiment, by transforming each frame of point cloud data to the first coordinate system, the calibration device can directly obtain the calibration results between the first coordinate system and the second coordinate system where the camera is located, without the need for an additional coordinate transformation process, thus improving the efficiency of extrinsic parameter calibration.

[0100] Specifically, when the position of the single-line lidar is not the position where the rotation angle is 0, the point cloud data is transformed to the first coordinate system according to the rotation angle corresponding to each frame of point cloud data to obtain the dense point cloud data corresponding to the calibration board. This includes: for each frame of point cloud data where the rotation angle is not 0, the rotation angle of the single-line lidar is determined by the data collected by the encoder; the representation of the rotation vector in the first coordinate system is determined according to the installation position of the single-line lidar on the rotating structure; based on the rotation vector and the rotation angle, the transformation relationship between the coordinate system of the rotating single-line lidar and the first coordinate system is determined; based on the transformation relationship, each frame of point cloud data collected by the rotating single-line lidar is transformed to the first coordinate system and superimposed to obtain the dense point cloud of the calibration board in the first coordinate system.

[0101] The conversion is performed according to the following formula:

[0102] v rot =vcosθ+(1-cosθ)(k·v)·k+sinθ·k×v (Formula 1);

[0103] Among them, v rot To transform to the point in the first coordinate system, v is the point obtained by the current single-line lidar scan, θ is the rotation angle measured by the encoder, and k is the representation of the rotation vector corresponding to the rotation axis in the single-line lidar coordinate system.

[0104] When the single-line lidar and encoder acquire data asynchronously, and the motor rotates at a preset speed at a constant speed, the calibration device obtains the rotation angle corresponding to each frame of point cloud data in the following ways:

[0105] For each frame of point cloud data, the timestamps of each rotation angle are compared with the timestamps of the point cloud data to obtain the timestamp difference; the angle difference is determined based on the minimum value of the timestamp difference and the preset speed; the rotation angle corresponding to the point cloud data is determined based on the rotation angle corresponding to the minimum value of the timestamp difference and the angle difference.

[0106] In one example, the calibration device flips each frame of point cloud data it acquires to transform it to the first coordinate system; or, the calibration device flips each frame of point cloud data separately to transform it to the first coordinate system after the motor has rotated at least one revolution. This embodiment does not limit the timing of the calibration device flipping the point cloud data.

[0107] Step 805: Control the camera to acquire images of the calibration board and obtain image data of the calibration board in the second coordinate system; the second coordinate system is established based on the position of the camera.

[0108] The second coordinate system is the camera coordinate system, which is a three-dimensional rectangular coordinate system established with the camera's focal center as the origin and the optical axis as the Z-axis.

[0109] Step 806: Match the dense point cloud data with the image data to obtain the extrinsic parameter calibration results of the first coordinate system and the second coordinate system.

[0110] In this embodiment, the calibration board is rectangular and includes black and white cells spaced apart. Accordingly, the dense point cloud data is matched with the image data to obtain the extrinsic parameter calibration results for the first and second coordinate systems. This includes: edge extraction of the dense point cloud data to obtain its edge contours; line estimation of the edge contours to obtain the edge corner points in the four first coordinate systems formed by the intersection of four adjacent edge lines; and using the intensity information of the dense point cloud data to determine the m*n cell corner points in the first coordinate systems for the black and white cells.

[0111] The process of using intensity information from dense point cloud data to determine the corner points of m*n cells in the first coordinate system for black and white cells includes: performing planar estimation on the dense point cloud data; projecting all points in the dense point cloud data onto the estimated plane; retrieving intensity information of other points within a certain range around each point; and determining the point as a cell corner point if the intensity information conforms to a preset rule.

[0112] The specific method involves performing a plane estimation on the dense point cloud, projecting all points onto this plane, and then calculating the normal vector of the plane. by Using the rotation direction, the intensity information of all points within a radius of r of each point is retrieved sequentially. If the intensity information changes more than the threshold 4 times, it is considered to be a corner point of the cell.

[0113] Identify the edges of the calibration plate in the image data to obtain four edge lines and four edge corner points in the second coordinate system; identify the black and white cells in the image data to obtain m*n cell corner points in the second coordinate system.

[0114] Matching edge lines in the first coordinate system with edge lines in the second coordinate system, matching edge corner points in the first coordinate system with edge corner points in the second coordinate system, and matching cell corner points in the first coordinate system with cell corner points in the second coordinate system, combined with camera intrinsic parameters, forms a Perspective-n-Point (PnP) problem. Solving this PnP problem yields the extrinsic parameter calibration results. These extrinsic parameter calibration results include the relative distance and rotation angle between the first and second coordinate systems.

[0115] The PnP problem describes how to estimate the rotation matrix and translation vector from the point cloud coordinate system to the camera coordinate system when the coordinates of n 3D points (in the point cloud coordinate system) and the pixel coordinates of these points are known. Formula 2 is as follows.

[0116]

[0117] Where p is the coordinate of the point in the pixel coordinate system, P C Let P be the coordinates of the point in the camera coordinate system. W Let ω be the coordinates of the point in the point cloud coordinate system, K be the depth of the point, and R be the intrinsic parameter matrix of the camera. CW and This refers to the pose transformation from the point cloud coordinate system to the camera coordinate system, which is also the extrinsic parameter between the single-line LiDAR and the camera.

[0118] There are four main algorithms for solving the PnP problem (direct linear transformation, P3P method, EPnP method, and speed of light difference method). This embodiment does not limit the solution method for the PnP problem.

[0119] Optionally, edge extraction is performed on the dense point cloud data to obtain the edge contours of the dense point cloud data, including: extracting the edge contours using a scatter contour (alpha shapes) algorithm.

[0120] The process of estimating the edge contour to obtain the edge line in the first coordinate system includes: using point cloud data located on the edge contour to perform line fitting to obtain the edge line in the first coordinate system. Then, the intersection of two adjacent edge lines in the first coordinate system is determined as the edge corner point in the first coordinate system.

[0121] The linear fitting methods include, but are not limited to, least squares method, gradient descent method, or Hough transform. This embodiment does not limit the linear fitting method.

[0122] Optionally, identifying the edge of the calibration board in the image data to obtain the second edge line and the second edge corner point includes: using a corner point response algorithm to obtain all cell corner points in the second coordinate system in the image data; and calculating the edge line in the second coordinate system and the edge corner point in the second coordinate system based on all cell corner points in the second coordinate system, according to the pre-measured width of the blank area of ​​the calibration board edge and the width of the square cell.

[0123] This embodiment uses the edges of rectangles, the corners of the edges, and the corners of black and white cells to match extrinsic parameters, which can further improve the accuracy of extrinsic parameter calibration.

[0124] In summary, the single-line LiDAR and camera extrinsic parameter calibration method provided in this embodiment pre-calibrates the camera's intrinsic parameters; activates the encoder and single-line LiDAR, controlling the motor to rotate; acquires the rotation angle of the rotating body collected by the encoder, and multi-frame point cloud data of the calibration board collected by the single-line LiDAR at each rotation angle; transforms the point cloud data to a first coordinate system according to the rotation angle corresponding to each frame of point cloud data, obtaining dense point cloud data in the first coordinate system corresponding to the calibration board; the first coordinate system is established based on the position of the single-line LiDAR during use; and controls the camera to perform image acquisition on the calibration board. The system collects image data of the calibration board in the second coordinate system, which is established based on the camera's position. It then matches the features of the dense point cloud data with the features of the image data to obtain the extrinsic parameter calibration results for the first and second coordinate systems. This solves the problem that traditional calibration methods cannot achieve extrinsic parameter calibration between a single-line LiDAR and a camera. Since a motor drives the single-line LiDAR to rotate to obtain dense point cloud data of the calibration board, and this dense point cloud data is used to perform extrinsic parameter calibration with the image data acquired by the camera, extrinsic parameter calibration between the single-line LiDAR and the camera can be achieved.

[0125] In addition, the accuracy of the calibration results can be improved by calibrating the extrinsic parameters by matching the edges of the rectangle, the corners of the edges, and the corners of the black and white cells.

[0126] In addition, using the pixel values ​​of each pixel in the image data to determine the edges and corners of the calibration board can reduce the difficulty of identifying edges and corners and save equipment resources.

[0127] In addition, by controlling the single-line lidar and encoder to collect data synchronously, the difficulty of obtaining the rotation angle corresponding to the point cloud data can be reduced, and the efficiency of obtaining the rotation angle can be improved.

[0128] In addition, rotating the single-line lidar point cloud data to the first coordinate system according to the rotation angle measured by the encoder and superimposing it into a dense point cloud increases the degree of freedom of single-line lidar data acquisition and improves the flexibility of single-line lidar data acquisition.

[0129] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the single-line lidar and camera extrinsic parameter calibration method of the above method embodiments.

[0130] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the single-line lidar and camera extrinsic parameter calibration method of the above method embodiments.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for calibrating the extrinsic parameters of a single-line lidar and camera, characterized in that, The single-line lidar and the calibration environment of the camera have a calibration plate and a rotating structure; the rotating structure includes a fixed end body, a motor mounted on the fixed end body, an encoder connected to the drive shaft of the motor, and a rotating body, and the single-line lidar is mounted on the rotating body; The encoder is used to detect the rotation angle of the drive shaft; the method includes: The camera's intrinsic parameters are pre-calibrated; The encoder and single-line lidar are activated to control the motor rotation; The rotation angle of the rotating body collected by the encoder, and the multi-frame point cloud data of the calibration board collected by the single-line lidar at each rotation angle are obtained. According to the rotation angle corresponding to each frame of point cloud data, the point cloud data is transformed to the first coordinate system to obtain the dense point cloud data in the first coordinate system corresponding to the calibration board; the first coordinate system is established based on the position of the single-line lidar when it is used. The camera is controlled to acquire images of the calibration board, obtaining image data of the calibration board in a second coordinate system; the second coordinate system is established based on the position of the camera. The dense point cloud data is matched with the image data to obtain the extrinsic parameter calibration results of the first coordinate system and the second coordinate system; The calibration board is rectangular and includes spaced black and white cells; the step of matching the dense point cloud data with the image data to obtain the extrinsic parameter calibration results of the first coordinate system and the second coordinate system includes: Edge extraction is performed on the dense point cloud data to obtain the edge contours of the dense point cloud data; The edge contour is estimated by line estimation to obtain the edge lines in the four first coordinate systems and the edge corner points in the four first coordinate systems formed by the intersection of each pair of adjacent edge lines. The intensity information of the dense point cloud data is used to determine the corner points of the m*n cells in the first coordinate system of the black cells and the white cells; The edges of the calibration plate in the image data are identified to obtain four edge lines in the second coordinate system and four edge corner points in the second coordinate system; Identify the black and white cells in the image data to obtain m*n cell corner points in the second coordinate system; Match the edge lines in the first coordinate system with the edge lines in the second coordinate system, match the edge corners in the first coordinate system with the edge corners in the second coordinate system, match the cell corners in the first coordinate system with the cell corners in the second coordinate system, and combine these with the camera intrinsic parameters to form a PnP problem. Solve the PnP problem to obtain the extrinsic parameter calibration results.

2. The method according to claim 1, characterized in that, The pre-calibration of the camera's intrinsic parameters includes: The intrinsic parameters of the camera are calibrated using the calibration methods provided by OpenCV to obtain the model parameters of the camera.

3. The method according to claim 1, characterized in that, The position of the single-line lidar during use is the position when the rotation angle is 0, and the first coordinate system is established based on this position; the step of transforming the point cloud data to the first coordinate system according to the rotation angle corresponding to each frame of point cloud data to obtain the dense point cloud data in the first coordinate system corresponding to the calibration board includes: For point cloud data with a non-zero rotation angle in each frame, the rotation angle of the single-line lidar is determined by the data collected by the encoder. Based on the installation position of the single-line lidar on the rotating structure, the representation of the rotation vector in the first coordinate system is determined; Based on the rotation vector and the rotation angle, the transformation relationship between the rotating single-line lidar coordinate system and the first coordinate system is determined; Based on the transformation relationship, each frame of point cloud data collected by the rotating single-line lidar is transformed to the first coordinate system and superimposed to obtain the dense point cloud of the calibration plate in the first coordinate system.

4. The method according to claim 1, characterized in that, The step of edge extraction of the dense point cloud data to obtain the edge contour of the dense point cloud data includes: The edge contours are extracted using the scatter contour alpha shapes algorithm.

5. The method according to claim 1, characterized in that, The step of using the intensity information of the dense point cloud data to determine the corner points of the m*n cells in the first coordinate system of the black and white cells includes: Perform planar estimation on the dense point cloud data; Project all points in the dense point cloud data onto the estimated plane; Retrieve intensity information of other points within a certain range around each point; If the intensity information conforms to the preset rules, the point is determined to be a cell corner point.

6. The method according to claim 1, characterized in that, The process of identifying the edges of the calibration plate in the image data to obtain four edge lines in the second coordinate system and four edge corner points in the second coordinate system includes: The corner response algorithm is used to obtain all cell corners in the second coordinate system of the image data; Based on all cell corner points in the second coordinate system, and according to the pre-measured width of the blank area at the edge of the calibration plate and the width of the square cell, the edge lines and edge corner points in the second coordinate system are calculated.

7. The method according to claim 1, characterized in that, The start encoder and single-line lidar include: The single-line lidar is controlled to synchronously acquire data with the encoder to obtain point cloud data for each frame and the corresponding rotation angle for each frame of point cloud data.

8. The method according to claim 1, characterized in that, The motor rotates at a preset speed at a constant speed, and the method further includes: For each frame of point cloud data, the timestamp of each rotation angle is compared with the timestamp of the point cloud data to obtain the difference in timestamps; The angle difference is determined based on the minimum difference in timestamps and the preset speed; The rotation angle corresponding to the point cloud data is determined based on the rotation angle corresponding to the minimum value of the timestamp difference and the angle difference.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement the single-line lidar and camera extrinsic parameter calibration method as described in any one of claims 1 to 8.

Images