A method, apparatus, equipment and medium for determining joint calibration parameters

Abstract

This application provides a method, apparatus, device, and medium for determining joint calibration parameters. In this method, a calibration board is placed in different poses. For each pose, based on the edge line of the calibration board in the image acquired by the camera at that pose, the first coordinates of a preset position on the calibration board in the image are determined. Then, the point cloud data at that pose is processed using a plane fitting algorithm to obtain a first plane corresponding to the calibration board in the point cloud data acquired by the LiDAR. Based on the laser point located on the edge line of the calibration board and the determined first plane, the second coordinates of the target laser point at the preset position on the calibration board are determined. Based on the determined first and second coordinates of each pose and a preset data registration algorithm, the parameters for joint calibration of the camera and LiDAR are comprehensively determined, mitigating the problem of low accuracy in determining calibration parameters caused by error accumulation.

Description

Technical Field

[0001] This application relates to the field of sensor calibration technology, and in particular to a method, apparatus, device and medium for determining joint calibration parameters. Background Technology

[0002] LiDAR can provide 3D structural information of a scene, assisting in subsequent tasks such as ranging, obstacle avoidance, and 3D reconstruction, but it suffers from high measurement noise. Cameras can provide rich and detailed information such as the color and appearance features of a scene, but they are easily affected by lighting conditions and cannot function properly in dark environments. Therefore, multi-sensor fusion technology combining LiDAR and cameras has been extensively studied in many technical fields. Generally, LiDAR and cameras are configured and used in the same multi-sensor device, and its calibration has become a research hotspot.

[0003] Some related technologies involve pre-setting multiple calibration boards in fixed poses in a laboratory environment. During the movement of a multi-sensor device, the camera within the device identifies the calibration boards to obtain the continuous motion trajectory of the multi-sensor device. This trajectory is then aligned with the motion-compensated point cloud data, and the relevant calibration parameters are iteratively solved. However, this method requires a pre-arranged laboratory environment, including the placement of each calibration board, placing excessive demands on the calibration site. Furthermore, the accuracy of 3D spatial positioning using camera-based calibration boards is inherently low, and the trajectory under continuous motion across multiple frames further amplifies the accumulated single-frame recognition error. Similarly, the single-frame ranging error of the point cloud data will also be reflected in the continuous motion trajectory of the lidar. Solving for the calibration parameters of the lidar and camera using a motion trajectory with inherent errors cannot achieve high accuracy.

[0004] Therefore, improving the accuracy of parameter determination during joint calibration of lidar and camera has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for determining joint calibration parameters, in order to solve the problem of low accuracy of calibration parameters determined when LiDAR and camera are jointly calibrated in the prior art.

[0006] In a first aspect, this application provides a method for determining joint calibration parameters, the method comprising:

[0007] Acquire images of the calibration board in each pose captured by the camera, and point cloud data of the calibration board in each pose captured by the lidar;

[0008] For each pose, the edge line of the calibration plate in the image of that pose is detected based on an edge detection algorithm; based on the edge line, the first coordinates of the preset position in the calibration plate in the image are determined; the point cloud data of that pose is processed based on a plane fitting algorithm to obtain a first plane corresponding to the calibration plate; based on the coordinates of adjacent laser points in the point cloud data, the laser point located on the edge line of the calibration plate is determined; based on the first plane and the laser point located on the edge line of the calibration plate, the second coordinates of the target laser point at the preset position in the calibration plate are determined.

[0009] Based on each of the first coordinates, each of the second coordinates, and a preset data registration algorithm, the parameters for joint calibration of the camera and the lidar are determined.

[0010] Furthermore, after detecting the edge line of the calibration plate in the image of the pose based on the edge detection algorithm, the method further includes determining the preset position of the calibration plate in the image before the first coordinate in the image based on the edge line:

[0011] The edge lines are thinned using a thinning algorithm.

[0012] Furthermore, after obtaining the first plane corresponding to the calibration plate, and before determining the laser point located on the edge line of the calibration plate based on the coordinates of adjacent laser points in the point cloud data, the method further includes:

[0013] For each laser point in the point cloud data, determine the distance between the laser point and the first plane; if the distance is within a preset distance range, determine that the laser point is a laser point in a subset of the point cloud data; otherwise, determine that the laser point is a laser point not in the subset of the point cloud data.

[0014] Based on the subset of point cloud data and the plane fitting algorithm, a second plane corresponding to the calibration board is determined; and the first plane is updated using the second plane.

[0015] Further, determining the second coordinates of the target laser point at the preset position in the calibration plate based on the first plane and the laser point located on the edge line of the calibration plate includes:

[0016] Based on the first plane, the laser points located on the edge lines of the calibration plate, and Lagrange's theorem, determine the equation of the straight line corresponding to each edge line of the calibration plate.

[0017] For each pair of adjacent edge lines, the midpoint of the common perpendicular segment of the two adjacent edge lines is determined according to the equation of the line corresponding to the two adjacent edge lines, and the midpoint of the common perpendicular segment is projected onto the first plane, and the projected midpoint is used as the vertex of the calibration plate.

[0018] Based on each vertex, determine the second coordinates of the target laser point at the preset position in the calibration plate.

[0019] Secondly, this application provides a joint calibration parameter determination device, the device comprising:

[0020] The acquisition module is used to acquire images of the calibration board in each pose captured by the camera, and point cloud data of the calibration board in each pose captured by the lidar.

[0021] The determination module is configured to: detect the edge line of the calibration board in the image of each pose based on an edge detection algorithm; determine the first coordinates of a preset position in the image of the calibration board based on the edge line; process the point cloud data of the pose based on a plane fitting algorithm to obtain a first plane corresponding to the calibration board; determine the laser point located on the edge line of the calibration board based on the coordinates of adjacent laser points in the point cloud data; determine the second coordinates of the target laser point at the preset position in the calibration board based on the first plane and the laser point located on the edge line of the calibration board; and determine the parameters for joint calibration of the camera and the lidar based on each first coordinate, each second coordinate, and a preset data registration algorithm.

[0022] Furthermore, the determining module is also used to refine the edge line based on a thinning algorithm.

[0023] Furthermore, the determining module is also configured to determine the distance between each laser point in the point cloud data and the first plane; if the distance is within a preset distance range, then the laser point is determined to be a laser point in a subset of the point cloud data; otherwise, the laser point is determined to be a laser point not in the subset of the point cloud data; determine the second plane corresponding to the calibration board according to the subset of the point cloud data and the plane fitting algorithm; and update the first plane using the second plane.

[0024] Further, the determining module is specifically used to determine the equation of the straight line corresponding to each edge line of the calibration plate based on the first plane, the laser point located on the edge line of the calibration plate, and Lagrange's theorem; for each pair of adjacent edge lines, the midpoint of the common perpendicular segment of the adjacent edge lines is determined based on the equation of the straight line corresponding to the two adjacent edge lines, and the midpoint of the common perpendicular segment is projected onto the first plane, and the projected midpoint is used as the vertex of the calibration plate; based on each vertex, the second coordinates of the target laser point at the preset position in the calibration plate are determined.

[0025] Thirdly, this application also provides an electronic device, which includes at least a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the steps of the joint calibration parameter determination method described above.

[0026] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described methods for determining joint calibration parameters.

[0027] In this embodiment, a calibration board is placed in different poses. For each pose, the first coordinates of a preset position on the calibration board in the image are determined based on the edge line of the calibration board in the image acquired by the camera in that pose. The point cloud data in that pose is then processed using a plane fitting algorithm to obtain the first plane corresponding to the calibration board in the point cloud data acquired by the LiDAR. The second coordinates of the target laser point at the preset position on the calibration board are determined based on the laser point on the edge line of the calibration board and the determined first plane. Based on the determined first and second coordinates of each pose and the preset data registration algorithm, the parameters for joint calibration of the camera and LiDAR are comprehensively determined. The first coordinates corresponding to the camera and the second coordinates corresponding to the LiDAR in different poses are determined respectively. The parameters for joint calibration are determined based on the determined first and second coordinates, which reduces the problem of low accuracy in determining calibration parameters caused by error accumulation. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram illustrating the process of determining joint calibration parameters provided in the embodiments of this application;

[0030] Figure 2aA schematic diagram of a binary image processed by an edge detection algorithm, provided in an embodiment of this application;

[0031] Figure 2b This is another schematic diagram of a binary image processed by an edge detection algorithm, provided in an embodiment of this application.

[0032] Figure 2c A schematic diagram of a refined binary image provided in an embodiment of this application;

[0033] Figure 3a This is an enlarged schematic diagram of the edge line before it is thinned, provided in an embodiment of this application.

[0034] Figure 3b This is an enlarged schematic diagram showing the refined edge lines provided in an embodiment of this application;

[0035] Figure 4a A schematic diagram of point cloud data provided in an embodiment of this application;

[0036] Figure 4b An enlarged schematic diagram provided for an embodiment of this application;

[0037] Figure 5a Statistical analysis diagram of point cloud data when the measurement distance is 1.73m, provided for embodiments of this application;

[0038] Figure 5b Statistical analysis diagram of point cloud data when the measurement distance is 2.7m, provided for embodiments of this application;

[0039] Figure 5c Statistical analysis diagram of point cloud data when the measurement distance is 3.12m, provided for an embodiment of this application;

[0040] Figure 5d Statistical analysis diagram of point cloud data when the measurement distance is 3.28m, provided for embodiments of this application;

[0041] Figure 6 This is a schematic diagram of point cloud data filtering provided in an embodiment of this application;

[0042] Figure 7 This is a schematic diagram of the calibration plate and fitted edge line provided in the embodiments of this application;

[0043] Figure 8 This is a schematic diagram of the reprojection of calibration results provided in the embodiments of this application;

[0044] Figure 9a The calibration board provided in this embodiment is a reprojection diagram at pose 1;

[0045] Figure 9b The calibration board provided in this embodiment is a schematic diagram of reprojection at pose 2;

[0046] Figure 9c The calibration board provided in this embodiment is a schematic diagram of reprojection at pose 3;

[0047] Figure 10 A schematic diagram illustrating the process of determining joint calibration parameters provided in the embodiments of this application;

[0048] Figure 11 A schematic diagram of the joint calibration parameter determination device provided in the embodiments of this application;

[0049] Figure 12 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0051] This application provides a method, apparatus, device, and medium for determining joint calibration parameters. The method acquires images of a calibration board in each pose captured by a camera, and point cloud data of the calibration board in each pose captured by a lidar. For each pose, an edge detection algorithm is used to detect the edge line of the calibration board in the image of that pose. Based on the edge line, the first coordinates of a preset position on the calibration board in the image are determined. The point cloud data of that pose is then processed using a plane fitting algorithm to obtain a first plane corresponding to the calibration board. Based on the coordinates of adjacent laser points in the point cloud data, laser points located on the edge line of the calibration board are determined. Based on the first plane and the laser points located on the edge line of the calibration board, the second coordinates of a target laser point at a preset position on the calibration board are determined. Based on the first coordinate, the second coordinate, and a preset data registration algorithm for each pose, the parameters for joint calibration of the camera and the lidar are determined.

[0052] Example 1:

[0053] Figure 1 This is a schematic diagram of the joint calibration parameter determination process provided in an embodiment of this application. The process specifically includes the following steps:

[0054] S101: Acquire images of the calibration board in each pose captured by the camera, and point cloud data of the calibration board in each pose captured by the LiDAR.

[0055] The joint calibration parameter determination process provided in this application embodiment is applicable to electronic devices, such as servers, PCs, and sensors.

[0056] To improve the accuracy of calibration parameter determination during joint calibration of LiDAR and camera, in this embodiment, images containing the calibration board captured by the camera at different poses can be acquired during joint calibration. These images can be taken by a user of an electronic device when pressing a button. Since the position of the calibration board relative to the LiDAR and camera remains constant in the same pose, in this embodiment, video captured by the camera for a preset time period can be acquired, and the image is a randomly selected frame from that video. Simultaneously, point cloud data containing the calibration board collected by the LiDAR within the preset time period at different poses is acquired.

[0057] In this embodiment, the calibration board can be of any size, as long as the camera and LiDAR can capture the entire calibration board. The shape of the calibration board can be any shape, such as rectangle, parallelogram, or triangle. In this embodiment, a rectangle is used as the calibration board for explanation. In this embodiment, the calibration board is a non-transparent board, and there are no restrictions on its size, shape, or material. Furthermore, there is no need to add special visual aids to the calibration board, such as Aruco codes or checkerboard patterns.

[0058] In this embodiment, a calibration board of a certain size can be selected and placed in front of the sensor. The sensor collects relevant data including the calibration board. The sensor includes a camera and a LiDAR, which can be a multi-line LiDAR. The mounting positions of the camera and LiDAR within the sensor are fixed. The relevant data including the calibration board collected by the sensor includes the image of the calibration board captured by the camera and the point cloud data of the calibration board captured by the LiDAR.

[0059] In this embodiment, when the sensor collects relevant data from the calibration board, it can collect relevant data from different positions and angles of the calibration board, that is, the sensor collects multiple sets of relevant data including the calibration board. Specifically, the camera and lidar in the sensor can each collect approximately 80 to 100 frames of relevant data at each position. Assuming a frame rate per second (FPS) of 30, the sensor only needs to collect data for 3 to 4 seconds at different positions.

[0060] In this embodiment of the application, there is no need to set up a calibration site in advance. The entire calibration process only requires placing the calibration board at different positions in front of the sensor and collecting multiple sets of relevant data including the calibration board.

[0061] In this embodiment of the application, if the electronic device is a server, PC or other device, it can acquire the image containing the calibration board corresponding to each pose collected by the sensor connected to it, and the point cloud data corresponding to each pose collected by the LiDAR. If the electronic device is a sensor, it can acquire the image containing the calibration board corresponding to each pose collected by its own camera, and the point cloud data corresponding to each pose collected by the LiDAR.

[0062] S102: For each pose, detect the edge line of the calibration plate in the image of that pose based on the edge detection algorithm; determine the first coordinate of the preset position of the calibration plate in the image based on the edge line.

[0063] After acquiring the image containing the calibration board corresponding to each pose captured by the camera, the first coordinates of the preset position of the calibration board contained in the image of each pose can be determined. In the embodiments of this application, the coordinates of each vertex of the calibration board in the image can be determined, and the first coordinates of the preset position can be determined based on the coordinates of each vertex of the calibration board.

[0064] When determining the coordinates of each vertex of the calibration board in an image, image recognition technology can be used to obtain the coordinates of the calibration board vertices in the image. However, since the vertices of the calibration board may be located between two pixels, or because the calibration board is far from the camera, resulting in too many transition pixels in the image and low image quality, it may be impossible to accurately obtain the pixel coordinates of the calibration board vertices. Alternatively, the coordinates may be incorrectly identified because the vertices of the calibration board are not sharp or are damaged. To further improve the accuracy of coordinate determination, edge detection algorithms can be used to detect the edge lines of the calibration board in the image at this pose. Specifically, the Canny edge detection algorithm can be used to process the image, obtaining a processed binary image of the pose. Figure 2a This is a schematic diagram of a binary image processed by an edge detection algorithm according to an embodiment of this application. Figure 2a The largest rectangle formed by the four straight lines represents the detected edge line of the calibration board. The several rectangles within this largest rectangle are merely for visual representation of the calibration board. In this embodiment, the calibration board does not have any specially designed visual aids. Therefore, in this embodiment, the image is processed using an edge detection algorithm, and the resulting binary image contains only the largest rectangle formed by the four straight lines. Figure 2b As shown, Figure 2b This is another schematic diagram of a binary image processed by an edge detection algorithm, provided in an embodiment of this application.

[0065] After determining the edge lines of the calibration board, the equation of the line corresponding to each edge line can be determined, and the coordinates of each vertex of the calibration board in the image can be obtained based on the obtained line equations. Since the coordinates of each vertex of the calibration board are known, the first coordinates of the preset position in the calibration board in the image can be determined. The preset position can be the center pixel of the calibration board or a pixel at a preset distance from a certain edge line.

[0066] Specifically, after detecting the edge lines of the calibration board in the image frame, the coordinates of each pixel on each edge line can be obtained. Then, based on the obtained pixel coordinates, a straight line is fitted using algorithms such as least squares or gradient descent to obtain the equation of the corresponding line. Assuming the calibration board is rectangular, four straight line equations can be obtained. Solving these four equations simultaneously yields the coordinates of the four vertices of the calibration board. After determining the four vertices, the diagonals of the calibration board can be determined, and the equation for each diagonal can be calculated. Finally, solving the equations of the diagonals simultaneously yields the first coordinate of the pixel at the center of the calibration board.

[0067] In this embodiment, edge extraction is performed based on the difference between the foreground and background using an edge detection algorithm. An edge thinning algorithm based on the skeleton extraction idea is used to achieve sub-pixel extraction of the edge lines of the calibration board, and finally the first coordinates of the preset position of the calibration board in the image with sub-pixel precision are obtained.

[0068] S103: Process the point cloud data of the pose based on the plane fitting algorithm to obtain the first plane corresponding to the calibration plate; determine the laser point located on the edge line of the calibration plate according to the coordinates of the adjacent laser points in the point cloud data; determine the second coordinates of the target laser point at the preset position in the calibration plate according to the first plane and the laser point located on the edge line of the calibration plate.

[0069] After determining the first coordinates, or simultaneously determining the first coordinates, the second coordinates of the target laser point at a preset position in the calibration board can be determined based on the point cloud data collected by the lidar in that pose. In this embodiment, the point cloud data of the frame can be processed based on a plane fitting algorithm to obtain the first plane corresponding to the calibration board collected by the lidar. Specifically, the Random Sample Consensus (Ransac) algorithm can be used for plane fitting.

[0070] Since the point cloud data collected by the lidar can represent the distance between the lidar and the object being measured, and each point cloud data corresponds to a lidar point, after determining the first plane, the coordinates of adjacent lidar points in the point cloud data can be used to determine which lidar points are located on the edge line of the calibration board. Specifically, since the point cloud data includes coordinates, the lidar points in the point cloud data can be displayed in a coordinate system. For each lidar point in the coordinate system, the difference between the coordinates of that lidar point and the coordinates of the next adjacent lidar point is calculated. If this difference is greater than a preset threshold, it can be determined that the next adjacent lidar point is located on the edge line of the calibration board. In other words, by utilizing the gradient changes and intensity abrupt changes in the point cloud data hit by the lidar on the edge line of the calibration board, lidar points located on the edge line of the calibration board can be filtered out.

[0071] Once the first plane corresponding to the calibration board acquired by the lidar and the laser point located on the edge line of the calibration board are determined, the equation of the straight line corresponding to the edge line of the calibration board acquired by the lidar can also be determined. Based on the equation of the straight line corresponding to the edge line of the calibration board, the second coordinates of the target laser point at the preset position in the calibration board can also be determined.

[0072] S104: Determine the parameters for joint calibration of the camera and the lidar based on each of the first coordinates, each of the second coordinates, and a preset data registration algorithm.

[0073] Since the image captured by the camera is only a two-dimensional plane image, the first coordinate is the coordinate of the preset position of the calibration plate in the image, which is a coordinate in a two-dimensional coordinate system. The second coordinate is the coordinate of the laser point at the preset position of the calibration plate captured by the lidar, which is a coordinate in a three-dimensional coordinate system. Therefore, after determining the first coordinate and the second coordinate corresponding to each pose, in order to facilitate the determination of the parameters for joint calibration of the camera and lidar, in this embodiment of the application, the first coordinate and the second coordinate can be converted into coordinates in the same coordinate system, that is, the first coordinate in the two-dimensional coordinate system can be converted into coordinates in the three-dimensional coordinate system, or the second coordinate in the three-dimensional coordinate system can be converted into coordinates in the two-dimensional coordinate system.

[0074] Specifically, the PnP (Perspective-n-Point) algorithm can be used to process each first coordinate to obtain the third coordinate in the three-dimensional coordinate system. That is, the first coordinate in the XY coordinate system is transformed into the third coordinate in the XYZ coordinate system. After determining the third coordinate, the corresponding first coordinate is updated using the third coordinate.

[0075] Based on each first and second coordinate and a preset data matching algorithm, the parameters for joint calibration of the camera and LiDAR are determined.

[0076] Specifically, the transformation matrix between each first coordinate and each second coordinate can be determined based on the Iterative Closest Point (ICP) algorithm, thereby determining the extrinsic parameters for joint calibration of the camera and LiDAR. Given each first coordinate of the camera and each second coordinate of the LiDAR, the process of determining the extrinsic parameters for joint calibration of the camera and LiDAR based on the ICP algorithm has been described in detail in related technologies and will not be repeated in this embodiment.

[0077] In this embodiment, a calibration board is placed in different poses. For each pose, the first coordinates of a preset position on the calibration board in the image are determined based on the edge line of the calibration board in the image acquired by the camera in that pose. The point cloud data in that pose is then processed using a plane fitting algorithm to obtain the first plane corresponding to the calibration board in the point cloud data acquired by the LiDAR. The second coordinates of the target laser point at the preset position on the calibration board are determined based on the laser point on the edge line of the calibration board and the determined first plane. Based on the determined first and second coordinates of each pose and the preset data registration algorithm, the parameters for joint calibration of the camera and LiDAR are comprehensively determined. The first coordinates corresponding to the camera and the second coordinates corresponding to the LiDAR in different poses are determined respectively. The parameters for joint calibration are determined based on the determined first and second coordinates, which reduces the problem of low accuracy in determining calibration parameters caused by error accumulation.

[0078] Example 2:

[0079] To further improve the accuracy of joint calibration parameter determination, based on the above embodiments, in this embodiment, after detecting the edge line of the calibration plate in the image of the pose using an edge detection algorithm, the method further includes determining the preset position of the calibration plate in the image before the first coordinate in the image based on the edge line:

[0080] The edge lines are thinned using a thinning algorithm.

[0081] Since the edge lines of the calibration board in the image detected by the edge detection algorithm are jagged, and the vertices of the detected calibration board edge lines only have integer pixel coordinates and the corners of the vertices are not obvious, in order to further improve the accuracy of the joint calibration parameter determination, in this embodiment of the application, after the edge lines of the calibration board in the image are detected by the edge detection algorithm, each detected edge line can be thinned, and the sub-pixel value of the vertex coordinates of the calibration board can be calculated based on the thinned edge lines.

[0082] Specifically, in this embodiment of the application, the skeleton of the binary image corresponding to the image obtained after edge detection algorithm processing is extracted, that is, the pixels on the central axis of the edge line are preserved as much as possible. The algorithm for skeleton extraction can be a binary image thinning algorithm. Figure 2c A schematic diagram of the refined binary image provided in the embodiments of this application, as shown below. Figure 2c As shown, Figure 2c The four straight lines that form the largest rectangle and Figure 2a The four straight lines that make up the largest rectangle have all become thinner. Figure 3a This is an enlarged schematic diagram of the edge line before it is thinned, provided in an embodiment of this application. Figure 3b This is a magnified schematic diagram of the refined edge lines provided in an embodiment of this application. Figure 3a and Figure 3b As can be seen, the edge line before thinning is composed of multiple pixels, while the edge line after thinning is composed of only pixels on the central axis.

[0083] Example 3:

[0084] To further improve the accuracy of joint calibration parameter determination, based on the above embodiments, in this embodiment, after obtaining the first plane corresponding to the calibration plate and before determining the laser point located on the edge line of the calibration plate according to the coordinates of adjacent laser points in the point cloud data, the method further includes:

[0085] For each laser point in the point cloud data, determine the distance between the laser point and the first plane; if the distance is within a preset distance range, determine that the laser point is a laser point in a subset of the point cloud data; otherwise, determine that the laser point is a laser point not in the subset of the point cloud data.

[0086] Based on the subset of point cloud data and the plane fitting algorithm, a second plane corresponding to the calibration board is determined; and the first plane is updated using the second plane.

[0087] Figure 4a This is a schematic diagram of point cloud data provided in an embodiment of this application, such as... Figure 4a As shown, Figure 4a This is a schematic diagram illustrating the display of collected point cloud data in a coordinate system. Figure 4a It can be clearly observed that the point cloud data ultimately presented in the coordinate system has a certain thickness, that is, it is not a straight line. Figure 4a Taking any diagonal line composed of point cloud data as an example, let's enlarge the diagonal line, such as... Figure 4b As shown, Figure 4b This is an enlarged schematic diagram provided for an embodiment of this application, by Figure 4bIt can be seen that the coordinates of each point cloud data that makes up the oblique line fluctuate within a certain range, which is caused by the systematic error in the ranging accuracy of the lidar.

[0088] Figure 5a The point cloud data statistical analysis diagram provided in this application embodiment shows the measurement distance as 1.73m, where the measurement distance is the distance between the lidar and the calibration board. Figure 5a As shown in the figure, the horizontal axis Distance represents distance, and the vertical axis Density represents the number of laser points. Although the actual distance between the lidar and the calibration board is 1.73m, due to the lidar's own error, each laser point in the collected point cloud data is not only concentrated at 1.73m, but also includes laser points at other distances. Based on the point cloud data when the measured distance is 1.73m, the mean of the measured distance can be determined to be 1.73555m, and the variance is 0.000880754.

[0089] Figure 5b The point cloud data statistical analysis diagram provided in this application embodiment shows that the mean of the measurement distance is 2.7m and the variance is 0.00107312.

[0090] Figure 5c The point cloud data statistical analysis diagram provided in this application embodiment is when the measurement distance is 3.12m. According to the point cloud data when the measurement distance is 3.12m, the mean of the measurement distance is 3.12159m and the variance is 0.00148407.

[0091] Figure 5d The point cloud data statistical analysis diagram provided in this application embodiment is for a measurement distance of 3.28m. Based on the point cloud data for a measurement distance of 3.28m, the mean of the measurement distance can be determined to be 3.28028m and the variance is 0.00326907.

[0092] according to Figures 5a-5d Analysis of point cloud data at different measurement distances shows that the mean and variance of each measurement distance conform to a log-normal distribution. Therefore, the maximum likelihood estimate of the point cloud data from the lidar can be expressed as:

[0093]

[0094] Where n represents the number of laser points in the point cloud data, x k This represents the actual distance of the k-th laser point from the lidar. The mean, Let Variance be the variance.

[0095] Based on the above analysis of point cloud data, in order to further improve the accuracy of joint calibration parameter determination, after determining the equation of the first plane corresponding to the calibration board acquired by the lidar, and before determining the laser points located on the edge line of the calibration board, the acquired point cloud data can be further filtered to obtain point cloud data of laser points with high confidence. In this embodiment, for each laser point in the point cloud data, the distance between the laser point and the first plane can be determined. If the distance is within a preset distance range, the laser point can be considered to have high confidence and is determined as a laser point in the subset of the point cloud data; otherwise, the laser point is determined as a laser point outside the subset of the point cloud data. The preset distance range can be...

[0096] Specifically, assuming the plane equation of the first plane is A1x + B1y + C1z + D1 = 0, the process of determining whether a laser point belongs to a subset of the point cloud data can be represented by the following formula:

[0097]

[0098] in, The mean of the point cloud data is determined based on formula (1.1). Let A1, B1, C1, and D1 be the variance of the point cloud data determined based on formula (1.1), and let P(x), P(y), and P(z) be the coefficients in the first plane equation.

[0099] In this embodiment, for each laser point in the point cloud data, the coordinates of that laser point are substituted into the above formula (1.2). When the above relationship is satisfied, the laser point can be determined to be a laser point in a subset of the point cloud data. The above data filtering process can be used... Figure 6 To illustrate, Figure 6 This is a schematic diagram of point cloud data filtering provided in an embodiment of this application. The white portion represents the set of point cloud data collected by the lidar, and its thickness represents the ranging fluctuation range of the lidar. The dark portion represents a subset of the point cloud data. The white and dark portions are the same size. For ease of display, the dimensions of the dark portion have been expanded. The lidar points included in the dark portion correspond to the ranging fluctuation range of the lidar.

[0100] After determining the subset of point cloud data, the second plane corresponding to the calibration plate is determined again based on the coordinates of each laser point in the subset of point cloud data and the plane fitting algorithm, and the first plane is updated using the second plane.

[0101] Example 4:

[0102] To further improve the accuracy of joint calibration parameter determination, based on the above embodiments, in this embodiment, determining the second coordinates of the target laser point at the preset position in the calibration plate according to the first plane and the laser point located on the edge line of the calibration plate includes:

[0103] Based on the first plane, the laser points located on the edge lines of the calibration plate, and Lagrange's theorem, determine the equation of the straight line corresponding to each edge line of the calibration plate.

[0104] For each pair of adjacent edge lines, the midpoint of the common perpendicular segment of the two adjacent edge lines is determined according to the equation of the line corresponding to the two adjacent edge lines, and the midpoint of the common perpendicular segment is projected onto the first plane, and the projected midpoint is used as the vertex of the calibration plate.

[0105] Based on each vertex, determine the second coordinates of the target laser point at the preset position in the calibration plate.

[0106] In determining the second coordinates of the target laser point at a preset position in the calibration plate based on the first plane and the laser point located on the edge line of the calibration plate, in this embodiment of the application, the equation of the straight line corresponding to each edge line of the calibration plate can be determined based on the first plane, the laser point located on the edge line of the calibration plate, and Lagrange's theorem.

[0107] Specifically, Figure 7 This is a schematic diagram of the calibration plate and fitted edge line provided in the embodiments of this application, as shown below. Figure 7 As shown, assume that l1 is the line intersecting plane1: ax + by + cz + d1 = 0 with the calibration plate plane (i.e., the first plane), and l2 is the line intersecting plane2: ax + by + cz + d2 = 0 with the calibration plate plane (i.e., the first plane), and that plane1 / / plane2. p1 , l p2 Let represent the set of laser points located on the edge line of the calibration plate. Given the known parallelism of opposite edges, the following Lagrangian function can be constructed:

[0108]

[0109] Where N1 represents the laser point set l p1 The number of laser points in the set, N2 represents the number of laser point sets l p2 The number of laser points in (x) 1,i ,y 1,i ,z 1,i ) represents the laser point set l p1 The coordinates of the laser point in the image, where x 1,i y 1,i z 1,iThese represent the distances of laser point i relative to the center point of the lidar in various directions within the three-dimensional coordinate system. The three-dimensional coordinate system is established around the center point of the lidar and consists of the X-axis, Y-axis, and Z-axis. The X-axis represents the left-right spatial direction, the Y-axis represents the up-down spatial direction, and the Z-axis represents the front-back spatial direction. 2,i ,y 2,i ,z 2,i ) represents the laser point set l p2 The coordinates of the laser point in the image.

[0110] Taking the partial derivatives of equation (1.3) with respect to d1 and d2 respectively, and combining them with the Lagrange extremum condition, we can obtain:

[0111]

[0112] Subtracting the mean from each coordinate point eliminates d1 and d2, simplifying the result to:

[0113]

[0114] in, They are coordinate point sets l p1 and l p2 The x-mean and y and z coordinates are similar. Combining equations (1.4) and (1.5), equation (1.3) can be rewritten as:

[0115]

[0116] According to the Lagrange multiplier rule for solving multivariable functions, by taking the partial derivatives with respect to each variable a, b, and c and setting them equal to zero, we obtain the following system of equations:

[0117]

[0118] The above system of equations can be written in the following form:

[0119]

[0120] in:

[0121]

[0122] As can be seen from equation (1.8), λ and (abc) T Given the eigenvalues ​​and eigenvectors of the coefficient matrix, select the smallest eigenvalue λ. min The corresponding eigenvector (a) min b min c min ) TThese are the normal vectors of planes plane1 and plane2. Since the normal vectors of the planes are known, the equations of planes plane1 and plane2 can be determined. By simultaneously solving the equations of plane1, plane2, and the first plane, the equations of the line lines l1 and l2 of the calibration plate's edge lines can be obtained. Similarly, the equations of the other two edge lines l3 and l4 of the calibration plate can be obtained, which will not be elaborated upon here.

[0123] Since the equation of each edge line is determined based on the coordinates of multiple discrete laser points, the determined equations may not lie on the same plane. In this embodiment, after determining the equation of each edge line, for every two adjacent edge lines, the midpoint of the common perpendicular segment of the two adjacent edge lines is determined based on the equations of the two adjacent edge lines. The midpoint of the common perpendicular segment is then projected onto the first plane, and the projected midpoint is used as the vertex of the calibration plate. Subsequently, the intersection of the diagonals is obtained, which is the second coordinate of the target laser point at the preset position in the calibration plate.

[0124] The following example illustrates the effect of calibrating a sensor based on defined calibration parameters. Figure 8 This is a schematic diagram of the reprojection of calibration results provided in an embodiment of this application. Figure 8 The result of reprojecting the multi-line lidar point cloud data onto the calibration board portion of the image based on the joint calibration parameters is shown in the right half of the figure (left and right in the figure). It can be seen that the laser points projected by the multi-line lidar coincide with the edge of the calibration board, and it can be considered that the reprojection error is basically at the pixel level. Figure 9a The calibration board provided in this embodiment is a reprojection diagram at pose 1. Figure 9b The calibration board provided in this embodiment is a schematic diagram of reprojection at pose 2. Figure 9c The calibration board provided in this embodiment is a reprojection diagram at pose 3. Figures 9a-9c The results are obtained by selecting any wooden board for reprojection experiments. The laser point projected by the lidar coincides with the edge of the calibration board, which shows that the generalization and accuracy of the joint calibration results meet the pixel-level accuracy requirements.

[0125] The process of determining the joint calibration parameters is described below with reference to a specific embodiment. Figure 10 A schematic diagram of the process for determining the joint calibration parameters provided in the embodiments of this application is shown below. Figure 10 As shown, the process includes the following steps:

[0126] S1001: The sensor collects relevant data corresponding to each pose, including images of the calibration board collected by the camera and point cloud data collected by the LiDAR.

[0127] S1002: For each pose, the edge lines of the calibration board in the image of that pose are detected based on the edge detection algorithm, and the edge lines are thinned based on the thinning algorithm.

[0128] S1003: Determine the first coordinates of the center of the calibration plate in the image based on the edge line.

[0129] S1004: For each pose, the point cloud data of the pose is processed based on a plane fitting algorithm to obtain the first plane corresponding to the calibration board.

[0130] S1005: Based on the coordinates of adjacent laser points in the point cloud data, determine the laser points located on the edge line of the calibration board, and based on the first plane and the laser points located on the edge line of the calibration board, determine the second coordinates of the target laser point at the center of the calibration board.

[0131] S1006: Determine the parameters for joint calibration of the camera and LiDAR based on the first and second coordinates corresponding to each pose and the preset data registration algorithm.

[0132] Example 5:

[0133] Figure 11 A schematic diagram of the joint calibration parameter determination device provided in the embodiments of this application is shown below. Figure 11 As shown, the device includes:

[0134] The acquisition module 1101 is used to acquire images of the calibration board in each pose acquired by the camera, and point cloud data of the calibration board in each pose acquired by the lidar.

[0135] The determination module 1102 is used to: detect the edge line of the calibration board in the image of each pose based on an edge detection algorithm; determine the first coordinates of a preset position in the image of the calibration board based on the edge line; process the point cloud data of the pose based on a plane fitting algorithm to obtain a first plane corresponding to the calibration board; determine the laser point located on the edge line of the calibration board based on the coordinates of adjacent laser points in the point cloud data; determine the second coordinates of the target laser point at the preset position in the calibration board based on the first plane and the laser point located on the edge line of the calibration board; and determine the parameters for joint calibration of the camera and the lidar based on each first coordinate, each second coordinate, and a preset data registration algorithm.

[0136] In one possible implementation, the determining module 1102 is further configured to refine the edge line based on a thinning algorithm.

[0137] In one possible implementation, the determining module 1102 is further configured to determine the distance between each laser point in the point cloud data and the first plane; if the distance is within a preset distance range, then the laser point is determined to be a laser point in a subset of the point cloud data; otherwise, the laser point is determined to be a laser point not in the subset of the point cloud data; determine the second plane corresponding to the calibration board according to the subset of the point cloud data and the plane fitting algorithm; and update the first plane using the second plane.

[0138] In one possible implementation, the determining module 1102 is specifically configured to: determine the equation of a straight line corresponding to each edge line of the calibration plate based on the first plane, the laser point located on the edge line of the calibration plate, and Lagrange's theorem; for each pair of adjacent edge lines, determine the midpoint of the common perpendicular segment of the adjacent edge lines based on the equation of the straight line corresponding to the two adjacent edge lines, and project the midpoint of the common perpendicular segment onto the first plane, using the projected midpoint as the vertex of the calibration plate; and determine the second coordinates of the target laser point at the preset position in the calibration plate based on each vertex.

[0139] Example 6:

[0140] Figure 12 This application provides a schematic diagram of an electronic device structure as an embodiment of the present application. Based on the above embodiments, the present application also provides an electronic device, such as... Figure 12 As shown, it includes: processor 1201, communication interface 1202, memory 1203 and communication bus 1204, wherein processor 1201, communication interface 1202 and memory 1203 communicate with each other through communication bus 1204.

[0141] The memory 1203 stores a computer program. When the program is executed by the processor 1201, the processor 1201 performs the following steps:

[0142] Acquire images of the calibration board in each pose captured by the camera, and point cloud data of the calibration board in each pose captured by the lidar;

[0143] For each pose, the edge line of the calibration plate in the image of that pose is detected based on an edge detection algorithm; based on the edge line, the first coordinates of the preset position in the calibration plate in the image are determined; the point cloud data of that pose is processed based on a plane fitting algorithm to obtain a first plane corresponding to the calibration plate; based on the coordinates of adjacent laser points in the point cloud data, the laser point located on the edge line of the calibration plate is determined; based on the first plane and the laser point located on the edge line of the calibration plate, the second coordinates of the target laser point at the preset position in the calibration plate are determined.

[0144] Based on each of the first coordinates, each of the second coordinates, and a preset data registration algorithm, the parameters for joint calibration of the camera and the lidar are determined.

[0145] In one possible implementation, the processor 1201 is further configured to perform thinning processing on the edge lines based on a thinning algorithm.

[0146] In one possible implementation, the processor 1201 is further configured to determine the distance between each laser point in the point cloud data and the first plane; if the distance is within a preset distance range, then the laser point is determined to be a laser point in a subset of the point cloud data; otherwise, the laser point is determined to be a laser point not in the subset of the point cloud data.

[0147] Based on the subset of point cloud data and the plane fitting algorithm, a second plane corresponding to the calibration board is determined; and the first plane is updated using the second plane.

[0148] In one possible implementation, the processor 1201 is further configured to determine the equation of a straight line corresponding to each edge line of the calibration plate based on the first plane, the laser points located on the edge lines of the calibration plate, and the Lagrange theorem.

[0149] For each pair of adjacent edge lines, the midpoint of the common perpendicular segment of the two adjacent edge lines is determined according to the equation of the line corresponding to the two adjacent edge lines, and the midpoint of the common perpendicular segment is projected onto the first plane, and the projected midpoint is used as the vertex of the calibration plate.

[0150] Based on each vertex, determine the second coordinates of the target laser point at the preset position in the calibration plate.

[0151] The communication bus mentioned in the above-mentioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 1202 is used for communication between the above-mentioned electronic device and other devices. The memory can include random access memory (RAM), or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor. The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processing unit (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0152] Example 7:

[0153] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by a processor. When the program is run on the processor, the processor executes the following steps:

[0154] Acquire images of the calibration board in each pose captured by the camera, and point cloud data of the calibration board in each pose captured by the lidar;

[0155] For each pose, the edge line of the calibration plate in the image of that pose is detected based on an edge detection algorithm; based on the edge line, the first coordinates of the preset position in the calibration plate in the image are determined; the point cloud data of that pose is processed based on a plane fitting algorithm to obtain a first plane corresponding to the calibration plate; based on the coordinates of adjacent laser points in the point cloud data, the laser point located on the edge line of the calibration plate is determined; based on the first plane and the laser point located on the edge line of the calibration plate, the second coordinates of the target laser point at the preset position in the calibration plate are determined.

[0156] Based on each of the first coordinates, each of the second coordinates, and a preset data registration algorithm, the parameters for joint calibration of the camera and the lidar are determined.

[0157] In one possible implementation, after detecting the edge line of the calibration plate in the image of the pose using an edge detection algorithm, and determining the preset position of the calibration plate in the image before the first coordinate in the image based on the edge line, the method further includes:

[0158] The edge lines are thinned using a thinning algorithm.

[0159] In one possible implementation, after obtaining the first plane corresponding to the calibration plate, and before determining the laser point located on the edge line of the calibration plate based on the coordinates of adjacent laser points in the point cloud data, the method further includes:

[0160] For each laser point in the point cloud data, determine the distance between the laser point and the first plane; if the distance is within a preset distance range, determine that the laser point is a laser point in a subset of the point cloud data; otherwise, determine that the laser point is a laser point not in the subset of the point cloud data.

[0161] Based on the subset of point cloud data and the plane fitting algorithm, a second plane corresponding to the calibration board is determined; and the first plane is updated using the second plane.

[0162] In one possible implementation, determining the second coordinates of the target laser point at the preset position in the calibration plate based on the first plane and the laser point located on the edge line of the calibration plate includes:

[0163] Based on the first plane, the laser points located on the edge lines of the calibration plate, and Lagrange's theorem, determine the equation of the straight line corresponding to each edge line of the calibration plate.

[0164] For each pair of adjacent edge lines, the midpoint of the common perpendicular segment of the two adjacent edge lines is determined according to the equation of the line corresponding to the two adjacent edge lines, and the midpoint of the common perpendicular segment is projected onto the first plane, and the projected midpoint is used as the vertex of the calibration plate.

[0165] Based on each vertex, determine the second coordinates of the target laser point at the preset position in the calibration plate.

[0166] In this embodiment, a calibration board is placed in different poses. For each pose, the first coordinates of a preset position on the calibration board in the image are determined based on the edge line of the calibration board in the image acquired by the camera in that pose. The point cloud data in that pose is then processed using a plane fitting algorithm to obtain the first plane corresponding to the calibration board in the point cloud data acquired by the LiDAR. The second coordinates of the target laser point at the preset position on the calibration board are determined based on the laser point on the edge line of the calibration board and the determined first plane. Based on the determined first and second coordinates of each pose and the preset data registration algorithm, the parameters for joint calibration of the camera and LiDAR are comprehensively determined. The first coordinates corresponding to the camera and the second coordinates corresponding to the LiDAR in different poses are determined respectively. The parameters for joint calibration are determined based on the determined first and second coordinates, which reduces the problem of low accuracy in determining calibration parameters caused by error accumulation.

[0167] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] For system / device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0169] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0172] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0173] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A joint calibration parameter determination method, characterized in that, The method includes: The image of the calibration board in each pose acquired by the camera and the point cloud data of the calibration board in each pose acquired by the lidar are obtained. The calibration board has no visual auxiliary marks, and the visual auxiliary marks are Aruco code and checkerboard. For each pose, the image of the pose is processed based on an edge detection algorithm to obtain a processed binary image. The edge line of the calibration plate in the image is determined based on the binary image. Based on the edge line, the first coordinates of a preset position in the calibration plate in the image are determined. The point cloud data of the pose is processed based on a plane fitting algorithm to obtain a first plane corresponding to the calibration plate. Based on the coordinates of adjacent laser points in the point cloud data, the laser points located on the edge line of the calibration plate are determined. Based on the first plane and the laser points located on the edge line of the calibration plate, the second coordinates of the target laser point at the preset position in the calibration plate are determined, wherein the preset position is the center pixel of the calibration plate. Based on each of the first coordinates, each of the second coordinates, and a preset data registration algorithm, the parameters for joint calibration of the camera and the lidar are determined.

2. The method of claim 1, wherein, After detecting the edge line of the calibration plate in the image of the pose using an edge detection algorithm, the method further includes determining the preset position of the calibration plate before the first coordinate in the image based on the edge line: The edge lines are thinned using a thinning algorithm.

3. The method of claim 1, wherein, After obtaining the first plane corresponding to the calibration plate, and before determining the laser point located on the edge line of the calibration plate based on the coordinates of adjacent laser points in the point cloud data, the method further includes: For each laser point in the point cloud data, determine the distance between the laser point and the first plane; if the distance is within a preset distance range, determine that the laser point is a laser point in a subset of the point cloud data; otherwise, determine that the laser point is a laser point not in the subset of the point cloud data. Based on the subset of point cloud data and the plane fitting algorithm, a second plane corresponding to the calibration board is determined; and the first plane is updated using the second plane.

4. The method of claim 1, wherein, The step of determining the second coordinates of the target laser point at the preset position in the calibration plate based on the first plane and the laser point located on the edge line of the calibration plate includes: Based on the first plane, the laser points located on the edge lines of the calibration plate, and Lagrange's theorem, determine the equation of the straight line corresponding to each edge line of the calibration plate. For each pair of adjacent edge lines, the midpoint of the common perpendicular segment of the two adjacent edge lines is determined according to the equation of the line corresponding to the two adjacent edge lines, and the midpoint of the common perpendicular segment is projected onto the first plane, and the projected midpoint is used as the vertex of the calibration plate. Based on each vertex, determine the second coordinates of the target laser point at the preset position in the calibration plate.

5. A combined calibration parameter determination apparatus, characterized by The device includes: The acquisition module is used to acquire images of the calibration board in each pose captured by the camera, and point cloud data of the calibration board in each pose captured by the lidar, wherein the calibration board has no visual auxiliary marks, and the visual auxiliary marks are Aruco code and checkerboard pattern. The determination module is used to process the image of each pose based on an edge detection algorithm to obtain a processed binary image; determine the edge line of the calibration board in the image based on the binary image; determine the first coordinates of a preset position in the calibration board in the image based on the edge line; process the point cloud data of the pose based on a plane fitting algorithm to obtain a first plane corresponding to the calibration board; determine the laser point located on the edge line of the calibration board based on the coordinates of adjacent laser points in the point cloud data; determine the second coordinates of the target laser point at the preset position in the calibration board based on the first plane and the laser point located on the edge line of the calibration board, wherein the preset position is the center pixel of the calibration board; and determine the parameters for joint calibration of the camera and the lidar based on each first coordinate, each second coordinate, and a preset data registration algorithm.

6. The apparatus of claim 5, wherein, The determining module is further configured to refine the edge lines based on a thinning algorithm.

7. The apparatus of claim 5, wherein, The determining module is further configured to determine the distance between each laser point in the point cloud data and the first plane; if the distance is within a preset distance range, then the laser point is determined to be a laser point in a subset of the point cloud data. Otherwise, the laser point is determined to be a laser point that is not in the subset of the point cloud data; Based on the subset of point cloud data and the plane fitting algorithm, a second plane corresponding to the calibration board is determined; and the first plane is updated using the second plane.

8. The apparatus of claim 5, wherein, The determining module is specifically used to determine the equation of the straight line corresponding to each edge line of the calibration plate based on the first plane, the laser point located on the edge line of the calibration plate, and Lagrange's theorem; for each pair of adjacent edge lines, the midpoint of the common perpendicular segment of the adjacent edge lines is determined based on the equation of the straight line corresponding to the two adjacent edge lines, and the midpoint of the common perpendicular segment is projected onto the first plane, and the projected midpoint is used as the vertex of the calibration plate; and the second coordinates of the target laser point at the preset position in the calibration plate are determined based on each vertex.

9. An electronic device, comprising: The electronic device includes at least a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the steps of the joint calibration parameter determination method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the steps of the joint calibration parameter determination method according to any one of claims 1-4.

Images