A calibration method and device, electronic equipment and storage medium

Abstract

The application discloses a calibration method and device, electronic equipment and storage medium, the method comprises: obtaining target point cloud data and target image data under the same acquisition time; based on the target point cloud data and the preset calibration data, the first feature point position information is determined, the first feature point position information is the two-dimensional coordinate information of the vertex of the calibration board in the first coordinate system; the target image data is extracted to obtain the second feature point position information; the third feature point position information is obtained; the first feature point position information, the second feature point position information and the third feature point position information are processed by coordinate transformation, and the target calibration data between the laser radar and the camera is generated. By using the embodiments of the application, the influence of the ranging error of the laser radar on the calibration result data can be effectively avoided, and the precision of the calibration data between the laser radar and the camera is improved.

Description

Technical Field

[0001] This invention relates to the field of measurement technology, and in particular to a calibration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Autonomous driving systems, as a fusion of automotive electronics, intelligent control, and internet technologies, acquire information about the vehicle itself and its surrounding driving environment through a perception system. This information then informs decisions and controls the execution system to enable vehicle movement. Various sensors have their own advantages and disadvantages due to their different operating principles and characteristics. To better achieve environmental perception, multi-sensor fusion is typically employed, such as the fusion of LiDAR and cameras. In the LiDAR and camera fusion process, the accuracy of the calibration data between the LiDAR and camera determines the accuracy of the environmental perception.

[0003] In existing technologies, LiDAR is often treated as a 3D measuring instrument. Pixels are directly extracted from camera images, and the 3D coordinates of the corresponding laser points are extracted from the LiDAR point cloud. Points on the edge of the calibration board are used to fit the four corner points of the calibration board, and then the camera detects these four corner points for calibration. LiDAR errors are mainly divided into two parts: angular error and ranging error. Both types of errors are calibrated before the LiDAR leaves the factory, and corresponding compensations are made to the output point cloud. Generally, its angular error is very small, while the ranging error is often on the order of centimeters. The ranging principle of radar is to calculate the distance from the first reflected wave to the radar by multiplying the flight time of the laser beam by the speed of light; therefore, the ranging error depends on the clock error. Existing technologies make it difficult to improve the ranging error accuracy by another order of magnitude, which becomes one of the main factors affecting the accuracy of the calibration data between the LiDAR and the camera, and consequently affecting the accuracy of the sensor's environmental perception. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention discloses a calibration method, apparatus, electronic device, and storage medium that can effectively avoid the interference of lidar ranging errors on calibration result data and improve the accuracy of calibration data between lidar and camera. The technical solution disclosed in this invention is as follows:

[0005] According to one aspect of the embodiments disclosed in this invention, a calibration method is provided, comprising:

[0006] Acquire target point cloud data and target image data at the same acquisition time. The target point cloud data is the point cloud data of the calibration board acquired by the lidar, and the target image data is the image data of the calibration board acquired by the camera. The lidar and the camera are located on the device to be calibrated.

[0007] Based on the target point cloud data and the preset calibration data, the position information of the first feature point is determined. The position information of the first feature point is the two-dimensional coordinate information of the vertex of the calibration board in the first coordinate system.

[0008] Feature points are extracted from the target image data to obtain second feature point position information, which is the two-dimensional coordinate information of the vertex of the calibration board in the second coordinate system.

[0009] Obtain the position information of the third feature point, which is the three-dimensional coordinate information of the vertex of the calibration board in the third coordinate system;

[0010] The coordinate transformation processing is performed on the first feature point position information, the second feature point position information and the third feature point position information to generate target calibration data between the lidar and the camera.

[0011] Optionally, the step of performing coordinate transformation processing on the first feature point position information, the second feature point position information, and the third feature point position information to generate target calibration data between the lidar and the camera includes:

[0012] When the target point cloud data and the target image data are data obtained from a single acquisition, coordinate transformation processing is performed on the first feature point position information and the third feature point position information to generate first calibration data;

[0013] The coordinate transformation process is performed on the position information of the second feature point and the position information of the third feature point to generate the second calibration data;

[0014] The target calibration data is determined based on the first calibration data and the second calibration data.

[0015] Optionally, the step of performing coordinate transformation processing on the first feature point position information, the second feature point position information, and the third feature point position information to generate target calibration data between the lidar and the camera includes:

[0016] When the target point cloud data and the target image data include data acquired multiple times, coordinate transformation processing is performed on the second feature point position information and the third feature point position information to generate the second calibration data;

[0017] Based on the second calibration data, the coordinate transformation processing of the third feature point position information is performed to obtain the fourth feature point position information, which is the three-dimensional coordinate information of the vertex of the calibration board in the second coordinate system.

[0018] The coordinate transformation processing is performed on the position information of the first feature point and the position information of the fourth feature point to generate the target calibration data.

[0019] Optionally, determining the location information of the first feature point based on the target point cloud data and preset calibration data includes:

[0020] Based on the target point cloud data and the preset calibration data, the position information of the fifth feature point is determined, and the position information of the fifth feature point is the two-dimensional coordinate information of the target point cloud data in the first coordinate system.

[0021] Based on the position information of the fifth feature point, the edge corresponding to the calibration board is constructed;

[0022] The position information of the first feature point is determined based on the intersection of the edges corresponding to the calibration plate.

[0023] Optionally, before determining the location information of the first feature point based on the target point cloud data and preset calibration data, the method further includes:

[0024] Feature points are extracted from the target point cloud data to obtain feature point cloud data;

[0025] Accordingly, determining the location information of the first feature point based on the target point cloud data and preset calibration data includes:

[0026] Based on the feature point cloud data and the preset calibration data, the location information of the first feature point is determined.

[0027] Optionally, the method further includes:

[0028] Based on the target calibration data, coordinate transformation processing is performed on the target point cloud data to generate target position information, which is the coordinate information of the target point cloud data in the second coordinate system.

[0029] Optionally, acquiring target point cloud data and target image data at the same acquisition time includes:

[0030] Acquire initial image data, which is image data captured by the camera;

[0031] If the initial image data includes the image data of the calibration board, the target point cloud data is acquired.

[0032] The initial image data is used as the target image data.

[0033] According to another aspect of the disclosed embodiments of the present invention, a calibration apparatus is provided, comprising:

[0034] The data acquisition module is used to acquire target point cloud data and target image data at the same acquisition time. The target point cloud data is the point cloud data of the calibration board acquired by the lidar, and the target image data is the image data of the calibration board acquired by the camera. The lidar and the camera are located on the device to be calibrated.

[0035] The first feature point location information determination module is used to determine the first feature point location information based on the target point cloud data and preset calibration data. The first feature point location information is the two-dimensional coordinate information of the vertex of the calibration board in the first coordinate system.

[0036] The second feature point location information determination module is used to extract feature points from the target image data to obtain second feature point location information, which is the two-dimensional coordinate information of the vertex of the calibration board in the second coordinate system.

[0037] The third feature point position information determination module is used to obtain the third feature point position information, which is the three-dimensional coordinate information of the vertex of the calibration board in the third coordinate system.

[0038] The target calibration data generation module is used to perform coordinate transformation processing on the position information of the first feature point, the position information of the second feature point, and the position information of the third feature point to generate target calibration data between the lidar and the camera.

[0039] According to another aspect of the embodiments disclosed in this invention, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the calibration method as described in any of the preceding claims.

[0040] According to another aspect of the disclosed embodiments of the present invention, a computer-readable storage medium is provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the calibration method described in any one of the disclosed embodiments of the present invention.

[0041] According to another aspect of the disclosed embodiments of the present invention, a computer program product containing instructions is provided that, when run on a computer, causes the computer to perform the calibration method described in any one of the disclosed embodiments of the present invention.

[0042] The technical solutions provided by the embodiments disclosed in this invention bring at least the following beneficial effects:

[0043] The calibration method provided by this invention acquires point cloud data collected by a lidar. By using preset calibration data, the three-dimensional coordinate information of the point cloud data in the lidar coordinate system is converted into two-dimensional coordinate information in the depth image coordinate system. This abstracts the lidar into a 2D angle measurement tool, independent of its ranging information, effectively avoiding the interference of the lidar's ranging error on the calibration result data. Furthermore, based on the two-dimensional coordinate information of the calibration board's vertex in the camera coordinate system, the three-dimensional coordinate information in the calibration board coordinate system, and the two-dimensional coordinate information in the depth image coordinate system, coordinate transformation processing is performed to generate calibration data between the lidar and the camera, improving the accuracy of the calibration data between the lidar and the camera.

[0044] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the disclosure of this invention and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit the scope of this disclosure.

[0046] Figure 1 This is a flowchart illustrating a calibration method according to an exemplary embodiment;

[0047] Figure 2 This is a flowchart illustrating a method for determining the location information of a first feature point according to an exemplary embodiment;

[0048] Figure 3 This is a block diagram illustrating a calibration device according to an exemplary embodiment;

[0049] Figure 4 This is a block diagram illustrating a terminal electronic device for calibration according to an exemplary embodiment;

[0050] Figure 5 This is a block diagram illustrating a server electronic device for calibration according to an exemplary embodiment. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions disclosed in this invention, the technical solutions in the disclosed embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention disclosed herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0053] Analysis of traditional camera-LiDAR calibration methods reveals that, due to the extremely small distance between the transmitting and receiving sections within the LiDAR compared to the measurement distance, the three degrees of freedom of the radar's measurements are not independent. For repetitive scanning LiDARs, the pitch and yaw angles of each beam are fixed at a specified phase of the cycle; for non-repetitive scanning LiDARs, when measuring distance at a specified transmission position, the pitch and yaw angles of the returned point are highly definite as long as the distance is two orders of magnitude larger than the internal distance of the radar's transmitting unit. Therefore, LiDAR can be modeled using a pinhole camera-like model, assuming that all measurement points of the LiDAR are emitted from a single geometric center. Based on this, the LiDAR can be viewed as a virtual camera, which can serve as a 2D measurement tool.

[0054] Specifically, the three-dimensional coordinate information of the point cloud in the lidar coordinate system can be represented as:

[0055]

[0056] Where dist represents the product of the return time of the lidar measurement beam and the speed of light, theta represents the pitch angle of the measurement beam, and yaw represents the yaw angle of the measurement beam.

[0057] After this transformation, the clock error is completely tied to dist, and independent of theta and yaw. As long as calibration can be performed using only theta and yaw, the calibration accuracy is completely independent of the clock error.

[0058] The present invention provides a calibration method that can be applied to the generation of calibration data between lidar and camera.

[0059] The calibration method provided by this invention creatively abstracts the lidar into a 2D angle measurement sensor. Its implementation is independent of the lidar type. By eliminating the unreliable constraints caused by ranging, it can eliminate the problem of inconsistency between the calibration board scale and the real scale caused by the internal clock error of the lidar. Therefore, it is particularly suitable for completing the calibration task with high accuracy requirements.

[0060] Figure 1 This is a flowchart illustrating a calibration method according to an exemplary embodiment, such as... Figure 1 As shown, the calibration method includes the following steps.

[0061] S101: Acquire target point cloud data and target image data at the same acquisition time.

[0062] In one specific embodiment, the target point cloud data can be the point cloud data of the calibration board collected by the lidar. The point cloud data can include point cloud location information. Specifically, the point cloud location information can be the three-dimensional coordinate information of the point cloud in the lidar coordinate system.

[0063] In one specific embodiment, the target image data can be image data of a calibration board acquired by the camera. The image data can include the position information of the vertices of the calibration board. Specifically, the position information of the vertices can be the two-dimensional coordinate information of the vertices in the camera coordinate system. The lidar and the camera are located on the same calibration device. Specifically, the calibration device can include a vehicle, robot, etc.

[0064] In one specific embodiment, the calibration plate can be rectangular in shape, specifically, it can be square in shape. The length s of the side of the calibration plate can be set according to the actual application, for example, s can be 60cm.

[0065] In one specific embodiment, the calibration plate may be coated with ArUco codes, the vertices of which coincide with the vertices of the calibration plate. The vertices of the calibration plate can be located in the camera image using the ArUco codes. Specifically, the ArUco codes can be a type of QR code used for visual positioning.

[0066] In practical applications, the calibration board can be placed in the common field of view of the LiDAR and the camera, and can be completely captured by the LiDAR and the camera.

[0067] In an optional embodiment, acquiring target point cloud data and target image data at the same acquisition time may include:

[0068] Obtain initial image data;

[0069] If the initial image data includes the image data of the calibration board, the target point cloud data is acquired.

[0070] The initial image data is used as the target image data.

[0071] In one specific embodiment, the initial image data may be image data captured by a camera.

[0072] S103: Based on the target point cloud data and preset calibration data, determine the location information of the first feature point.

[0073] In one specific embodiment, the preset calibration data can be the preset calibration data of the LiDAR. Specifically, the preset calibration data can convert the three-dimensional coordinate information of any point in the LiDAR coordinate system into the two-dimensional coordinate information of that point in the depth image coordinate system, which can be the virtual camera coordinate system.

[0074] In one specific embodiment, the first feature point position information is the two-dimensional coordinate information of the vertex of the calibration plate in the first coordinate system. Specifically, the first coordinate system can be the aforementioned depth image coordinate system.

[0075] In one specific embodiment, the two-dimensional coordinates of any point in the first coordinate system can be determined according to the following formula:

[0076]

[0077]

[0078] in, This represents the two-dimensional coordinate information of any point in the first coordinate system. This represents the three-dimensional coordinate information of any point in the lidar coordinate system. Indicates preset parameters. This indicates the above-mentioned preset calibration data. Describing the first coordinate system Focal length along the axial direction, Describing the first coordinate system Focal length along the axial direction, Indicates the photocenter is Offset in the axial direction, Indicates the photocenter is Offset in the axial direction.

[0079] Specifically, preset parameters The settings can be customized according to the actual application. For example, the preset parameters can take the following values. ;parameter , , as well as The settings can be customized according to the actual application. Specifically, the parameters... , , as well as The camera's intrinsic parameters can be set based on the aforementioned camera intrinsic parameters, which are related to the camera's attributes.

[0080] Correspondingly, the two-dimensional coordinate information of any of the above points in the first coordinate system can be represented by the pitch angle and yaw angle of the lidar measuring beam at that point.

[0081] In the above embodiments, the lidar is abstracted as a 2D measurement tool, and the three-dimensional point cloud coordinate information is converted into two-dimensional coordinate information that is only related to the lidar angle information through preset calibration data, and is unrelated to the ranging information. This effectively avoids the interference of the lidar ranging error on the subsequent calibration result data and improves the accuracy of the calibration data between the lidar and the camera.

[0082] In an optional embodiment, Figure 2 This is a flowchart illustrating a method for determining the location information of a first feature point according to an exemplary embodiment, such as... Figure 2 As shown, determining the location information of the first feature point based on the target point cloud data and preset calibration data may include:

[0083] S301: Based on the target point cloud data and the preset calibration data, determine the location information of the fifth feature point.

[0084] In one specific embodiment, the fifth feature point location information is the two-dimensional coordinate information of the target point cloud data in the first coordinate system.

[0085] In one specific embodiment, determining the location information of the fifth feature point based on the target point cloud data and the preset calibration data may include multiplying the target point cloud data with the preset calibration data to determine the location information of the fifth feature point.

[0086] S303: Based on the position information of the fifth feature point, construct the edge corresponding to the calibration board.

[0087] In one specific embodiment, the edge corresponding to the calibration plate can be the edge corresponding to the calibration plate constructed in the first coordinate system, which can be considered as drawing the edge corresponding to the calibration plate on the lidar image.

[0088] In one specific embodiment, in the lidar image, the four sides L1, L2, L3, and L4 of the calibration board are drawn respectively, where L1 represents the line connecting the first and second points of the ArUco code, L2 represents the line connecting the second and third points, and so on.

[0089] In one specific embodiment, constructing the edge corresponding to the calibration board based on the position information of the fifth feature point may include connecting lines based on the position information of the fifth feature point to construct the edge corresponding to the calibration board.

[0090] S305: Determine the position information of the first feature point based on the intersection of the edges corresponding to the calibration plate.

[0091] In one specific embodiment, determining the position information of the first feature point based on the intersection of the edges corresponding to the calibration plate may include using the position information of the intersection of the edges corresponding to the calibration plate as the position information of the first feature point.

[0092] In one specific embodiment, the position information of the above-mentioned intersection points p1, p2, p3, and p4 on the lidar image can be obtained by connecting the four sides of the calibration plate to the intersection points p1, p2, p3, and p4. That is, the two-dimensional coordinate information of the vertex of the calibration plate in the above-mentioned depth image coordinate system.

[0093] In the above embodiments, the edges corresponding to the calibration board are constructed in the first coordinate system, and the coordinate information of the calibration board vertices is determined based on the intersection of the edges, which can improve the accuracy of the calibration board vertex position information.

[0094] In an optional embodiment, before determining the location information of the first feature point based on the target point cloud data and preset calibration data, the method further includes:

[0095] Feature points are extracted from the target point cloud data to obtain feature point cloud data;

[0096] Accordingly, determining the location information of the first feature point based on the target point cloud data and preset calibration data includes:

[0097] Based on the feature point cloud data and the preset calibration data, the location information of the first feature point is determined.

[0098] In one specific embodiment, extracting feature points from the target point cloud data to obtain feature point cloud data may include: determining edge point cloud data based on the target point cloud data; and performing sparsification processing on the edge point cloud data to obtain feature point cloud data.

[0099] In one specific embodiment, the edge point cloud data can be the point cloud data of the calibration board edge collected by the lidar.

[0100] In one specific embodiment, the sparsification of the edge point cloud data to obtain feature point cloud data may include sparsifying the edge point cloud data based on a preset distance to obtain feature point cloud data; specifically, point cloud data with a distance between two adjacent points exceeding the preset distance are selected as feature point cloud data. The preset distance can be set according to the actual application situation, for example, the preset distance can be 2 meters.

[0101] In the above embodiments, feature point extraction of the target point cloud data can effectively remove the interference of abnormal collection points, making the coordinate information of the calibration board vertices obtained later more accurate.

[0102] S105: Extract feature points from the target image data to obtain the second feature point location information.

[0103] In one specific embodiment, the second feature point position information can be the two-dimensional coordinate information of the vertex of the calibration board in the second coordinate system. Specifically, the second coordinate system can be the camera coordinate system.

[0104] In one specific embodiment, the four vertices of the calibration board in the target image data are q1, q2, q3, and q4, which correspond to the aforementioned intersection points p1, p2, p3, and p4 on the lidar image.

[0105] In one specific embodiment, extracting feature points from the target image data to obtain the second feature point location information may include extracting feature points from the target image data based on a preset feature extraction algorithm to obtain the second feature point location information; specifically, the preset feature extraction algorithm may be an algorithm capable of corner detection and extraction, such as the Harris corner detection algorithm, the Shi-Tomasi corner detection algorithm, etc.

[0106] S107: Obtain the location information of the third feature point.

[0107] In one specific embodiment, the third feature point position information can be the three-dimensional coordinate information of the vertex of the calibration board in the third coordinate system.

[0108] In one specific embodiment, the third coordinate system can be the calibration plate coordinate system. Specifically, the calibration plate coordinate system can take any vertex of the calibration plate as its origin, and the two edges of the calibration plate connected to that vertex as its origin. shaft and Axis, with Axis cross product Axis as The axes are used to establish the calibration; for example, if the four vertices of the calibration plate are denoted as r1, r2, r3, and r4 respectively, then... The axis direction can be taken as r1 pointing to r2. The axis direction can be taken as r1 pointing to r4. Take the axial direction Axis cross product The three-dimensional coordinate information of the calibration plate vertex in the calibration plate coordinate system can be represented as: (0,0,0), (s,0,0), (s,s,0), ​​(0,s,0), ​​where s is the side length of the calibration plate.

[0109] S109: Perform coordinate transformation processing on the first feature point position information, the second feature point position information and the third feature point position information to generate target calibration data between the lidar and the camera.

[0110] In one specific embodiment, the target calibration data can convert the coordinate information of any point in the lidar coordinate system into the coordinate information of that point in the camera coordinate system.

[0111] In an optional embodiment, the step of performing coordinate transformation processing on the first feature point position information, the second feature point position information, and the third feature point position information to generate target calibration data between the lidar and the camera may include:

[0112] When the target point cloud data and the target image data are data obtained from a single acquisition, coordinate transformation processing is performed on the first feature point position information and the third feature point position information to generate first calibration data;

[0113] The coordinate transformation process is performed on the position information of the second feature point and the position information of the third feature point to generate the second calibration data;

[0114] The target calibration data is determined based on the first calibration data and the second calibration data.

[0115] In one specific embodiment, the first calibration data can be the calibration data between the calibration board and the lidar. Specifically, the first calibration data can be used to convert the coordinate information of any point in the lidar coordinate system to the coordinate information of that point in the calibration board coordinate system.

[0116] In one specific embodiment, the second calibration data can be the calibration data between the calibration board and the camera. Specifically, the second calibration data can be used to convert the coordinate information of any point in the camera coordinate system to the coordinate information of that point in the calibration board coordinate system.

[0117] In an optional embodiment, when the target point cloud data and the target image data are data acquired in a single acquisition, performing coordinate transformation processing on the first feature point position information and the third feature point position information to generate first calibration data may include: when the target point cloud data and the target image data are data acquired in a single acquisition, performing coordinate transformation processing on the first feature point position information and the third feature point position information based on a preset algorithm and preset calibration data to generate first calibration data.

[0118] In one specific embodiment, the aforementioned preset algorithm can be an algorithm capable of coordinate transformation; specifically, the preset algorithm can be PnP (Perspective P). n Point (Perspective Multi-Point Projection) algorithm.

[0119] In an optional embodiment, performing coordinate transformation processing on the second feature point position information and the third feature point position information to generate second calibration data may include: performing coordinate transformation processing on the second feature point position information and the third feature point position information based on a preset algorithm and camera intrinsic parameter data to generate second calibration data.

[0120] In an optional embodiment, the target calibration data can be determined according to the following formula:

[0121]

[0122] in, This refers to the first calibration data mentioned above. This refers to the second calibration data mentioned above.

[0123] In an optional embodiment, the step of performing coordinate transformation processing on the first feature point position information, the second feature point position information, and the third feature point position information to generate target calibration data between the lidar and the camera may further include:

[0124] When the target point cloud data and the target image data include data acquired multiple times, coordinate transformation processing is performed on the second feature point position information and the third feature point position information to generate the second calibration data;

[0125] Based on the second calibration data, coordinate transformation processing is performed on the position information of the third feature point to obtain the position information of the fourth feature point.

[0126] The coordinate transformation processing is performed on the position information of the first feature point and the position information of the fourth feature point to generate the target calibration data.

[0127] In one specific embodiment, the fourth feature point position information can be the three-dimensional coordinate information of the vertex of the calibration board in the second coordinate system.

[0128] In an optional embodiment, performing coordinate transformation processing on the first feature point position information and the fourth feature point position information to generate the target calibration data may include: performing coordinate transformation processing on the first feature point position information and the fourth feature point position information based on a preset algorithm, preset calibration data, and camera intrinsic parameter data to generate the target calibration data.

[0129] In practical applications, the above target calibration data can be generated by coordinate transformation processing using the computer vision software OpenCV. Furthermore, the influence of outliers can be removed using the solve PnP Ransac method in OpenCV, making the target calibration data more accurate, robust, and less susceptible to the influence of small angular errors.

[0130] In the above embodiments, when the target point cloud data and target image data include data acquired multiple times, an overall coordinate transformation is performed on the multiple acquired data to improve the generation efficiency of calibration data.

[0131] In an optional embodiment, the method further includes:

[0132] Based on the target calibration data, coordinate transformation processing is performed on the target point cloud data to generate target location information.

[0133] In one specific embodiment, the target location information may be the coordinate information of the target point cloud data in the second coordinate system.

[0134] In one specific embodiment, generating target location information by performing coordinate transformation processing on target point cloud data based on target calibration data may include multiplying the target calibration data with the target point cloud data to obtain the target location information.

[0135] As can be seen from the technical solutions provided in the embodiments of this specification above, this specification converts the three-dimensional coordinate information of point cloud data in the lidar coordinate system into two-dimensional coordinate information in the depth image coordinate system by pre-setting calibration data. This abstracts the lidar into a 2D angle measurement tool, independent of its ranging information, effectively avoiding the interference of lidar ranging errors on the calibration results. Furthermore, coordinate transformation is performed based on the two-dimensional coordinate information of the calibration board's vertices in the camera coordinate system, the three-dimensional coordinate information in the calibration board coordinate system, and the two-dimensional coordinate information in the depth image coordinate system to generate calibration data between the lidar and the camera. This improves the accuracy of the calibration data between the lidar and the camera, while also enhancing operational convenience. In addition, the two-dimensional coordinate information in the depth image coordinate system is only related to the lidar angle information, and the angle information is independent of the lidar type, exhibiting good universality. Moreover, feature point extraction from the point cloud data effectively removes interference from abnormal acquisition points, making the coordinate information of the calibration board vertices more accurate.

[0136] Figure 3 This is a block diagram illustrating a calibration device according to an exemplary embodiment. (Refer to...) Figure 3 The device includes:

[0137] The data acquisition module 410 is used to acquire target point cloud data and target image data at the same acquisition time. The target point cloud data is the point cloud data of the calibration board acquired by the lidar, and the target image data is the image data of the calibration board acquired by the camera. The lidar and the camera are located on the device to be calibrated.

[0138] The first feature point location information determination module 420 is used to determine the first feature point location information based on the target point cloud data and preset calibration data. The first feature point location information is the two-dimensional coordinate information of the vertex of the calibration board in the first coordinate system.

[0139] The second feature point position information determination module 430 is used to extract feature points from the target image data to obtain second feature point position information, wherein the second feature point position information is the two-dimensional coordinate information of the vertex of the calibration plate in the second coordinate system.

[0140] The third feature point position information determination module 440 is used to obtain the third feature point position information, which is the three-dimensional coordinate information of the vertex of the calibration plate in the third coordinate system.

[0141] The target calibration data generation module 450 is used to perform coordinate transformation processing on the first feature point position information, the second feature point position information and the third feature point position information to generate target calibration data between the lidar and the camera.

[0142] Optionally, the target calibration data generation module 450 may include:

[0143] The first calibration data generation unit is used to perform coordinate transformation processing on the first feature point position information and the third feature point position information to generate first calibration data when the target point cloud data and the target image data are data obtained from a single acquisition.

[0144] The second calibration data generation unit is used to perform coordinate transformation processing on the second feature point position information and the third feature point position information to generate second calibration data.

[0145] The first target calibration data generation unit is used to determine the target calibration data based on the first calibration data and the second calibration data.

[0146] Optionally, the target calibration data generation module 450 may also include:

[0147] A coordinate transformation processing unit is used to perform coordinate transformation processing on the second feature point position information and the third feature point position information when the target point cloud data and the target image data include data acquired multiple times, to generate the second calibration data.

[0148] The fourth feature point position information determination unit is used to perform coordinate transformation processing on the third feature point position information based on the second calibration data to obtain the fourth feature point position information, wherein the fourth feature point position information is the three-dimensional coordinate information of the vertex of the calibration board in the second coordinate system;

[0149] The second target calibration data generation unit is used to perform coordinate transformation processing on the position information of the first feature point and the position information of the fourth feature point to generate the target calibration data.

[0150] Optionally, the first feature point location information determination module 420 may include:

[0151] The fifth feature point location information determination unit is used to determine the fifth feature point location information based on the target point cloud data and the preset calibration data. The fifth feature point location information is the two-dimensional coordinate information of the target point cloud data in the first coordinate system.

[0152] The construction unit is used to construct the edge corresponding to the calibration board based on the position information of the fifth feature point;

[0153] The intersection point determination unit is used to determine the position information of the first feature point based on the intersection point of the edge corresponding to the calibration plate.

[0154] Optionally, the device may further include:

[0155] The feature point cloud data determination module is used to extract feature points from the target point cloud data to obtain feature point cloud data.

[0156] Correspondingly, the first feature point location information determination module 420 may also include:

[0157] The first feature point location information determination unit is used to determine the location information of the first feature point based on the feature point cloud data and the preset calibration data.

[0158] Optionally, the device may further include:

[0159] The target location information generation module is used to perform coordinate transformation processing on the target point cloud data based on the target calibration data to generate target location information, wherein the target location information is the coordinate information of the target point cloud data in the second coordinate system.

[0160] Optionally, the data acquisition module 410 may include:

[0161] An initial image data acquisition unit is used to acquire initial image data, which is image data captured by the camera.

[0162] The target point cloud data acquisition unit is used to acquire the target point cloud data when the initial image data includes the image data of the calibration board;

[0163] A target image data determination unit is used to use the initial image data as the target image data.

[0164] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0165] Figure 4 This is a block diagram illustrating a terminal electronic device for calibration according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a calibration method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0166] Figure 5 This is a block diagram illustrating a server electronic device for calibration according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a calibration method.

[0167] Those skilled in the art will understand that Figure 4 or Figure 5 The structures shown are merely block diagrams of some structures related to the disclosed solutions of this invention, and do not constitute a limitation on the electronic devices to which the disclosed solutions of this invention are applied. Specific electronic devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0168] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the calibration method as disclosed in the embodiments of the present invention.

[0169] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the calibration method in the disclosed embodiments of the present invention.

[0170] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the calibration method in the disclosed embodiments of the present invention.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0172] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles disclosed herein and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0173] It should be understood that the present invention is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A calibration method characterized by, The method comprises: acquiring target point cloud data and target image data at the same acquisition time, the target point cloud data being point cloud data of a calibration board collected by a laser radar, and the target image data being image data of the calibration board collected by a camera; determining first feature point position information based on the target point cloud data and preset calibration data, the first feature point position information being two-dimensional coordinate information of vertices of the calibration board in a first coordinate system; the first coordinate system being a depth image coordinate system; extracting feature points from the target image data to obtain second feature point position information, the second feature point position information being two-dimensional coordinate information of the vertices of the calibration board in a second coordinate system; the second coordinate system being a camera coordinate system; acquiring third feature point position information, the third feature point position information being three-dimensional coordinate information of the vertices of the calibration board in a third coordinate system; the third coordinate system being a calibration board coordinate system; performing coordinate transformation processing on the first feature point position information, the second feature point position information and the third feature point position information to generate target calibration data between the laser radar and the camera, including: in the case that the target point cloud data and the target image data comprise data acquired multiple times, performing coordinate transformation processing on the second feature point position information and the third feature point position information to generate second calibration data; based on the second calibration data, performing coordinate transformation processing on the third feature point position information to obtain fourth feature point position information, the fourth feature point position information being three-dimensional coordinate information of the vertices of the calibration board in the second coordinate system; performing coordinate transformation processing on the first feature point position information and the fourth feature point position information to generate the target calibration data.

2. The method of claim 1, wherein The coordinate transformation processing on the first feature point position information, the second feature point position information and the third feature point position information to generate the target calibration data between the laser radar and the camera further comprises: in the case that the target point cloud data and the target image data are data acquired once, performing coordinate transformation processing on the first feature point position information and the third feature point position information to generate first calibration data; performing coordinate transformation processing on the second feature point position information and the third feature point position information to generate second calibration data; determining the target calibration data based on the first calibration data and the second calibration data.

3. The method of claim 1, wherein The determination of the first feature point position information based on the target point cloud data and preset calibration data comprises: determining fifth feature point position information based on the target point cloud data and the preset calibration data, the fifth feature point position information being two-dimensional coordinate information of the target point cloud data in the first coordinate system; constructing edges corresponding to the calibration board based on the fifth feature point position information; determining the first feature point position information based on intersection points of the edges corresponding to the calibration board.

4. The method of claim 1, wherein Before the determination of the first feature point position information based on the target point cloud data and preset calibration data, the method further comprises: Feature point extraction is performed on the target point cloud data to obtain feature point cloud data; The first feature point position information is determined based on the target point cloud data and preset calibration data. The first feature point position information is determined based on the feature point cloud data and the preset calibration data.

5. The method of claim 1, wherein The method further includes: The target point cloud data and the target image data obtained at the same acquisition time include:

6. The method of claim 1, wherein The initial image data is obtained, and the initial image data is image data acquired by a camera; In a case where the initial image data includes image data of the calibration board, the target point cloud data is obtained; The initial image data is taken as the target image data. The method includes:

7. A calibration device, characterized by A data acquisition module is configured to obtain target point cloud data and target image data at the same acquisition time, the target point cloud data being point cloud data of a calibration board acquired by a laser radar, and the target image data being image data of the calibration board acquired by a camera, the laser radar and the camera being located on a device to be calibrated; A first feature point position information determination module is configured to determine first feature point position information based on the target point cloud data and preset calibration data, the first feature point position information being two-dimensional coordinate information of vertices of the calibration board in a first coordinate system; the first coordinate system being a depth image coordinate system; A second feature point position information determination module is configured to perform feature point extraction on the target image data to obtain second feature point position information, the second feature point position information being two-dimensional coordinate information of the vertices of the calibration board in a second coordinate system; the second coordinate system being a camera coordinate system; A third feature point position information determination module is configured to obtain third feature point position information, the third feature point position information being three-dimensional coordinate information of the vertices of the calibration board in a third coordinate system; The third coordinate system is a calibration board coordinate system; A target calibration data generation module is configured to perform coordinate transformation processing on the first feature point position information, the second feature point position information, and the third feature point position information to generate target calibration data between the laser radar and the camera; The target calibration data generation module includes: A coordinate transformation processing unit is configured to perform coordinate transformation processing on the second feature point position information and the third feature point position information to generate the second calibration data in a case where the target point cloud data and the target image data include data obtained through multiple acquisitions; A fourth feature point position information determination unit is configured to perform coordinate transformation processing on the third feature point position information based on the second calibration data to obtain fourth feature point position information, the fourth feature point position information being three-dimensional coordinate information of the vertices of the calibration board in the second coordinate system; ​ The second target calibration data generation unit is configured to perform coordinate transformation on the first feature point position information and the fourth feature point position information to generate the target calibration data.

8. An electronic device, comprising: Comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the calibration method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the calibration method of any one of claims 1 to 6.

Images