An external parameter calibration method, device and electronic equipment

Abstract

The present application relates to a kind of external parameter calibration method, device and electronic equipment, the external parameter calibration method includes: obtaining multiple frames containing texture feature plane point cloud and multiple frames containing texture feature plane image;According to multiple images, estimate three-dimensional structure point and the initial pose of camera;Plane extraction is carried out to three-dimensional structure point, and the plane pixel point in multiple images is obtained, wherein, plane pixel point is the pixel point generated by texture feature plane;According to the plane pixel point in multiple images, optimize initial pose to obtain camera pose, and obtain the first plane equation in world coordinate system;Plane extraction is carried out to point cloud, and the second plane equation is obtained;According to the first plane equation and the second plane equation and camera pose, the external parameter of laser radar camera is obtained, to obtain more accurate external parameter of laser radar camera.

Description

Technical Field

[0001] This invention relates to the field of external parameter calibration technology, and in particular to an external parameter calibration method, apparatus and electronic device. Background Technology

[0002] In robotic systems, LiDAR and cameras are the most commonly used sensors, complementing each other to compensate for texture information and the 3D measurement environment. Extrinsic parameter calibration of LiDAR and cameras is crucial for placing texture information and the 3D measurement environment into the same coordinate system, especially suitable for scenarios performing automated tasks in complex environments. Many existing extrinsic parameter calibration methods rely on pre-prepared special calibration objects, such as checkerboard patterns. Summary of the Invention

[0003] This invention provides an external parameter calibration method, apparatus, and electronic device that can obtain more accurate external parameters of laser radar and cameras.

[0004] To solve the above-mentioned technical problems, one technical solution adopted in this invention is: providing an extrinsic parameter calibration method applied to an electronic device, wherein the electronic device is equipped with a camera and a lidar, and the same textured plane exists within the camera's imaging range and the lidar's scanning range. The extrinsic parameter calibration method includes:

[0005] Acquire point clouds and images containing texture feature planes from multiple frames;

[0006] The three-dimensional structure points and the initial pose of the camera are estimated based on the images from multiple frames.

[0007] The three-dimensional structural points are extracted by plane to obtain planar pixels in multiple frames of the image, wherein the planar pixels are pixels generated by the texture feature plane;

[0008] Based on the planar pixels in the multiple frames of the image, the initial pose is optimized to obtain the camera pose, and the first plane equation in the world coordinate system is obtained.

[0009] The point cloud is subjected to planar extraction to obtain the second plane equation;

[0010] The extrinsic parameters of the laser radar to the camera are obtained based on the first plane equation, the second plane equation, and the camera pose.

[0011] In some embodiments, the step of performing planar extraction on the three-dimensional structural points to obtain planar pixels in multiple frames of the image includes:

[0012] Planar structure points are obtained by performing planar extraction on the three-dimensional structure points, wherein the planar structure points are the points corresponding to the texture feature plane in the three-dimensional structure points;

[0013] The three-dimensional plane and the planar pixel points are obtained based on the planar structure points.

[0014] In some embodiments, optimizing the initial pose to obtain the camera pose based on planar pixels in multiple frames of the images includes:

[0015] Based on the initial pose, the planar pixels in the previous frame image are projected onto the current frame image to obtain the two-dimensional projection points in the current frame image;

[0016] Minimize the pixel error between the planar pixels of the current frame and the two-dimensional projection points to optimize the initial pose and obtain the camera pose.

[0017] In some embodiments, obtaining the extrinsic parameters of the lidar to the camera based on the first plane equation, the second plane equation, and the camera pose includes:

[0018] Based on the camera pose, the first plane equation in the world coordinate system is transformed to the camera coordinate system to obtain the third plane equation;

[0019] Based on the second and third plane equations, the extrinsic parameters of the laser radar to the camera are obtained.

[0020] To solve the above-mentioned technical problems, another technical solution adopted in the embodiments of the present invention is: providing an extrinsic parameter calibration device applied to an electronic device, the electronic device being equipped with a camera and a lidar, wherein the camera's imaging range and the lidar's scanning range share the same textured plane, the extrinsic parameter calibration device comprising:

[0021] The acquisition module is used to acquire point clouds and images containing texture feature planes from multiple frames.

[0022] An estimation module is used to estimate the three-dimensional structure points and the initial pose of the camera based on multiple frames of the image.

[0023] The first planar extraction module is used to extract the three-dimensional structural points into a planar shape to obtain planar pixels in multiple frames of the image, wherein the planar pixels are pixels generated by the texture feature plane;

[0024] An optimization module is used to optimize the initial pose based on planar pixels in multiple frames of the image to obtain the camera pose and obtain the first plane equation in the world coordinate system.

[0025] The second plane extraction module is used to extract the plane from the point cloud to obtain the second plane equation;

[0026] The calculation module is used to obtain the extrinsic parameters of the lidar to the camera based on the first plane equation, the second plane equation, and the camera pose.

[0027] In some embodiments, the first planar extraction module includes:

[0028] A planar extraction unit is used to obtain planar structure points by performing planar extraction on the three-dimensional structure points, wherein the planar structure points are points corresponding to the texture feature plane in the three-dimensional structure points;

[0029] The acquisition unit is used to obtain the three-dimensional plane and the plane pixel points based on the planar structure points.

[0030] In some embodiments, the optimization module includes:

[0031] The projection unit is used to project planar pixels from the previous frame image onto the current frame image according to the initial pose, so as to obtain two-dimensional projection points in the current frame image.

[0032] An optimization unit is used to minimize the pixel error between the planar pixels of the current frame and the two-dimensional projection points in order to optimize the initial pose to obtain the camera pose.

[0033] In some embodiments, the computing module includes:

[0034] The transformation unit is used to transform the first plane equation in the world coordinate system to the camera coordinate system to obtain the third plane equation based on the camera pose.

[0035] The calculation unit is used to obtain the extrinsic parameters of the laser radar to the camera based on the second plane equation and the third plane equation.

[0036] To solve the above-mentioned technical problems, another technical solution adopted in the embodiments of the present invention is: to provide an electronic device, comprising:

[0037] Cameras and LiDAR;

[0038] At least one processor, which is communicatively connected to both the camera and the lidar;

[0039] The memory communicatively connected to the at least one processor, wherein,

[0040] The memory stores instructions that can be executed by the at least one processor, which enable the at least one processor to perform the external parameter calibration method as described above.

[0041] To solve the above-mentioned technical problems, another technical solution adopted in the embodiments of the present invention is to provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, the computer-executable instructions being used to cause an electronic device to perform the method described above.

[0042] Unlike related technologies, this invention provides an extrinsic parameter calibration method, apparatus, and electronic device. The extrinsic parameter calibration method includes: acquiring multiple frames of point clouds containing texture feature planes and multiple frames of images containing texture feature planes; estimating three-dimensional structure points and the initial pose of the camera based on the multiple frames of images; performing plane extraction on the three-dimensional structure points to obtain planar pixels in the multiple frames of images, wherein the planar pixels are pixels generated by the texture feature planes; optimizing the initial pose based on the planar pixels in the multiple frames of images to obtain the camera pose and obtaining a first plane equation in the world coordinate system; performing plane extraction on the point cloud to obtain a second plane equation; and obtaining the extrinsic parameters of the LiDAR to the camera based on the first plane equation, the second plane equation, and the camera pose. This method enables the acquisition of more accurate extrinsic parameters of the LiDAR to the camera. Attached Figure Description

[0043] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0044] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart illustrating an external parameter calibration method provided in an embodiment of the present invention;

[0046] Figure 3 This is a flowchart illustrating the method for extracting planar pixels from multiple frames of images by performing planar extraction of three-dimensional structural points according to an embodiment of the present invention.

[0047] Figure 4 This is a flowchart illustrating a method for obtaining a camera pose by optimizing the initial pose based on planar pixels in multiple frames of images, as provided in an embodiment of the present invention.

[0048] Figure 5 This is a flowchart illustrating a method for obtaining the pose of a laser radar to a camera based on a first plane equation, a second plane equation, and the camera pose, according to an embodiment of the present invention.

[0049] Figure 6 This is a schematic diagram of the structure of an external parameter calibration device provided in an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of the structure of the first planar extraction module provided in an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the structure of the optimization module provided in an embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram of the structure of the computing module provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] It should be noted that, unless otherwise specified, the various features in the embodiments of the present invention can be combined with each other, and all are within the protection scope of the present invention. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device schematic diagram or the order in the flowchart.

[0055] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0056] Camera-radar extrinsic parameter calibration methods are mainly divided into calibration with calibration objects and calibration without calibration objects. Calibration without calibration objects does not rely on a pre-made calibration board, but instead seeks common scene features in the radar and camera's field of view for extrinsic parameter calibration. For example, calibration can be performed using mutual information, Renyi entropy, or SFM-based (3D reconstruction-based) and motion-based methods. These methods have certain requirements for the scene. For example, when using mutual information methods for calibration, data from the LiDAR's intensity (reflection intensity) channel is required, and the measurement accuracy needs to be high, requiring rich intensity information from both the radar and camera in the scene. When using SFM-based methods for calibration, dense mapping is required, and the texture information of objects in the scene also needs to be richer. These methods also have requirements for the sensor's motion trajectory. For example, when using motion-based and SFM-based methods for calibration, the camera and radar need to rotate and translate sufficiently in 3D space. However, in most cases, autonomous vehicles can only move on a two-dimensional plane in smaller scenarios. In this situation, the rotation in the roll and pitch directions is insufficient, and the translation in the yaw direction is also insufficient. This may lead to the inability to calculate the extrinsic parameters. Furthermore, in low-cost autonomous driving, low-line-count LiDAR is a commonly used sensor, while some of the methods mentioned above are based on high-line-count LiDAR for extrinsic parameter calibration. Applying them to low-line-count LiDAR will significantly reduce accuracy.

[0057] Currently, much of the work focuses on extrinsic parameter calibration methods based on sparse radar (e.g., Velodyne), which achieve good calibration results on such sparse radars. However, with the development of sensors in the field of autonomous driving, radar resolution has increased significantly, and high-resolution radars (e.g., Livox radar) are now commonly used. Existing extrinsic parameter calibration methods based on sparse radar are not suitable for dense radar. First, directly applying previous calibration methods based on sparse radar to such high-resolution radar presents certain challenges. Because the data structure of dense radar is different, applying methods originally used for sparse radar may result in the loss of the advantages of dense data. Second, the installation positions of the LiDAR and camera need to be adjusted according to actual perception requirements, which can also cause some methods to fail. Third, some methods that utilize boundary information in the environment for extrinsic parameter calibration require the LiDAR to be tilted upwards, but this is not applicable to downward-mounted LiDARs.

[0058] Therefore, embodiments of the present invention provide an extrinsic parameter calibration method, apparatus, and electronic device. The extrinsic parameters of the lidar to the camera can be obtained by simultaneously scanning and calculating a texture feature plane with a camera and a lidar. Embodiments of the present invention are applicable to dense lidar, and can automatically update the extrinsic parameters of the lidar to the camera even when the positions of the lidar and the camera are adjusted. It is also applicable to lidars installed downwards.

[0059] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 200 includes: a camera 201, a lidar 202, at least one processor 203, and a memory 204. Figure 1 Taking a bus connection and a processor as an example, the processor 203 is communicatively connected to the camera 201 and the lidar 202 respectively.

[0060] Specifically, the camera 201 and the lidar 202 are fixed to the body of the electronic device 200. In this embodiment, the lidar 202 can be a dense lidar, such as a Livox lidar. The camera 201 can be a monocular camera, and the electronic device 200 can be a car or a robot, etc. Before the lidar 202 scans the point cloud and the camera 201 captures the image, there is a common plane with textured features within the imaging range of the camera 201 and the scanning range of the lidar 202. In some embodiments, this plane can be a floor tile, a textured wall, etc. In other embodiments, this plane can also be artificially constructed.

[0061] The electronic device 200 can be any type of electronic device with computing capabilities, such as an autonomous vehicle or a robot.

[0062] The memory 204 communicates with at least one processor 203, wherein the memory 204 stores instructions that can be executed by at least one processor 203, the instructions being executed by at least one processor 203 to enable at least one processor 203 to perform the extrinsic calibration method as described below.

[0063] The communication connection can be a wired connection, such as fiber optic cable or serial communication bus (CAN bus), or a wireless communication connection, such as WIFI connection, Bluetooth connection, 4G wireless communication connection, 5G wireless communication connection, etc.

[0064] The processor 203 is used to provide computing and control capabilities to control the electronic device 200 to perform corresponding tasks, such as controlling the electronic device 200 to perform any of the external parameter calibration methods provided in the following embodiments of the invention.

[0065] It is understood that processor 203 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0066] Memory 204, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the extrinsic parameter calibration method in the embodiments of the present invention. Processor 203 can implement the extrinsic parameter calibration method in any of the following method embodiments by running the non-transitory software programs, instructions, and modules stored in memory 204. Specifically, memory 204 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 204 may also include memory remotely located relative to processor 203, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0067] The following provides a detailed description of the external parameter calibration method provided in the embodiments of the present invention. Please refer to [link / reference]. Figure 2 This extrinsic parameter calibration method is applied to an electronic device equipped with a camera and a lidar, wherein the same textured plane exists within the camera's field of view and the lidar's scanning range. The extrinsic parameter calibration method includes, but is not limited to, the following steps:

[0068] Step S1: Obtain point clouds containing texture feature planes in multiple frames and images containing texture feature planes in multiple frames.

[0069] The texture feature plane refers to the plane with texture features mentioned above. This texture feature typically refers to a repeating pattern or repeating texture. The pattern can be a rectangle, rhombus, circle, or other shapes; it can also be a straight line, a wavy line, or other patterns.

[0070] The textured surface can be a textured surface, a textured wall, or other similar surface.

[0071] LiDAR uses laser light to illuminate a target (texture feature plane) and can collect a point cloud. This point cloud reflects the spatial coordinates of each laser spot hitting the texture feature plane. From this point cloud, the shape features of the texture feature plane and the distance between the LiDAR and the texture feature plane can be obtained.

[0072] It is understandable that the camera captures the texture feature plane, thereby acquiring an image of that texture feature plane.

[0073] It is important to note that for this textured plane, the position and angle of the sensor need to be varied to acquire multiple frames of point cloud and multiple frames of image, and the textured plane needs to be clearly visible in each frame of point cloud and each frame of image.

[0074] In practical applications, if the LiDAR used is the aforementioned Livox LiDAR, then when collecting point clouds, the LiDAR needs to be placed stationary in front of the texture feature plane for several seconds to obtain dense point clouds.

[0075] Step S2: Estimate the 3D structure points and the initial pose of the camera based on multiple frames of images.

[0076] Here, the 3D structure point is the point corresponding to multiple pixels in each frame of the image in the world coordinate system. The initial pose of the camera is the rotation matrix from the world coordinate system to the camera coordinate system in each frame of the image.

[0077] Specifically, 3D reconstruction is performed on each of the multiple frames to obtain the 3D structural points and the initial pose of the camera.

[0078] It is understandable that an image is composed of a matrix of pixels. For multiple frames of images from different angles, by analyzing the differences in the pixel matrix presented in the images from different angles, it is possible to deduce the spatial location information of the three-dimensional structure reflected in this set of two-dimensional images.

[0079] Specifically, the SFM (Structure From Motion) algorithm can be used to reconstruct the three-dimensional images from these multiple frames.

[0080] Step S3: Perform planar extraction on the three-dimensional structure points to obtain planar pixels in multiple frames of images, where the planar pixels are pixels generated by the texture feature plane.

[0081] The three-dimensional structural points include those on the texture feature plane and those on non-texture feature planes. In this embodiment, firstly, the three-dimensional structural points on the texture feature plane need to be extracted, that is, planar extraction is performed on the three structural points. Then, based on the three-dimensional structural points on the texture feature plane, the planar pixels in each frame of the image are obtained.

[0082] Step S4: Based on the planar pixels in multiple frames of images, optimize the initial pose to obtain the camera pose, and obtain the first plane equation in the world coordinate system.

[0083] The camera pose is the rotation matrix from the world coordinate system to the camera coordinate system in each frame of the image obtained after optimizing the initial pose.

[0084] The first plane equation is the plane equation of the texture feature plane in the world coordinate system.

[0085] Step S5: Extract the plane from the point cloud to obtain the second plane equation.

[0086] The second plane equation is the plane equation of the texture feature plane in the radar coordinate system.

[0087] Each frame of point cloud contains point clouds corresponding to the texture feature plane and point clouds not corresponding to the texture feature plane.

[0088] For multi-frame point clouds, firstly, it is necessary to extract the point cloud corresponding to the texture feature plane in each frame, that is, to perform plane extraction on the point cloud. Then, based on the point cloud corresponding to the texture feature plane, the equation of the second plane is obtained.

[0089] Step S6: Obtain the extrinsic parameters of the laser radar to the camera based on the first plane equation, the second plane equation, and the camera pose.

[0090] The extrinsic parameter calibration method provided in this invention involves simultaneously acquiring multiple frames of images and multiple frames of point clouds on a texture feature plane using a camera and a lidar. Based on the multiple frames of images, an optimized camera pose and a first plane equation are obtained. Then, a second plane equation is obtained based on the multiple frames of point clouds. Finally, the extrinsic parameters of the lidar to the camera are calculated based on the camera pose, the first plane equation, and the second plane equation. This extrinsic parameter calibration method does not rely on a pre-prepared special calibration object but instead depends on a texture feature plane, making it more widely applicable. Furthermore, this extrinsic parameter calibration method is also suitable for dense lidar systems.

[0091] In some embodiments, please refer to Figure 3 Step S3 specifically includes:

[0092] Step S31: Obtain planar structure points by performing planar extraction on the three-dimensional structure points, where the planar structure points are the points corresponding to the texture feature planes in the three-dimensional structure points.

[0093] Because our plane is a plane with texture features, we can extract the planar structure points corresponding to the texture feature plane from the three-dimensional structure points based on the texture features of the texture feature plane.

[0094] Specifically, the Random Sample Consensus Algorithm (RANSAC) can be used to extract three-dimensional structural points from an image in a plane.

[0095] Step S32: Obtain the three-dimensional plane and planar pixel points based on the planar structure points.

[0096] Based on planar structural points, a three-dimensional plane can be reconstructed, that is, the texture feature plane in the world coordinate system.

[0097] The planar structure point is the point corresponding to the texture feature plane in the world coordinate system. Based on the planar structure point, the pixel point corresponding to the planar structure point in the image can be extracted. The pixel point corresponding to the planar structure point in the image is the planar pixel point.

[0098] In some embodiments, please refer to Figure 4 Step S4 specifically includes:

[0099] Step S41: Project the planar pixels in the previous frame image onto the current frame image according to the initial pose to obtain the two-dimensional projection points in the current frame image.

[0100] After initialization of the 3D reconstruction, planar structural points, planar pixels in each frame, and the initial pose of the camera are obtained.

[0101] In this embodiment, the initial pose of the camera needs to be optimized. First, the planar pixels in the previous frame are used to obtain planar structural points corresponding to the planar pixels based on the camera's initial pose. Then, these planar structural points corresponding to the planar pixels are projected onto the current frame image based on the initial pose to obtain the two-dimensional projection points in the current frame image. At this point, we have the planar pixels and two-dimensional projection points of the current frame image.

[0102] Step S41: Minimize the pixel error between the planar pixels and the two-dimensional projection points of the current frame to optimize the initial pose and obtain the camera pose.

[0103] In this embodiment, the camera pose can be set as an unknown, and the pixel error can be represented by the camera pose, the coordinates of the planar pixels in the current frame, and the coordinates of the two-dimensional projection points in the current frame. The camera pose obtained by minimizing the planar pixels and the two-dimensional projection points in the current frame image is the optimized camera pose.

[0104] In some embodiments, please refer to Figure 5 Step S6 specifically includes:

[0105] Step S61: Based on the camera pose, transform the equation of the first plane in the world coordinate system to the equation of the third plane in the camera coordinate system.

[0106] The third plane equation is the plane equation of the texture feature plane in the camera coordinate system.

[0107] Specifically, the camera pose is a rotation matrix from the world coordinate system to the camera coordinate system, the first plane equation is the plane equation in the world coordinate system, and the plane equation in the camera coordinate system, i.e., the third plane equation, can be calculated by combining the first plane equation and the camera pose.

[0108] Step S62: Obtain the extrinsic parameters of the laser radar to the camera based on the second plane equation and the third plane equation.

[0109] The extrinsic parameters of the lidar to the camera consist of the rotation matrix and the translation matrix of the lidar to the camera.

[0110] Because the geometric relationships between multiple planes can uniquely constrain the extrinsic parameters—that is, after rotation and translation of the extrinsic parameters from the lidar to the camera—the plane normal vector of the second plane equation and the plane direction vector of the third plane equation should be equal. Furthermore, the center point of the texture feature plane in the radar coordinate system, after transformation based on the lidar-to-camera extrinsic parameters, lies on the texture feature plane in the camera coordinate system. Therefore, when calculating the extrinsic parameters from the lidar to the camera, the extrinsic parameters from the lidar to the camera can be set as unknowns, establishing the relationship between the center point of the texture feature plane in the radar coordinate system, the texture feature plane in the camera coordinate system, and the extrinsic parameters from the lidar to the camera.

[0111] Among them, the rotation matrix of lidar to camera can be obtained by singular value decomposition to obtain a closed-form solution.

[0112] In some embodiments, the extrinsic parameter calibration method further optimizes the extrinsic parameters of the lidar to the camera.

[0113] The first optimization can be PBA (plane-constrained bundle adjustment).

[0114] The planes used in the above optimization of camera pose are extracted from the 3D structural points after the image is 3D reconstructed, while the planes used in the extrinsic parameter calibration are extracted from the point cloud.

[0115] Then, a relationship can be established between the planar pixels on the texture feature plane extracted from the image and the intersection of the texture feature plane in the radar coordinate system, and the extrinsic parameters of the LiDAR to the camera. The extrinsic parameters of the LiDAR to the camera can be optimized based on this relationship.

[0116] The camera pose and the extrinsic parameters of the LiDAR to the camera can all be optimized first, so as to obtain more accurate camera pose and extrinsic parameters of the LiDAR to the camera.

[0117] In some embodiments, the extrinsic parameter calibration method further includes a second optimization of the extrinsic parameters of the lidar to the camera.

[0118] To perform a second optimization of the extrinsic parameters of the lidar to the camera, firstly, measurement noise is acquired, extracted from the texture feature plane in the point cloud. Then, a maximum likelihood estimator is constructed, incorporating all measurement data. Next, the Levenberg-Maguire method is used to iterate through the maximum likelihood estimator, substituting all measurement data, to obtain the optimized extrinsic parameters of the lidar to the camera.

[0119] The camera pose and the extrinsic parameters of the LiDAR and the camera can all be optimized a second time.

[0120] The extrinsic parameter calibration method provided in this invention first acquires multiple frames of point clouds containing texture feature planes and multiple frames of images containing texture feature planes. Second, it estimates the initial pose of the 3D structure points and the camera based on the multiple frames of images. Next, it extracts planes from the 3D structure points to obtain planar pixels in the multiple frames of images, where the planar pixels are generated by the texture feature planes. Then, based on the planar pixels in the multiple frames of images, it optimizes the initial pose to obtain the camera pose and derives a first plane equation in the world coordinate system. Next, it extracts planes from the point cloud to obtain a second plane equation. Finally, it obtains the extrinsic parameters of the LiDAR to the camera based on the first and second plane equations and the camera pose. This extrinsic parameter calibration method can calculate the extrinsic parameters of the LiDAR to the camera by scanning the same texture feature plane with both the camera and the LiDAR, thus broadening its application range. Furthermore, by optimizing the camera pose, it optimizes the extrinsic parameters of the LiDAR to the camera, resulting in more accurate extrinsic parameters.

[0121] Please see Figure 6 The extrinsic parameter calibration device 100 is applied to an electronic device equipped with a camera and a lidar, wherein the camera's imaging range and the lidar's scanning range share the same textured plane. The extrinsic parameter calibration device 100 includes:

[0122] The acquisition module 10 is used to acquire point clouds containing texture feature planes in multiple frames and images containing texture feature planes in multiple frames.

[0123] The estimation module 20 is used to estimate the three-dimensional structure points and the initial pose of the camera based on multiple frames of images.

[0124] The first plane extraction module 30 is used to extract three-dimensional structural points into planes to obtain planar pixels in multiple frames of images, wherein the planar pixels are pixels generated by the texture feature plane.

[0125] The optimization module 40 is used to optimize the initial pose based on the planar pixels in multiple frames of images to obtain the camera pose and the first plane equation in the world coordinate system.

[0126] The second plane extraction module 50 is used to extract the plane from the point cloud to obtain the second plane equation.

[0127] The calculation module 60 is used to obtain the extrinsic parameters of the lidar to the camera based on the first plane equation, the second plane equation, and the camera pose.

[0128] In some embodiments, please refer to Figure 7 The first planar extraction module 30 includes:

[0129] The planar extraction unit 301 is used to obtain planar structure points by performing planar extraction on the three-dimensional structure points, wherein the planar structure points are the points corresponding to the texture feature plane in the three-dimensional structure points.

[0130] The acquisition unit 302 is used to obtain the three-dimensional plane and planar pixel points based on the planar structure points.

[0131] In some embodiments, please refer to Figure 8 The optimization module 40 includes:

[0132] The projection unit 401 is used to project planar pixels from the previous frame image onto the current frame image according to the initial pose, so as to obtain two-dimensional projection points in the current frame image.

[0133] The optimization unit 402 is used to minimize the pixel error between the planar pixels and the two-dimensional projection points of the current frame in order to optimize the initial pose and obtain the camera pose.

[0134] In some embodiments, please refer to Figure 9 The computing module 60 includes:

[0135] The transformation unit 601 is used to transform the first plane equation in the world coordinate system to the third plane equation in the camera coordinate system according to the camera pose.

[0136] The calculation unit 602 is used to obtain the extrinsic parameters of the laser radar to the camera based on the second plane equation and the third plane equation.

[0137] This invention provides an extrinsic parameter calibration device applied to an electronic device. The electronic device is equipped with a camera and a lidar, and the same textured plane exists within the camera's field of view and the lidar's scanning range. The extrinsic parameter calibration device includes: an acquisition module for acquiring multiple frames of point clouds containing the textured plane and multiple frames of images containing the textured plane; an estimation module for estimating three-dimensional structure points and the camera's initial pose based on the multiple frames of images; a first plane extraction module for extracting planes from the three-dimensional structure points to obtain planar pixels in the multiple frames of images, wherein the planar pixels are pixels generated by the textured plane; an optimization module for optimizing the initial pose based on the planar pixels in the multiple frames of images to obtain the camera pose and obtain a first plane equation in the world coordinate system; a second plane extraction module for extracting planes from the point cloud to obtain a second plane equation; and a calculation module for obtaining the extrinsic parameters from the lidar to the camera based on the first plane equation, the second plane equation, and the camera pose. This extrinsic parameter calibration device can calculate the extrinsic parameters of the lidar to the camera by scanning the same textured plane with both the camera and the lidar, thus broadening its application range. Furthermore, by optimizing the camera pose, it optimizes the extrinsic parameters of the lidar to the camera, thereby obtaining more accurate extrinsic parameters of the lidar to the camera.

[0138] It should be noted that the above-described external parameter calibration device can execute the external parameter calibration method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the external parameter calibration device can be found in the external parameter calibration method provided in the embodiments of the present invention.

[0139] Another embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer-executable instructions for causing an electronic device to perform the extrinsic parameter calibration method as described in any of the above embodiments.

[0140] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for calibrating external parameters, characterized in that, The method for extrinsic parameter calibration is applied to an electronic device equipped with a camera and a lidar, wherein the camera's field of view and the lidar's scanning range share the same textured plane. Acquire point clouds and images containing texture feature planes from multiple frames; The three-dimensional structure points and the initial pose of the camera are estimated based on the images from multiple frames. The planar extraction of the three-dimensional structural points yields planar pixels in multiple frames of the image. This includes obtaining planar structural points by performing planar extraction on the three-dimensional structural points, and obtaining the three-dimensional plane and the planar pixels based on the planar structural points. The planar structural points are points corresponding to texture feature planes within the three-dimensional structural points, and the planar pixels are pixels generated by the texture feature planes. Based on the initial pose, the planar pixels in the previous frame image are projected onto the current frame image to obtain the two-dimensional projection points in the current frame image. The pixel error between the planar pixels in the current frame and the two-dimensional projection points is minimized to optimize the initial pose and obtain the camera pose. The first plane equation in the world coordinate system is obtained. The first plane equation is the plane equation of the texture feature plane in the world coordinate system. The point cloud is subjected to planar extraction to obtain the second plane equation; The extrinsic parameters of the lidar to the camera are obtained based on the first plane equation, the second plane equation, and the camera pose. A second optimization of the extrinsic parameters of the lidar to the camera is performed, including extracting measurement noise from the texture feature plane in the point cloud; a maximum likelihood estimator is constructed, wherein the maximum likelihood estimation includes all measurement data, and the Levenberg-Maguire method is used to substitute all measurement data into the maximum likelihood estimator for iteration to obtain the optimized extrinsic parameters of the lidar to the camera.

2. The external parameter calibration method according to claim 1, characterized in that, The step of obtaining the extrinsic parameters of the lidar to the camera based on the first plane equation, the second plane equation, and the camera pose includes: Based on the camera pose, the first plane equation in the world coordinate system is transformed to the camera coordinate system to obtain the third plane equation; Based on the second and third plane equations, the extrinsic parameters of the laser radar to the camera are obtained.

3. An external parameter calibration device, characterized in that, An external parameter calibration device is applied to an electronic device equipped with a camera and a lidar, wherein the camera's field of view and the lidar's scanning range share the same textured plane. The acquisition module is used to acquire point clouds and images containing texture feature planes from multiple frames. An estimation module is used to estimate the three-dimensional structure points and the initial pose of the camera based on multiple frames of the image. The first plane extraction module is used to perform plane extraction on the three-dimensional structure points to obtain planar pixels in multiple frames of the image. The module includes obtaining planar structure points by performing plane extraction on the three-dimensional structure points, and obtaining the three-dimensional plane and the planar pixels based on the planar structure points. The planar structure points are the points corresponding to the texture feature plane in the three-dimensional structure points, and the planar pixels are the pixels generated by the texture feature plane. An optimization module is used to project planar pixels from the previous frame image onto the current frame image based on the initial pose, so as to obtain two-dimensional projection points in the current frame image, minimize the pixel error between the planar pixels in the current frame and the two-dimensional projection points, optimize the initial pose to obtain the camera pose, and obtain the first plane equation in the world coordinate system. The first plane equation is the plane equation of the texture feature plane in the world coordinate system. The second plane extraction module is used to extract the plane from the point cloud to obtain the second plane equation; The computation module is used to obtain the extrinsic parameters of the lidar to the camera based on the first plane equation, the second plane equation, and the camera pose; and to perform a second optimization on the extrinsic parameters of the lidar to the camera, including extracting measurement noise from the texture feature plane in the point cloud; constructing a maximum likelihood estimator, wherein the maximum likelihood estimation includes all measurement data, and using the Levenberg-Maguire method to substitute all measurement data into the maximum likelihood estimator for iteration to obtain the optimized extrinsic parameters of the lidar to the camera.

4. The external parameter calibration device according to claim 3, characterized in that, The computing module includes: The transformation unit is used to transform the first plane equation in the world coordinate system to the camera coordinate system to obtain the third plane equation based on the camera pose. The calculation unit is used to obtain the extrinsic parameters of the laser radar to the camera based on the second plane equation and the third plane equation.

5. An electronic device, characterized in that, include: Cameras and LiDAR; At least one processor, which is communicatively connected to both the camera and the lidar; The memory communicatively connected to the at least one processor, wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the external parameter calibration method according to claim 1 or 2.

6. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions for causing an electronic device to perform the method as described in claim 1 or 2.

Images