A dual-circle-based external parameter calibration method, device and equipment

Abstract

The present disclosure relates to the technical field of environment perception, and provides a double-circle landmark-based extrinsic parameter calibration method, device and equipment. The method comprises: fitting a double-ellipse initial curve according to a camera shooting image; obtaining a double-ellipse optimized curve by using a consistency constraint condition, and determining two-dimensional feature points containing a tangent point and a circle center projection image according to the double-ellipse optimized curve; obtaining landmark feature points of two target circular landmarks; determining extrinsic parameters between cameras based on the two-dimensional feature points and the landmark feature points; transforming the landmark feature points to a laser radar coordinate system to obtain three-dimensional feature points conforming to internal consistency; determining extrinsic parameters between the camera and the laser radar based on the two-dimensional feature points and the three-dimensional feature points; and determining extrinsic parameters between the laser radars based on the three-dimensional feature points. The technical scheme provided by one or more embodiments of the present disclosure can achieve high-precision extrinsic parameter calibration between the cameras and between the camera and the laser radar.

Description

Technical Field

[0001] This disclosure relates to the field of environmental sensing technology, specifically to a method, apparatus, and equipment for extrinsic parameter calibration based on double-circle markers. Background Technology

[0002] Autonomous driving technology (also known as self-driving technology) refers to the technology that allows vehicles (or other vehicles) to perceive their driving environment, autonomously decide on their driving path, and control themselves to reach their destination without human intervention, using onboard sensors. Autonomous driving technology integrates multiple technologies, primarily including real-time perception and localization, path planning, communication and data interaction, and intelligent vehicle control.

[0003] Similar to traditional driving, real-time perception of the vehicle's surroundings is a necessary prerequisite for decision-making and control in autonomous driving systems. Consequently, environmental perception technologies have received significant attention from both academia and industry. Compared to sensors like LiDAR and millimeter-wave radar, which only return geometric and velocity information, visible light cameras can perceive the richest range of external information, such as the color, structure, texture, and semantic information of the surrounding environment (e.g., roads, pedestrians, traffic signs). Thanks to breakthroughs in computer vision using deep learning, this camera-perceived information can be identified, tracked, and even used for 3D reconstruction, making vision-based environmental perception systems a key support for the successful implementation of autonomous driving technology.

[0004] To reduce the annotation cost and improve the efficiency of deep learning annotation, the concept of 4D annotation has been introduced into environmental perception systems. Using camera- and LiDAR-based environmental perception technologies, a dense map can be constructed from an image and point cloud sequence. Annotators only need to complete the desired annotation within a unified dense map in one go, and the annotation results can be back-projected back onto the original image using precise extrinsic parameters, achieving consistent annotation of the original image. In multi-sensor systems, extrinsic parameters specifically refer to the rigid body transformation parameters required to transform data from one sensor coordinate system to another. For the above annotation process to proceed smoothly, a prerequisite is having precise extrinsic parameters between different sensors, as well as between the camera and LiDAR; otherwise, back-projection errors can accumulate over time, causing annotation drift. Summary of the Invention

[0005] In view of this, one or more embodiments of this disclosure provide a method, apparatus and device for extrinsic parameter calibration based on double circle markers, which can achieve high-precision extrinsic parameter calibration between cameras and between cameras and lidar.

[0006] Firstly, this disclosure provides an extrinsic parameter calibration method based on dual-circle markers. The method is used for extrinsic parameter calibration between multiple target sensors, including a target camera and a target lidar. The method includes: acquiring a camera-captured image of a target dual-circle calibration plate; fitting a dual-ellipse initial curve based on the camera-captured image; the target dual-circle calibration plate containing two target circular markers with identical patterns; determining, using a target feature extraction algorithm, the initial common tangent, initial tangent point, initial vanishing point formed by mutually parallel initial common tangents, initial vanishing line formed by different initial vanishing points, and the initial center projection image of the target circular markers based on the dual-ellipse initial curve; optimizing the dual-ellipse initial curve using consistency constraints constructed based on the initial vanishing point and initial vanishing line to obtain an optimized dual-ellipse curve, and then... The double-ellipse optimization curve utilizes the target feature extraction algorithm to determine two-dimensional feature points, including optimized tangent points and optimized circle center projection images; it acquires the marker feature points of the two target circular markers, including the marker center and the marker tangent point; based on the two-dimensional feature points and the marker feature points, it determines the extrinsic parameters between each of the target cameras; according to the point cloud edge detection results of the two target circular markers, it optimizes the initial pose of the target double-circle calibration board in the lidar coordinate system, and based on the optimization results of the initial pose, it transforms the marker feature points to the lidar coordinate system to obtain three-dimensional feature points that conform to intrinsic consistency; based on the two-dimensional feature points and the three-dimensional feature points, it determines the extrinsic parameters between the target camera and the target lidar; based on the three-dimensional feature points, it determines the extrinsic parameters between each of the target lidars.

[0007] Secondly, this disclosure provides an extrinsic parameter calibration device based on dual-circle markers. The device is used for extrinsic parameter calibration between multiple target sensors, including a target camera and a target lidar. The device includes: an elliptic curve fitting unit, used to acquire camera images of a target dual-circle calibration plate and fit an initial curve of a dual ellipse based on the camera images; the target dual-circle calibration plate contains two target circular markers with identical patterns; a two-dimensional feature extraction unit, used to determine, based on the initial curve of the dual ellipse and using a target feature extraction algorithm, the initial common tangent, initial tangent point, initial vanishing point formed by mutually parallel initial common tangents, initial vanishing line formed by different initial vanishing points, and the initial center projection image of the target circular markers; and a two-dimensional feature optimization unit, used to optimize the initial curve of the dual ellipse using consistency constraints constructed based on the initial vanishing point and initial vanishing line to obtain an optimized curve of the dual ellipse, and using the optimized curve of the dual ellipse to... The target feature extraction algorithm is used to determine two-dimensional feature points, which include optimized tangent points and optimized center projection images; a marker feature acquisition unit is used to acquire marker feature points of two target circular markers, which include marker center and marker tangent points; a first extrinsic parameter calibration unit is used to determine the extrinsic parameters between each target camera based on the two-dimensional feature points and the marker feature points; a three-dimensional feature extraction unit is used to optimize the initial pose of the target dual-circle calibration plate in the lidar coordinate system according to the point cloud edge detection results of the two target circular markers, and transform the marker feature points to the lidar coordinate system according to the optimization results of the initial pose to obtain three-dimensional feature points that conform to intrinsic consistency; a second extrinsic parameter calibration unit is used to determine the extrinsic parameters between the target camera and the target lidar based on the two-dimensional feature points and the three-dimensional feature points; a third extrinsic parameter calibration unit is used to determine the extrinsic parameters between each target lidar based on the three-dimensional feature points.

[0008] Thirdly, this disclosure provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program, and when the computer program is executed by the processor, it implements the above-described external parameter calibration method based on double-circle markers.

[0009] Fourthly, this disclosure provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described extrinsic parameter calibration method based on double-circle markers.

[0010] This disclosure provides one or more embodiments of an extrinsic parameter calibration method based on dual-circle markers, primarily applied to multi-sensor extrinsic parameter calibration tasks. By designing a dual-circle marker calibration board and corresponding calibration algorithms, extrinsic parameter calibration between different cameras, between a camera and a LiDAR, and between different LiDARs can be efficiently achieved using only a single frame of data. Furthermore, the dual-circle marker-based extrinsic parameter calibration method provided in one or more embodiments of this disclosure considers the consistency between the markers on the calibration board during the camera and LiDAR detection phase, further improving the accuracy of the calibration results.

[0011] This disclosure provides a technical solution through one or more embodiments, enabling multi-sensor extrinsic parameter calibration tasks to be completed using a single calibration board containing a simple double-circle marker pattern. This eliminates the need for separate calibration boards to calibrate extrinsic parameters between cameras, between LiDARs, and between cameras and LiDARs, thus improving extrinsic parameter calibration efficiency. Furthermore, the double-circle calibration board can tolerate the absence of detection components, enhancing the robustness of extrinsic parameter calibration.

[0012] This disclosure provides a technical solution through one or more embodiments, and designs a unique calibration algorithm suitable for dual-circle calibration boards. It can calibrate the extrinsic parameters between multiple sensors using single-frame data, and achieves higher calibration accuracy between sensors due to the implementation of feature consistency constraints. Attached Figure Description

[0013] The features and advantages of the embodiments of this disclosure will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the present disclosure in any way. In the drawings:

[0014] Figure 1 This illustration shows a step diagram of an extrinsic parameter calibration method based on double-circle markers in one embodiment of the present disclosure;

[0015] Figure 2 A schematic diagram of a target double-circle calibration plate is shown in one embodiment of this disclosure;

[0016] Figure 3 A schematic diagram of a double ellipse candidate curve is shown in one embodiment of this disclosure;

[0017] Figure 4 A schematic diagram of the functional units of an extrinsic parameter calibration device based on a double-circle marker is shown in one embodiment of this disclosure;

[0018] Figure 5 A schematic diagram of the structure of an electronic device according to one embodiment of the present disclosure is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] In related technologies, industry-standard multi-sensor extrinsic parameter calibration schemes typically require the preparation of calibration boards with different markings. For example, a checkerboard calibration board is used between cameras, while a circular marking calibration board is used between a camera and a LiDAR. This necessitates maintaining different calibration algorithms. More importantly, the detection of each marking on the calibration board by each sensor is independent, without considering the consistency between the markings on the calibration board. This makes it extremely easy to introduce errors during the calibration process.

[0021] In view of this, the extrinsic parameter calibration method based on dual-circle markers provided in one or more embodiments of this disclosure is mainly applied to multi-sensor extrinsic parameter calibration tasks. By designing a dual-circle marker calibration board and a corresponding calibration algorithm, extrinsic parameter calibration between different cameras, between a camera and a LiDAR, and between different LiDARs can be efficiently achieved using only a single frame of data. Furthermore, the extrinsic parameter calibration method based on dual-circle markers provided in one or more embodiments of this disclosure considers the consistency between the markers on the calibration board during the detection stage of the calibration board by the camera and LiDAR, further improving the accuracy of the calibration results.

[0022] Please see Figure 1 This disclosure provides an embodiment of an extrinsic parameter calibration method based on dual-circle markers, which can be used for extrinsic parameter calibration between multiple target sensors, including a target camera and a target lidar. The method may include steps S1 to S8.

[0023] Step S1: Obtain a camera image of the target double-circle calibration plate, and fit an initial curve of a double ellipse based on the camera image. The target double-circle calibration plate contains two target circular marks with the same pattern.

[0024] In this embodiment, a schematic diagram of the target double-circle calibration plate can be found in [reference needed]. Figure 2 Its key feature is that the double-circle marks on the calibration plate are large enough. Thus, even when the calibration plate is far from the target sensor, or when the double-circle marks on the calibration plate are obscured by edges, the calibration plate can still provide sufficient edge detection data to combat detection errors.

[0025] In this embodiment, when using the target dual-circle calibration plate for extrinsic parameter calibration, the intrinsic parameters of each target sensor are known in advance. The camera images of the target dual-circle calibration plate obtained by the target camera, or the radar point cloud data of the target dual-circle calibration plate obtained by the target lidar, can be preprocessed before extrinsic parameter calibration to remove distorted data in the detection data.

[0026] In this embodiment, the target circular mark typically appears as two elliptical patterns in the camera-captured image (unless the camera's optical axis is perpendicular to the plane of the circular mark, in which case the image is still a circular pattern, but the circular pattern can also be considered a special type of elliptical pattern). Using common edge extraction methods (such as Canny, Sobel, etc.), the edge information of these two elliptical patterns can be obtained from the camera-captured image. Then, using the least squares method, the elliptical curves of the two elliptical patterns can be fitted. It should be noted that if a conic section has the form of... any point on it Must meet .

[0027] Step S2: Based on the initial curve of the double ellipse, use the target feature extraction algorithm to determine the initial common tangent, initial tangent point, initial vanishing point formed by the parallel initial common tangents, initial vanishing line formed by different initial vanishing points, and the initial center projection image of the target circular mark.

[0028] In this embodiment, unlike traditional methods that only use the projected image of the center of the target circular mark as feature points, this scheme also extracts the tangent points corresponding to the common tangent of the two ellipses as feature points. This provides sufficient point information for subsequent single-frame data recovery of camera and LiDAR extrinsic parameters. Using the tangent points corresponding to the common tangent of the two ellipses as feature points is a clever design made by this disclosure after considering the property of perspective transformation preserving intersection points.

[0029] In this embodiment, the target feature extraction algorithm can not only robustly calculate the common tangent and common tangent point of the double elliptic curve, but also robustly solve for the central projection image of the target circular mark. The determination of the vanishing point and vanishing line improves the computational efficiency of the central projection image and facilitates further optimization of the initial solution results.

[0030] In some implementations, the target feature extraction algorithm includes: transforming a bielliptic candidate curve in point space into a dual curve in line space; and based on the dual curve, performing eigenvalue decomposition and linear solution to determine the common tangent of the bielliptic candidate curve.

[0031] Specifically, according to projective geometry theory, for two candidate elliptical curves in point space... Finding the common tangent can be transformed into finding their dual curves in the linear space. Find the intersection points, where ,symbol This indicates equivalence, meaning the quantities on both sides are the same or differ only by a scale factor. To avoid simultaneous equations... Then, solve for the problem about To address the numerical instability caused by higher-order equations, this disclosure presents a more robust computational method based on eigenvalue decomposition and linear solution. (Robustness refers to the ability of a system or model to maintain stable performance or resist disturbances when faced with disturbances in input data, environmental conditions, or model parameters.)

[0032] In some implementations, determining the common tangent of the candidate bielliptic curves by performing eigenvalue decomposition and linear solution based on the dual curves may include: ... Perform eigenvalue decomposition to obtain real eigenvalues. , making The rank is 2; Decomposed into two points With n, such that ;right Perform eigenvalue decomposition so that Seeking , ;right Perform eigenvalue decomposition and determine the eigenvectors corresponding to the real eigenvalues ​​as candidate vector solutions; if the candidate vector solutions pass... If the verification is successful, then the candidate vector solution is determined as the common tangent of the bielliptic candidate curves; where, The first candidate ellipse curve The dual curve, For the second ellipse candidate curve The dual curve.

[0033] Specifically, for Perform eigenvalue decomposition, and find its corresponding real eigenvalues. Make The rank of is 2, which means that It can be decomposed into two points , making Geometrically, this means that the common tangent of the two ellipses obtained according to the target double-circle calibration plate must pass through one of these two points.

[0034] right Perform eigenvalue decomposition so that Seeking , Here, and It is the only degenerate parameter in the elliptic bundle that can give two distinct real points; the other intermediate roots are either repeated roots or have no real geometric meaning.

[0035] The common tangent of the two ellipses passes through the point. For example, the next step is to solve... And confirm its solution Also satisfies In the calculation process, firstly... Eigenvalue decomposition essentially involves finding the eigenvectors corresponding to the eigenvalues, which are the eigenvectors of the preceding system of equations. The solution, then substitute it into The results are then verified. Finally, the common tangents of the candidate bielliptic curves can be obtained.

[0036] In some embodiments, the common tangent of the bielliptic candidate curves comprises four common tangents. , , and The target feature extraction algorithm further includes: the tangent points of the double elliptical candidate curves are... The first vanishing point of the candidate double ellipse curve is ;make Solve The second vanishing point of the candidate double ellipse curve is obtained. The vanishing curve of the candidate double ellipse is... The central projection image of the target circular marker in the double elliptical candidate curve is: .

[0037] Specifically, please refer to Figure 3 Candidate curves of double ellipses and and their common tangent , , and The calculation of the tangent point is as follows: , Figure 3 medium curve The four tangent points on the curve are clearly marked. The four tangent points on the ellipse can be determined by marking the four common tangent lines. Based on the centers of the two ellipses ( Figure 3 In This allows for the sorting of all tangent points to ensure the correspondence between 2D and 3D feature points in subsequent method steps. In the Euclidean plane... and If the two obviously parallel common tangents to the target circular marker are the vanishing point at their intersection at infinity in the projective plane, then... .at the same time, , The lines that form the equations are all parallel to each other, and the symbol is... Let represent the cross product. The cross product of two points represents the line passing through those two points, and the cross product of two lines represents the intersection of those two lines. Let Solve This will give you another vanishing point. The final vanishing line is calculated as follows: Calculation of the projected image at the center of the circle .

[0038] Optionally, and The other two common tangents to the target circular mark not only intersect in the Euclidean plane, but also exist in the projective plane. Figure 3 The intersection point p shown can also serve as a supplementary feature point to further improve the accuracy of extrinsic parameter calibration between different sensors.

[0039] Step S3: Optimize the initial curve of the double ellipse using the consistency constraints constructed based on the initial vanishing point and the initial vanishing line to obtain the optimized curve of the double ellipse. Based on the optimized curve of the double ellipse, determine the two-dimensional feature points using the target feature extraction algorithm. The two-dimensional feature points include the optimized tangent point and the optimized circle center projection image.

[0040] In this embodiment, unlike traditional image detection feature methods that generally consider each marker pattern independently, this disclosure specifically considers the inherent constraints between two target circular markers. Considering that the markers on the calibration board are repeating patterns, they generally satisfy a point-to-point linear transformation, called planar homology. For the target double-circle calibration board, its planar homology transformation is translation, that is, translating one target circular marker to the position of the other target circular marker; thus, the two target circular markers obviously coincide. Furthermore, under perspective transformation, the double circles in the camera-captured image should also satisfy this translation transformation.

[0041] In some implementations, optimizing the initial double-ellipse curve using consistency constraints constructed based on the initial vanishing point and initial vanishing line to obtain the optimized double-ellipse curve includes: substituting the initial double-ellipse curve into the candidate double-ellipse curve; and establishing a homography matrix for translation transformation based on the initial vanishing point and initial vanishing line. So that ; Optimize using the LM algorithm and ;Will and The optimization result is determined as the double ellipse optimization curve.

[0042] Specifically, the translation transformation on the projective plane of the image captured by the camera can be denoted as: ,and The target double-circle calibration plate can be modeled as follows: (5 degrees of freedom) and (4 degrees of freedom). Among them, the vanishing point... and disappearing line The initial value was calculated in step S2 above. It is the identity matrix. Let be the scalar value representing the translation change. Then, the LM algorithm is used for optimization. and Consistency can eventually be achieved. and .

[0043] Step S4: Obtain the feature points of the two target circular markers, the feature points of which include the center of the marker circle and the tangent point of the marker.

[0044] In this embodiment, unlike traditional methods that only use the center of the target circular marker as a feature point, this scheme also selects the tangent points corresponding to the two target circular markers as feature points. This provides sufficient point information for subsequent single-frame data recovery of camera and lidar extrinsic parameters. The marker feature points are fixed features of the target dual-circle calibration plate. In addition to being calculated temporarily during extrinsic parameter calibration, they can also be calculated by reading pre-stored calculation results.

[0045] In some implementations, obtaining the feature points of the two target circular markers includes: substituting the conic curves of the two target circular markers into the double ellipse candidate curve; and using the target feature extraction algorithm to determine the tangent points of the two target circular markers.

[0046] Specifically, assume that the centers of the two circles on the target double-circle calibration plate are respectively and So, with Let be the origin of the coordinate system, with Connected to The direction is the coordinate system In the axial direction, the normal vector of the calibration plate is in the coordinate system. The axes allow for the establishment of a calibration plate coordinate system. Subsequently, based on the physical information of the double circular marks on the calibration plate, the conic sections of the two target circular marks can be calculated. and The specific form of the target circular markers can be determined based on the target feature extraction algorithm described above.

[0047] Step S5: Based on the two-dimensional feature points and the marker feature points, determine the extrinsic parameters between each of the target cameras.

[0048] In this embodiment, a set of two-dimensional feature points can be obtained from a single frame of image captured by each target camera. Each set of two-dimensional feature points also has a projection relationship with a marker feature point. Therefore, based on these two-dimensional feature points and the marker feature point, classical computer vision methods for solving camera pose (e.g., the Perspective-n-Point (PnP) algorithm) can be used to determine the extrinsic parameters between the target cameras. Optionally, the Levenberg-Marquardt (LM) algorithm can be used to further optimize the extrinsic parameter solution results.

[0049] Step S6: Based on the point cloud edge detection results of the two target circular markers, optimize the initial pose of the target dual-circle calibration plate in the lidar coordinate system, and based on the optimization results of the initial pose, transform the marker feature points to the lidar coordinate system to obtain three-dimensional feature points that conform to the inherent consistency.

[0050] In this embodiment, some common point cloud feature extraction methods are used to extract the dual-circle calibration plate region from the radar point cloud, and further extract the edges of the dual circles based on the reflection intensity information of the point cloud, fitting the initial curve of the dual circles. Based on the initial curve of the dual circles, the common tangent point and center of the dual circles in the lidar coordinate system can be calculated using the same method described above. Thus, based on the correspondence between two-dimensional feature points in the image and three-dimensional feature points in the lidar system, the extrinsic parameters between the camera and the lidar are calculated using the PnP algorithm. Traditional methods generally stop here. However, this method, based on the center of the circle and the common tangent obtained in the lidar coordinate system, can obtain the initial pose R,t of the calibration plate according to the above-mentioned calibration plate coordinate system setting method. In order to further obtain the inherently consistent marker feature points, an optimization step of the initial pose is added. The optimized R,t is used to transform the pre-calculated inherently consistent marker feature points on the calibration plate to the lidar coordinate system.

[0051] In some embodiments, optimizing the initial pose of the target dual-circle calibration plate in the lidar coordinate system based on the point cloud edge detection results of the two target circular markers includes: mapping the two target circular markers to two-dimensional conic sections in the calibration plate coordinate system. Transformed into a three-dimensional conical surface Using the initial pose, the conical surface in the calibration plate coordinate system is... Conical surface converted to lidar coordinate system Based on the point cloud edge detection results, construct constraints. The initial pose is optimized using the LM algorithm.

[0052] Specifically, first, the two-dimensional conic section on the calibration plate coordinate system... Transformed into a three-dimensional conical surface The formula is expressed as follows:

[0053] .

[0054] Then, the three-dimensional conical surface in the calibration plate coordinate system Convert to a three-dimensional conical surface in the lidar coordinate system The formula is expressed as follows:

[0055] .

[0056] At this point, constraints can be constructed based on the point cloud edge detection results. The LM algorithm was used to optimize the initial pose of the target dual-circle calibration plate in the lidar coordinate system. Finally, based on the optimization results, the marker feature points can be transformed to the lidar coordinate system to obtain three-dimensional feature points that conform to inherent consistency.

[0057] Step S7: Based on the two-dimensional feature points and the three-dimensional feature points, determine the extrinsic parameters between the target camera and the target lidar.

[0058] In this embodiment, a set of three-dimensional feature points can be obtained from the single-frame point cloud data obtained from a single scan of each target LiDAR. Each set of three-dimensional feature points has a projection relationship with the marker feature points. As explained above, each set of two-dimensional feature points also has a projection relationship with the marker feature points. Therefore, based on these two-dimensional feature points, three-dimensional feature points, and marker feature points, some classic computer vision methods for solving pose (such as the PnP algorithm) can be used to obtain the extrinsic parameters between the target camera and the target LiDAR. Optionally, the LM (Levenberg–Marquardt) algorithm can be used to further optimize the extrinsic parameter solution results.

[0059] Step S8: Based on the three-dimensional feature points, determine the extrinsic parameters between each of the target lidars.

[0060] In this embodiment, based on the projection relationship between 3D feature points and marker feature points, some classic methods for solving LiDAR pose (e.g., Iterative Closest Point, or ICP algorithm) can be used to solve for the initial values ​​of the extrinsic parameters between the LiDARs. Then, the ICP algorithm is used again on the entire single-frame point cloud data to optimize these extrinsic parameters.

[0061] It should be further explained that the key work of this disclosed technical solution is "determining what kind of feature points (i.e., the projection of the circle center and the common tangent point) and how to determine the feature points (i.e., the target feature extraction algorithm and the optimization algorithm that considers consistency)", rather than how algorithms such as PnP and ICP use feature points to calculate extrinsic parameters.

[0062] This disclosure provides a technical solution through one or more embodiments, enabling multi-sensor extrinsic parameter calibration tasks to be completed using a single calibration board containing a simple double-circle marker pattern. This eliminates the need for separate calibration boards to calibrate extrinsic parameters between cameras, between LiDARs, and between cameras and LiDARs, thus improving extrinsic parameter calibration efficiency. Furthermore, the double-circle calibration board can tolerate the absence of detection components, enhancing the robustness of extrinsic parameter calibration.

[0063] This disclosure provides a technical solution through one or more embodiments, and designs a unique calibration algorithm suitable for dual-circle calibration boards. It can calibrate the extrinsic parameters between multiple sensors using single-frame data, and achieves higher calibration accuracy between sensors due to the implementation of feature consistency constraints.

[0064] Please see Figure 4 This disclosure also provides an extrinsic parameter calibration device based on dual-circle markers. The device can be used for extrinsic parameter calibration among multiple target sensors, including a target camera and a target lidar. The device includes:

[0065] Elliptic curve fitting unit 100 is used to acquire camera images of the target double-circle calibration plate and fit an initial curve of the double ellipse based on the camera images. The target double-circle calibration plate contains two target circular marks with the same pattern.

[0066] The two-dimensional feature extraction unit 200 is used to determine the initial common tangent, initial tangent point, initial vanishing point formed by the parallel initial common tangents, initial vanishing line formed by different initial vanishing points, and the initial center projection image of the target circular mark based on the initial curve of the double ellipse and using a target feature extraction algorithm.

[0067] The two-dimensional feature optimization unit 300 is used to optimize the initial curve of the double ellipse by using the consistency constraints constructed based on the initial vanishing point and the initial vanishing line, to obtain the optimized curve of the double ellipse, and to determine the two-dimensional feature points based on the optimized curve of the double ellipse by using the target feature extraction algorithm. The two-dimensional feature points include the optimized tangent point and the optimized circle center projection image.

[0068] The feature acquisition unit 400 is used to acquire feature points of the two target circular signs, wherein the feature points include the center of the sign circle and the tangent point of the sign;

[0069] The first extrinsic parameter calibration unit 500 is used to determine the extrinsic parameters between each of the target cameras based on the two-dimensional feature points and the marker feature points;

[0070] The three-dimensional feature extraction unit 600 is used to optimize the initial pose of the target double-circle calibration plate in the lidar coordinate system based on the point cloud edge detection results of the two target circular marks, and transform the mark feature points to the lidar coordinate system based on the optimization results of the initial pose to obtain three-dimensional feature points that conform to the inherent consistency.

[0071] The second extrinsic parameter calibration unit 700 is used to determine the extrinsic parameters between the target camera and the target lidar based on the two-dimensional feature points and the three-dimensional feature points;

[0072] The third extrinsic parameter calibration unit 800 is used to determine the extrinsic parameters between each of the target lidars based on the three-dimensional feature points.

[0073] In one embodiment, the target feature extraction algorithm includes: transforming a bielliptic candidate curve in point space into a dual curve in line space; and based on the dual curve, performing eigenvalue decomposition and linear solution to determine the common tangent of the bielliptic candidate curve.

[0074] In one implementation, the step of determining the common tangent of the candidate bielliptic curves by performing eigenvalue decomposition and linear solution based on the dual curves includes: ... Perform eigenvalue decomposition to obtain real eigenvalues. , making The rank is 2; Decomposed into two points With n, such that ;right Perform eigenvalue decomposition so that Seeking , ;right Perform eigenvalue decomposition and determine the eigenvectors corresponding to the real eigenvalues ​​as candidate vector solutions; if the candidate vector solutions pass... If the verification is successful, then the candidate vector solution is determined as the common tangent of the bielliptic candidate curves; where, The first candidate ellipse curve The dual curve, For the second ellipse candidate curve The dual curve.

[0075] In one implementation, the common tangent of the dual elliptic candidate curves comprises four common tangents. , , and The target feature extraction algorithm further includes: the tangent points of the double elliptical candidate curves are... The first vanishing point of the candidate double ellipse curve is ;make Solve The second vanishing point of the candidate double ellipse curve is obtained. The vanishing curve of the candidate double ellipse is... The central projection image of the target circular marker in the double elliptical candidate curve is: .

[0076] In one embodiment, the two-dimensional feature optimization unit 300 is specifically used to: substitute the initial double ellipse curve into the candidate double ellipse curve; and establish a homography matrix for translation transformation based on the initial vanishing point and the initial vanishing line. So that ; Optimize using the LM algorithm and ;Will and The optimization result is determined as the double ellipse optimization curve.

[0077] In one embodiment, the marker feature acquisition unit 400 is specifically used to: substitute the conic curves of the two target circular markers into the double ellipse candidate curve; and use the target feature extraction algorithm to determine the marker tangent points of the two target circular markers.

[0078] In one embodiment, the three-dimensional feature extraction unit 600 is specifically used to: extract the two target circular marks into two-dimensional conic sections in the calibration plate coordinate system. Transformed into a three-dimensional conical surface Using the initial pose, the conical surface in the calibration plate coordinate system is... Conical surface converted to lidar coordinate system Based on the point cloud edge detection results, constraints are established. The initial pose is optimized using the LM algorithm.

[0079] The various units described in the above embodiments can be implemented by a computer chip or by a product with a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0080] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0081] Please see Figure 5 This disclosure also provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program, and when the computer program is executed by the processor, it implements the above-described external parameter calibration method based on double-circle markers.

[0082] This disclosure also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described extrinsic parameter calibration method based on double-circle markers.

[0083] The processor can be a central processing unit (CPU). It can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0084] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above-described embodiments.

[0085] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0086] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0087] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of apparatus, devices, and storage media are basically similar to method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0088] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0089] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for extrinsic parameter calibration based on double-circle markers, characterized in that, The method is used for extrinsic parameter calibration among multiple target sensors, including a target camera and a target lidar, and the method includes: Acquire a camera image of the target double-circle calibration plate, and fit an initial curve of a double ellipse based on the camera image. The target double-circle calibration plate contains two target circular marks with the same pattern. Based on the initial curve of the double ellipse, the target feature extraction algorithm is used to determine the initial common tangent, initial tangent point, initial vanishing point formed by the parallel initial common tangents, initial vanishing line formed by different initial vanishing points, and the initial center projection image of the target circular mark. The initial curve of the double ellipse is optimized by using the consistency constraints constructed based on the initial vanishing point and the initial vanishing line to obtain the optimized double ellipse curve. Based on the optimized double ellipse curve, the target feature extraction algorithm is used to determine the two-dimensional feature points, which include the optimized tangent point and the optimized circle center projection image. Obtain the feature points of the two target circular markers, wherein the feature points include the center of the circle and the tangent point of the circle; Based on the two-dimensional feature points and the marker feature points, the extrinsic parameters between each of the target cameras are determined; Based on the point cloud edge detection results of the two target circular markers, the initial pose of the target double-circle calibration plate in the lidar coordinate system is optimized, and based on the optimization results of the initial pose, the marker feature points are transformed to the lidar coordinate system to obtain three-dimensional feature points that conform to the inherent consistency. Based on the two-dimensional feature points and the three-dimensional feature points, the extrinsic parameters between the target camera and the target lidar are determined; Based on the three-dimensional feature points, the extrinsic parameters between each of the target lidars are determined.

2. The method according to claim 1, characterized in that, The target feature extraction algorithm includes: Transform the candidate double ellipse curve in the point space into the dual curve in the line space; Based on the dual curves, eigenvalue decomposition and linear solution are performed to determine the common tangent of the candidate double ellipse curves.

3. The method according to claim 2, characterized in that, The step of determining the common tangent of the candidate bielliptic curves by performing eigenvalue decomposition and linear solution based on the dual curves includes: right Perform eigenvalue decomposition to obtain real eigenvalues. , making The rank is 2; Will Decomposed into two points With n, such that ; right Perform eigenvalue decomposition so that Seeking , ; right Perform eigenvalue decomposition and determine the eigenvectors corresponding to the real eigenvalues ​​as candidate vector solutions; If the candidate vector solution passes If the verification is successful, the candidate vector solution will be determined as the common tangent of the candidate double ellipse curve; in, The first candidate ellipse curve The dual curve, For the second ellipse candidate curve The dual curve.

4. The method according to claim 3, characterized in that, The common tangent of the candidate bielliptic curves comprises four common tangents. , , and The target feature extraction algorithm further includes: The tangent points of the candidate double ellipse curve are: ; The first vanishing point of the candidate hyperellipse curve is: ; make Solve The second vanishing point of the candidate double ellipse curve is obtained. ; The vanishing curve of the candidate double ellipse is ; The central projection image of the target circular mark in the double ellipse candidate curve is: .

5. The method according to claim 4, characterized in that, The process of optimizing the initial curve of the double ellipse using consistency constraints constructed based on the initial vanishing point and initial vanishing line to obtain the optimized double ellipse curve includes: Substitute the initial curve of the double ellipse into the candidate curve of the double ellipse; Based on the initial vanishing point and the initial vanishing line, establish the homography matrix of the translation transformation. So that , It is the identity matrix. A scalar value representing a translational change; Optimize using the LM algorithm and ; Will and The optimization result is determined as the double ellipse optimization curve.

6. The method according to claim 5, characterized in that, The step of obtaining the feature points of the two target circular markers includes: Substitute the conic sections of the two target circular markers into the candidate double ellipse curve; Using the target feature extraction algorithm, the tangent points of the two target circular markers are determined.

7. The method according to claim 1, characterized in that, The step of optimizing the initial pose of the target dual-circle calibration plate in the lidar coordinate system based on the point cloud edge detection results of the two target circular markers includes: The two target circular marks are represented by a two-dimensional conic section in the calibration plate coordinate system. Transformed into a three-dimensional conical surface ; Using the initial pose, the conical surface in the calibration plate coordinate system is... Conical surface converted to lidar coordinate system ; Based on the point cloud edge detection results, constraints are constructed. The initial pose is optimized using the LM algorithm.

8. An extrinsic parameter calibration device based on double-circle markers, characterized in that, The device is used for extrinsic parameter calibration among multiple target sensors, including a target camera and a target lidar. The device includes: An elliptic curve fitting unit is used to acquire camera images of the target double-circle calibration plate and fit an initial curve of the double ellipse based on the camera images. The target double-circle calibration plate contains two target circular marks with the same pattern. The two-dimensional feature extraction unit is used to determine the initial common tangent, initial tangent point, initial vanishing point formed by the parallel initial common tangents, initial vanishing line formed by different initial vanishing points, and the initial center projection image of the target circular mark based on the initial curve of the double ellipse using a target feature extraction algorithm. The two-dimensional feature optimization unit is used to optimize the initial curve of the double ellipse by using the consistency constraints constructed based on the initial vanishing point and the initial vanishing line, to obtain the optimized curve of the double ellipse, and to determine the two-dimensional feature points based on the optimized curve of the double ellipse by using the target feature extraction algorithm. The two-dimensional feature points include the optimized tangent point and the optimized circle center projection image. The feature acquisition unit is used to acquire feature points of the two target circular signs, wherein the feature points include the center of the circle and the tangent point of the circle; The first extrinsic parameter calibration unit is used to determine the extrinsic parameters between each of the target cameras based on the two-dimensional feature points and the marker feature points; The three-dimensional feature extraction unit is used to optimize the initial pose of the target double-circle calibration plate in the lidar coordinate system based on the point cloud edge detection results of the two target circular marks, and transform the mark feature points to the lidar coordinate system based on the optimization results of the initial pose to obtain three-dimensional feature points that conform to the inherent consistency. The second extrinsic parameter calibration unit is used to determine the extrinsic parameters between the target camera and the target lidar based on the two-dimensional feature points and the three-dimensional feature points; The third extrinsic parameter calibration unit is used to determine the extrinsic parameters between each of the target lidars based on the three-dimensional feature points.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images