Multi-sensor fusion external parameter calibration method and device and storage medium

Abstract

The invention provides a multi-sensor fusion external parameter calibration method and device and a storage medium, and the method comprises the steps: obtaining an image of a calibration reference object in a target scene through a first sensor, and extracting the first position information of a key point; acquiring first point cloud data in the target scene through a second sensor, performing ground point cloud segmentation, and extracting second point cloud data of the calibration reference object; projecting the second point cloud data into the image to obtain second position information of the projection point set; determining a first distance according to the first position information and the second position information, and establishing a corresponding matching relation between the first sensor and the second sensor based on the first distance; and under the matching relation, external parameter matrix traversal is carried out, and an external parameter matrix with the largest projection point number in the image area of the calibration reference object projected by the second point cloud data is determined as a target external parameter matrix of the first sensor and the second sensor. According to the embodiment of the invention, the precision of multi-sensor fusion external parameter calibration is improved.

Description

Technical Field

[0001] The present application relates to the technical field of positioning and mapping, and particularly relates to a multi-sensor fusion extrinsic parameter calibration method, device, and storage medium. Background Art

[0002] The extrinsic parameter calibration between a camera and a radar is a key technology for multi-sensor fusion SLAM (Simultaneous Localization and Mapping), multi-modal perception, and navigation. The accuracy and robustness of the calibration will affect the perception accuracy, positioning, and mapping accuracy. Currently, the extrinsic parameter calibration between a camera and a radar is usually based on a calibration board. The corresponding point and line features of the calibration board in the camera and the radar are extracted, and then the extrinsic parameters are estimated by matching. The camera image has a high resolution and rich color information, and the results of extracting point and line features are relatively accurate. However, the radar point cloud is sparse, and the error of extracting point and line features is large, resulting in inaccurate extrinsic parameter calibration results. Summary of the Invention

[0003] The present application provides a multi-sensor fusion extrinsic parameter calibration method, device, and storage medium, aiming to improve the accuracy of multi-sensor fusion extrinsic parameter calibration.

[0004] To achieve the above object, the present application provides a multi-sensor fusion extrinsic parameter calibration method, including:

[0005] Obtaining, by a first sensor, an image of a calibration reference object in a target scene, and extracting first position information of key points of the image of the calibration reference object;

[0006] Obtaining, by a second sensor, first point cloud data in the target scene, performing ground point cloud segmentation processing on the first point cloud data, and extracting second point cloud data corresponding to the calibration reference object in the target scene;

[0007] Projecting the second point cloud data onto the image of the calibration reference object to obtain second position information of a projection point set;

[0008] Determining a first distance between the key points of the calibration reference object and the projection point set according to the first position information and the second position information, and establishing a corresponding matching relationship between the first sensor and the second sensor based on the first distance;

[0009] Under the matching relationship, traversing an extrinsic parameter matrix based on a preset extrinsic parameter space, and determining, as the target extrinsic parameter matrix between the first sensor and the second sensor, the extrinsic parameter matrix with the largest number of projection points when the second point cloud data is projected into the image area of the calibration reference object.

[0010] In addition, to achieve the above object, the present application further provides a multi-sensor fusion extrinsic parameter calibration device, which includes a processor and a memory. The memory stores a computer program executable by the processor. When the computer program is executed by the processor, the steps of the multi-sensor fusion extrinsic parameter calibration method as described above are implemented.

[0011] In addition, to achieve the above object, the present application further provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the multi-sensor fusion extrinsic parameter calibration method as described above.

[0012] The multi-sensor fusion extrinsic parameter calibration method, device and storage medium provided by the embodiments of the present application obtain an image of a calibration reference object in a target scene through a first sensor, extract first position information of key points of the image of the calibration reference object, and obtain first point cloud data in the target scene through a second sensor. Perform ground point cloud segmentation processing on the first point cloud data, extract second point cloud data corresponding to the calibration reference object in the target scene, project the second point cloud data onto the image of the calibration reference object, obtain second position information of the projection point set, and determine a first distance between the key points of the calibration reference object and the projection point set according to the first position information and the second position information. Based on the first distance, establish a corresponding matching relationship between the first sensor and the second sensor. Under the matching relationship, perform an extrinsic parameter matrix traversal based on a preset extrinsic parameter space, and determine the extrinsic parameter matrix with the largest number of projection points when the second point cloud data is projected into the image area of the calibration reference object as the target extrinsic parameter matrix between the first sensor and the second sensor, avoiding the problem of large errors in extracting sparse radar point cloud point and line features and improving the calibration accuracy.

[0013] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0015] Figure 1 is a flowchart of a multi-sensor fusion extrinsic parameter calibration method provided by an embodiment of the present application;

[0016] Figure 2 is a schematic diagram of a calibration room provided by an embodiment of the present application;

[0017] Figure 3 It is a schematic diagram of the corner points of the calibration board in the camera image extracted in the embodiment of the present application;

[0018] Figure 4 It is a schematic flowchart of extracting the first position information of the key points of the image of the calibration reference object provided by the embodiment of the present application;

[0019] Figure 5 It is a schematic flowchart of performing ground point cloud segmentation processing on the first point cloud data provided by the embodiment of the present application;

[0020] Figure 6 It is a schematic diagram of the calibration board plane point cloud provided by the embodiment of the present application;

[0021] Figure 7 It is a schematic flowchart of projecting the second point cloud data into the image of the calibration reference object to obtain the second position information of the projection point set provided by the embodiment of the present application;

[0022] Figure 8 It is a schematic diagram of the calibration board radar points projected onto the camera image provided by the embodiment of the present application;

[0023] Figure 9 It is the final target extrinsic matrix T of the camera and radar provided by the embodiment of the present application cl Schematic diagram of the projection map of projecting the radar point cloud onto the camera image;

[0024] Figure 10 It is a schematic block diagram of a multi-sensor fusion extrinsic calibration device provided by the embodiment of the present application. Detailed implementation manners

[0025] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0026] The flowcharts shown in the accompanying drawings are only illustrative examples, and do not necessarily include all the contents and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can also be decomposed, combined, or partially merged. Therefore, the actual execution order may change according to the actual situation.

[0027] It should be understood that the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. As used in the specification of this application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0028] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] Embodiments of this application provide a multi-sensor fusion extrinsic parameter calibration method, device and storage medium for improving the accuracy of multi-sensor fusion extrinsic parameter calibration.

[0030] Please refer to Figure 1 , Figure 1 which is a schematic flow chart of the multi-sensor fusion extrinsic parameter calibration method provided by an embodiment of this application. This method can be applied to a multi-sensor fusion extrinsic parameter calibration device or other devices. The application scenario of this method is not limited in this application.

[0031] As Figure 1 shown, the multi-sensor fusion extrinsic parameter calibration method specifically includes steps S101 to S105.

[0032] S101. Obtain an image of a calibration reference object in a target scene through a first sensor, and extract first position information of key points of the image of the calibration reference object.

[0033] In the following, taking the first sensor corresponding to a camera, the second sensor corresponding to a radar, the target scene being a calibration room, and the calibration reference object being a calibration board as an example, the multi-sensor fusion extrinsic parameter calibration method will be introduced in detail. It should be noted that the first sensor and the second sensor are not limited to a camera and a radar, and can also be other types of sensors, and the calibration reference object is not limited to a calibration board.

[0034] A calibration board is pre-set in the calibration room, and the calibration board is photographed by the camera to obtain a camera image, that is, an image of the calibration reference object. Exemplarily, multiple calibration boards are fixedly arranged in the calibration room, and the calibration boards in the camera image include multiple. For example, as Figure 2 shown, multiple calibration boards are used to build the calibration room. Therefore, there is no need to move the calibration board, and only single-frame camera and radar data need to be collected to perform the extrinsic parameter calibration between the camera and the radar, improving the calibration efficiency.

[0035] For a camera image, extract the key points of the calibration board in the camera image. Among them, the key points of the calibration board include, but are not limited to, the center point of the calibration board. For example, the key points of the calibration board also include multiple corner points, etc. For example, use the Apriltag algorithm to extract the corner points of the calibration board in the camera image. For example, as Figure 3 shown, the black dots on the calibration board in the figure are the corner points of the calibration board.

[0036] In practical applications, the camera image may be distorted. Exemplarily, perform undistortion processing on the camera image. For example, use the cv2.undistort function to perform undistortion processing on the camera image, and then extract the key points of the calibration board in the camera image. For example, use algorithms such as Apriltag and Harris to extract the corner points of the calibration board in the camera image, and use the cv2.findChessboardCorners function or other methods to extract the corner points on the calibration board. By performing undistortion processing on the camera image, the accuracy of extracting the key points of the calibration board is ensured.

[0037] For the key points of the calibration board in the camera image, obtain the position information of the key points of the calibration board. For the convenience of distinguishing and describing, the position information of the key points of the calibration board will be referred to as the first position information hereinafter. Among them, the first position information includes, but is not limited to, two-dimensional coordinates.

[0038] In some embodiments, as Figure 4 shown, step S101 may include sub-step S1011 and sub-step S1012.

[0039] S1011. Obtain the three-dimensional coordinates of the key points of the calibration reference object and the homography matrix from the three-dimensional space to the two-dimensional image plane;

[0040] S1012. Based on the three-dimensional coordinates and the homography matrix, determine the two-dimensional coordinates of the key points of the image of the calibration reference object.

[0041] Exemplarily, according to the mechanical dimensions of the calibration board, such as the length, width, diagonal length, etc. of the calibration board, construct the three-dimensional coordinates corresponding to each corner point on the calibration board. For example, assume that the length of the calibration board is L and the width is W. According to the mechanical dimensions of the calibration board, the three-dimensional coordinates of the four corner points and the center point on the calibration board can be calculated. Among them, the center point of the calibration board is the geometric center of the calibration board. For example, if the three-dimensional coordinates of the four corner points P1, P2, P3, and P4 of the calibration board are P1 = [0, 0, 0] T , P2 = [L, 0, 0] T , P3 = [L, W, 0] T , P4 = [0, W, 0] T , the three-dimensional coordinates of the center point P center of the calibration board are Pcenter = [L / 2, W / 2, 0] T , the three-dimensional coordinates of these points correspond to the actual physical space.

[0042] The three-dimensional coordinates of the corner points and the center point on the calibration board will be paired with the two-dimensional coordinates of the corresponding points in the camera image to obtain a corresponding set of three-dimensional to two-dimensional point pairs. Based on the set of three-dimensional to two-dimensional point pairs, the homography matrix from the three-dimensional space to the two-dimensional image plane is estimated. For example, the findHomography function of OpenCV is used to estimate the homography matrix.

[0043] According to the obtained homography matrix, the three-dimensional coordinates on the calibration board can be converted into two-dimensional image coordinates to obtain the two-dimensional coordinates of the corresponding points in the camera image.

[0044] Exemplarily, the two-dimensional coordinates of the key points of the calibration reference object in the image of the calibration reference object are obtained through the following formula (1):

[0045]

[0046] where P is the three-dimensional coordinates of the key point, H is the homography matrix, and p is the two-dimensional coordinates of the key point of the calibration reference object in the image of the calibration reference object. For example, p is the two-dimensional coordinates of the point corresponding to the center point of the calibration board in the camera image.

[0047] Based on the mechanical dimensions of the calibration board, additional information constraints are added during the calibration process to make the solution more robust. By introducing the calibration board size information, some common noise and error sources can be countered, such as illumination changes, blurring, and non-linear distortion in the image, thereby improving the robustness of the calibration in practical applications.

[0048] S102. Obtain the first point cloud data in the target scene through the second sensor, perform ground point cloud segmentation processing on the first point cloud data, and extract the second point cloud data corresponding to the calibration reference object in the target scene.

[0049] The radar point cloud obtained by the radar is the first point cloud data. In the calibration room, since the calibration board is off the ground, by removing the ground points in the radar point cloud, that is, performing ground point cloud segmentation processing on the first point cloud data, the calibration board plane point cloud is extracted, that is, the second point cloud data. Based on the second point cloud data, the external parameters of the camera and the radar are calibrated, thereby further improving the calibration accuracy.

[0050] In some embodiments, as Figure 5 shown, step S102 may include sub-steps S1021 to S1024.

[0051] S1021. Select points within a preset angle range from the first point cloud data to obtain a candidate point set;

[0052] S1022. Determine ground candidate points from the candidate point set, where the height coordinate of the ground candidate points is less than the average height coordinate of the candidate point set;

[0053] S1023. Perform plane fitting on the ground candidate points to obtain a ground candidate point plane, and obtain the normal vector of the ground candidate point plane and the second distance from the ground candidate point plane to the origin;

[0054] S1024. Determine the ground points in the first point cloud data based on the normal vector and the second distance, and remove the ground points.

[0055] Each point P in the radar point cloud i has three-dimensional coordinates P i = [x, y, z] T , and the heading angle Φ and pitch angle θ of each point P are calculated through the following formula (2): i The heading angle Φ and pitch angle θ of

[0056]

[0057] Exemplarily, the angle ranges of the heading angle Φ and pitch angle θ are preset in advance. Based on the heading angle Φ and pitch angle θ of each point P i , the radar points that meet the preset angle ranges of the heading angle Φ and pitch angle θ are selected as candidate points, and a candidate point set is obtained based on the selected candidate points.

[0058] After that, the average value of the height coordinates z of all points in the candidate point set is calculated to obtain the average height coordinate z'. The points with z values less than the average value z' are selected as ground candidate points.

[0059] Perform plane fitting on the determined ground candidate points to obtain a ground candidate point plane, which can effectively describe the characteristics of the ground. For example, use the RANSAC (Random Sample Consensus) algorithm to fit the ground candidate points to obtain a ground candidate point plane, and obtain the normal vector n = [n x , n y , n z of the ground candidate point plane T and the second distance d from the ground candidate point plane to the origin. Determine the ground points in the radar point cloud based on the normal vector n and the second distance d, and remove the ground points.

[0060] Exemplarily, the ground points in the first point cloud data are determined through the following formula (3):

[0061] |n T P i+d∣<D g (3)

[0062] where n is the normal vector, d is the second distance, and P i is the three-dimensional coordinate of a point in the first point cloud data, and D g is the first preset distance threshold. The specific value of D g can be flexibly set according to the actual situation and is not specifically limited in this application.

[0063] After removing the ground points in the lidar point cloud, clustering and plane fitting are performed on the remaining lidar points. For example, set the distance threshold to 0.08 m, the minimum number of points to 200, and the maximum number of points to 1000. Use PCL (Point Cloud Library) for Euclidean distance clustering, fit the plane of each cluster point set based on RANSAC, remove non-plane points, and obtain the calibration board plane point cloud, that is, the second point cloud data. For example, as Figure 6 shown, it is the calibration board plane point cloud obtained. Exemplarily, the accuracy of the obtained calibration board plane point cloud can be verified by comparing with the mechanical dimensions of the actual calibration board.

[0064] S103. Project the second point cloud data onto the image of the calibration reference object to obtain the second position information of the projection point set.

[0065] Exemplarily, project the center point of the calibration board of the calibration board plane point cloud onto the camera image to obtain the position information of the projection point of the center point of the calibration board projected onto the camera image. For the convenience of distinguishing and describing, the position information of the projection point will be referred to as the second position information below. Among them, the second position information includes but is not limited to two-dimensional coordinates.

[0066] In some embodiments, as Figure 7 shown, step S103 may include sub-step S1031 and sub-step S1032.

[0067] S1031. Calculate the coordinate average value of the second point cloud data as the third position information of the key point of the calibration reference object of the second point cloud data;

[0068] S1032. Determine the second position information based on the third position information, the internal parameters of the first sensor, and the mechanical external parameters between the first sensor and the second sensor.

[0069] Among them, the second position information includes but is not limited to two-dimensional coordinates, and the third position information includes but is not limited to three-dimensional coordinates. Exemplarily, calculate the coordinate average value of the three-dimensional coordinates of each point in the calibration board plane point cloud as the three-dimensional coordinate P o of the center point P o of the calibration board in the calibration board plane point cloud T。

[0070] Exemplarily, the two-dimensional coordinates of the projection point are calculated by the following formula (4):

[0071]

[0072] where P o is the three-dimensional coordinate of the key point of the calibration reference object of the second point cloud data, such as the three-dimensional coordinate of the center point of the calibration board in the point cloud of the calibration board plane, K is the internal parameter of the first sensor, such as the internal parameter of the camera, and T cl_init is the mechanical external parameter between the first sensor and the second sensor, such as the mechanical external parameter between the camera and the radar, and p o is the two-dimensional coordinate of the projection point.

[0073] S104. Determine the first distance between the key point of the calibration reference object and the set of projection points according to the first position information and the second position information, and establish the corresponding matching relationship between the first sensor and the second sensor based on the first distance.

[0074] Exemplarily, according to the two-dimensional coordinates of the center point of the calibration board in the camera image and the two-dimensional coordinates of the projection point, calculate the first distance between the center point of the calibration board in the camera image and the projection point, and establish the corresponding matching relationship between the camera and the radar based on the magnitude of the first distance. Exemplarily, if the first distance between the center point of the calibration board in the camera image and the projection point is less than the second preset distance threshold, it is determined that the camera and the radar are correspondingly calibrated with the calibration board. Among them, the specific value of the second preset distance threshold can be flexibly set according to the actual situation, and no specific limitation is made in this application.

[0075] S105. Under the matching relationship, perform an external parameter matrix traversal based on the preset external parameter space, and determine the target external parameter matrix between the first sensor and the second sensor as the external parameter matrix with the largest number of projection points when the second point cloud data is projected into the image area of the calibration reference object.

[0076] Among them, the preset external parameter space includes a preset window, a preset step size, etc. The preset window and the preset step size can be flexibly set according to the actual situation, and no specific limitation is made in this application.

[0077] For example, the preset window is set to [-2°, 2°] and the preset step size is set to 0.05° in advance. Take the mechanical external parameter T cl_init between the camera and the radar as the initial value, and take the Euler angle corresponding to the mechanical external parameter T cl_init between the camera and the radar as the center, and traverse all Euler angles based on the preset window [-2°, 2°] and the preset step size 0.05°. Use the external parameter matrix corresponding to each Euler angle to project all calibration board radar points into the camera image. For example, as Figure 8As shown, the quadrilateral frame represents the calibration board area, the red dots are the points outside the quadrilateral frame, and the blue dots are the points inside the quadrilateral frame. By judging whether the calibration board radar points are inside the quadrilateral frame, the external parameter matrix with the largest number of points inside the quadrilateral frame is taken as the final target external parameter matrix T of the camera and the radar. cl .

[0078] In some embodiments, traversing the external parameter matrix based on a preset external parameter space, and determining the external parameter matrix with the largest number of projection points when projecting the second point cloud data into the image area of the calibration reference object as the target external parameter matrix of the first sensor and the second sensor, includes: traversing the Euler angles corresponding to the external parameter matrix based on a preset window and a preset step size, adjusting the Euler angles according to a preset perturbation factor, projecting the points corresponding to the adjusted Euler angles onto the image of the calibration reference object, and determining the external parameter matrix with the largest number of projection points projected into the image area of the calibration reference object as the target external parameter matrix.

[0079] During the process of traversing the Euler angles, in order to avoid falling into a local optimal solution, a small perturbation factor is added to each Euler angle to expand the search space. For example, when the angle values selected during the current traversal are (φ0, θ0, ψ0), a random perturbation factor δ is added to each base angle to adjust the angle values, and the adjusted angle values are (φ, θ, ψ) = (φ0 + δ φ , θ0 + δ θ , ψ0 + δ ψ ).

[0080] Exemplarily, the generation method of the perturbation factor δ can adopt a uniform distribution or a normal distribution:

[0081] For the perturbation factor generated by the uniform distribution method, the perturbation factor δ is added to each angle value, δ ∈ [-Δ, Δ], where Δ is the predefined maximum perturbation amplitude. For example, it is preset that Δ = 0.5°, and of course, it can also be set to other values, which is not specifically limited in this application.

[0082] For the perturbation factor generated by the normal distribution method, the perturbation factor δ is generated according to a given standard deviation σ, making the perturbation more concentrated near the center of the angle. For example, the perturbation factor δ ~ N(0, σ) is generated, where the value of σ is not specifically limited in this application. For example, σ can be set to a value less than 1°, or σ can be set to a value less than 0.5°.

[0083] By introducing the perturbation factor, the local optimal problem caused by inaccurate initial guesses during the search process is reduced, and without affecting the global function search ability, the local angle adjustment can be more finely optimized, making the estimation result of the external parameters more accurate and stable.

[0084] Project the radar points corresponding to the adjusted angle values (φ, θ, ψ) onto the calibration plate area in the camera image, calculate the projection error, and record the projection accuracy.

[0085] Exemplarily, when the current projection error reaches a certain threshold, reduce the perturbation amplitude to narrow the perturbation range for more refined optimization.

[0086] For example, when the projection error is less than a certain threshold (e.g., 1 cm), automatically adjust the perturbation amplitude Δ from Δ = 0.5° to Δ = 0.2°, and continue the search within a smaller range.

[0087] Exemplarily, in the initial stage of the search, adopt a larger perturbation amplitude to expand the search range as much as possible and avoid falling into local optimal solutions. A larger perturbation amplitude can accelerate the search process, especially when the search space is large, improving the search efficiency. As the search progresses, gradually reduce the perturbation amplitude to precisely adjust the optimal solution, improve the accuracy of the final result, and designing random perturbations can jump out of local optima and enhance the effect of global optimization.

[0088] By adaptively and dynamically adjusting the perturbation amplitude, the optimization process can be made more efficient, reducing the computational amount while ensuring the accuracy of the final result.

[0089] Determine the final target extrinsic matrix T of the camera and the radar cl After that, use the final target extrinsic matrix T of the camera and the radar cl The projection of the radar point cloud onto the camera image is as Figure 9 shown. It can be seen from the figure that the projection error is small, indicating high calibration accuracy.

[0090] In the above embodiments, by obtaining the first position information of the key points of the calibration plate in the camera image, and projecting the radar point cloud onto the camera image after removing the ground points to obtain the second position information of the projection points, according to the first position information and the second position information, determine the first distance between the key points of the calibration plate in the camera image and the corresponding projection points, establish the corresponding matching relationship between the camera and the radar based on the first distance, and under the matching relationship, perform an extrinsic matrix traversal based on a preset window and a preset step size, and determine the extrinsic matrix with the largest number of calibration plate radar points projected into the calibration plate area in the camera image as the final target extrinsic matrix of the camera and the radar, thus avoiding extracting sparse radar point cloud point and line features and falling into local optimal solutions, improving the calibration accuracy and robustness. And by introducing an angle perturbation strategy, it is possible to effectively avoid falling into local optimal solutions and improve the accuracy of the optimal extrinsic parameters. The angle perturbation strategy can generate perturbation factors through uniform distribution or normal distribution, and dynamically adjust the perturbation amplitude within global and local ranges. This strategy can ensure high precision of the radar point cloud projected into the quadrilateral frame of the camera image, improving the accuracy and robustness of calibration.

[0091] Please refer to Figure 10 , Figure 10 which is a schematic block diagram of a multi-sensor fusion extrinsic parameter calibration device provided by an embodiment of the present application. The multi-sensor fusion extrinsic parameter calibration device can be configured in a computer device to execute the foregoing multi-sensor fusion extrinsic parameter calibration method.

[0092] As Figure 10 shown, the multi-sensor fusion extrinsic parameter calibration device 200 may include a processor 210 and a memory 220. Among them, the processor 210 is connected to the memory 220 through a bus, and this bus is, for example, an I2C (Inter-integrated Circuit) bus.

[0093] Specifically, the processor 210 may be a micro-control unit (MCU), a central processing unit (CPU), a digital signal processor (DSP), etc.

[0094] Specifically, the memory 220 may be a Flash chip, a read-only memory (ROM), a magnetic disk, an optical disc, a USB flash drive, or a mobile hard disk, etc. Various computer programs for the processor 210 to execute are stored in the memory 220.

[0095] Among them, the processor 210 is used to run the computer program stored in the memory and, when executing the computer program, implement:

[0096] Obtain an image of a calibration reference object in the target scene through a first sensor, and extract first position information of key points of the image of the calibration reference object;

[0097] Obtain first point cloud data in the target scene through a second sensor, perform ground point cloud segmentation processing on the first point cloud data, and extract second point cloud data corresponding to the calibration reference object in the target scene;

[0098] Project the second point cloud data onto the image of the calibration reference object to obtain second position information of a set of projection points;

[0099] According to the first position information and the second position information, determine a first distance between the key points of the calibration reference object and the set of projection points, and establish a corresponding matching relationship between the first sensor and the second sensor based on the first distance;

[0100] Under the matching relationship, the external parameter matrix is ​​traversed based on the preset external parameter space, and the external parameter matrix with the largest number of projection points projected onto the second point cloud data within the image area of ​​the calibration reference object is determined as the target external parameter matrix of the first sensor and the second sensor.

[0101] In some embodiments, the first position information includes two-dimensional coordinates, and the key points include the center point and the corner points of the calibration reference object. When the processor 210 extracts the first position information of the key points of the image of the calibration reference object, it is used to implement:

[0102] Obtaining the three-dimensional coordinates of the key points of the calibration reference object and a homography matrix from the three-dimensional space to the two-dimensional image plane;

[0103] Based on the three-dimensional coordinates and the homography matrix, the two-dimensional coordinates of the key points of the image of the calibration reference object are determined.

[0104] In some embodiments, when the processor 210 determines the two-dimensional coordinates of the key points of the image of the calibration reference object based on the three-dimensional coordinates and the homography matrix, it is configured to implement:

[0105] Based on the formula Calculate and obtain the two-dimensional coordinates;

[0106] Wherein, P is the three-dimensional coordinate, H is the homography matrix, and p is the two-dimensional coordinate.

[0107] In some embodiments, when implementing the ground point cloud segmentation processing for the first point cloud data, the processor 210 is used to implement:

[0108] Selecting points within a preset angle range from the first point cloud data to obtain a candidate point set;

[0109] Determine a ground candidate point from the candidate point set, wherein the height coordinate of the ground candidate point is less than an average value of the height coordinates of the candidate point set;

[0110] Performing plane fitting processing on the ground candidate point to obtain a ground candidate point plane, and acquiring a normal vector of the ground candidate point plane and a second distance from the ground candidate point plane to the origin;

[0111] A ground point in the first point cloud data is determined based on the normal vector and the second distance, and the ground point is eliminated.

[0112] In some embodiments, when implementing the determining of the ground point in the first point cloud data based on the normal vector and the second distance, the processor 210 is configured to implement:

[0113] If the three-dimensional coordinates of the points in the first point cloud data satisfy |n T P i + d| < D g , then the points in the first point cloud data are determined as ground points;

[0114] where n is the normal vector, d is the second distance, and P i is the three-dimensional coordinates of the points in the first point cloud data, and D g is the first preset distance threshold.

[0115] In some embodiments, when the processor 210 implements projecting the second point cloud data into the image of the calibration reference object and obtaining the second position information of the set of projection points, it is used to implement:

[0116] Calculate the coordinate average value of the second point cloud data as the third position information of the key points of the calibration reference object of the second point cloud data;

[0117] Based on the third position information, the internal parameters of the first sensor, and the mechanical external parameters between the first sensor and the second sensor, determine the second position information.

[0118] In some embodiments, when the processor 210 implements establishing the corresponding matching relationship between the first sensor and the second sensor based on the first distance, it is used to implement:

[0119] If the first distance is less than the second preset distance threshold, it is determined that the first sensor and the second sensor are correspondingly calibrated with the reference object.

[0120] In some embodiments, the second position information includes two-dimensional coordinates, the third position information includes three-dimensional coordinates, and when the processor 210 implements determining the second position information based on the third position information, the internal parameters of the first sensor, and the mechanical external parameters between the first sensor and the second sensor, it is used to implement:

[0121] Based on the formula Calculate the two-dimensional coordinates of the projection points;

[0122] where P o is the three-dimensional coordinates of the key points of the calibration reference object of the second point cloud data, K is the internal parameters of the first sensor, T cl_init is the mechanical external parameters between the first sensor and the second sensor, and p o is the two-dimensional coordinates of the projection points.

[0123] In some embodiments, when the processor 210 traverses the extrinsic parameter matrix based on a preset extrinsic parameter space and determines the extrinsic parameter matrix with the largest number of projection points when projecting the second point cloud data into the image area of the calibration reference object as the target extrinsic parameter matrix between the first sensor and the second sensor, it is used to implement:

[0124] Traverse the Euler angles corresponding to the extrinsic parameter matrix based on a preset window and a preset step size, adjust the Euler angles according to a preset perturbation factor, project the points corresponding to the adjusted Euler angles onto the image of the calibration reference object, and determine the extrinsic parameter matrix with the largest number of projection points projected into the image area of the calibration reference object as the target extrinsic parameter matrix.

[0125] In some embodiments, a plurality of the calibration reference objects are fixedly arranged in the target scene, and the calibration reference objects in the image of the calibration reference object include a plurality.

[0126] The multi-sensor fusion extrinsic parameter calibration device 200 can execute the multi-sensor fusion extrinsic parameter calibration method provided by the embodiments of the present application. Therefore, the beneficial effects that can be achieved by the multi-sensor fusion extrinsic parameter calibration method provided by the embodiments of the present application can be realized. For details, see the previous embodiments and will not be repeated here.

[0127] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-sensor fusion extrinsic parameter calibration method as described above are implemented.

[0128] Among them, the computer-readable storage medium may be an internal storage unit of the multi-sensor fusion extrinsic parameter calibration device or computer device described in the foregoing embodiments, such as the hard disk or memory of the multi-sensor fusion extrinsic parameter calibration device or computer device. The computer-readable storage medium may also be an external storage device of the multi-sensor fusion extrinsic parameter calibration device or computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital Card (SDCard), a Flash Card, etc. equipped on the multi-sensor fusion extrinsic parameter calibration device or computer device.

[0129] Since the computer program stored in this storage medium can execute any of the multi-sensor fusion extrinsic parameter calibration methods provided by the embodiments of the present application, the beneficial effects that can be achieved by any of the multi-sensor fusion extrinsic parameter calibration methods provided by the embodiments of the present application can be realized. For details, see the previous embodiments and will not be repeated here.

[0130] It should be noted that in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or system. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or system comprising such element.

[0131] As described above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily conceive of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present application.

Claims

1. A multi-sensor fusion extrinsic parameter calibration method, characterized in that, include: Acquire an image of a calibration reference object in a target scene through a first sensor, and extract first position information of key points of the image of the calibration reference object; Acquire first point cloud data in the target scene through a second sensor, perform ground point cloud segmentation processing on the first point cloud data, and extract second point cloud data corresponding to the calibration reference object in the target scene; Projecting the second point cloud data into the image of the calibration reference object to obtain second position information of the projection point set; Determine a first distance between the key point of the calibration reference object and the set of projection points according to the first position information and the second position information, and establish a corresponding matching relationship between the first sensor and the second sensor based on the first distance; Under the matching relationship, the external parameter matrix is ​​traversed based on the preset external parameter space, and the external parameter matrix with the largest number of projection points projected onto the second point cloud data within the image area of ​​the calibration reference object is determined as the target external parameter matrix of the first sensor and the second sensor.

2. The multi-sensor fusion external parameter calibration method according to claim 1, wherein The first position information includes two-dimensional coordinates, the key points include the center point and corner points of the calibration reference object, and the first position information of the key points of the image of the calibration reference object is extracted, including: Obtaining the three-dimensional coordinates of the key points of the calibration reference object and a homography matrix from the three-dimensional space to the two-dimensional image plane; Based on the three-dimensional coordinates and the homography matrix, the two-dimensional coordinates of the key points of the image of the calibration reference object are determined.

3. The multi-sensor fusion extrinsic parameter calibration method according to claim 2, wherein The step of determining the two-dimensional coordinates of the key points of the image of the calibration reference object based on the three-dimensional coordinates and the homography matrix includes: Based on the formula the two-dimensional coordinates are calculated; Wherein, P is the three-dimensional coordinate, H is the homography matrix, and p is the two-dimensional coordinate.

4. The multi-sensor fusion extrinsic parameter calibration method according to claim 1, characterized in that The performing ground point cloud segmentation processing on the first point cloud data includes: Selecting points within a preset angle range from the first point cloud data to obtain a candidate point set; Determine a ground candidate point from the candidate point set, wherein the height coordinate of the ground candidate point is less than an average value of the height coordinates of the candidate point set; Performing plane fitting processing on the ground candidate point to obtain a ground candidate point plane, and acquiring a normal vector of the ground candidate point plane and a second distance from the ground candidate point plane to the origin; A ground point in the first point cloud data is determined based on the normal vector and the second distance, and the ground point is eliminated.

5. The method for calibrating the external parameters of multi-sensor fusion according to claim 4, wherein The determining of the ground point in the first point cloud data based on the normal vector and the second distance comprises: If the three-dimensional coordinates of the points in the first point cloud data satisfy |n T P i + d| < D g , then the points in the first point cloud data are determined as ground points; where n is the normal vector, d is the second distance, and P i is the three-dimensional coordinate of a point in the first point cloud data, and D g is the first preset distance threshold.

6. The multi-sensor fusion external parameter calibration method according to claim 1, characterized in that The step of projecting the second point cloud data onto the image of the calibration reference object to obtain second position information of the projection point set includes: Calculating an average coordinate value of the second point cloud data as third position information of a key point of a calibration reference object of the second point cloud data; The second position information is determined based on the third position information, the first sensor internal parameters, and the first sensor and the second sensor mechanical external parameters.

7. The multi-sensor fusion extrinsic parameter calibration method according to claim 6, wherein The establishing a corresponding matching relationship between the first sensor and the second sensor based on the first distance includes: If the first distance is less than a second preset distance threshold, it is determined that the calibration reference objects corresponding to the first sensor and the second sensor match.

8. The multi-sensor fusion extrinsic parameter calibration method according to claim 6, characterized in that The second position information includes two-dimensional coordinates, and the third position information includes three-dimensional coordinates. Determining the second position information based on the third position information, the internal parameters of the first sensor, and the mechanical external parameters between the first sensor and the second sensor includes: Based on the formula Calculate the two-dimensional coordinates of the projection point; Among them, P o is the three-dimensional coordinate of the calibration reference key point of the second point cloud data, K is the internal parameter of the first sensor, and T cl_init is the mechanical external parameter between the first sensor and the second sensor, and p o is the two-dimensional coordinate of the projection point.

9. The multi-sensor fusion extrinsic parameter calibration method according to claim 1, wherein Performing a traversal of the external parameter matrix based on a preset external parameter space, and determining the external parameter matrix with the largest number of projection points when projecting the second point cloud data into the image area of the calibration reference object, including: Traversing the Euler angles corresponding to the external parameter matrix based on a preset window and a preset step size, adjusting the Euler angles according to a preset perturbation factor, projecting the points corresponding to the adjusted Euler angles onto the image of the calibration reference object, and determining the external parameter matrix with the largest number of projection points projected into the image area of the calibration reference object as the target external parameter matrix.

10. The multi-sensor fusion extrinsic parameter calibration method according to any one of claims 1 to 9, characterized in that, A plurality of the calibration reference objects are fixedly arranged in the target scene, and the calibration reference objects in the image of the calibration reference object include a plurality.

11. A multi-sensor fusion external parameter calibration device, characterized in that The multi-sensor fusion external parameter calibration device includes a processor and a memory. The memory stores a computer program executable by the processor. When the computer program is executed by the processor, the steps of the multi-sensor fusion external parameter calibration method according to any one of claims 1 to 10 are implemented.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the multi-sensor fusion external parameter calibration method according to any one of claims 1 to 10.

Images