External parameter calibration method and device, electronic equipment and storage medium
By acquiring the captured images and point cloud data of the camera and radar, determining the initial external parameters and calibrating it, the problem of impossible to accurately calibrate external parameters in the prior art is solved, and the accurate information fusion of radar and camera is achieved.
Patent Information
- Application Number
- CN202411468016.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-02-28
AI Technical Summary
In the prior art, external parameter calibration cannot be accurately performed, resulting in the inability of information fusion between radar and camera.
By acquiring the captured images and point cloud data of the same target acquisition area of the camera and radar, the initial external parameters of the camera and radar are determined, and calibrated according to the initial external parameters, the first target corner point and the second target corner point are determined, and the target external parameters of the camera and radar are determined.
Accurate external parameter calibration of camera and radar is realized, so that radar and camera can accurately fusion information, avoiding the failure of information fusion caused by inability to accurately diametric calibration.
Smart Images
Figure CN119540368A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to an external parameter calibration method, device, electronic device and storage medium. Background Art
[0002] Cameras and LiDAR are two very important sensors on autonomous mobile devices (such as self-driving vehicles, robots, etc.). By fusing the information of the two sensors, the accuracy and robustness of algorithms in key links such as map creation, positioning, and environmental perception can be effectively improved. In order to achieve information fusion of the two sensors, good external parameter calibration is one of the important prerequisites.
[0003] The commonly used camera radar extrinsic calibration method is the feature point matching method. In the actual calibration process, the point feature matching method often fails to extrinsic calibration due to mismatching caused by inaccurately selected points. Summary of the invention
[0004] In view of this, the embodiments of the present application provide an external parameter calibration method, device, electronic device and storage medium to solve the problem in the prior art that the external parameter calibration cannot be accurately performed, resulting in the inability to perform information fusion between the radar and the camera.
[0005] In a first aspect of an embodiment of the present application, a method for calibrating external parameters is provided, the method comprising: obtaining captured images and point cloud data obtained by capturing the same target acquisition area by a camera and a radar, wherein a target object for calibrating the external parameters of the camera and the radar is provided in the target acquisition area; determining initial external parameters corresponding to the camera and the radar according to the captured images corresponding to the camera and the point cloud data corresponding to the radar, the initial external parameters being used to characterize the initial position and initial direction of the camera in the three-dimensional space where the radar is located; calibrating the camera and the radar according to the initial external parameters, and when the calibration is completed, determining a first target corner point from the captured image corresponding to the camera, and determining a second target corner point from the point cloud data corresponding to the radar; determining target external parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point, the target external parameters being used to characterize the target position and target direction of the camera in the three-dimensional space where the radar is located.
[0006] According to a second aspect of an embodiment of the present application, an external parameter calibration device is provided, which includes: an acquisition module, configured to acquire captured images and point cloud data obtained by a camera and a radar shooting the same target acquisition area, wherein a target object for calibrating the external parameters of the camera and the radar is provided in the target acquisition area; a calibration module, configured to determine initial external parameters corresponding to the camera and the radar according to the captured images corresponding to the camera and the point cloud data corresponding to the radar, wherein the initial external parameters are used to characterize the initial position and initial direction of the camera in the three-dimensional space where the radar is located; a corner point module, configured to calibrate the camera and the radar according to the initial external parameters, and when the calibration is completed, determine a first target corner point from the captured image corresponding to the camera, and determine a second target corner point from the point cloud data corresponding to the radar; the calibration module is also configured to determine target external parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point, wherein the target external parameters are used to characterize the target position and target direction of the camera in the three-dimensional space where the radar is located.
[0007] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0008] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0009] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the method in the embodiments of the present application obtains captured images and point cloud data obtained by capturing the same target acquisition area by a camera and a radar; determines the initial external parameters corresponding to the camera and the radar according to the captured images corresponding to the camera and the point cloud data corresponding to the radar; calibrates the camera and the radar according to the initial external parameters, and when the calibration is completed, determines the first target corner point from the captured image corresponding to the camera, and determines the second target corner point from the point cloud data corresponding to the radar; determines the target external parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point; the present application first calibrates the camera and the radar according to the initial external parameters, so as to set the camera and the radar to the same viewing angle, and then determines the target external parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point, thereby realizing the accurate determination of the target external parameters of the camera and the radar through the corner points, so that the camera and the radar can subsequently accurately perform information fusion, thereby avoiding the problem in the related art that the external parameters cannot be accurately calibrated, resulting in the inability of the radar and the camera to perform information fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0011] Figure 1 It is a flow chart of an external parameter calibration method provided in an embodiment of the present application;
[0012] Figure 2 It is a flow chart of another external parameter calibration method provided in an embodiment of the present application;
[0013] Figure 3 It is a flowchart of another external parameter calibration method provided in an embodiment of the present application;
[0014] Figure 4 This is a basic schematic diagram of a target object and a shadow overlap provided by an embodiment of the present application;
[0015] Figure 5 This is a basic schematic diagram of target object and shadow segmentation provided in an embodiment of the present application;
[0016] Figure 6 It is a flowchart of another external parameter calibration method provided in an embodiment of the present application;
[0017] Figure 7 This is a basic schematic diagram of a target object obscured by an obstruction provided in an embodiment of the present application;
[0018] Figure 8 It is a flowchart of another optional external parameter calibration method provided in an embodiment of the present application;
[0019] Fig. 9 This is a basic schematic diagram of a boundary line rotating around an intersection point to obtain a new intersection point provided in an embodiment of the present application;
[0020] Fig.10 It is a flowchart of another optional external parameter calibration method provided in an embodiment of the present application;
[0021] Fig.11 This is a basic schematic diagram of radar corresponding point cloud data provided by an embodiment of the present application;
[0022] Fig.12 This is a basic schematic diagram of plane fitting of point cloud data provided by an embodiment of the present application;
[0023] Fig.13 It is a flowchart of another optional external parameter calibration method provided in an embodiment of the present application;
[0024] Fig.14 It is a structural schematic diagram of an external parameter calibration device provided in an embodiment of the present application;
[0025] Fig.15 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0027] A method and device for calibrating external parameters according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0028] Figure 1 This is an external parameter calibration method provided in the embodiment of the present application, such as Figure 1 As shown, the method includes:
[0029] S101, obtaining images and point cloud data obtained by shooting the same target acquisition area by a camera and a radar;
[0030] S102, determining initial external parameters corresponding to the camera and the radar according to the captured image corresponding to the camera and the point cloud data corresponding to the radar;
[0031] S103, calibrating the camera and the radar according to the initial external parameters, and when the calibration is completed, determining the first target corner point from the captured image corresponding to the camera, and determining the second target corner point from the point cloud data corresponding to the radar;
[0032] S104: Determine target extrinsic parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point.
[0033] It can be understood that the external parameter calibration method provided in this example is applied to an autonomous mobile device (such as an autonomous driving vehicle, a robot, etc.), which is provided with a radar and a camera; it should be noted that in this application, the installation position of the laser radar and the camera can be installed up and down, that is, the radar and the camera can be stacked up and down on a monitoring pole or device; in some examples, the radar and the camera can be installed left and right, that is, the radar and the camera can be stacked up and down on a monitoring pole or device.
[0034] In addition, in order to enable the radar and camera to collect data in the same target collection area, when installing the camera and radar, the radar field of view angle should be limited according to the camera's field of view angle (or the camera's field of view angle should be limited according to the radar field of view angle) so that the radar and camera can collect data in the same target collection area.
[0035] It can be understood that the extrinsic parameter calibration method provided in the embodiment of the present application can realize the extrinsic parameter calibration for any number of cameras and any number of radars. That is to say, except for the cases where the number is particularly emphasized or a specific object is referred to, the "camera" and "radar" described in the present application are general terms and can be used to refer to any one or more cameras and radars on an autonomous mobile device, respectively.
[0036] Among them, the radar mentioned in this application includes but is not limited to laser radar.
[0037] In this application, the external parameter calibration of the camera and radar is divided into initial calibration and precise calibration. In the process of initial calibration, the captured images and point cloud data obtained by the camera and radar shooting the same target acquisition area are first obtained; it can be understood that the target object referred to in this application refers to any object that can be used for calibration and appears in the target acquisition area. The target object has corner point features. The target object may refer to an object with obvious corner point information such as a truck or bus perpendicular to the ground, and the angle between the normal vector of the side of the target object facing the camera and radar and the camera optical axis is less than 30 degrees. Among them, the corner point refers to the point that belongs to the two sides of the object in the object structure; the two sides should have different normal vectors. For example, the corners of a car; the end of the handlebar of a tricycle, etc. The above-mentioned target object can also be an object with corner point information set by relevant personnel, such as a rectangular object set by relevant personnel, a triangular object set by relevant personnel, etc.
[0038] Continuing with the above example, the radar collects point cloud data in the target collection area to obtain point cloud data; the camera photographs the target collection area to obtain photographed images; it can be understood that in order to enable the autonomous mobile device to obtain accurate information, the radar and the camera collect data in the target collection area at the same time, that is, the point cloud data and the photographed images are acquired at the same time.
[0039] In some examples, after acquiring the captured images and point cloud data, the present application determines the initial extrinsic parameters corresponding to the camera and the radar based on the captured images corresponding to the camera and the point cloud data corresponding to the radar. The initial extrinsic parameters are used to characterize the initial position and initial direction of the camera in the three-dimensional space where the radar is located.
[0040] In some examples, after obtaining the initial external parameters, the present application calibrates the camera and radar according to the initial external parameters, thereby rotating the camera and radar to the same viewing angle.
[0041] When the camera and radar are calibrated according to the initial external parameters, the application completes the initial calibration of the camera and radar and enters the precise calibration step. In the precise calibration, this example determines the first target corner point from the captured image corresponding to the camera, and then determines the second target corner point from the point cloud data corresponding to the radar, and then determines the target external parameters corresponding to the camera and radar based on the first target corner point and the second target corner point. It can be understood that the method of determining the target external parameters in the precise calibration is the same as the method of determining the initial external parameters in the initial calibration, and will not be repeated here.
[0042] According to the technical solution provided in the embodiment of the present application, captured images and point cloud data obtained by capturing the same target acquisition area by a camera and a radar are obtained; initial external parameters corresponding to the camera and the radar are determined according to the captured images corresponding to the camera and the point cloud data corresponding to the radar; the camera and the radar are calibrated according to the initial external parameters, and when the calibration is completed, the first target corner point is determined from the captured image corresponding to the camera, and the second target corner point is determined from the point cloud data corresponding to the radar; the target external parameters corresponding to the camera and the radar are determined according to the first target corner point and the second target corner point; the present application first calibrates the camera and the radar according to the initial external parameters, so as to set the camera and the radar to the same viewing angle, and then determines the target external parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point, thereby realizing the accurate determination of the target external parameters of the camera and the radar through the corner points, so that the camera and the radar can accurately perform information fusion later, thereby avoiding the problem in the related art that the external parameters cannot be accurately calibrated, resulting in the inability of the radar and the camera to perform information fusion.
[0043] In some embodiments, Figure 2 As shown, according to the captured image corresponding to the camera and the point cloud data corresponding to the radar, the initial external parameters corresponding to the camera and the radar are determined, including:
[0044] S201, determining a first calibration point from a captured image, and determining a second calibration point from point cloud data, wherein a distance between the first calibration point and the second calibration point is lower than a preset distance threshold;
[0045] S202, determining the coordinates of the first calibration point in the captured image, and determining the coordinates of the second calibration point in the point cloud data;
[0046] S203, obtaining camera intrinsic parameters corresponding to the camera, and determining initial extrinsic parameters according to the coordinates corresponding to the first calibration point, the coordinates corresponding to the second calibration point, and the camera intrinsic parameters.
[0047] It can be understood that in the process of initial calibration of the camera and radar, the present application needs to select matching point pairs, and then determine the initial external parameters through PnP (Perspective-n-Point) calculation.
[0048] Specifically, this example first determines a first calibration point from the captured image and determines a second calibration point from the point cloud data. The interval distance between the first calibration point and the second calibration point is lower than a preset distance threshold, thereby making the first calibration point and the second calibration point closer in reality.
[0049] It can be understood that the first calibration point and the second calibration point can be a pair of points selected manually according to actual conditions, or they can be a pair of points determined automatically. The first calibration point and the second calibration point can be roughly matched without excessive requirements on accuracy.
[0050] In some examples, the present application determines the coordinates of the first calibration point in the captured image and determines the coordinates of the second calibration point in the point cloud data, wherein, since the image captured by the camera is a plane image, the coordinates of the first calibration point in the captured image are marked as (u, v); and since the point cloud data acquired by the radar is three-dimensional data, the coordinates of the second calibration point in the point cloud data are marked as (x, y, z).
[0051] In some examples, after obtaining the coordinates of the first calibration point in the captured image and the coordinates of the second calibration point in the point cloud data, PnP calculation can be performed to determine the initial external parameters; specifically, the camera intrinsic parameters corresponding to the camera are obtained, and the initial external parameters are determined based on the coordinates corresponding to the first calibration point, the coordinates corresponding to the second calibration point, and the camera intrinsic parameters.
[0052] Following the above example, the camera internal parameters are recorded as The relationship between the first calibration point and the second calibration point is as follows:
[0053]
[0054] Among them, the above R is the rotation matrix, T is the translation matrix, R and T together constitute the initial external parameters, the coordinates of the first calibration point in the captured image (u, v) the coordinates of the second calibration point in the point cloud data (x, y, z) and the camera intrinsic parameters Substituting the above relationship formula into the initial external parameters can determine the initial external parameters.
[0055] It is understandable that in order to more accurately calculate the above initial external parameters, this example will obtain multiple groups of point pairs, each group of point pairs consists of a first calibration point and a second calibration point. Preferably, in order to accurately obtain the above initial external parameters and avoid excessive calculation leading to waste of resources, the number of the above point pairs is in the range of 6-10.
[0056] After obtaining the above initial external parameters, this application will calibrate the camera and radar according to the initial external parameters, and then rotate the camera and radar to the same viewing angle.
[0057] According to the technical solution provided in the embodiment of the present application, a first calibration point is determined from the captured image, and a second calibration point is determined from the point cloud data, and the interval distance between the first calibration point and the second calibration point is lower than a preset distance threshold; the coordinates of the first calibration point in the captured image are determined, and the coordinates of the second calibration point in the point cloud data are determined; the camera intrinsic parameters corresponding to the camera are obtained, and the initial extrinsic parameters are determined according to the coordinates corresponding to the first calibration point, the coordinates corresponding to the second calibration point, and the camera intrinsic parameters. The present application determines the position and posture of the camera through the first calibration point and the second calibration point, determines the initial extrinsic parameters, and then realizes rotating the camera and the radar to the same viewing angle, providing a basis for subsequent precise calibration.
[0058] In some examples, such as Figure 3 As shown, determining the first target corner point from the captured image corresponding to the camera includes:
[0059] S301, determining a target object from a captured image, and extracting a boundary of the target object to obtain a boundary line corresponding to the target object;
[0060] S302: Obtain intersection points of different boundary lines, and use the intersection points of different boundary lines as first target corner points.
[0061] After the initial calibration is completed, the application will enter the process of precise calibration of the camera and radar; it can be understood that in the precise calibration process, the captured images and point cloud data obtained during the initial calibration process can be used, or the same target acquisition area can be re-shot to obtain the captured images and point cloud data again, and this example is not limited to this.
[0062] In order to accurately obtain the corner points of the target object, this example will determine the target object from the captured image, and extract the boundary of the target object to obtain the boundary line corresponding to the target object. After obtaining the boundary line, this application obtains the first target corner point by the method of finding the intersection point of a straight line.
[0063] It can be understood that the present application adopts a boundary line determination model to determine the boundary pixel points of the target object in the captured image, and then fits the boundary pixel points into a straight line, and finally obtains the first target corner point by the method of finding the intersection of the straight lines. Among them, the training process of the boundary line determination model can be determined by relevant personnel according to actual needs, and this example will not be repeated here.
[0064] In some examples, due to the influence of lighting, there will be a large error in directly extracting points or lines from the image. In some cases, the extracted lines are not the boundaries of the objects themselves, but the shadows of the light. Figure 4 As shown, at this time, the target object and the shadow are merged. If the boundary of the target object is directly extracted, the boundary line extraction will be inaccurate, resulting in inaccurate determination of the first target corner point.
[0065] In order to avoid the above situation, determining the target object from the captured image includes: determining a shadow image containing the target object from the captured image, and segmenting the shadow image to obtain the target object. Specifically, Figure 5 As shown, Figure 5 As shown in the figure, a shadow image containing a target object is determined. The present application first segments the shadow image based on a segmentation network to obtain the target object, and then uses a boundary line determination model to determine the boundary pixel points of the target object in the captured image, and then fits the boundary pixel points into a straight line, and finally obtains the first target corner point by the method of finding the intersection point of the straight lines.
[0066] Among them, the present application can accurately extract the target object in the captured image through the segmentation method, thereby accurately obtaining the boundary line of the target object, effectively reducing interference factors, and achieving a high-precision first target corner point.
[0067] It can be understood that the above steps of determining the shadow and segmenting the target object and the shadow can also be performed using a model. The specific model training process can be determined by relevant personnel according to actual needs, and this example will not be repeated here.
[0068] According to the technical solution provided in the embodiment of the present application, the target object is determined from the captured image, and the boundary of the target object is extracted to obtain the boundary line corresponding to the target object; the intersection points of different boundary lines are obtained, and the intersection points of different boundary lines are used as the first target corner points. The present application determines the intersection points by using multiple boundary lines to achieve a high-precision first target corner point.
[0069] In some embodiments, Figure 6 As shown, the intersection points of different boundary lines are obtained, including:
[0070] S601, determining whether the target object is blocked by an obstruction;
[0071] S602: If the target object is blocked by an obstruction, determine a boundary line blocked by the obstruction, and extend the boundary line blocked by the obstruction to obtain intersection points of different boundary lines.
[0072] In some examples, when the camera shoots the target acquisition area, it is very likely that the target object will be blocked. When the target object is blocked, the number of first target corner points extracted will be reduced, affecting the accuracy of subsequent determination of target external parameters. To compensate for this defect, when the target object is blocked, the virtual intersection is solved by extending the straight line to fill in the missing corner points.
[0073] Specifically, the present application determines whether the target object is blocked by an obstruction, such as Figure 7As shown, if the target object is blocked, the number of corner points will be reduced. To avoid the above problem, when the target object is blocked by the blocker, the boundary line blocked by the blocker is determined, and the boundary line blocked by the blocker is extended to obtain the intersection points of different boundary lines. If the target object is not blocked, the intersection point of the boundary line can be directly obtained as the first target corner point.
[0074] According to the technical solution provided in the embodiment of the present application, it is determined whether the target object is obscured by the obstruction; if the target object is obscured by the obstruction, the boundary line obscured by the obstruction is determined, and the boundary line obscured by the obstruction is extended to obtain the intersection of different boundary lines, thereby avoiding the problem of too few first target corner points caused by the target object being obscured.
[0075] In some embodiments, Figure 8 As shown, after obtaining the intersection points of different boundary lines and taking the intersection points of different boundary lines as the first target corner points, the method further includes:
[0076] S801, comparing the determined number of first target corner points with a preset corner point threshold;
[0077] S802. When the number of the first target corner points is lower than the corner point threshold, rotate at least one boundary line around the intersection point corresponding to the boundary line to obtain a new boundary line, determine the intersection point between the new boundary line and a different boundary line, and use the determined intersection point as the first target corner point.
[0078] In some examples, due to the simple structure of the target object and the small number of corner points, it is impossible to obtain enough first target corner points; for example, when the target object is a triangular object, at most 3 corner points can be extracted from the captured image or point cloud data. If the number of corner points is too small, it will lead to a large error in the target external parameters.
[0079] In order to avoid the problem that the number of first target corner points is too small, resulting in a large error in the target extrinsic parameters, this example compares the determined number of first target corner points with the pre-set corner point threshold. If the number of first target corner points is lower than the corner point threshold, it indicates that there are too few first target corner points, which may easily lead to a large error in the target extrinsic parameters; if the number of first target corner points is not lower than the corner point threshold, it indicates that there are sufficient first target corner points.
[0080] In order to solve the problem of too few first target corner points, the present application rotates any boundary line around a corner point to generate a virtual boundary line, uses the virtual boundary line intersection to obtain virtual first target corner points, and uses virtual first target corner points to supplement the number; specifically, when the number of first target corner points is lower than the corner point threshold, rotate at least one boundary line around the intersection point corresponding to the boundary line to obtain a new boundary line, determine the intersection point between the new boundary line and different boundary lines, and use the determined intersection point as the first target corner point. Fig. 9 As shown, the present application rotates the boundary line around the intersection point corresponding to the boundary line, determines the intersection point between the new boundary line and the different boundary lines, and uses the determined intersection point as the first target corner point until the number of the first target corner points is greater than or equal to the corner point threshold.
[0081] According to the technical solution provided in the embodiment of the present application, the number of determined first target corner points is compared with a pre-set corner point threshold; when the number of first target corner points is lower than the corner point threshold, at least one boundary line is rotated around the intersection point corresponding to the boundary line to obtain a new boundary line, and the intersection point between the new boundary line and the different boundary lines is determined, and the determined intersection point is used as the first target corner point. The present application realizes the determination of new first target corner points by rotating the boundary line, thereby increasing the number of first target corner points, avoiding the problem that the target object has a simple structure and fewer corner points, resulting in the inability to obtain enough first target corner points, and the subsequent inability to accurately determine the target external parameters based on the first target corner points.
[0082] In some embodiments, Fig.10 As shown, determining the second target corner point from the point cloud data corresponding to the radar includes:
[0083] S1001, performing plane fitting on the point cloud data corresponding to the radar to obtain at least two fitting planes;
[0084] S1002, determining a boundary line corresponding to each fitting plane according to the intersection position of at least two fitting planes;
[0085] S1003: Obtain intersection points of different boundary lines, and use the intersection points of different boundary lines as second target corner points.
[0086] It is understandable that the point cloud generated by the radar contains noise, which will cause the boundary line of the target generated by the radar to be unclear, and thus the corner points of the point cloud are unclear, such as Fig.11 As shown; since the accurate second target corner point of the target object cannot be obtained in the point cloud data, in order to obtain the accurate second target corner point, the present application first fits a plane in the point cloud data and obtains the accurate boundary line by plane intersection.
[0087] Specifically, plane fitting is performed according to the point cloud data corresponding to the radar to obtain at least two fitting planes, such as Fig.12 As shown, the boundary line corresponding to each fitting plane is then determined according to the intersection position of at least two fitting planes, and finally the intersection points of different boundary lines are obtained and used as the second target corner points.
[0088] It is understandable that the field of view of the radar is larger than that of the camera, and many corner points may be extracted from the point cloud data. In order to accurately match the second target corner point extracted from the point cloud data with the first target corner point extracted from the captured image, the corner points in the point cloud data need to be screened. The clustering method is used here. The target object can be distinguished in the point cloud data through clustering. After the clustering is completed, only the corner points on the target object are retained. Similarly, if the target object is obscured, the missing corner points can be supplemented by extending the boundary line to solve the virtual corner point. If there are relatively few corner points, this application also rotates the straight line extracted from the point cloud data by a certain angle with the corner point as the center to generate a virtual line, and increases the number of matching point pairs by finding the intersection point of the virtual line.
[0089] It can be understood that after obtaining the first target corner point and the second target corner point, the present application will match the first target corner point and the second target corner point; wherein, since the radar and the camera have been initially calibrated, the point cloud data has the same viewing angle as the captured image, and it is only necessary to take points on the target object in a certain order to match the second target corner point in the point cloud data with the first target corner point in the captured image, and subsequently the target extrinsic parameters can be calculated based on the matched first target corner point and the second target corner point.
[0090] According to the technical solution provided in the embodiment of the present application, a plane is fitted according to the point cloud data corresponding to the radar to obtain at least two fitting planes; the boundary line corresponding to each fitting plane is determined according to the intersection position of the at least two fitting planes; the intersection points of different boundary lines are obtained, and the intersection points of different boundary lines are used as the second target corner points. The position of the target can be determined by point cloud clustering, and then the plane is extracted. The straight line is obtained by intersecting the planes, and the point point is obtained by intersecting the straight lines. The corner point of the point cloud target can be obtained; the present application can determine the position of the target object by point cloud clustering, and then the plane of the target object is extracted. The straight line is obtained by intersecting the planes, and the intersection point is obtained by intersecting the straight lines. The second target corner point in the point cloud data can be obtained by intersecting the straight lines.
[0091] In some embodiments, Fig.13 As shown, after determining the target extrinsic parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point, the method further includes:
[0092] S1301, projecting the point cloud data obtained by radar photography onto the captured image obtained by camera photography according to the target external parameter;
[0093] S1302, determining a projection error after the point cloud data corresponding to the radar is projected onto the captured image corresponding to the camera;
[0094] S1303: Compare the projection error with a preset error threshold. If the projection error is greater than the error threshold, redefine the first target corner point and the second target corner point, and calibrate the camera and the radar based on the redetermined first target corner point and the second target corner point.
[0095] Among them, since the autonomous mobile device may shake during movement, the positions of the camera and radar in the autonomous mobile device are not completely fixed, so this application needs to regularly check the accuracy of the target external parameters.
[0096] This application projects the point cloud data obtained by radar photography onto the captured image obtained by the camera according to the target extrinsic parameters; specifically, by projecting the points in the radar onto the pixel plane of the camera according to the target extrinsic parameters [R|T] and the camera's intrinsic parameters K and distortion coefficient d.
[0097] After the projection is completed, the projection error after the point cloud data corresponding to the radar is projected onto the captured image corresponding to the camera is determined, and then the projection error is compared with the preset error threshold. If the projection error is greater than the error threshold, it proves that the accuracy of the target extrinsic parameters is too low and the target extrinsic parameters need to be re-determined. Therefore, if the projection error is greater than the error threshold, the first target corner point and the second target corner point are re-determined, and the camera and radar are calibrated according to the re-determined first target corner point and second target corner point; if the projection error is greater than the error threshold, it proves that the accuracy of the target extrinsic parameters is too low and the target extrinsic parameters need to be re-determined. Therefore, if the projection error is less than or equal to the error threshold, the subsequent process is stopped until the next target extrinsic parameter is detected.
[0098] It is understandable that if the projection error is greater than the error threshold, the autonomous mobile device can also issue a voice warning to remind the user that the target external parameter is being recalibrated. During the calibration process, the existing target external parameter is used as the initial value and the calibration can be completed by following the precise calibration steps.
[0099] According to the technical solution provided in the embodiment of the present application, the point cloud data obtained by radar is projected onto the captured image obtained by the camera according to the target extrinsic parameters; the projection error after the point cloud data corresponding to the radar is projected onto the captured image corresponding to the camera is determined; the projection error is compared with a preset error threshold, and if the projection error is greater than the error threshold, the first target corner point and the second target corner point are re-determined, and the camera and radar are calibrated according to the re-determined first target corner point and the second target corner point. The present application achieves the accuracy of detecting the target extrinsic parameters through the above steps, and ensures the accuracy of information fusion between the camera and the radar.
[0100] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0101] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0102] This embodiment also provides an external parameter calibration device, such as Fig.14 As shown, the device comprises:
[0103] The acquisition module 1401 is configured to acquire images and point cloud data obtained by a camera and a radar shooting the same target acquisition area, where a target object for extrinsic calibration of the camera and the radar is provided in the target acquisition area;
[0104] The calibration module 1402 is configured to determine initial external parameters corresponding to the camera and the radar according to the captured image corresponding to the camera and the point cloud data corresponding to the radar, where the initial external parameters are used to characterize the initial position and initial direction of the camera in the three-dimensional space where the radar is located;
[0105] The corner point module 1403 is configured to calibrate the camera and the radar according to the initial external parameters, and when the calibration is completed, determine the first target corner point from the captured image corresponding to the camera, and determine the second target corner point from the point cloud data corresponding to the radar;
[0106] The calibration module 1402 is also configured to determine target extrinsic parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point, where the target extrinsic parameters are used to characterize the target position and target direction of the camera in the three-dimensional space where the radar is located.
[0107] In some examples, the calibration module 1402 is also configured to determine a first calibration point from the captured image, determine a second calibration point from the point cloud data, and the interval distance between the first calibration point and the second calibration point is lower than a preset distance threshold; determine the coordinates of the first calibration point in the captured image, and determine the coordinates of the second calibration point in the point cloud data; obtain the camera intrinsic parameters corresponding to the camera, and determine the initial external parameters based on the coordinates corresponding to the first calibration point, the coordinates corresponding to the second calibration point, and the camera intrinsic parameters.
[0108] In some examples, the corner point module 1403 is further configured to determine a target object from the captured image, and extract the boundary of the target object to obtain a boundary line corresponding to the target object; obtain the intersection of different boundary lines, and use the intersection of different boundary lines as the first target corner point.
[0109] In some examples, the corner point module 1403 is further configured to determine whether the target object is obstructed by an obstruction; if the target object is obstructed by the obstruction, the boundary line obstructed by the obstruction is determined, and the boundary line obstructed by the obstruction is extended to obtain the intersection of different boundary lines.
[0110] In some examples, the corner point module 1403 is also configured to compare the number of determined first target corner points with a pre-set corner point threshold; when the number of first target corner points is lower than the corner point threshold, at least one boundary line is rotated around the intersection point corresponding to the boundary line to obtain a new boundary line, and the intersection point between the new boundary line and the different boundary line is determined, and the determined intersection point is used as the first target corner point.
[0111] In some examples, the corner point module 1403 is also configured to perform plane fitting based on the point cloud data corresponding to the radar to obtain at least two fitting planes; determine the boundary line corresponding to each fitting plane based on the intersection position of the at least two fitting planes; obtain the intersection points of different boundary lines, and use the intersection points of different boundary lines as the second target corner points.
[0112] In some examples, the calibration module 1402 is also configured to project the point cloud data obtained by the radar into the captured image obtained by the camera according to the target external parameters; determine the projection error after the point cloud data corresponding to the radar is projected onto the captured image corresponding to the camera; compare the projection error with a preset error threshold, and if the projection error is greater than the error threshold, re-determine the first target corner point and the second target corner point, and calibrate the camera and radar based on the re-determined first target corner point and the second target corner point.
[0113] According to the technical solution provided in the embodiment of the present application, the external parameter calibration device provided in the present embodiment obtains the captured images and point cloud data obtained by the camera and the radar when shooting the same target acquisition area; determines the initial external parameters corresponding to the camera and the radar according to the captured images corresponding to the camera and the point cloud data corresponding to the radar; calibrates the camera and the radar according to the initial external parameters, and when the calibration is completed, determines the first target corner point from the captured image corresponding to the camera, and determines the second target corner point from the point cloud data corresponding to the radar; determines the target external parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point; the present application first calibrates the camera and the radar according to the initial external parameters, so as to set the camera and the radar to the same viewing angle, and then determines the target external parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point, thereby realizing the accurate determination of the target external parameters of the camera and the radar through the corner points, so that the camera and the radar can accurately perform information fusion later, thereby avoiding the problem in the related art that the external parameter calibration cannot be accurately performed, resulting in the inability of the radar and the camera to perform information fusion.
[0114] Fig.15Schematic diagram of an electronic device 15 provided in an embodiment of the present application. Fig.15 As shown, the electronic device 15 of this embodiment includes: a processor 1501, a memory 1502, and a computer program 1503 stored in the memory 1502 and executable on the processor 1501. When the processor 1501 executes the computer program 1503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1501 executes the computer program 1503, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0115] The electronic device 15 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 15 may include, but is not limited to, a processor 1501 and a memory 1502. Those skilled in the art will appreciate that Fig.15 The electronic device 15 is merely an example and does not limit the electronic device 15 , and may include more or less components than those shown in the figure, or different components.
[0116] Processor 1501 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0117] The memory 1502 may be an internal storage unit of the electronic device 15, for example, a hard disk or memory of the electronic device 15. The memory 1502 may also be an external storage device of the electronic device 15, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 15. The memory 1502 may also include both an internal storage unit of the electronic device 15 and an external storage device. The memory 1502 is used to store computer programs and other programs and data required by the electronic device.
[0118] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0119] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of regional requirements and patent practice. For example, in some areas, according to regional requirements and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.
[0120] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for calibrating an external parameter, characterized in that: The method comprises: Acquire captured images and point cloud data obtained by capturing the same target acquisition area with a camera and a radar, wherein a target object for extrinsic calibration of the camera and the radar is provided in the target acquisition area; Determine initial extrinsic parameters corresponding to the camera and the radar according to the captured image corresponding to the camera and the point cloud data corresponding to the radar, wherein the initial extrinsic parameters are used to characterize an initial position and an initial direction of the camera in the three-dimensional space where the radar is located; Calibrate the camera and the radar according to the initial external parameters, and when the calibration is completed, determine a first target corner point from the captured image corresponding to the camera, and determine a second target corner point from the point cloud data corresponding to the radar; According to the first target corner point and the second target corner point, target extrinsic parameters corresponding to the camera and the radar are determined, and the target extrinsic parameters are used to characterize a target position and a target direction of the camera in the three-dimensional space where the radar is located.
2. The method according to claim 1, characterized in that: Determining initial external parameters corresponding to the camera and the radar according to the captured image corresponding to the camera and the point cloud data corresponding to the radar, including: Determine a first calibration point from the captured image, and determine a second calibration point from the point cloud data, wherein a distance between the first calibration point and the second calibration point is less than a preset distance threshold; Determine the coordinates of the first calibration point in the captured image, and determine the coordinates of the second calibration point in the point cloud data; Obtain camera intrinsic parameters corresponding to the camera, and determine the initial extrinsic parameters according to coordinates corresponding to the first calibration point, coordinates corresponding to the second calibration point, and the camera intrinsic parameters.
3. The method according to claim 1, characterized in that Determining a first target corner point from the captured image corresponding to the camera includes: Determine the target object from the captured image, and extract the boundary of the target object to obtain a boundary line corresponding to the target object; The intersection points of different boundary lines are obtained, and the intersection points of different boundary lines are used as the first target corner points.
4. The method according to claim 3, characterized in that Obtaining the intersection points of different boundary lines, including: Determining whether the target object is blocked by an obstruction; If the target object is blocked by the blocking object, a boundary line blocked by the blocking object is determined, and the boundary line blocked by the blocking object is extended to obtain intersection points of different boundary lines.
5. The method according to claim 3, characterized in that: After obtaining the intersection points of different boundary lines and taking the intersection points of different boundary lines as the first target corner points, the method further includes: Comparing the determined number of the first target corner points with a preset corner point threshold; When the number of the first target corner points is lower than the corner point threshold, at least one of the boundary lines is rotated around the intersection point corresponding to the boundary line to obtain a new boundary line, and the intersection point between the new boundary line and the different boundary lines is determined, and the determined intersection point is used as the first target corner point.
6. The method according to claim 1, characterized in that Determining a second target corner point from point cloud data corresponding to the radar includes: Performing plane fitting on the point cloud data corresponding to the radar to obtain at least two fitting planes; Determining a boundary line corresponding to each of the fitting planes according to an intersection position of at least two of the fitting planes; The intersection points of different boundary lines are obtained, and the intersection points of different boundary lines are used as the second target corner points.
7. The method according to claim 1, characterized in that After determining target extrinsic parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point, the method further includes: Projecting the point cloud data obtained by the radar onto the captured image obtained by the camera according to the target external parameter; Determine a projection error after projecting the point cloud data corresponding to the radar onto the captured image corresponding to the camera; The projection error is compared with a preset error threshold. If the projection error is greater than the error threshold, the first target corner point and the second target corner point are re-determined, and the camera and the radar are calibrated according to the re-determined first target corner point and the second target corner point.
8. An external parameter calibration device, characterized in that: The device comprises: An acquisition module is configured to acquire captured images and point cloud data obtained by a camera and a radar shooting the same target acquisition area, wherein a target object for extrinsic calibration of the camera and the radar is provided in the target acquisition area; A calibration module is configured to determine initial external parameters corresponding to the camera and the radar according to the captured image corresponding to the camera and the point cloud data corresponding to the radar, wherein the initial external parameters are used to characterize the initial position and initial direction of the camera in the three-dimensional space where the radar is located; A corner point module is configured to calibrate the camera and the radar according to the initial external parameters, and when the calibration is completed, determine a first target corner point from the captured image corresponding to the camera, and determine a second target corner point from the point cloud data corresponding to the radar; The calibration module is also configured to determine target extrinsic parameters corresponding to the camera and the radar according to the first target corner point and the second target corner point, wherein the target extrinsic parameters are used to characterize a target position and a target direction of the camera in the three-dimensional space where the radar is located.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.