An external parameter calibration method and device, and an intelligent vehicle

Abstract

The present application relates to the technical field of external parameter calibration, and particularly relates to an external parameter calibration method and device, and an intelligent vehicle, the external parameter calibration method is applied to the intelligent vehicle, the intelligent vehicle is fixedly provided with a camera and a radar, and the method comprises the following steps: obtaining initial external parameters according to the pose of the radar and the pose of the camera; extracting a boundary point set in a point cloud diagram obtained by the radar and a line segment set in an image obtained by the camera respectively, and obtaining a corresponding relationship between the boundary point set and the line segment set; projecting the boundary point set in the point cloud diagram to the image according to the initial external parameters; and optimizing the initial external parameters according to the distance between the boundary point set and the line segment set, so as to obtain the external parameters of the radar and the camera. The method and device and the intelligent vehicle can be applied to a low-line-number radar, reduce the use cost, expand the application range, have low requirements on the motion track of the sensor, can be applied to the case that the sensor moves in a 2D plane in a small scene, improve the case that the external parameter result cannot be calculated in this case, and enable the calibration process to proceed normally.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of external parameter calibration, and in particular to an external parameter calibration method and device and an intelligent vehicle. BACKGROUND

[0002] In an unmanned driving system, cameras and lidars are the most commonly used sensors, which are usually used for 2D / 3D object detection, depth map completion, real-time positioning and other perception detection tasks. However, the above tasks can be completed on the premise that accurate external parameters can be obtained between the camera and the lidar. In the camera-lidar external parameter calibration method, it is mainly divided into a calibration object calibration method and a calibration object-free calibration method. The calibration object calibration method uses a calibration board or other calibration objects prepared in advance, calculates the attitude of the radar and the camera relative to the calibration object according to the prior information of the calibration object, and thus obtains the external parameters of the camera and the radar. The calibration object-free calibration method does not rely on a calibration board prepared in advance, but finds common scene features in the field of view of the radar and the camera to calibrate the external parameters.

[0003] However, the above calibration object-free calibration method has certain requirements for the motion trajectory of the scene and the sensor, and is not suitable for the case that the existing unmanned vehicle can only move in a 2D plane in a small scene. At the same time, in a low-cost unmanned driving system, the commonly used sensor is generally a low-line laser radar, and the existing calibration object-free calibration method is based on a high-line radar to calibrate the external parameters, so that the precision and accuracy of the low-line radar will be greatly reduced. SUMMARY

[0004] The present application provides an external parameter calibration method and device and an intelligent vehicle, which aims to solve the problem that the existing calibration object-free external parameter calibration method is not suitable for the case that the existing unmanned vehicle can only move in a 2D plane in a small scene, and is not suitable for a low-line laser radar.

[0005] To solve the above technical problems, one technical scheme adopted by the embodiment of the present application is to provide an external parameter calibration method, which is applied to an intelligent vehicle, the intelligent vehicle is fixedly provided with a camera and a radar, and the method comprises the following steps: obtaining an initial external parameter according to a pose of the radar and a pose of the camera; extracting a boundary point set in a point cloud image obtained by the radar and a line segment set in an image obtained by the camera, respectively, and obtaining a corresponding relationship between the boundary point set and the line segment set; projecting the boundary point set in the point cloud image into the image according to the initial external parameter; and optimizing the initial external parameter according to a distance between the boundary point set and the line segment set, to obtain the external parameters of the radar and the camera.

[0006] In some embodiments, the step of extracting the boundary point set from the point cloud map obtained by the radar and the line segment set from the image obtained by the camera, and obtaining the correspondence between the boundary point set and the line segment set, comprises: separating continuous scanning point clouds from the point cloud in the point cloud map; taking the first and last two of the continuous scanning point clouds as candidate boundary points; projecting the candidate boundary points to the image, extracting the line segment set in the image, and finding the candidate boundary points corresponding to the line segment set; filtering the candidate boundary points to obtain a boundary point set; comparing the line segment set with the boundary point set to obtain the correspondence between the boundary point set and the line segment set.

[0007] In some embodiments, the step of comparing the line segment set with the boundary point set to obtain the correspondence between the boundary point set and the line segment set comprises: calculating the included angle between the straight line where the boundary point set is located and the straight line where the line segment set is located; calculating the Euler distance of the first and last two boundary points of the boundary point set and the Euler distance of the two end points of the line segment set, respectively; calculating the overlap rate of the boundary point set and the line segment set; and obtaining the correspondence between the boundary point set and the line segment set according to the included angle, the Euler distance of the first and last two boundary points of the boundary point set, the Euler distance of the two end points of the line segment set, and the overlap rate.

[0008] In some embodiments, the step of filtering the candidate boundary points to obtain a boundary point set specifically comprises: obtaining the radar emission angle of any two candidate boundary points; obtaining the candidate boundary points with a radar emission angle less than a first threshold; and based on the obtained candidate boundary points, filtering out the candidate boundary points far away from the radar to obtain a boundary point set.

[0009] In some embodiments, the step of optimizing the initial extrinsic parameter according to the distance between the boundary point set and the line segment set to obtain the extrinsic parameter of the radar and the camera comprises: calculating the distance from each boundary point in the boundary point set to the line segment corresponding to the line segment set, respectively; obtaining a loss function according to the distance; and optimizing the initial extrinsic parameter according to the loss function to obtain the extrinsic parameter of the radar and the camera.

[0010] In some embodiments, after the step of obtaining an initial extrinsic parameter according to the pose of the radar and the pose of the camera, the method further comprises: extracting a ground point cloud according to a ground normal vector; obtaining an offset vector according to the ground point cloud; and modifying the initial extrinsic parameter based on the offset vector.

[0011] To solve the above technical problems, another technical scheme adopted by the embodiment of the present application is to provide an external parameter calibration device, which is applied to an intelligent vehicle, the intelligent vehicle is fixedly provided with a camera and a radar, and the device comprises an initialization module, a corresponding relationship determination module, a projection module and an external parameter determination module, the initialization module is used to obtain an initial external parameter according to a pose of the radar and a pose of the camera; the corresponding relationship determination module is used to extract a boundary point set in a point cloud diagram obtained by the radar and a line segment set in an image obtained by the camera respectively, and obtain a corresponding relationship between the boundary point set and the line segment set; the projection module is used to project the boundary point set in the point cloud diagram into the image according to the initial external parameter; and the external parameter determination module is used to optimize the initial external parameter according to distances between the boundary point set and the line segment set, and obtain an external parameter of the radar and the camera.

[0012] In some embodiments, the corresponding relationship determination module comprises a separation module, a marking module, a finding module, a filtering module and a comparison module, the separation module is used to separate continuous scanning point clouds from point clouds in the point cloud diagram; the marking module is used to mark the first and last two point clouds in the continuous scanning point clouds as candidate boundary points; the finding module is used to project the candidate boundary points into the image, extract a line segment set in the image, and find candidate boundary points corresponding to the line segment set; the filtering module is used to filter the candidate boundary points to obtain a boundary point set; and the comparison module is used to compare the line segment set with the boundary point set to obtain the corresponding relationship between the boundary point set and the line segment set.

[0013] In some embodiments, the external parameter determination module comprises a distance calculation module, a loss function module and an external parameter optimization module, the distance calculation module is used to calculate distances from each boundary point in the boundary point set to a line segment corresponding to the line segment set respectively; the loss function module is used to obtain a loss function according to the distances; and the external parameter optimization module is used to optimize the initial external parameter according to the loss function to obtain an external parameter of the radar and the camera.

[0014] To solve the above technical problems, still another technical scheme adopted by the embodiment of the present application is to provide an intelligent vehicle, which comprises at least one processor and a memory, the memory is in communication connection with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0015] Different from the related art, the external parameter calibration method and device, and the intelligent vehicle, by matching and aligning the line segments in the image collected by the camera and the point cloud collected by the radar, the initial external parameter is optimized to obtain the final external parameter, the method and device, and the intelligent vehicle can be applied to a low-line-number radar, reduce the use cost, expand the application range, and have low requirements on the motion track of the sensor, can be applied to the case of 2D plane motion in a small scene, improve the case that the external parameter result cannot be calculated in this case, make the normal calibration process. BRIEF DESCRIPTION OF DRAWINGS

[0016] One or more embodiments are illustrated by way of example in the figures that are not intended to be limiting of the embodiments. Like numbers refer to like elements throughout the drawings, and any reference to first and second, top and bottom, and the like, indicate desired orientations during use. Elements having the same numbering can have the same structure and / or function, or similar structure and / or function, unless otherwise specified.

[0017] Figure 1 is a hardware structure schematic diagram of the intelligent vehicle provided by the embodiment of the present application;

[0018] Figure 2 is a flowchart of the external parameter calibration method provided by the embodiment 1 of the present application;

[0019] Figure 3 is Figure 2 is a specific flowchart of step S22 in the embodiment 2;

[0020] Figure 4 is an algorithm provided by the embodiment 2 of the present application;

[0021] Figure 5 is Figure 3 is a specific flowchart of step S224 in the embodiment 2;

[0022] Figure 6 is Figure 3 is a specific flowchart of step S225 in the embodiment 2;

[0023] Figure 7 is Figure 2 is a specific flowchart of step S24 in the embodiment 2;

[0024] Figure 8 is a flowchart of the external parameter calibration method provided by the embodiment 3 of the present application;

[0025] Figure 9 is Figure 8 is a specific flowchart of step S35 in the embodiment 3;

[0026] Figure 10 is a structure schematic diagram of the external parameter calibration device provided by the embodiment 4 of the present application;

[0027] Figure 11 isFigure 10 Structure diagram of the correspondence determination module;

[0028] Figure 12 is Figure 10 Structure diagram of the extrinsic parameter determination module. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0030] It should be noted that the various features of the embodiments of the present application can be combined with each other without conflict, and are within the protection scope of the present application. In addition, although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device schematic diagram or the order in the flowchart.

[0031] Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used in the present application includes any and all combinations of one or more related listed items.

[0032] In the camera-radar extrinsic parameter calibration method, it is mainly divided into calibration methods with calibration objects and calibration methods without calibration objects. The extrinsic parameter calibration method without calibration objects does not depend on the calibration board made in advance, but finds the common scene features in the radar and camera field of view for extrinsic parameter calibration. For example, mutual information, Renyi entropy are used for calibration, or SFM-based method, motion-based method, etc.

[0033] The above methods have certain requirements for the scene. For example, when mutual information is used for calibration, the data of the laser radar intensity channel is required, and the measurement accuracy is required to be high, and the intensity information of the radar and the camera in the scene needs to be rich. When the SFM-based method is used for calibration, dense mapping needs to be performed, and the texture information of the objects in the scene also needs to be richer.

[0034] The methods mentioned above also have requirements for the motion trajectory of the sensors. For example, when using motion-based and SFM-based methods for calibration, the camera and radar need to rotate and translate sufficiently in 3D space. However, in most cases, autonomous vehicles can only move in a 2D plane in small scenes. In this case, the rotation in the roll and pitch angles is insufficient, and the translation in the z-axis direction is also insufficient, which may lead to the inability to calculate the extrinsic parameters.

[0035] Furthermore, low-line-count lidar is a commonly used sensor in low-cost autonomous driving, while the methods mentioned above are based on high-line-count lidar for extrinsic parameter calibration. Applying them to low-line-count lidar will severely reduce accuracy.

[0036] Example 1

[0037] Figure 1 This is a schematic diagram of the hardware structure of the intelligent vehicle provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the intelligent vehicle 100 includes:

[0038] At least one processor 110 and memory 120, Figure 1 Take a processor 110 as an example.

[0039] The memory 120 is communicatively connected to the at least one processor 110. The processor 110 and the memory 120 can be connected via a bus or other means. Figure 1 Taking the example of a connection between China and Israel via a bus.

[0040] The memory 120 stores instructions executable by the at least one processor 110, which, when executed by the at least one processor 110, enable the at least one processor 110 to perform the external parameter calibration method of the method embodiment.

[0041] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the external parameter calibration device embodiment of this invention (e.g., Figure 10 (The functions of modules 410-440 in the memory). The processor 110 executes various functional applications and data processing of the intelligent vehicle 100 by running non-volatile software programs, instructions, and modules stored in the memory 120, that is, implementing the external parameter calibration method of the method embodiment.

[0042] The memory 120 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function, and the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 120 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 120 can optionally include a memory 120 that is remotely arranged with respect to the processor 110, and these remote memories can be connected to the terminal device through a network. The network includes but is not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The one or more modules are stored in the memory 120, and when executed by the one or more processors 110, perform the modeling information recommendation method in any of the method embodiments described above, for example, perform the method steps S21 to S24 in Figure 2 , to realize the functions of the 410-440 modules in Figure 10 .

[0044] The intelligent vehicle 100 can perform the external parameter calibration method provided by the method embodiments, and has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the intelligent vehicle 100 embodiment can be referred to the external parameter calibration method provided by the method embodiments.

[0045] The intelligent vehicle 100 can be in communication connection with a cloud server, the cloud server receives the instructions and information sent by the intelligent vehicle 100, and analyzes and processes the instructions and the information, and then sends feedback instructions and feedback information to the intelligent vehicle 100. The cloud server can be a single server or a server cluster.

[0046] The intelligent vehicle 100 of the embodiment of the application obtains initial external parameters according to the poses of the camera and the radar by setting the camera and the radar, and then optimizes the initial external parameters according to the data collected by the camera and the radar to obtain final external parameters. According to the external parameter calibration method of the embodiment of the application, the intelligent vehicle 100 of the embodiment of the application can be applied to a low-line radar, reduces the use cost, expands the application range, has low requirements for the motion trajectory of the sensor, can be applied to the case of 2D plane motion in a small scene, improves the case that the external parameter result cannot be calculated under this condition, and enables the normal progress of the calibration process.

[0047] Embodiment 2

[0048] As shown in Figure 2 , Figure 2is a flowchart of an external parameter calibration method provided by Embodiment 1 of the present application, and the external parameter calibration method provided by the present application is applied to an intelligent vehicle 100, the intelligent vehicle 100 is fixedly provided with a camera and a radar, and the method comprises the following steps:

[0049] S21: obtaining an initial external parameter according to the pose of the radar and the pose of the camera.

[0050] The external parameter calibration of the mechanical arm and the camera is a very classic hand-eye calibration problem, and the hand-eye calibration can also be applied to other sensors that can be positioned, for example, the external parameter calibration between the camera and the radar.

[0051] The poses of the camera and the radar can be obtained through a SLAM algorithm, specifically, the camera can be positioned by using ORB-SLAM2, and the radar can be positioned by using LOAM.

[0052] First, the orientation and displacement of the camera and the radar are given:

[0053] The orientation of the camera:

[0054]

[0055] wherein,

[0056] The displacement of the camera:

[0057]

[0058] The orientation of the radar:

[0059]

[0060] The displacement of the radar:

[0061]

[0062] Then, according to the hand-eye calibration formula: AX=XB, the external parameter X can be calculated, wherein A and B are the relative poses of the camera and the radar respectively.

[0063] Since the rotation matrix is not differentiable and is expressed redundantly, a quaternion can be used for calculation; since the pose of the camera has a scale transformation, a scale factor s can be introduced to obtain the following equation:

[0064]

[0065]

[0066] Because the rotation matrix is calculated first and then the translation vector is calculated based on the rotation matrix, the solution of the translation vector is greatly affected in the process, and the error of the rotation matrix in the rotation axis direction also greatly affects the solution of the translation vector, so separate solutions are not applicable to the motion hand-eye calibration on a 2D plane, and simultaneous solutions of the rotation matrix and the translation matrix can reduce errors and obtain the following equations:

[0067]

[0068]

[0069] The above two equations are converted into an unconstrained minimization formula as follows:

[0070]

[0071] where λ3(q T q-1) 2 The optimal solution of the quaternion can guarantee that the quaternion is a unit quaternion, and the Gauss-Newton method is used for iteration to solve the above minimization problem, that is, to obtain the initial external parameter.

[0072] S22: Extract the boundary point set in the point cloud obtained by the radar and the line segment set in the image obtained by the camera respectively, and obtain the correspondence between the boundary point set and the line segment set.

[0073] By matching and aligning the first contour line of the object in the image with the second contour line extracted from the point cloud in the point cloud, the initial external parameter can be optimized. For example, in a city scene, there are a large number of buildings, public places and other structured scenes. According to the characteristics of these structured scenes, the first contour line corresponding to the object in the structured scene in the image can be extracted, and the second contour line corresponding to the object in the structured scene in the point cloud can be extracted. Aligning the first contour line and the second contour line corresponding to the same object can optimize the initial external parameter.

[0074] In actual scenes, the extraction and matching of straight line features are easier than other shapes, so in this step, the extraction and matching of straight line features are taken as an example.

[0075] As shown in Figure 3 , step S22 specifically includes:

[0076] S221: Separate the continuously scanned point cloud from the point cloud in the point cloud.

[0077] According to the characteristics of depth discontinuity in the radar scanning process, continuous scanning point clouds can be extracted. A threshold of the number of continuous scanning point clouds can be given to filter out a series of continuous points whose number is less than the threshold, thereby reducing the interference caused by textures on the object.

[0078] For example, there is a straight line texture on the target object, which is represented as two continuous scanning point clouds in the point cloud map. At this time, the threshold is set to 4, and the interference caused by the straight line texture can be filtered out.

[0079] S222: The first and last two of the continuous scanning point clouds are recorded as candidate boundary points.

[0080] The continuous scanning point clouds are connected as a polyline, and the first and last two endpoints of the polyline are recorded as candidate boundary points.

[0081] S223: Project the candidate boundary points onto the image, extract the line segment set in the image, and find the candidate boundary points corresponding to the line segment set.

[0082] Using a deep hough network, the object straight line contour in the image is extracted to obtain a line segment set. Compared with traditional canny operator and hough transform methods, unnecessary straight lines can be reduced, such as straight line textures in the image.

[0083] Based on the coordinate system where the candidate boundary points are located, the candidate boundary points in the xy plane are divided into multiple sectors, such as 6 equal angle sectors. Then, the point clouds in the sectors are projected onto the image. Next, ransac can be used to find the point clouds corresponding to the line segments in the image. Finally, the candidate boundary points for which no corresponding line segment can be found are filtered out.

[0084] Further, when the same point cloud in the sector corresponds to multiple line segments, the point cloud is filtered out.

[0085] Further, when multiple first line segments are connected end to end to form a closed figure, and the closed figure contains a second line segment, the second line segment is filtered out.

[0086] The algorithm of this step is shown in Figure 4 .

[0087] S224: Filter the candidate boundary points to obtain a boundary point set.

[0088] In the candidate boundary points, there may be boundary occlusion, so pseudo-boundaries caused by occlusion need to be filtered out.

[0089] As shown in Figure 5 , step S224 specifically includes:

[0090] S2241: Obtain the radar emission angle of any two of the candidate boundary points.

[0091] The radar emission angle of any two points can be calculated by the position of the point cloud in the radar coordinate system, that is, the radar emission angle.

[0092] S2242: Obtain the candidate boundary points with the radar emission angle less than the first threshold value.

[0093] By setting the first threshold value, the candidate boundary points with the radar emission angle less than the first threshold value are obtained and recorded as the first candidate boundary points, which are used for preparing the filtering process, and the remaining candidate boundary points are recorded as the second candidate boundary points.

[0094] S2243: Based on the obtained candidate boundary points, the candidate boundary points far away from the radar are filtered out to obtain a boundary point set.

[0095] Based on the points in the obtained candidate boundary points, that is, the points in the first candidate boundary points, according to the distance between the radar, compare two by two, discard the farther points, and retain the closer points to obtain the third candidate boundary points. Then, the second candidate boundary points and the third candidate boundary points are combined to obtain the boundary point set.

[0096] S225: Compare the line segment set with the boundary point set to obtain the correspondence between the boundary point set and the line segment set.

[0097] Project the boundary point set onto the image, calculate the deviation of the points in the boundary point set and the line segments in the line segment set, accumulate the deviation between each boundary point and each line segment, and perform minimum deviation processing to obtain the correspondence between the boundary point set and the line segment set.

[0098] As shown in Figure 6 , step S225 includes:

[0099] S2251: Calculate the angle between the straight line where the boundary point set is located and the straight line where the line segment set is located.

[0100] Calculate the angle between the straight line formed by the connection of the boundary points and the line segment, accumulate the angle to obtain the total angle deviation.

[0101] S2252: Calculate the Euler distance of the first and last boundary points of the boundary point set and the Euler distance of the two end points of the line segment set, respectively.

[0102] Calculate the Euler distance of the first and last end points of the boundary points and the two end points of the line segment, accumulate the Euler distance to obtain the total Euler distance deviation.

[0103] S2253: Calculate the overlap rate of the boundary point set and the line segment set.

[0104] Calculate the overlap rate of the boundary point and the line segment, accumulate the overlap rate, and obtain the total overlap rate deviation.

[0105] S2254: Obtain the correspondence relationship between the boundary point set and the line segment set according to the included angle, the Euler distance between the first and last boundary points of the boundary point set, the Euler distance between the two end points of the line segment set, and the overlap rate.

[0106] Accumulate the total included angle deviation, the total Euler distance deviation, and the total overlap rate deviation to obtain the total deviation, minimize the total deviation to obtain the positional relationship between the boundary point set and the line segment set, that is, the correspondence relationship between the boundary point set and the line segment set.

[0107] Further, the size transformation relationship between the boundary point set and the line segment set can also be obtained.

[0108] S23: Project the boundary point set in the point cloud image into the image according to the initial extrinsic parameter.

[0109] The initial extrinsic parameter, that is, the correspondence relationship between the point cloud image and the image, directly projects the boundary point set into the image, and the boundary point and the corresponding line segment have a large deviation, wherein the deviation includes a spatial position deviation.

[0110] Further, the deviation also includes a size transformation deviation.

[0111] S24: Optimize the initial extrinsic parameter according to the distance between the boundary point set and the line segment set to obtain the extrinsic parameter of the radar and the camera.

[0112] Each boundary point in the boundary point set has a displacement deviation with the corresponding line segment in the line segment set, the displacement deviation is accumulated and minimized, and a series of correction parameters are obtained, and the initial extrinsic parameter can be optimized through the correction parameters.

[0113] As shown in Figure 7 , step S24 includes:

[0114] S241: Calculate the distance of each boundary point in the boundary point set to the corresponding line segment in the line segment set.

[0115] Calculate the distance deviation between the corresponding boundary point and the line segment, which can be represented as a vector.

[0116] S242: obtaining a loss function according to the distance.

[0117] According to the distance deviation, a loss function is constructed, and the loss function is as follows:

[0118]

[0119] wherein, two endpoints of the line segment, the boundary point.

[0120] S243: optimizing the initial extrinsic parameter according to the loss function to obtain the extrinsic parameter of the radar and the camera.

[0121] Minimizing the loss function obtains a series of correction parameters, and the initial extrinsic parameter can be corrected by the correction parameters to obtain the extrinsic parameter of the camera and the radar.

[0122] The extrinsic parameter calibration method of the application optimizes the initial extrinsic parameter to obtain the final extrinsic parameter by matching and aligning the line segment in the image collected by the camera and the point cloud collected by the radar. This method can be applied to low-line-number radars, reducing the use cost, expanding the application range, and having low requirements for the motion trajectory of the sensor, which can be applied to the case of 2D plane motion in a small scene, improving the case where the extrinsic parameter result cannot be calculated under this condition, and enabling the normal calibration process.

[0123] Embodiment 3

[0124] As Figure 8 shown, Figure 8 is a flowchart of the extrinsic parameter calibration method provided by Embodiment 3 of the application. The extrinsic parameter calibration method provided by the embodiment is applied to an intelligent vehicle 100, and the intelligent vehicle 100 is fixedly provided with a camera and a radar. The method comprises the following steps:

[0125] S31: obtaining an initial extrinsic parameter according to the pose of the radar and the pose of the camera.

[0126] S32: extracting a boundary point set in a point cloud obtained by the radar and a line segment set in an image obtained by the camera, respectively, and obtaining the corresponding relationship between the boundary point set and the line segment set.

[0127] S33: projecting the boundary point set in the point cloud to the image according to the initial extrinsic parameter.

[0128] S34: optimizing the initial extrinsic parameter according to the distance between the boundary point set and the line segment set to obtain the extrinsic parameter of the radar and the camera.

[0129] S35: correcting the initial extrinsic parameter according to the ground normal vector.

[0130] Since the camera and the radar move on a plane, it is impossible to determine the translation of the camera and the radar in the direction of the rotation axis, at this time the offset vector in the direction of the rotation axis can be calculated through the ground point cloud, and the initial external parameter is corrected through the offset vector.

[0131] In some examples, step S35 only needs to be after step S31, and there is no sequence relationship between steps S32 to S34.

[0132] In one example, as shown in Figure 9 step S35 specifically includes:

[0133] S351: Extract the ground point cloud according to the ground normal vector.

[0134] In the plane rotation of the camera and the radar, the direction of the rotation axis is the direction of the ground normal vector, and the ground point cloud can be extracted using ransac according to the ground normal vector, specifically: rotating the point cloud map to a standard right-handed coordinate system, taking the part with the minimum value in the z-axis direction as the ground candidate points, for example, taking the first 10% of the part with the minimum value in the z-axis direction as the ground candidate points, and then using ransac to extract the ground points.

[0135] Further, the extraction of the ground point cloud is performed by a loop iteration method, specifically: using the ground normal vector used last time for the extraction of the ground points this time, which can reduce the error in the direction of the rotation axis, and improve the problem of inaccurate extraction of the ground candidate points due to the angle between the local ground normal vector and the rotation axis caused by a too large scene.

[0136] S352: Obtain the offset vector according to the ground point cloud.

[0137] According to the ground point cloud, the ground equation coefficients in the camera coordinate system and the radar coordinate system are calculated respectively:

[0138] [a c , b c , c c , d c ], [a l , b l , c l , d l ]

[0139] wherein ax+by+cz+d=0, ||[a.b, c]|| 2 =1

[0140] According to the ground equation coefficients, the offset vector is calculated:

[0141] t=tinit + n * (d c - d l )

[0142] wherein n is a unit normal vector of a local position of the ground.

[0143] S353: correcting the initial extrinsic parameter based on the offset vector.

[0144] According to the obtained offset vector, the error in the rotation axis direction of the initial extrinsic parameter is corrected, and the initial extrinsic parameter is further optimized.

[0145] The extrinsic parameter calibration method of the application can reduce the error in the rotation axis direction of the camera and the radar by extracting the ground points and correcting the initial extrinsic parameter, and further optimize the extrinsic parameter between the camera and the radar.

[0146] Embodiment 4

[0147] As Figure 10 shown, Figure 10 is a structural schematic diagram of an extrinsic parameter calibration device provided by Embodiment 4 of the application. The extrinsic parameter calibration device 400 provided by the embodiment of the application is applied to an intelligent vehicle 100, the intelligent vehicle 100 is fixedly provided with a camera and a radar, and the device comprises an initialization module 410, a corresponding relationship determination module 420, a projection module 430 and an extrinsic parameter determination module 440. The initialization module 410 is used to obtain an initial extrinsic parameter according to the pose of the radar and the pose of the camera. The corresponding relationship determination module 420 is used to extract a boundary point set in a point cloud diagram obtained by the radar and a line segment set in an image obtained by the camera, respectively, and obtain the corresponding relationship between the boundary point set and the line segment set. The projection module 430 is used to project the boundary point set in the point cloud diagram into the image according to the initial extrinsic parameter. The extrinsic parameter determination module 440 is used to optimize the initial extrinsic parameter according to the distance between the boundary point set and the line segment set, and obtain the extrinsic parameter of the radar and the camera.

[0148] As Figure 11As shown, for the correspondence determining module 420, the correspondence determining module 420 comprises a separation module 421, a marking module 422, a finding module 423, a filtering module 424 and a comparison module 425, the separation module 421 is configured to separate the continuously scanned point cloud from the point cloud in the point cloud image; the marking module 422 is configured to mark the first and last two points in the continuously scanned point cloud as candidate boundary points; the finding module 423 is configured to project the candidate boundary points to the image, extract a line segment set in the image, and find candidate boundary points corresponding to the line segment set; the filtering module 424 is configured to filter the candidate boundary points to obtain a boundary point set; and the comparison module 425 is configured to compare the line segment set with the boundary point set to obtain the correspondence between the boundary point set and the line segment set.

[0149] As shown, Figure 12 for the extrinsic parameter determining module 440, the extrinsic parameter determining module 440 comprises a distance calculation module 441, a loss function module 442 and an extrinsic parameter optimization module 443, the distance calculation module 441 is configured to calculate the distance from each boundary point in the boundary point set to the line segment corresponding to the line segment set respectively; the loss function module 442 is configured to obtain a loss function according to the distance; and the extrinsic parameter optimization module 443 is configured to optimize the initial extrinsic parameter according to the loss function to obtain the extrinsic parameter of the radar and the camera.

[0150] It should be noted that the above-mentioned extrinsic parameter calibration device 400 can execute the extrinsic parameter calibration method provided by the embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the embodiment of the extrinsic parameter calibration device 400 can be referred to the extrinsic parameter calibration method provided by the embodiment of the present application.

[0151] Embodiment 5

[0152] The embodiment of the present application provides a computer program product, the computer program product comprises a computer program stored on a non-volatile computer readable storage medium, the computer program comprises program instructions, when the program instructions are executed by a computer, the computer executes the extrinsic parameter calibration method. For example, the method steps S21 to S24 in Figure 2 are executed.

[0153] Embodiment 6

[0154] The embodiment of the present application provides a non-volatile computer readable storage medium, the computer readable storage medium stores computer executable instructions, the computer executable instructions are used to make the computer execute the extrinsic parameter calibration method. For example, the method steps S21 to S24 in Figure 2 are executed.

[0155] It should be noted that the above-described apparatus embodiments are merely illustrative, and the modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, i.e., can be located in one place or distributed on multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the method of the embodiments can be completed by computer program instruction related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the embodiments of the method. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0157] Finally, it should be noted that: the above embodiments are used to illustrate the technical solutions of the present application, and not to limit them; under the idea of the present application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of the different aspects of the present application as described above. In order to be brief, they are not provided in detail; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An external parameter calibration method, applied to an intelligent vehicle, wherein the intelligent vehicle is fixedly equipped with a camera and radar, characterized in that, The method includes the following steps: Initial extrinsic parameters are obtained based on the pose of the radar and the pose of the camera; Ground point cloud is extracted based on ground normal vectors using an iterative method. This iterative method includes using the previously used ground normal vector as the current extraction vector for ground points; obtaining an offset vector from the ground point cloud; and correcting the initial extrinsic parameters based on the offset vector. The boundary point set in the point cloud map obtained by the radar and the line segment set in the image obtained by the camera are extracted respectively, and the correspondence between the boundary point set and the line segment set is obtained. Based on the initial extrinsic parameters, the boundary point set in the point cloud map is projected into the image; The initial extrinsic parameters are optimized based on the distance between the boundary point set and the line segment set to obtain the extrinsic parameters of the radar and the camera. This includes: calculating the distance from each boundary point in the boundary point set to the corresponding line segment in the line segment set; obtaining a loss function based on the distance; and optimizing the initial extrinsic parameters based on the loss function to obtain the extrinsic parameters of the radar and the camera. The loss function is as follows: ； The orientation of the camera is as follows: ； The displacement of the camera is: ; The radar is oriented as follows: ; The displacement of the radar is: ; for The normal vector; is the identity matrix; K is the intrinsic parameter of the camera; , The two endpoints of the line segment are... Let be the boundary point.

2. The method according to claim 1, characterized in that, The step of extracting the boundary point set from the point cloud image obtained by the radar and the line segment set from the image obtained by the camera, and obtaining the correspondence between the boundary point set and the line segment set, includes: Separate the continuously scanned point cloud from the point cloud image; The first and last two points in the continuously scanned point cloud are recorded as candidate boundary points; Project the candidate boundary points onto the image, extract the line segment set in the image, and find the candidate boundary points corresponding to the line segment set; Filter the candidate boundary points to obtain the boundary point set; By comparing the set of line segments with the set of boundary points, the correspondence between the set of boundary points and the set of line segments is obtained.

3. The method according to claim 2, characterized in that, The step of comparing the set of line segments with the set of boundary points to obtain the correspondence between the set of boundary points and the set of line segments includes: Calculate the angle between the line containing the boundary point set and the line containing the line segment set; Calculate the Eulerian distance between the first and last boundary points of the boundary point set and the Eulerian distance between the two endpoints of the line segment set, respectively. Calculate the overlap rate between the boundary point set and the line segment set; The correspondence between the boundary point set and the line segment set is obtained based on the included angle, the Eulerian distance between the first and last boundary points of the boundary point set, the Eulerian distance between the two endpoints of the line segment set, and the overlap rate.

4. The method according to claim 2, characterized in that, The step of filtering the candidate boundary points to obtain the boundary point set specifically includes: Obtain the radar emission angle between any two candidate boundary points; Obtain the candidate boundary points whose radar emission angle is less than a first threshold; Based on the acquired candidate boundary points, candidate boundary points that are far from the radar are filtered out to obtain a set of boundary points.

5. An external parameter calibration device, applied to an intelligent vehicle, wherein the intelligent vehicle is fixedly equipped with a camera and radar, characterized in that, The device includes: The initialization module obtains initial extrinsic parameters based on the pose of the radar and the pose of the camera; The correction module extracts ground point clouds based on ground normal vectors using an iterative method; obtains offset vectors based on the ground point clouds; and corrects the initial extrinsic parameters based on the offset vectors. The correspondence determination module extracts the boundary point set from the point cloud map obtained by the radar and the line segment set from the image obtained by the camera, and obtains the correspondence between the boundary point set and the line segment set. The projection module projects the set of boundary points in the point cloud image onto the image based on the initial extrinsic parameters. The extrinsic parameter determination module optimizes the initial extrinsic parameters based on the distance between the boundary point set and the line segment set, thereby obtaining the extrinsic parameters of the radar and the camera. The external parameter determination module includes: The distance calculation module calculates the distance from each boundary point in the boundary point set to the corresponding line segment in the line segment set. The loss function module obtains the loss function based on the distance. The extrinsic parameter optimization module optimizes the initial extrinsic parameters according to the loss function to obtain the extrinsic parameters of the radar and the camera; The loss function is as follows: ； The orientation of the camera is as follows: ； The displacement of the camera is: ; The radar is oriented as follows: ; The displacement of the radar is: ; for The normal vector; is the identity matrix; K is the intrinsic parameter of the camera; , The two endpoints of the line segment are... Let be the boundary point.

6. The apparatus according to claim 5, characterized in that, The correspondence determination module includes: The separation module separates the continuously scanned point cloud from the point cloud in the point cloud image; The marking module records the first and last two points in the continuously scanned point cloud as candidate boundary points; The module projects the candidate boundary points onto the image, extracts the set of line segments in the image, and finds the candidate boundary points corresponding to the set of line segments. The filtering module filters the candidate boundary points to obtain a set of boundary points; The comparison module compares the set of line segments with the set of boundary points to obtain the correspondence between the set of boundary points and the set of line segments.

7. An intelligent vehicle, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-4.

Images