Camera External Parameter Calibration Method, Device, Equipment and Storage Medium

Abstract

An embodiment of the present application provides a method, device, equipment and storage medium for calibrating the external parameters of a camera. The method includes: determining a first feature of a set calibration board corresponding to a first sensor based on the point cloud data of the set calibration board detected by the first sensor, where the set calibration board is a QR code calibration board, and the first feature includes a first normal vector and a first center point; determining a second feature obtained by a camera to be calibrated based on the set calibration board, where the second feature includes a second normal vector and a second center point; and obtaining an external parameter matrix corresponding to the camera to be calibrated based on a feature constraint equation, the first feature and the second feature. The embodiment of the present application solves the problems of insufficient accuracy and reliability of environmental detection of autonomous vehicles, and ensures the accuracy and reliability of environmental detection of autonomous vehicles.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of autonomous driving, and in particular, to a method, device, equipment and storage medium for calibrating the external parameters of a camera. Background Art

[0002] With the continuous development of driverless technology, the driving environment of vehicles is becoming more open and complex. To ensure the safety of vehicle driving, it is necessary to continuously detect the environment where the vehicle is located. In terms of environmental detection, the data of a single sensor such as a lidar or a camera is difficult to meet the requirements. Therefore, in related technologies, multi-sensor data fusion algorithms are usually used to give full play to the advantages of each sensor and meet the perception requirements of complex scenarios. In multi-sensor data fusion processing, it is necessary to first ensure that the coordinate systems corresponding to the data obtained by the camera and other sensors are unified. Therefore, it is necessary to calibrate the external parameter matrix used to convert data between different coordinate systems in the camera, that is, external parameter calibration.

[0003] Due to the large volume of driverless vehicles, the external parameter calibration methods in related technologies usually use a checkerboard calibration board for calibration. However, due to the uneven sparse distribution of the point cloud data measured by the lidar, it is easy to have a situation where there are very few points scanned on a certain edge of the checkerboard, resulting in a large fitting error of the calibration board and insufficient calibration accuracy, leading to problems of insufficient accuracy and reliability of the vehicle's environmental detection. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, equipment and storage medium for calibrating the external parameters of a camera to solve the problems of insufficient accuracy and reliability of the environmental detection of autonomous driving vehicles.

[0005] In a first aspect, the present application provides a method for calibrating the external parameters of a camera, and the method for calibrating the external parameters of a camera includes:

[0006] Based on the point cloud data of a set calibration board detected by a first sensor, determine the first features of the set calibration board corresponding to the first sensor, the set calibration board is a two-dimensional code calibration board, and the first features include a first normal vector and a first center point;

[0007] Determine the second features obtained by the camera to be calibrated based on the set calibration board, and the second features include a second normal vector and a second center point;

[0008] Based on the feature constraint equation, the first features and the second features, obtain the external parameter matrix corresponding to the camera to be calibrated.

[0009] It can be seen that, based on the point cloud data of the set calibration board detected by the first sensor, the first feature of the set calibration board corresponding to the first sensor is determined, and the second feature obtained by the camera to be calibrated based on the set calibration board is determined. Then, based on the feature constraint equation, the first feature and the second feature, the external parameter matrix corresponding to the camera to be calibrated is obtained. Thus, the external parameter calibration of the camera to be calibrated can be performed by using the calibration board features obtained by a sensor of a different type from the camera to be calibrated. By using a flat plate containing a QR code as the calibration board for calibration, there is no need for complex processing such as thickening or hollowing out the calibration board, which saves costs. At the same time, by accurately detecting the QR code by the camera, the accuracy of the second feature measured by the camera is guaranteed, and further the accuracy of the external parameter matrix is guaranteed, thereby ensuring the accuracy and reliability of the environmental detection of the autonomous driving vehicle.

[0010] Optionally, the first sensor is a laser sensor.

[0011] It can be seen that, by using the laser sensor as the first sensor cooperating with the camera to be calibrated, and taking advantage of the simple structure when the set calibration board is a QR code calibration board, it is convenient for laser detection, guarantees the accuracy and reliability of the first feature obtained by the first sensor, and further guarantees the accuracy of the external parameter matrix obtained by calibration.

[0012] Optionally, obtaining the external parameter matrix corresponding to the camera to be calibrated based on the feature constraint equation, the first feature and the second feature includes: obtaining the first feature and the second feature of a set number of groups; substituting the first feature and the second feature of the set number of groups into the feature constraint equation, and determining the parameter matrix when the direction consistency, direction matching degree and distance matching degree between the first sensor and the camera to be calibrated are optimized, so as to determine the external parameter matrix.

[0013] It can be seen that, by taking the direction consistency, direction matching degree and distance matching degree between the first sensor and the camera to be calibrated as indicators for optimization, it is ensured that the obtained external parameter matrix can guarantee that the detection results of the camera to be calibrated and the first sensor are optimal in these indicators, and further guarantee the maximum unification of the coordinate systems of the camera and the first sensor, thereby ensuring the accuracy and reliability of the environmental detection of the autonomous driving vehicle.

[0014] Optionally, the feature constraint equation includes:

[0015]

[0016] T(R,t)=argmin T(R,t) (e d +e r +e t );

[0017] Wherein, e d is the direction consistency index, e r is the direction matching degree index, et is the distance matching degree index, M is the set number of groups, T(R,t) is the external parameter matrix, R is the rotation matrix in the external parameter matrix, t is the translation matrix in the external parameter matrix, o l is the coordinate of the first center point, n l is the first normal vector, o c is the coordinate of the second center point, n c is the second normal vector.

[0018] It can be seen that since the sum of the three indexes of direction consistency, direction matching degree and distance matching degree is a non - negative value, when the sum is the smallest, that is, when the three indexes of the first feature obtained by the first sensor and the second feature obtained by the camera to be calibrated are the best, thus, it can be ensured that the external parameter matrix migrates the second feature to the same coordinate system as the first feature, achieving the maximum unification of the coordinate system, thereby ensuring the accuracy and reliability of the environmental detection of the autonomous driving vehicle.

[0019] Optionally, based on the point cloud data of the set calibration board detected by the first sensor, determining the first feature of the set calibration board corresponding to the first sensor includes: based on the point cloud data, determining the plane equation of the plane where the set calibration board is located and the first normal vector corresponding to the plane; based on the plane equation, determining the edge points in the point cloud data; based on the edge points, determining the first center point in the set calibration board.

[0020] It can be seen that by determining the plane equation and the first direction vector of the set calibration board through the point cloud data, and finding the edge points, and then determining the center point based on the edge points, thus, it is possible to conveniently determine the first feature of the set calibration board through the point cloud data, so as to combine with the second feature to perform external parameter calibration on the camera to be calibrated. And the calculation amount is small, the calculation efficiency is high, and the structural accuracy is strong.

[0021] Optionally, based on the plane equation, determining the edge points in the point cloud data includes: projecting the point cloud data onto the plane where the set calibration board is located to obtain the projected point cloud; determining the projected points in the projected point cloud that are different from two adjacent projected points as the edge points, and determining the coordinates of the edge points.

[0022] It can be seen that by projecting the point cloud data onto the plane where the set calibration board is located, all the projected points are located on the same plane. At this time, the relative positions of the projected points corresponding to the points originally on the same plane will remain unchanged, while the relative positions of the projected points corresponding to the points not on the same plane will change. Thus, it is possible to conveniently distinguish the edge points from the non - edge points and quickly identify the edge points to determine the first feature accordingly.

[0023] Optionally, set the calibration board as a circular calibration board, and the circular calibration board includes a QR code; based on the edge points, determine the first center point in the set calibration board, including: determining a target point that is equidistant from all edge points and lies in the plane where the set calibration board is located; determining the target point as the first center point.

[0024] It can be seen that by setting the set calibration board as circular, the target point can be conveniently determined directly according to the center of the circle formed by all the edge points of the set calibration board, and then the first center point can be determined, reducing the calculation difficulty of the first center point and improving the calculation efficiency, so as to determine the first feature accordingly.

[0025] Optionally, determining the second feature obtained by the camera to be calibrated based on the set calibration board includes: determining the corresponding positioning algorithm according to the type of the set calibration board; determining the second feature based on the positioning algorithm and the image data collected by the camera to be calibrated.

[0026] It can be seen that by determining the corresponding camera positioning algorithm according to the type of the set calibration board, the existing camera positioning algorithm can be used to quickly obtain the second feature, which is convenient to process and is adapted to the type of the set calibration board, ensuring the accuracy of the camera measurement result.

[0027] In a second aspect, the present application provides a camera extrinsic parameter calibration device, and the camera extrinsic parameter calibration device includes:

[0028] A first detection module, configured to determine the first feature of the set calibration board corresponding to the first sensor based on the point cloud data of the set calibration board detected by the first sensor;

[0029] A second detection module, configured to determine the second feature obtained by the camera to be calibrated based on the set calibration board, where the second feature includes a second normal vector and a second center point;

[0030] A processing module, configured to obtain the extrinsic parameter matrix corresponding to the camera to be calibrated based on the feature constraint equation, the first feature, and the second feature.

[0031] Optionally, the first detection module specifically includes that the first sensor is a laser sensor.

[0032] Optionally, the processing module is specifically configured to: obtain a set number of first features and second features; substitute the set number of first features and second features into the feature constraint equation, and determine the parameter matrix when the direction consistency, direction matching degree, and distance matching degree between the first sensor and the camera to be calibrated are optimized, and determine the extrinsic parameter matrix.

[0033] Optionally, the processing module specifically includes that the feature constraint equation includes:

[0034]

[0035] T(R, t) = argmin T(R,t) (e d + e r + e t );

[0036] Wherein, e d is the direction consistency index, e r is the direction matching degree index, e t is the distance matching degree index, M is the set number of groups, T(R, t) is the external parameter matrix, R is the rotation matrix in the external parameter matrix, t is the translation matrix in the external parameter matrix, o l is the coordinate of the first center point, n l is the first normal vector, o c is the coordinate of the second center point, n c is the second normal vector.

[0037] Optionally, the first detection module is specifically configured to determine the plane equation of the plane where the set calibration plate is located and the first normal vector corresponding to the plane based on the point cloud data; determine the edge points in the point cloud data based on the plane equation; and determine the first center point in the set calibration plate based on the edge points.

[0038] Optionally, the first detection module is specifically configured to project the point cloud data onto the plane where the set calibration plate is located to obtain the projected point cloud; determine the edge points as the points in the projected point cloud that are different from two adjacent projected points and determine the coordinates of the edge points.

[0039] Optionally, the first detection module is specifically configured to, if the set calibration plate is a circular calibration plate and the circular calibration plate contains a QR code, determine the target points that are equidistant from all edge points and are located on the plane where the set calibration plate is located; and determine the target points as the first center points.

[0040] Optionally, the second detection module is specifically configured to determine the corresponding positioning algorithm according to the type of the set calibration plate; and determine the second feature based on the positioning algorithm and the image data collected by the camera to be calibrated.

[0041] In a third aspect, the present application further provides a control device, which includes:

[0042] At least one processor;

[0043] And a memory communicatively connected to the at least one processor;

[0044] Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the control device executes the camera external parameter calibration method corresponding to any one of the embodiments in the first aspect of the present application.

[0045] Fourthly, the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the camera external parameter calibration method according to any one of the first aspects of the present application.

[0046] Fifthly, the present application further provides a computer program product, which includes computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the camera external parameter calibration method according to any one of the corresponding embodiments of the first aspect of the present application. Description of the Drawings

[0047] Figure 1 It is an application scenario diagram of the camera external parameter calibration method provided by an embodiment of the present application;

[0048] Figure 2 It is a flowchart of the camera external parameter calibration method provided by an embodiment of the present application;

[0049] Figure 3a It is a flowchart of the camera external parameter calibration method provided by another embodiment of the present application;

[0050] Figure 3b It is Figure 3a the flowchart of the method for determining edge points provided in the illustrated embodiment;

[0051] Figure 3c It is Figure 3a the flowchart of the method for determining the first center point provided in the illustrated embodiment;

[0052] Figure 4 It is a schematic structural diagram of the camera external parameter calibration device provided by another embodiment of the present application;

[0053] Figure 5 It is a schematic structural diagram of the control device provided by another embodiment of the present application. Detailed Embodiments

[0054] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0055] The following uses specific embodiments to elaborate in detail on the technical solutions of the embodiments of the present application and how the technical solutions of the embodiments of the present application solve the above technical problems. The following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0056] With the continuous development of autonomous driving technology, the driving environment of vehicles is becoming more open and complex. To ensure the safety of vehicle driving, it is necessary to continuously detect the environment where the vehicle is located. In terms of environmental detection, it is difficult for the data of a single sensor such as a lidar or a camera to meet the requirements. Therefore, in related technologies, a multi-sensor data fusion algorithm is usually used to give full play to the advantages of each sensor and meet the perception requirements of complex scenarios. In the multi-sensor data fusion process, it is necessary to first ensure that the coordinate systems corresponding to the data obtained by the camera and other sensors are unified. Therefore, it is necessary to calibrate the external parameter matrix used to convert data between different coordinate systems in the camera, that is, external parameter calibration.

[0057] Common external parameter calibration methods include calibration through calibration rooms and calibration through checkerboard calibration plates. The former is carried out in an indoor calibration room. Since autonomous vehicles are large in size and have high requirements for the measurement range, it is not suitable for calibration in calibration rooms. Therefore, the external parameter calibration methods in related technologies usually use checkerboard calibration plates for calibration. By scanning the points on each side of the calibration plate, fitting each side, and determining its intersection points, the calibration plate detection is completed. However, due to the uneven sparse distribution of the point cloud data measured by the lidar, it is easy to have a situation where there are very few points scanned on a certain side of the checkerboard, resulting in large errors when fitting the corresponding sides and intersection points of the calibration plate, insufficient calibration accuracy, and problems with the accuracy and reliability of vehicle environmental detection.

[0058] To solve the above problems, the embodiments of the present application provide a camera external parameter calibration method. By using a QR code calibration plate, the first sensor and the camera to be calibrated are respectively used for detection and positioning, and then based on the optimization of the detection results, the external parameter matrix is determined to complete the external parameter calibration and improve the camera detection accuracy.

[0059] Figure 1 This is an application scenario diagram of the camera external parameter calibration method provided by the embodiments of the present application. As Figure 1 shown, in the process of camera external parameter calibration, the first sensor 100 and the camera 110 to be calibrated are respectively used to obtain the feature data of the calibration plate 120, and combined with the feature data, the external parameter matrix corresponding to the camera to be calibrated is obtained to achieve camera external parameter calibration.

[0060] It should be noted that Figure 1In the illustrated scenario, only one first sensor, a camera to be calibrated, and a calibration board are taken as an example for illustration, but the embodiments of the present application are not limited thereto. That is to say, the numbers of the first sensor, the camera to be calibrated, and the calibration board can be arbitrary.

[0061] The method for calibrating the external parameters of a camera provided by the present application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0062] Figure 2 It is a flowchart of the method for calibrating the external parameters of a camera provided by an embodiment of the present application. As Figure 2 shown, the method for calibrating the external parameters of a camera provided in this embodiment includes, but is not limited to, the following steps:

[0063] Step S201: Based on the point cloud data of the set calibration board detected by the first sensor, determine the first features of the set calibration board corresponding to the first sensor.

[0064] Among them, the set calibration board is a two-dimensional code calibration board, and the first features include a first normal vector and a first center point.

[0065] Specifically, the first sensor is a sensor that cooperates with the camera to be calibrated and is jointly used to detect the external environment data of an autonomous vehicle. According to different requirements of actual measurement applications, the first sensor can be different types of vehicle-mounted sensors, such as common lidar sensors, or different types of sensors such as a pre-calibrated structured light camera and a stereo vision camera. Any vehicle-mounted sensor that can obtain point cloud data in the external environment can be used as the first sensor and is applicable to this solution.

[0066] Since the first sensor can obtain the point cloud data in the external environment, it can obtain the point cloud data in the pre-prepared set calibration board.

[0067] The calibration board is a pre-prepared calibration board containing patterns for positioning. Different from the existing checkerboard calibration board or hollow calibration board, the calibration boards used in this solution are all QR code calibration boards. And according to the different positioning algorithms used by the camera to be calibrated, the patterns of the QR codes in the specific QR code calibration boards also vary, such as Apriltag codes, aruco calibration boards, charuco calibration boards, or any other existing QR code calibration boards. By using a QR code calibration board instead of a hollow calibration board, the QR code calibration board only requires the first sensor to be able to distinguish the boundary between the area where the QR code pattern is located and the edge area (i.e., the edge points), and only a small number of boundary points are required (such as at least three edge points) to determine the corresponding first feature, so as to solve the problem that when the first sensor is a lidar sensor, the scanned point cloud distribution may be uneven, resulting in errors in the first feature measured.

[0068] The specific type of the first feature is the type of feature that the camera to be calibrated can also detect, so as to determine the external parameter matrix required to transform the first feature to the second feature by comparing the first feature and the second feature collected by the camera to be calibrated.

[0069] The first feature is usually the first normal vector and the first center point (coordinates). The first normal vector is the normal vector of the plane where the calibration board is located obtained by the first sensor, and the first center point is the center point of the QR code pattern in the calibration board.

[0070] Step S202: Determine the second feature obtained by the camera to be calibrated based on the calibration board.

[0071] Among them, the second feature includes the second normal vector and the second center point.

[0072] Specifically, the camera to be calibrated can obtain the second feature in the calibration board based on the positioning algorithm through the image of the calibration board captured. According to the different types of QR codes in the calibration board, the specific positioning algorithms may vary. Its principle is mainly to identify and process the captured image, find the border contours in the QR code, and then determine the coordinates of the center point of the QR code pattern (i.e., the second center point) based on the coordinates of the border contours of all patterns in the QR code pattern in the captured image. Then, according to the deformation state of the quadrilateral contour in the QR code pattern, combined with the center point of the QR code pattern, the second normal vector of the calibration board is determined.

[0073] The second normal vector is the normal vector of the plane where the calibration board is located obtained by the camera to be calibrated, and the second center point is the center point of the QR code pattern in the calibration board.

[0074] Step S203: Based on the feature constraint equation, the first feature and the second feature, obtain the external parameter matrix corresponding to the camera to be calibrated.

[0075] Specifically, the feature constraint equation is an equation used to define the relationship between the first feature and the second feature from the perspective of set indicators. The set indicators can be the similarities between the first feature and the second feature in different dimensions, such as direction consistency, distance matching degree, etc.

[0076] From the perspective of the set indicators, the smaller the difference obtained by comparing the second feature after parameter matrix transformation with the first feature, the closer the corresponding set indicator is to the ideal result.

[0077] To ensure the coordinate systems of the first sensor and the camera to be calibrated are unified, the set indicators in the feature constraint equation need to reach the optimal value. At this time, the parameter matrix is the external parameter matrix. Given the first feature and the second feature, by constructing a feature constraint equation system with multiple set indicators and combining the goal of the optimal set indicators, the external parameter matrix can be solved, and then the external parameter calibration of the camera to be calibrated can be completed.

[0078] The camera external parameter calibration method provided in the embodiments of the present application determines the first feature of the set calibration board corresponding to the first sensor based on the point cloud data of the set calibration board detected by the first sensor, and determines the second feature obtained by the camera to be calibrated based on the set calibration board. Then, based on the feature constraint equation, the first feature, and the second feature, the external parameter matrix corresponding to the camera to be calibrated is obtained. Thus, the external parameter calibration of the camera to be calibrated can be performed using the calibration board features obtained by a sensor of a different type from the camera to be calibrated. By using a flat plate containing a QR code as the calibration board for calibration, there is no need for complex processing such as thickening and hollowing out the calibration board, saving costs. At the same time, by accurately detecting the QR code by the camera, the accuracy of the second feature measured by the camera is ensured, and then the accuracy of the external parameter matrix is ensured, thereby ensuring the accuracy and reliability of the environmental detection of the autonomous driving vehicle.

[0079] Figure 3a It is a flowchart of the camera external parameter calibration method provided in another embodiment of the present application. As Figure 3a shown, the camera external parameter calibration method includes:

[0080] Step S301: Based on the point cloud data, determine the plane equation of the plane where the set calibration board is located and the first normal vector corresponding to the plane.

[0081] Among them, the first sensor is a laser sensor.

[0082] Specifically, since the laser sensor is the sensor that cooperates the most with the camera in the field of environmental detection of autonomous driving vehicles, in this embodiment, the first sensor is described as a laser sensor. However, in fact, the first sensor can also be other sensors except the laser sensor, referring to the corresponding description in the Figure 2 shown embodiment.

[0083] After obtaining the point cloud data, the plane equation of the plane where the set calibration plate is located can be determined through the algorithm of fitting a plane with the point cloud. There are various existing conventional algorithms for the algorithm of fitting a plane with the point cloud, and those skilled in the art can arbitrarily select from them according to actual needs, which will not be elaborated here.

[0084] The expression of the plane equation is usually: ax + by + cz + d = 0, where (a, b, c, d) are the parameters of the plane equation. At this time, (a, b, c) is the normal vector of this plane.

[0085] Thus, after determining the plane equation of the set calibration plate through the algorithm of fitting a plane with the point cloud, its corresponding normal vector, that is, the first normal vector, can be conveniently determined.

[0086] Step S302: Determine the edge points in the point cloud data based on the plane equation.

[0087] Specifically, the point cloud detected by the laser sensor includes not only the point cloud located on the set calibration plate but also the point cloud located outside the set calibration plate. The points located on the edge of the set calibration plate and adjacent to the point cloud outside the set calibration plate are the edge points. By determining the edge points, the center point of the set calibration plate can be conveniently determined.

[0088] Furthermore, as Figure 3b shown, it is the flowchart of the method for determining the edge points, and its specific steps include:

[0089] Step S3021: Project the point cloud data onto the plane where the set calibration plate is located to obtain the projected point cloud.

[0090] Specifically, all the point clouds monitored by the laser sensor are projected onto the plane where the set calibration plate is located. At this time, the points originally located on the set calibration plate coincide with their projected points, while the points not on the set calibration plate do not coincide with their projected points.

[0091] The calculation method of the projected point cloud can directly substitute the point cloud data into the plane equation of the set calibration plate to obtain the point cloud of the corresponding projected point.

[0092] Step S3022: Determine the projected points in the projected point cloud that are different from two adjacent projected points as edge points and determine the coordinates of the edge points.

[0093] Specifically, due to the sparsity of the point cloud, the distance between the projected points of the points not on the set calibration plate and the adjacent points on the set calibration plate is usually greater than the distance between the adjacent points on the set calibration plate (and since the objects corresponding to the points not on the set calibration plate are usually not objects parallel to the plane where the set calibration plate is located, the distances between the points not on the set calibration plate are usually also different from each other). Therefore, it can be judged according to the distances between the projected points and the adjacent points in the projected point cloud.

[0094] Exemplarily, when the distances from the first projection point to two adjacent projection points (the second projection point and the third projection point) are the same (denoted as the first distance), and the distance from the second projection point to the fourth projection point adjacent to the second projection point (denoted as the second distance) is different from the first distance, the second projection point can be determined as the edge point, the first projection point and the third projection point are non-edge points on the set calibration plate, and the fourth projection point is the projection point of the point on the non-set calibration plate.

[0095] Step S303: Determine the first center point in the set calibration plate based on the edge points.

[0096] Specifically, when the shape of the set calibration plate is known, the center point of the set calibration plate, that is, the first center point, can be determined according to the edge points.

[0097] Furthermore, as Figure 3c shown, it is a method flow chart for determining the first center point, and its specific steps include:

[0098] Step S3031: Determine the target point that has the same distance to all edge points and lies in the plane where the set calibration plate is located.

[0099] Among them, the set calibration plate is a circular calibration plate, and the circular calibration plate contains a QR code.

[0100] Specifically, when the set calibration plate is a circular calibration plate, only the coordinates of three edge points need to be known, and the coordinates of the point target point can be determined in combination with the plane equation of the set calibration plate.

[0101] Step S3032: Determine the target point as the first center point.

[0102] Specifically, the target point is the first center point. This determination method can be not restricted by the pattern shape in the set calibration plate. As long as it is a laser sensor, it can be conveniently and directly calculated, with low calculation difficulty, high calculation efficiency, and strong reliability.

[0103] Step S304: Determine the corresponding positioning algorithm according to the type of the set calibration plate.

[0104] Specifically, the QR code of the set calibration plate can be of various different types. In the prior art, there are various QR code calibration plates for camera positioning detection. Each QR code has a corresponding positioning algorithm. Those skilled in the art can select the type of the QR code calibration plate and the corresponding positioning algorithm according to the actual situation and requirements, which are not limited here.

[0105] Step S305: Determine the second feature based on the positioning algorithm and the image data collected by the camera to be calibrated.

[0106] Specifically, the camera to be calibrated can determine the normal vector and the center point of the set calibration board, that is, the second normal vector and the second center point, based on the positioning algorithm corresponding to the QR code of the set calibration board and the collected image data.

[0107] Step S306: Obtain a set number of first features and second features.

[0108] Specifically, repeat Step S301 to Step S305 until a set number of first features and second features are obtained, so as to calculate the external parameter matrix more accurately through multiple groups of first features and second features. The larger the set number, the higher the accuracy and reliability when calculating the external parameter matrix.

[0109] Step S307: Substitute the set number of first features and second features into the feature constraint equation, and determine the external parameter matrix with the parameter matrix when the direction consistency, direction matching degree, and distance matching degree between the first sensor and the camera to be calibrated are optimized.

[0110] Specifically, substitute the first features and second features into the feature constraint equation, and optimize based on the objective function in the feature constraint equation to calculate the external parameter matrix.

[0111] In some embodiments, the feature constraint equation includes:

[0112]

[0113] T(R,t)=argmin T(R,t) (e d +e r +e t );

[0114] Where e d is the direction consistency index, e r is the direction matching degree index, e t is the distance matching degree index, M is the set number, T(R,t) is the external parameter matrix, R is the rotation matrix in the external parameter matrix, t is the translation matrix in the external parameter matrix, o l is the coordinate of the first center point, n l is the first normal vector, o c is the coordinate of the second center point, n c is the second normal vector.

[0115] T(R,t) is the objective function, Argmin is the value of the objective function when (e d +e r +e t ) is the smallest. Substitute the set number of first features and second features into the above formula, and the corresponding values of R and t can be calculated, and then the external parameter matrix can be obtained.

[0116] The camera extrinsic parameter calibration method provided by an embodiment of this application determines the plane equation of the plane where the set calibration board is located and the first normal vector corresponding to the plane based on point cloud data, then determines the edge points in the point cloud data based on the plane equation, and then determines the first center point in the set calibration board based on the edge points. According to the type of the set calibration board, the corresponding positioning algorithm is determined to determine the second feature accordingly. Finally, the obtained first features and second features of the set number of groups are substituted into the feature constraint equation to determine the extrinsic parameter matrix. Thus, by using the circular set calibration board, the computational data requirements during the detection of the laser sensor can be reduced, and the detection accuracy can be improved. By using the positioning algorithm corresponding to the QR code pattern, the detection accuracy of the camera to be calibrated is ensured, and further the accuracy of the extrinsic parameter matrix determined by the feature constraint equation is ensured, and the accuracy and reliability of the environmental detection of the autonomous driving vehicle are ensured.

[0117] Figure 4 It is a schematic structural diagram of a camera extrinsic parameter calibration device provided by an embodiment of this application. As Figure 4 shown, the camera extrinsic parameter calibration device 400 includes: a first detection module 410, a second detection module 420, and a processing module 430. Among them:

[0118] The first detection module 410 is configured to determine the first feature of the set calibration board corresponding to the first sensor based on the point cloud data of the set calibration board detected by the first sensor;

[0119] The second detection module 420 is configured to determine the second feature obtained by the camera to be calibrated based on the set calibration board, and the second feature includes a second normal vector and a second center point;

[0120] The processing module 430 is configured to obtain the extrinsic parameter matrix corresponding to the camera to be calibrated based on the feature constraint equation, the first feature, and the second feature.

[0121] Optionally, the first detection module 410 specifically includes that the first sensor is a laser sensor.

[0122] Optionally, the processing module 430 is specifically configured to obtain the first features and second features of the set number of groups; substitute the first features and second features of the set number of groups into the feature constraint equation, and determine the extrinsic parameter matrix when the direction consistency, direction matching degree, and distance matching degree between the first sensor and the camera to be calibrated are optimized.

[0123] Optionally, the processing module 430 specifically includes that the feature constraint equation includes:

[0124]

[0125] T(R,t)=argmin T(R,t) (ed +e r +e t );

[0126] Among them, e d is the direction consistency index, e r is the direction matching degree index, e t is the distance matching degree index, M is the set number of groups, T(R, t) is the external parameter matrix, R is the rotation matrix in the external parameter matrix, t is the translation matrix in the external parameter matrix, o l is the coordinate of the first center point, n l is the first normal vector, o c is the coordinate of the second center point, n c is the second normal vector.

[0127] Optionally, the first detection module 410 is specifically configured to determine the plane equation of the plane where the set calibration board is located and the first normal vector corresponding to the plane based on the point cloud data; determine the edge points in the point cloud data based on the plane equation; and determine the first center point in the set calibration board based on the edge points.

[0128] Optionally, the first detection module 410 is specifically configured to project the point cloud data onto the plane where the set calibration board is located to obtain the projected point cloud; determine the edge points as the projected points that are different or the same from the two adjacent projected points in the projected point cloud, and determine the coordinates of the edge points.

[0129] Optionally, when the set calibration board is a circular calibration board and the circular calibration board contains a QR code, the first detection module 410 is specifically configured to determine the target points that are equidistant from all edge points and located in the plane where the set calibration board is located; and determine the target points as the first center points.

[0130] Optionally, the second detection module 420 is specifically configured to determine the corresponding positioning algorithm according to the type of the set calibration board; and determine the second feature based on the positioning algorithm and the image data collected by the camera to be calibrated.

[0131] In this embodiment, the camera external parameter calibration device solves the problems of insufficient accuracy and reliability in environmental detection of existing autonomous vehicles through the combination of each module, and ensures the accuracy and reliability of environmental detection of autonomous vehicles.

[0132] Figure 5 is a schematic structural diagram of a control device provided by an embodiment of the present application. As Figure 5 shown, the control device 500 includes: a memory 510 and a processor 520.

[0133] Among them, the memory 510 stores a computer program that can be executed by at least one processor 520. The computer program is executed by at least one processor 520 to enable the control device to implement the camera extrinsic parameter calibration method provided in any of the above embodiments.

[0134] Among them, the memory 510 and the processor 520 can be connected through a bus 530.

[0135] For relevant descriptions, reference can be made to the corresponding descriptions and effects in the method embodiments, which will not be elaborated here.

[0136] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the camera extrinsic parameter calibration method of any of the above embodiments.

[0137] Among them, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0138] An embodiment of the present application provides a computer program product, which includes computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the camera extrinsic parameter calibration method of any corresponding Figures 2 to 3a embodiment.

[0139] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules is only a logical function division. In actual implementation, there can be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or modules can be in an electrical, mechanical or other form.

[0140] Those skilled in the art will readily think of other implementation schemes of the present application after considering the specification and practicing the application here. The present application aims to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include the common general knowledge or conventional technical means in the technical field not claimed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope of the present application is pointed out by the claims.

[0141] It should be understood that the present application is not limited to the exact structure already described and shown in the drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for calibrating the external parameters of a camera, characterized in that, Including: Based on the point cloud data of the set calibration board detected by the first sensor, determine the first feature of the set calibration board corresponding to the first sensor. The set calibration board is a QR code calibration board, and the first feature includes a first normal vector and a first center point; Determine the second feature obtained by the camera to be calibrated based on the set calibration board. The second feature includes a second normal vector and a second center point; Obtain a set number of the first features and the second features; Substitute the set number of the first features and the second features into the feature constraint equation, and determine the parameter matrix when the direction consistency, direction matching degree, and distance matching degree between the first sensor and the camera to be calibrated are optimized as the external parameter matrix; The feature constraint equation includes: ， ； Among them, e d is the direction consistency index, e r is the direction matching degree index, e t is the distance matching degree index, M is the set number, T(R,t) is the external parameter matrix, R is the rotation matrix in the external parameter matrix, t is the translation matrix in the external parameter matrix, o l is the coordinate of the first center point, n l is the first normal vector, o c is the coordinate of the second center point, n c is the second normal vector.

2. The camera extrinsic parameter calibration method according to claim 1, characterized in that, The first sensor is a laser sensor.

3. The camera extrinsic parameter calibration method according to any one of claims 1 to 2, characterized in that The determining the first feature of the set calibration board corresponding to the first sensor based on the point cloud data of the set calibration board detected by the first sensor includes: Based on the point cloud data, determine the plane equation of the plane where the set calibration board is located and the first normal vector corresponding to the plane; Based on the plane equation, determine the edge points in the point cloud data; Based on the edge points, determine the first center point in the set calibration board.

4. The camera extrinsic parameter calibration method according to claim 3, wherein, The determining the edge points in the point cloud data based on the plane equation includes: Project the point cloud data onto the plane where the set calibration board is located to obtain the projected point cloud; Determine the projected points in the projected point cloud with different distances from two adjacent projected points as edge points, and determine the coordinates of the edge points.

5. The method for calibrating the external parameters of a camera according to claim 3, characterized in that, The set calibration board is a circular calibration board, and the circular calibration board contains a QR code; The determining the first center point in the set calibration board based on the edge points includes: Determine a target point that is equidistant from all edge points and lies in the plane where the set calibration board is located; Determine the target point as the first center point.

6. The camera extrinsic parameter calibration method according to any one of claims 1 to 2, characterized in that, The determining the second feature obtained by the camera to be calibrated based on the set calibration board includes: According to the type of the set calibration board, determine the corresponding positioning algorithm; Based on the positioning algorithm and the image data collected by the camera to be calibrated, determine the second feature.

7. An external camera calibration device, characterized in that, Including: A first detection module, configured to determine the first feature of the set calibration board corresponding to the first sensor based on the point cloud data of the set calibration board detected by the first sensor. The set calibration board is a QR code calibration board, and the first feature includes a first normal vector and a first center point; A second detection module, configured to determine the second feature obtained by the camera to be calibrated based on the set calibration board. The second feature includes a second normal vector and a second center point; A processing module, configured to obtain the external parameter matrix corresponding to the camera to be calibrated based on the feature constraint equation, the first feature, and the second feature; The processing module is specifically configured to obtain a set number of the first features and the second features; Substitute the set number of the first features and the second features into the feature constraint equation, and determine the parameter matrix when the direction consistency, direction matching degree, and distance matching degree between the first sensor and the camera to be calibrated are optimized as the external parameter matrix; The feature constraint equation includes: ， ； Among them, e d is the direction consistency index, e r is the direction matching degree index, e t is the distance matching degree index, M is the set number, T(R,t) is the external parameter matrix, R is the rotation matrix in the external parameter matrix, t is the translation matrix in the external parameter matrix, o l is the coordinate of the first center point, n l is the first normal vector, o c is the coordinate of the second center point, n c is the second normal vector.

8. A control device, characterized in that, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the control device to execute the camera extrinsic parameter calibration method according to any one of claims 1 to 6.

Images