Vehicle multi-camera extrinsic parameter calibration method and device and storage medium

Abstract

This application relates to the field of multi-sensor fusion calibration technology, and provides a method, apparatus, and storage medium for extrinsic parameter calibration of multiple cameras in a vehicle. The method includes: acquiring multiple sets of data pairs, each set including image data of a calibration device acquired by a camera to be calibrated and point cloud data of the calibration device acquired by a LiDAR group, wherein the camera to be calibrated and the LiDAR group are adjacent, and the calibration device is located within the common visual area of ​​both the camera to be calibrated and the LiDAR group; extracting pixel coordinates of feature points corresponding to the calibration device from the image data, wherein the feature points include at least corner points of the calibration device and the center point of identification holes on the calibration device; extracting point cloud coordinates of the feature points corresponding to the calibration device from the point cloud data; and performing extrinsic parameter estimation based on the pixel coordinates and point cloud coordinates in each set of data pairs to obtain the same set of extrinsic parameters shared by all cameras to be calibrated. This achieves extrinsic parameter calibration for multiple cameras that do not share a common visual area.

Description

Technical Field

[0001] This application relates to the field of multi-sensor fusion calibration technology, and in particular to a method, apparatus and storage medium for calibrating the external parameters of multiple cameras in a vehicle. Background Technology

[0002] With the rapid development of autonomous driving and Advanced Driver Assistance Systems (ADAS) technologies, heavy-duty unmanned vehicles in ports are typically equipped with multiple cameras to achieve 360° all-around environmental perception. The accuracy of the fusion of this perception data directly determines the reliability of driving decisions. The extrinsic parameters of a multi-camera system are a core prerequisite for achieving multi-source image data fusion. Only by aligning the coordinate systems of all cameras to the same common coordinate system can the images acquired by different cameras be accurately aligned in spatial position, thereby enabling core functions such as obstacle detection, lane recognition, and path planning. In practical applications, the extrinsic parameter calibration of a multi-camera system requires a common field of view between the cameras; however, extrinsic parameter calibration cannot be performed between multiple cameras that do not share a common field of view. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus and storage medium for calibrating the extrinsic parameters of multiple cameras to be calibrated in a camera system that do not have a common field of view, thereby unifying the coordinate systems of all cameras to be calibrated to the same common coordinate system.

[0004] This application provides a method, apparatus, and storage medium for calibrating the extrinsic parameters of a vehicle's multi-camera system. The following description covers various aspects of this application, and the embodiments and beneficial effects described herein can be referenced interchangeably.

[0005] In a first aspect, this application provides a method for calibrating the extrinsic parameters of multiple cameras in a vehicle, applicable to a vehicle, wherein the vehicle is equipped with at least two cameras to be calibrated that do not share a common field of view, and at least one lidar group disposed between the two cameras to be calibrated, and the lidar group includes at least one lidar; the method includes:

[0006] Multiple sets of data pairs are acquired. Each set of data pairs includes image data of the calibration device collected by the camera to be calibrated and point cloud data of the calibration device collected by the LiDAR group. The camera to be calibrated and the LiDAR group are adjacent to each other, and the calibration device is located in the common visual area of ​​the camera to be calibrated and the LiDAR group.

[0007] Extract the pixel coordinates of feature points corresponding to the calibration device from the image data, wherein the feature points include at least the corner points of the calibration device and the center point of the identification hole on the calibration device;

[0008] Extract the point cloud coordinates of feature points corresponding to the calibration device from the point cloud data;

[0009] Extrinsic parameter estimation is performed based on the pixel coordinates and point cloud coordinates in each data pair to obtain the same set of extrinsic parameters shared by all cameras to be calibrated.

[0010] According to the embodiments of this application, the above-described technical solution of this application has at least the following beneficial effects:

[0011] By jointly acquiring multiple sets of image data and point cloud data of the calibration device within the common field of view using the camera to be calibrated and the LiDAR group, the pixel coordinates and point cloud coordinates of various feature points such as the corner points of the calibration device and the center point of the identification circular hole on the calibration device are extracted, and the same set of extrinsic parameters are shared by all the cameras to be calibrated, thus realizing the extrinsic parameter calibration of multiple cameras.

[0012] In one possible implementation of the first aspect above, extracting the point cloud coordinates of feature points corresponding to the calibration device from the point cloud data includes:

[0013] Filter the point cloud data to obtain valid point cloud data;

[0014] Geometric fitting is performed on the effective point cloud data to obtain the point cloud coordinates.

[0015] According to the embodiments of this application, the original point cloud data is filtered to remove invalid information, and the valid point cloud data corresponding to the calibration device is selected. Then, the valid point cloud data is geometrically fitted to obtain accurate point cloud coordinates.

[0016] In one possible implementation of the first aspect above, extrinsic parameters are estimated based on the pixel coordinates and point cloud coordinates in each data pair to obtain the same set of extrinsic parameters shared by all cameras to be calibrated, including:

[0017] The pixel coordinates and point cloud coordinates in each data pair are paired to determine the pixel coordinates of the common-view feature points, the point cloud coordinates of the common-view feature points, and the number of common-view feature points in each data pair. The common-view feature points are the feature points on the calibration device that can be observed simultaneously by the camera to be calibrated and the lidar group.

[0018] The extrinsic parameters are obtained based on the pixel coordinates of the common-view feature points, the point cloud coordinates of the common-view feature points, and the number of common-view feature points.

[0019] According to the implementation method of this application, by pairing the pixel coordinates and point cloud coordinates in each data pair, the common-view feature points and their number are identified, and extrinsic parameter estimation is performed, which effectively improves the stability and accuracy of extrinsic parameter solution and realizes the solution of unified extrinsic parameters for multiple cameras.

[0020] In one possible implementation of the first aspect above, extrinsic parameters are obtained based on the pixel coordinates of the common-view feature points, the point cloud coordinates of the common-view feature points, and the number of common-view feature points, including:

[0021] Based on the pixel coordinates of the common-view feature points, the preset actual coordinates of the common-view feature points, and the preset camera intrinsic parameters, the first pose corresponding to the camera to be calibrated is obtained.

[0022] Based on the point cloud coordinates of the common-view feature points and the actual coordinates of the common-view feature points, the second pose and point cloud registration error corresponding to the lidar group are obtained.

[0023] Based on the pixel coordinates of the common-view feature points, the actual coordinates of the common-view feature points, the camera intrinsic parameters, and the first pose, the reprojection error corresponding to the camera is obtained.

[0024] The extrinsic parameters are obtained based on the first pose, the second pose, the point cloud registration error, the reprojection error, and the number of common feature points.

[0025] According to the implementation method of this application, by solving the first pose of the camera to be calibrated and the second pose of the lidar group, and then combining the multi-dimensional information of the number of shared feature points, point cloud registration error and reprojection error to jointly estimate the extrinsic parameters, the extrinsic parameters can be accurately solved.

[0026] In one possible implementation of the first aspect above, extrinsic parameters are obtained based on the first pose, the second pose, the point cloud registration error, the reprojection error, and the number of common-view feature points, including:

[0027] Based on the first pose, the second pose, the number of common-view feature points, the point cloud registration error, and the reprojection error, a heterogeneous sensor pose map is constructed.

[0028] The extrinsic parameters are estimated based on the attitude diagram of the heterogeneous sensor.

[0029] According to the implementation of this application, by fusing the first pose of the camera to be calibrated, the second pose of the lidar group, and multi-dimensional constraint information such as the number of common-view feature points, point cloud registration error, and reprojection error to construct a heterogeneous sensor pose map, and then performing extrinsic parameter estimation based on this pose map, the pose and error information of multi-source, heterogeneous sensors can be uniformly modeled and globally optimized, thereby significantly improving the accuracy, stability, and consistency of extrinsic parameter solution.

[0030] In one possible implementation of the first aspect above, a heterogeneous sensor pose map is constructed based on the first pose, the second pose, the number of common-view feature points, the point cloud registration error, and the reprojection error, including:

[0031] Treat each first pose and second pose as a node;

[0032] If it is determined that two nodes share a common view relationship, an edge is set between the two nodes;

[0033] The heterogeneous sensor attitude map is obtained by using the number of common-view feature points, point cloud registration error, and reprojection error corresponding to the two nodes as the weights of the edges.

[0034] According to the implementation method of this application, the poses of the camera to be calibrated and the LiDAR group are used as nodes, and connecting edges are constructed based on the common-view relationship. The number of common-view feature points, point cloud registration error, and reprojection error are used as the weights of the edges to form a heterogeneous sensor pose graph. This graph can intuitively and quantitatively express the constraint relationship between the poses of each sensor, realize the global unified modeling of pose information of multi-source heterogeneous sensors, effectively improve the constraint rationality and optimization stability when estimating external parameters, and further improve the accuracy and reliability of unified external parameters of multiple cameras.

[0035] In one possible implementation of the first aspect above, extrinsic parameters are estimated based on the heterogeneous sensor attitude map to obtain the extrinsic parameters, including:

[0036] Based on the preset initial node and edge weights, the heterogeneous sensor attitude graph is traversed to obtain the minimum spanning tree;

[0037] Determine the relative poses of all nodes in the minimum spanning tree, and use the relative poses as initial extrinsic parameters;

[0038] Based on the preset objective function, the initial extrinsic parameters are optimized to obtain the extrinsic parameters.

[0039] According to the implementation method of this application, by traversing the attitude map of heterogeneous sensors to generate a minimum spanning tree, using its relative pose as the initial extrinsic parameter, and combining it with the objective function for optimization, stable and reliable initial extrinsic parameters can be obtained quickly, avoiding local optima and error accumulation. Furthermore, global optimization is used to further improve the accuracy, making the final extrinsic parameters more stable and more accurate.

[0040] Secondly, this application provides an external parameter calibration device for a vehicle multi-camera system, comprising:

[0041] The data acquisition module is used to acquire multiple sets of data pairs. Each set of data pairs includes image data of the calibration device acquired by the camera to be calibrated and point cloud data of the calibration device acquired by the lidar group. The camera to be calibrated and the lidar group are adjacent to each other, and the calibration device is located in the common visual area of ​​the camera to be calibrated and the lidar group.

[0042] The pixel coordinate extraction module is used to extract the pixel coordinates of feature points corresponding to the calibration device from the image data, wherein the feature points include at least the corner points of the calibration device and the center point of the identification circular hole on the calibration device;

[0043] The point cloud coordinate extraction module is used to extract the point cloud coordinates of feature points corresponding to the calibration device from the point cloud data;

[0044] The parameter estimation module is used to estimate the extrinsic parameters based on the pixel coordinates and point cloud coordinates in each data pair, so as to obtain the same set of extrinsic parameters shared by all cameras to be calibrated.

[0045] Thirdly, this application provides an electronic device including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the extrinsic parameter calibration method for a vehicle multi-camera as disclosed in the first aspect and any possible implementation thereof.

[0046] Fourthly, this application provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the extrinsic parameter calibration method for a vehicle multi-camera as disclosed in the first aspect and any possible implementation thereof.

[0047] Fifthly, this application provides a computer program product comprising: computer instructions that, when executed on an electronic device, cause the electronic device to perform the extrinsic parameter calibration method for a vehicle multi-camera as disclosed in the first aspect and any possible implementation thereof.

[0048] The beneficial effects of the second to fifth aspects can be found in the first aspect and the beneficial effects of any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0049] Figure 1 This is a diagram showing the sensor location distribution in the embodiments of this application;

[0050] Figure 2 This is a schematic diagram of the multi-camera extrinsic parameter calibration process in an embodiment of this application;

[0051] Figure 3 This is a flowchart illustrating the external parameter calibration process in the embodiments of this application.

[0052] Figure 4 This is a schematic diagram of the calibration device in the embodiments of this application;

[0053] Figure 5 This is a line drawing schematic diagram of the calibration device in the embodiments of this application;

[0054] Figure 6 This is a flowchart of the point cloud coordinate extraction process in the embodiments of this application;

[0055] Figure 7 This is a flowchart illustrating the calculation of common-view feature point related parameters in the embodiments of this application;

[0056] Figure 8 This is a detailed flowchart of parameter calculation in the embodiments of this application;

[0057] Figure 9 Flowchart for constructing the attitude map of heterogeneous sensors in the embodiments of this application;

[0058] Figure 10 A detailed flowchart of the construction of the heterogeneous sensor attitude diagram in the embodiments of this application;

[0059] Figure 11 This is a flowchart of the process for calculating extrinsic parameters by traversing the attitude map of heterogeneous sensors in the embodiments of this application;

[0060] Figure 12 This is a block diagram of the electronic device in the embodiments of this application;

[0061] Figure 13 This is a block diagram of a system-on-chip (SoC) in the embodiments of this application.

[0062] Figure label:

[0063] 1. Checkerboard pattern; 2. Identify round holes; 3. Corner points; 4. Center point. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0065] The technical problems to be solved by the embodiments of this application will be described below.

[0066] As described in the background technology above, in the existing port heavy-duty unmanned vehicle multi-camera extrinsic parameter calibration technology, the extrinsic parameter calibration of the multi-camera system requires that the cameras to be calibrated have a common field of view. However, it is impossible to perform extrinsic parameter calibration between cameras that do not have a common field of view.

[0067] Therefore, to address the aforementioned issues, this application provides a method, apparatus, and storage medium for calibrating the extrinsic parameters of multiple cameras in a vehicle. At least one lidar group is added between cameras to be calibrated that do not share a common field of view. Each lidar group contains at least one lidar. Data from the calibration device is jointly collected by the cameras to be calibrated and the lidar groups. Multiple feature points, including corner points and center points, are extracted and their coordinates are matched to obtain the same set of extrinsic parameters shared by all cameras to be calibrated. This unifies the coordinate systems of the multiple cameras, effectively reducing the accumulation of calibration errors, improving the accuracy and robustness of the extrinsic parameter calibration for multiple cameras, simplifying the perception data fusion process, and providing a reliable data foundation for autonomous driving environmental perception.

[0068] To better understand the extrinsic parameter calibration method for vehicle multi-cameras according to the embodiments of this application, the following description is provided in conjunction with the appendix to the specification. Figure 1 The sensor location distribution of the embodiments of this application will be described in detail.

[0069] refer to Figure 1 , Figure 1 A sensor location distribution diagram is shown in an embodiment of this application.

[0070] like Figure 1 As shown, currently, two cameras, camera 1 and camera 2, are mounted on the front of the heavy-duty large unmanned vehicles in the port. Camera 1 and camera 2 are located in the same position on the vehicle and are arranged vertically. Similarly, two cameras, camera 3 and camera 4, are also located in the same position at the rear of the vehicle, and they are also arranged vertically.

[0071] To better inform vehicle driving decisions, the extrinsic parameters of cameras 1, 2, 3, and 4 need to be standardized. Common camera extrinsic parameter calibration requires a shared field of view between cameras, using image data from a calibration device within that shared field of view. However, while cameras 1 and 2, and cameras 3 and 4 share a shared field of view, cameras 1 and 2 do not share a shared field of view with cameras 3 and 4. Therefore, relying solely on these cameras themselves is insufficient to calibrate the extrinsic parameters of all cameras to be calibrated.

[0072] Therefore, this application uses LiDAR 1 and LiDAR 2 installed at the front and rear of the vehicle respectively, and uses LiDAR 1 and LiDAR 2 as relay sensors between the cameras to be calibrated, so that all external parameters to be calibrated are consistent.

[0073] The following is in conjunction with the appendix Figure 2-11 The methods and apparatus in the embodiments of this application will be described in detail.

[0074] Reference Figures 2 to 5 , Figure 2 This is a schematic diagram of the multi-camera extrinsic parameter calibration process in an embodiment of this application. Figure 3This is a flowchart illustrating the external parameter calibration process in this application. Figure 4 This is a schematic diagram of the calibration device in the embodiments of this application. Figure 5 This is a line drawing schematic diagram of the calibration device in the embodiments of this application.

[0075] like Figure 2 As shown in the embodiment of this application, a method for calibrating the extrinsic parameters of a vehicle multi-camera system is applied to a vehicle. The vehicle is equipped with at least two cameras to be calibrated that do not have a common field of view, and at least one lidar group is disposed between the two cameras to be calibrated. The lidar group includes at least one lidar. The method includes steps S10-S40.

[0076] S10 involves acquiring multiple sets of data pairs.

[0077] like Figure 3 As shown, step S10 corresponds to step S1, which involves acquiring image data and point cloud data.

[0078] It should be noted that each data pair includes image data of the calibration device collected by the camera to be calibrated and point cloud data of the calibration device collected by the LiDAR group. The camera to be calibrated and the LiDAR group are located adjacent to each other, and the calibration device is located in the common visual area of ​​the camera to be calibrated and the LiDAR group.

[0079] For example, such as Figure 1 As shown, the image data acquired by camera 1 and the point cloud data acquired by lidar 1 form a data pair. Similarly, the image data and point cloud data acquired by other cameras and lidars with a common field of view also form a data pair.

[0080] like Figure 4 As shown, the calibration device includes multiple checkerboard textures 1 and multiple recognition holes 2. The checkerboard textures 1 and recognition holes 2 are used to quickly and stably extract the pixel coordinates and point cloud coordinates of feature points on the calibration device from image data and point cloud data. Furthermore, the checkerboard textures 1 can also be used for manual detection of whether the image data acquired by the current camera is clear.

[0081] S20: Extract the pixel coordinates of feature points corresponding to the calibration device from the image data.

[0082] like Figure 3 As shown, step S20 corresponds to step S2, which extracts the pixel coordinates of the feature points.

[0083] like Figure 5 As shown, by way of example, the aforementioned feature points may include at least the corner point 3 of the calibration device and the center point 4 of the identification hole on the calibration device.

[0084] In some embodiments, the pixel coordinates of the corner points on the calibration device are obtained from the image data by a corner detection algorithm, and the pixel coordinates of the center point of the identification hole are extracted by a circle detection or contour fitting algorithm.

[0085] S30, extract the point cloud coordinates of feature points corresponding to the calibration device from the point cloud data.

[0086] like Figure 3 As shown, step S30 above corresponds to step S3, which extracts the point cloud coordinates of feature points.

[0087] In some embodiments, point cloud coordinates of feature points corresponding to the calibration device are extracted from point cloud data through geometric fitting.

[0088] S40, based on the pixel coordinates and point cloud coordinates in each data pair, perform extrinsic parameter estimation to obtain the same set of extrinsic parameters shared by all cameras to be calibrated.

[0089] like Figure 3 As shown, step S40 corresponds to step S4. First, point-to-point registration is performed, then the spatial transformation relationship is calculated, and finally, evaluation and optimization are performed to output extrinsic parameters.

[0090] Understandably, the aforementioned extrinsic parameters can unify the coordinate systems of all cameras to the same common coordinate system. By acquiring multiple sets of image data and point cloud data from the calibration device collected by the camera and LiDAR within the shared field of view, the pixel coordinates and point cloud coordinates of feature points such as corner points and center points of the calibration device are extracted. Based on these coordinates, extrinsic parameter estimation is performed to obtain unified extrinsic parameters applicable to all cameras, thus unifying the coordinate systems of multiple cameras to the same common coordinate system.

[0091] The following provides a detailed explanation of steps S30-S40.

[0092] refer to Figure 6 , Figure 6 A flowchart illustrating the point cloud coordinate extraction process in an embodiment of this application is shown.

[0093] like Figure 6 As shown, in some embodiments, the point cloud coordinates of feature points corresponding to the calibration device are extracted from the point cloud data, including steps S310-S320.

[0094] S310 filters the point cloud data to obtain valid point cloud data.

[0095] In some embodiments, point cloud data is filtered based on reflection intensity and a preset reflection intensity threshold to obtain valid point cloud data.

[0096] It should be noted that reflection intensity reflects the target surface material's ability to reflect laser light. Surfaces of different materials, colors, and angles exhibit significant differences in reflection intensity. This characteristic can be utilized to eliminate noise, separate targets, or enhance specific features. The specific filtering process can be found in existing technologies and will not be elaborated upon here.

[0097] Understandably, by filtering point cloud data to remove noise points, redundant points, and non-target area points, effective point cloud data can be obtained. This can reduce the amount of point cloud data processing, improve the efficiency and accuracy of feature point extraction, avoid interference from noise and invalid points on feature recognition and external parameter estimation of the calibration device, and improve the accuracy of point cloud feature extraction and the stability of the calibration process.

[0098] S320 performs geometric fitting on the effective point cloud data to obtain the point cloud coordinates.

[0099] In some embodiments, the corner points can be obtained by plane fitting, and the center point of the identification hole can be obtained by circle fitting.

[0100] By filtering point cloud data to remove noise, redundancy, and invalid point clouds, pure and effective point cloud data is obtained. Then, geometric fitting is performed on it to obtain accurate point cloud coordinates. This can significantly reduce the impact of interference information on feature extraction, improve the accuracy and uniqueness of point cloud feature point coordinates, and provide stable and reliable three-dimensional geometric constraints for subsequent extrinsic parameter calculation, thereby improving the accuracy of camera and lidar extrinsic parameter calibration.

[0101] refer to Figure 7-11 , Figure 7 A flowchart illustrating the calculation of common-view feature point related parameters in an embodiment of this application is shown. Figure 8 A detailed parameter calculation flowchart is shown in the embodiments of this application. Figure 9 A flowchart illustrating the heterogeneous sensor attitude map construction process in an embodiment of this application is shown. Figure 10 This paper illustrates a detailed flowchart of the construction process for the heterogeneous sensor attitude map in an embodiment of this application. Figure 11 A flowchart illustrating the calculation of extrinsic parameters by traversing the attitude diagram of heterogeneous sensors in an embodiment of this application is shown.

[0102] like Figure 7 As shown in the embodiment of this application, extrinsic parameters are estimated based on the pixel coordinates and point cloud coordinates in each data pair to obtain uniform extrinsic parameters applicable to all cameras, including steps S410-S420.

[0103] In S410, the pixel coordinates and point cloud coordinates in each data pair are paired to determine the pixel coordinates of the common-view feature points, the point cloud coordinates of the common-view feature points, and the number of common-view feature points in each data pair.

[0104] It is understandable that common-view feature points are feature points on the calibration device that can be observed simultaneously by the camera to be calibrated and the lidar group.

[0105] In some embodiments, based on the preset geometric topology of the calibration device, a correlation between pixel coordinates and point cloud coordinates is established to determine whether the two types of coordinates correspond to the same physical feature point on the calibration device. Feature points that simultaneously satisfy the preset geometric topology are selected, and their corresponding pixel coordinates are the common-view feature point pixel coordinates, and their corresponding point cloud coordinates are the common-view feature point point cloud coordinates. The total number of common-view feature points obtained by the above selection in each data pair is counted, which is the number of common-view feature points in that data pair.

[0106] It should be noted that geometric topological relationships include the spatial distribution of feature points, relative distances, and numbering correspondence rules.

[0107] S420 derives extrinsic parameters based on the pixel coordinates of common-view feature points, the point cloud coordinates of common-view feature points, and the number of common-view feature points.

[0108] It is understandable that by matching the pixel coordinates and point cloud coordinates in each data pair, the pixel coordinates, point cloud coordinates, and number of common-view feature points can be determined. Combined with the actual coordinates of the feature points, extrinsic parameter estimation can be carried out, which can effectively improve the stability and accuracy of extrinsic parameter solution and realize the solution of unified extrinsic parameters for multiple cameras.

[0109] like Figure 8 As shown, in some embodiments, extrinsic parameters are obtained based on the pixel coordinates of the common-view feature points, the point cloud coordinates of the common-view feature points, and the number of common-view feature points, including steps S4210-S4240.

[0110] S4210 obtains the pose (i.e. the first pose mentioned above) corresponding to the camera to be calibrated based on the pixel coordinates of the common-view feature points, the preset actual coordinates of the common-view feature points, and the preset camera intrinsic parameters.

[0111] In some embodiments, the pose corresponding to the camera to be calibrated is obtained by using the perspective-n-point (PnP) algorithm based on the pixel coordinates of the common-view feature points, the actual coordinates of the common-view feature points, and the intrinsic parameters of the camera.

[0112] S4220, based on the point cloud coordinates of the common-view feature points and the actual coordinates of the common-view feature points, obtains the pose (i.e. the second pose mentioned above) and point cloud registration error corresponding to the lidar group.

[0113] In some embodiments, the pose corresponding to the lidar is obtained by geometric transformation based on the point cloud coordinates of the common-view feature points and the actual coordinates of the common-view feature points.

[0114] It should be noted that the point cloud registration error is the Euclidean distance between the point cloud coordinates of the common-view feature points and the actual coordinates of the common-view feature points.

[0115] S4230, based on the pixel coordinates of the common-view feature points, the actual coordinates of the common-view feature points, the camera intrinsic parameters, and the pose corresponding to the camera to be calibrated, obtains the reprojection error corresponding to the camera to be calibrated.

[0116] It should be noted that the above steps can refer to existing technologies, and will not be repeated in this article.

[0117] S4240 obtains extrinsic parameters based on the pose corresponding to the camera to be calibrated, the pose corresponding to the lidar group, the point cloud registration error, the reprojection error, and the number of common-view feature points.

[0118] It should be noted that the beneficial effects of the embodiments of this application can be referred to above, and will not be repeated here.

[0119] like Figure 9 As shown, in some embodiments, extrinsic parameters are obtained based on the pose corresponding to the camera, the pose corresponding to the lidar, the point cloud registration error, the reprojection error, and the number of common-view feature points, including steps S42410-S42420.

[0120] Among them, S42410 constructs a heterogeneous sensor attitude map based on the pose corresponding to the camera to be calibrated, the pose corresponding to the lidar group, the point cloud registration error, the reprojection error, and the number of common-view feature points.

[0121] like Figure 10 As shown, in some embodiments, a heterogeneous sensor pose map is constructed based on the pose corresponding to the camera, the pose corresponding to the lidar, the point cloud registration error, the reprojection error, and the number of common-view feature points, including steps S42411-S42413.

[0122] S42411 uses each pose corresponding to the camera to be calibrated and the pose corresponding to the lidar group as a node.

[0123] S42412, where an edge is set between two nodes if it is determined that there is a common view relationship between them.

[0124] It should be noted that when the corresponding camera and lidar group of two nodes have a common field of view, it can be confirmed that the two nodes have a common field of view relationship.

[0125] S42413 uses the number of common-view feature points, point cloud registration error, and reprojection error corresponding to the two nodes as the weights of the edges to obtain the heterogeneous sensor attitude map.

[0126] Understandably, the poses of the camera and the LiDAR array to be calibrated are defined as nodes, and connecting edges are constructed based on their shared-view relationships. Simultaneously, the number of shared-view feature points, point cloud registration error, and reprojection error are used as weight parameters for the edges. This method constructs a heterogeneous sensor pose graph, which can present the constraint relationships between the poses of each sensor in an intuitive and quantitative form. Based on this, global unified modeling of pose information from multiple heterogeneous sensors can be achieved, effectively enhancing the rationality of constraints during extrinsic parameter estimation, improving the stability of the optimization process, and further improving the accuracy and reliability of unified extrinsic parameters for multiple cameras.

[0127] S42420 performs extrinsic parameter estimation based on the attitude diagram of heterogeneous sensors to obtain the extrinsic parameters.

[0128] Understandably, by using the poses of the camera and lidar group to be calibrated as nodes, constructing connecting edges based on common-view relationships, and using the number of common-view feature points, point cloud registration error, and reprojection error as edge weights, a heterogeneous sensor pose graph can be constructed. This can intuitively and quantitatively represent the constraint relationships between the poses of each sensor, realize global unified modeling of pose information of multi-source heterogeneous sensors, improve the rationality of constraints and optimization stability in the external parameter estimation process, and thus improve the accuracy and reliability of unified external parameters of multiple cameras.

[0129] like Figure 11 As shown, in some embodiments, extrinsic parameters are estimated based on the heterogeneous sensor attitude diagram to obtain extrinsic parameters, including steps S42421-S42423.

[0130] In S42421, based on the preset initial nodes and edge weights, the heterogeneous sensor attitude graph is traversed to obtain the minimum spanning tree.

[0131] For example, the preset initial node can be as follows: Figure 1 The node corresponding to camera 1 shown is used as the initial node. Nodes corresponding to other cameras can also be used as initial nodes; this paper does not impose restrictions on the setting of the initial node.

[0132] In some embodiments, the attitude maps of heterogeneous sensors are traversed using a breadth-first traversal algorithm.

[0133] It should be noted that, compared to the traditional breadth-first search algorithm, the breadth-first search algorithm in this embodiment first selects edges based on the number of common view feature points when traversing the heterogeneous sensor attitude map. However, during the traversal, when multiple edges have the same number of common view feature points, the edge with the smaller point cloud registration error is selected. Similarly, when multiple edges have equal point cloud registration errors, the edge with the smaller reprojection error is selected. Besides the breadth-first search algorithm, the depth-first search algorithm can also be used.

[0134] S42422 determines the relative poses of all nodes in the minimum spanning tree and uses the relative poses as initial extrinsic parameters.

[0135] It should be noted that the above steps can refer to existing technologies, and will not be repeated in this article.

[0136] S42423 optimizes the initial extrinsic parameters based on a preset objective function to obtain the extrinsic parameters.

[0137] In some embodiments, the objective function E can be expressed as:

[0138]

[0139] Where k is the number of feature points on the calibration device, T is the initial extrinsic parameter, p(*) is the robust kernel function (e.g., Huber kernel or Cauchy kernel) used to reduce the impact of mismatches or noisy data on the optimization results and enhance the robustness of the calibration process, a(*) is the camera projection model function that projects the actual coordinates of the feature points onto the image plane, X k Let x be the actual coordinates of the k-th feature point on the calibration device. k Y represents the pixel coordinates of the k-th feature point on the calibration device, b is a weighting coefficient used to balance the contributions of visual reprojection error and point cloud registration error to the overall objective function, ensuring they are numerically equivalent and preventing one from dominating the optimization process. i Z represents the actual coordinates of the i-th feature point on the calibration device. i Let be the point cloud coordinates of the i-th feature point on the calibration device.

[0140] It should be noted that when optimizing the initial extrinsic parameters based on the objective function, the objective function needs to be minimized using a nonlinear least squares algorithm to obtain the aforementioned extrinsic parameters.

[0141] Secondly, this application provides an external parameter calibration device for a vehicle multi-camera system, comprising:

[0142] The data acquisition module is used to acquire multiple sets of data pairs. Each set of data pairs includes image data of the calibration device acquired by the camera to be calibrated and point cloud data of the calibration device acquired by the lidar group. The camera to be calibrated and the lidar group are adjacent to each other, and the calibration device is located in the common visual area of ​​the camera to be calibrated and the lidar group.

[0143] The pixel coordinate extraction module is used to extract the pixel coordinates of feature points corresponding to the calibration device from the image data, wherein the feature points include at least the corner points of the calibration device and the center point of the identification circular hole on the calibration device;

[0144] The point cloud coordinate extraction module is used to extract the point cloud coordinates of feature points corresponding to the calibration device from the point cloud data;

[0145] The parameter estimation module is used to estimate the extrinsic parameters based on the pixel coordinates and point cloud coordinates in each data pair, so as to obtain the same set of extrinsic parameters shared by all cameras to be calibrated.

[0146] In the apparatus of this application embodiment, each module executes the method of the above embodiment, and its specific functions and corresponding technical effects can be referred to the above embodiment. Figures 1-11 The methods explained will not be elaborated here.

[0147] This application provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program. When the processor loads and executes the instruction or program, the electronic device performs the extrinsic parameter calibration method for a multi-camera system described in the above embodiments. Its specific functions and corresponding technical effects can be found in the above embodiments. Figures 1-11 The method for calibrating the extrinsic parameters of a vehicle's multi-camera system, as explained above, will not be repeated here. The following section will combine... Figure 12 The electronic devices described in the embodiments of this application will be described in detail.

[0148] refer to Figure 12 The diagram shows a block diagram of an electronic device 1200 according to one embodiment of this application. The electronic device 1200 may include one or more processors 1201 coupled to a controller hub 1203. In at least one embodiment, the controller hub 1203 communicates with the processor 1201 via a multi-branch bus such as a front side bus (FSB) 1210, a point-to-point interface such as a quick path interconnect (QPI), or a similar connection. The processor 1201 executes instructions controlling general types of data processing operations. In one embodiment, the controller hub 1203 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0149] Electronic device 1200 may also include a coprocessor 1202 and a memory 1204 coupled to a controller hub 1203. Alternatively, one or both of the memory and the GMCH may be integrated within the processor (as described in this application), with memory 1204 and coprocessor 1202 directly coupled to processor 1201 and controller hub 1203, which resides on a single chip with the IOH. Memory 1204 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of both. In one embodiment, coprocessor 1202 is a dedicated processor, such as, for example, a high-throughput MIC (many integerized core) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on GPU (GPGPU), or an embedded processor, etc. Optional properties of coprocessor 1202 are indicated by dashed lines. Figure 12 middle.

[0150] As a computer-readable storage medium, memory 1204 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, memory 1204 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device such as one or more hard-disk drives (HDDs), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0151] In one embodiment, electronic device 1200 may further include a network interface controller (NIC) 1206. Network interface 1206 may include a transceiver for providing a radio interface for electronic device 1200 to communicate with any other suitable device, such as a front-end module, antenna, etc. In various embodiments, network interface 1206 may be integrated with other components of electronic device 1200. Network interface 1206 can implement the functions of the communication unit in the above embodiments.

[0152] Electronic device 1200 may further include input / output (I / O) device 1205. I / O device 1205 may include: a user interface designed to enable a user to interact with electronic device 1200; a peripheral component interface designed to enable peripheral components to also interact with electronic device 1200; and / or sensors designed to determine environmental conditions and / or location information related to electronic device 1200.

[0153] It is worth noting that, Figure 12 This is merely an example. That is, although... Figure 12 The electronic device 1200 shown includes multiple devices such as a processor 1201, a coprocessor 1202, a controller hub 1203, and a memory 1204. However, in practical applications, devices using the methods of this application may include only a portion of the devices in the electronic device 1200. For example, it may include only the processor 1201 and the network interface 1206. Figure 12 The properties of the optional devices are shown in dashed lines. According to some embodiments of this application, the memory 1204, which is a computer-readable storage medium, stores instructions or programs that, when executed on a computer, perform the vehicle multi-camera extrinsic parameter calibration method described in the above embodiments. Specific details can be found in the methods described in the above embodiments, and will not be repeated here.

[0154] Now for reference Figure 13 The diagram shown is a block diagram of a system-on-chip (SoC) 1300 according to an embodiment of this application. Figure 13 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 13 In this SoC 1300, the following are included: an interconnect unit 1350 coupled to an application processor 1310; a system proxy unit 1380; a bus controller unit 1390; an integrated memory controller unit 1340; a group or one or more coprocessors 1320, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1330; and a direct memory access (DMA) unit 1360. In one embodiment, the coprocessor 1320 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0155] The static random access memory (SRAM) cell 1330 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 1300 to perform the extrinsic parameter calibration method for a vehicle multi-camera according to the above embodiments, as detailed in the methods described above, which will not be repeated here.

[0156] This application provides a computer-readable storage medium storing at least one instruction or at least one program. The instruction or program is loaded and executed by a processor to implement the vehicle multi-camera extrinsic parameter calibration method described in the above embodiments. Its specific functions and corresponding technical effects can be found in the above embodiments. Figures 1-11 The method for calibrating the extrinsic parameters of the vehicle's multi-camera system, as explained above, will not be repeated here.

[0157] This application provides a computer program product, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device causes the electronic device to implement the extrinsic parameter calibration method for a multi-camera system described in the above embodiments. Its specific functions and corresponding technical effects can be found in the above embodiments. Figures 1-11 The method for calibrating the extrinsic parameters of the vehicle's multi-camera system, as explained above, will not be repeated here.

[0158] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0159] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0160] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0161] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0162] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0163] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0164] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0165] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0166] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A method for calibrating the extrinsic parameters of a vehicle multi-camera system, characterized in that, The method is applied to the vehicle, which is equipped with at least two cameras to be calibrated that do not share a common field of view, and at least one lidar group disposed between the two cameras to be calibrated, wherein the lidar group includes at least one lidar. Multiple sets of data pairs are acquired. Each set of data pairs includes image data of the calibration device acquired by the camera to be calibrated and point cloud data of the calibration device acquired by the lidar group. The camera to be calibrated and the lidar group are adjacent to each other, and the calibration device is located in the common visual area of ​​the camera to be calibrated and the lidar group. The pixel coordinates of feature points corresponding to the calibration device are extracted from the image data, wherein the feature points include at least the corner points of the calibration device and the center point of the identification hole on the calibration device; Extract the point cloud coordinates of the feature points corresponding to the calibration device from the point cloud data; Based on the pixel coordinates and point cloud coordinates in each set of data pairs, extrinsic parameter estimation is performed to obtain the same set of extrinsic parameters shared by all the cameras to be calibrated.

2. The method according to claim 1, characterized in that, Extracting the point cloud coordinates of feature points corresponding to the calibration device from the point cloud data includes: The point cloud data is filtered to obtain valid point cloud data; Geometric fitting is performed on the effective point cloud data to obtain the point cloud coordinates.

3. The method according to claim 1, characterized in that, The step of estimating extrinsic parameters based on the pixel coordinates and point cloud coordinates in each set of data pairs yields the same set of extrinsic parameters shared by all the cameras to be calibrated, including: The pixel coordinates and point cloud coordinates in each set of data pairs are paired to determine the pixel coordinates of common-view feature points, the point cloud coordinates of common-view feature points, and the number of common-view feature points in each set of data pairs. The common-view feature points are the feature points on the calibration device that can be observed simultaneously by the camera to be calibrated and the lidar group. The extrinsic parameters are obtained based on the pixel coordinates of the common-view feature points, the point cloud coordinates of the common-view feature points, and the number of common-view feature points.

4. The method according to claim 3, characterized in that, The extrinsic parameters are obtained based on the pixel coordinates of the shared feature points, the point cloud coordinates of the shared feature points, and the number of shared feature points, including: Based on the pixel coordinates of the common-view feature points, the preset actual coordinates of the common-view feature points, and the preset camera intrinsic parameters, the first pose corresponding to the camera to be calibrated is obtained. Based on the point cloud coordinates of the common-view feature points and the actual coordinates of the common-view feature points, the second pose and point cloud registration error corresponding to the lidar group are obtained. Based on the pixel coordinates of the common-view feature points, the actual coordinates of the common-view feature points, the camera intrinsic parameters, and the first pose, the reprojection error corresponding to the camera is obtained. The extrinsic parameters are obtained based on the first pose, the second pose, the point cloud registration error, the reprojection error, and the number of common feature points.

5. The method according to claim 4, characterized in that, The extrinsic parameters are obtained based on the first pose, the second pose, the point cloud registration error, the reprojection error, and the number of common feature points, including: Based on the first pose, the second pose, the number of co-view feature points, the point cloud registration error, and the reprojection error, a heterogeneous sensor pose map is constructed. The extrinsic parameters are obtained by estimating the extrinsic parameters based on the attitude diagram of the heterogeneous sensor.

6. The method according to claim 5, characterized in that, The construction of a heterogeneous sensor pose map based on the first pose, the second pose, the number of common-view feature points, the point cloud registration error, and the reprojection error includes: Each of the first pose and the second pose is treated as a node; If it is determined that there is a shared view relationship between two nodes, an edge is set between the two nodes; The heterogeneous sensor attitude map is obtained by using the number of common-view feature points corresponding to the two nodes, the point cloud registration error, and the reprojection error as the weights of the edges.

7. The method according to claim 5, characterized in that, The extrinsic parameter estimation based on the heterogeneous sensor attitude map, to obtain the extrinsic parameters, includes: Based on the preset initial node and edge weights, the heterogeneous sensor attitude graph is traversed to obtain the minimum spanning tree; Determine the relative poses of all nodes in the minimum spanning tree, and use the relative poses as initial extrinsic parameters; Based on a preset objective function, the initial extrinsic parameters are optimized to obtain the extrinsic parameters.

8. A device for calibrating the external parameters of a vehicle multi-camera system, characterized in that, include: The data acquisition module is used to acquire multiple sets of data pairs. Each set of data pairs includes image data of the calibration device acquired by the camera to be calibrated and point cloud data of the calibration device acquired by the lidar group. The camera to be calibrated and the lidar group are adjacent to each other, and the calibration device is located in the common visual area of ​​the camera to be calibrated and the lidar group. A pixel coordinate extraction module is used to extract the pixel coordinates of feature points corresponding to the calibration device from the image data, wherein the feature points include at least the corner points of the calibration device and the center point of the identification circular hole on the calibration device; The point cloud coordinate extraction module is used to extract the point cloud coordinates of feature points corresponding to the calibration device from the point cloud data. The parameter estimation module is used to perform extrinsic parameter estimation based on the pixel coordinates and point cloud coordinates in each set of data pairs, so as to obtain the same set of extrinsic parameters shared by all the cameras to be calibrated.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the extrinsic parameter calibration method for a vehicle multi-camera as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, include: Computer instructions, when executed on an electronic device, cause the electronic device to perform the extrinsic parameter calibration method for a vehicle multi-camera according to any one of claims 1 to 7.

Images