Multi-camera external parameter simultaneous calibration method and system based on ground point cloud

Abstract

The invention provides a multi-camera external parameter simultaneous calibration method and system based on ground point clouds, and belongs to the field of robot vision and sensor calibration, and the method comprises the steps: enabling a plurality of depth cameras to be fixedly disposed on a robot, enabling the robot to be located at any position, carrying out the parameter initialization, and obtaining the point cloud data collected by the plurality of cameras; performing ground segmentation on the point cloud data to obtain ground point cloud; synchronously adjusting the Euler angle parameter of each camera based on the normal vector feature of the ground point cloud to obtain the Euler angle parameter of the adjusted camera; performing synchronous adjustment on the height parameter of each camera based on the centroid feature of the ground point cloud to obtain the height parameter of the adjusted camera; and the calibration program of the robot simultaneously updates the external parameter data of the plurality of cameras based on the adjusted Euler angle parameters of the cameras and the adjusted height parameters of the cameras to obtain the calibrated parameters of the cameras. According to the invention, external parameter calibration is carried out on multiple cameras at the same time, camera calibration time is shortened, and calibration accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot vision and sensor calibration technology, and in particular to a method and system for simultaneous calibration of multi-camera extrinsic parameters based on ground point clouds. Background Art

[0002] Camera extrinsic calibration is a crucial step in computer vision and robotics, directly impacting the registration accuracy of each camera in a multi-camera system and the resulting perception quality. Traditional extrinsic calibration methods typically require manual adjustment of camera position and posture, or rely on complex calibration plates, extracting their feature points through image processing algorithms, and then calculating the relative positional relationships between cameras. While these methods can provide a certain degree of accuracy, they have many limitations.

[0003] Conventional calibration methods are often time-consuming, especially in multi-camera systems. These methods require high operator expertise and are susceptible to changes in ambient lighting and device placement, leading to unstable calibration results. Furthermore, traditional calibration methods become particularly cumbersome and inefficient when frequent camera repositioning or configuration updates are required.

[0004] Therefore, finding a method that can not only quickly calibrate a camera but also improve the accuracy of camera calibration is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] The present invention provides a method and system for simultaneous calibration of multi-camera extrinsic parameters based on ground point clouds, which is used to solve the defects of the existing technology such as long calibration time, complex operation and inadaptability to environmental changes. It realizes the automatic calibration of ground point cloud data using multiple cameras, and then quickly and accurately completes the synchronous calibration of camera extrinsic parameters.

[0006] The present invention provides a method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud, comprising the following steps:

[0007] S1. Fix multiple depth cameras on a robot, position the robot at any position, initialize parameters, obtain configuration parameters of each camera from the robot's parameter server, and obtain point cloud data collected by the multiple cameras;

[0008] S2. performing ground segmentation on the point cloud data collected by each of the multiple cameras to obtain a ground point cloud corresponding to each camera;

[0009] S3. Synchronously adjust the Euler angle parameters of each camera based on the normal vector features of the ground point cloud to obtain the adjusted Euler angle parameters of the camera;

[0010] S4. Synchronously adjust the height parameters of each camera based on the centroid feature of the ground point cloud to obtain the adjusted height parameters of the camera;

[0011] S5. The robot calibration program simultaneously updates the external parameter data of the multiple cameras based on the adjusted Euler angle parameters of the cameras and the adjusted height parameters of the cameras to obtain calibrated parameters of the cameras.

[0012] According to a method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud provided by the present invention, step S2 specifically includes:

[0013] S21, taking the point cloud data collected by each camera as a set of initial point cloud data, and constructing a KD tree based on each set of point cloud data;

[0014] S22, performing a k-nearest neighbor search for each point in each set of point cloud data based on the KD tree to determine a neighborhood point set for each point in each set of point cloud data;

[0015] S23, calculating the covariance matrix of the neighborhood point set of each point, and performing eigenvalue decomposition on the covariance matrix to obtain the normal vector of each point;

[0016] S24. Use the RANSAC algorithm to iteratively calculate the plane model parameters based on the three-dimensional coordinates of each point in each set of point cloud data and the normal vector of each point, mark the points in each set of point cloud data that conform to the plane model as inliers, and form a ground point cloud corresponding to each camera based on the inliers in each set of point cloud data.

[0017] According to a method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud provided by the present invention, step S24 specifically includes:

[0018] Step 1: Select any point p in each set of point cloud data i ;

[0019] Step 2: Calculate point p i The distance to the plane is calculated as follows:

[0020]

[0021] Among them, dis represents the distance from the point to the plane, n represents the normal vector of the plane, and p represents the point p i The coordinate vector, n T represents the transpose of the normal vector n, d represents the offset of the plane on the coordinate axis, ‖·‖ represents the norm of the vector, and |·| represents the absolute value operation;

[0022] Step 3: Point p i The distance to the plane is compared with the preset distance threshold. If point p i If the distance to the plane is less than the preset distance threshold, point pi Marked as inliers;

[0023] Step 4: Repeat steps 1 to 3 until all points in each set of point cloud data are traversed to obtain an inlier set, and form a ground point cloud corresponding to the camera based on the inlier set.

[0024] According to a method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud provided by the present invention, step S3 specifically includes:

[0025] The point cloud ground normal vector calculated from the camera content coordinate system based on the ground point cloud And calculate the transformation matrix from the camera content coordinate system to the base coordinate system

[0026] Using the transformation matrix The ground normal vector in the base coordinate system Convert to the camera content coordinate system to get the ground normal vector in the camera content coordinate system

[0027] According to the point cloud ground normal vector in the camera content coordinate system Construct the rotation matrix of the camera content coordinate system and the rotation matrix of the base coordinate system;

[0028] Calculate the rotation matrix difference between the rotation matrix of the camera content coordinate system and the rotation matrix of the base coordinate system to form a difference matrix, and extract the Euler angle of the camera based on the difference matrix;

[0029] The Euler angles from the calibration base coordinate system to the camera coordinate system are calibrated according to the rotation matrix difference to obtain the Euler angle parameters of the adjusted camera.

[0030] According to a method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud provided by the present invention, the calculation formula of the Euler angle is:

[0031]

[0032] ry=arctan(-R diff (2,0));

[0033]

[0034] Among them, rx represents the rotation angle of the difference matrix around the x-axis, ry represents the rotation angle of the difference matrix around the y-axis, rz represents the rotation angle of the difference matrix around the z-axis, R diff represents the difference matrix, R diff (2,1) represents the element in the 3rd row and 2nd column of the difference matrix, R diff (2,2) represents the element in the 3rd row and 3rd column of the difference matrix, R diff(2,0) represents the element in the 3rd row and 1st column of the difference matrix, R diff (1,0) represents the element in the 2nd row and 1st column of the difference matrix, R diff (0,0) represents the element in the first row and first column of the difference matrix. Represented by the transpose of the rotation matrix of the camera content coordinate system, R base_link Represents the rotation matrix of the base coordinate system.

[0035] According to a method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud provided by the present invention, step S4 specifically includes:

[0036] Calculate the average coordinates of all points in the ground point cloud in the camera content coordinate system to obtain the centroid of the ground point cloud;

[0037] Converting the ground point cloud centroid from the camera content coordinate system to the robot base coordinate system to obtain the ground point cloud centroid in the base coordinate system;

[0038] Calculate the difference between the height value of the centroid of the ground point cloud in the base coordinate system and the preset reference height value in the base coordinate system to obtain the height adjustment offset;

[0039] The height parameter of the camera is adjusted according to the height adjustment offset to obtain the adjusted height parameter of the camera.

[0040] According to a method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud provided by the present invention, the centroid calculation formula of the ground point cloud in the base coordinate system is:

[0041]

[0042] in, Represents the centroid coordinates of the ground point cloud in the base coordinate system, represents the transformation matrix from the camera coordinate system to the base coordinate system, represents the transformation matrix from the camera content coordinate system to the camera coordinate system, Represents the centroid coordinates of the ground point cloud in the camera content coordinate system.

[0043] The present invention also provides a multi-camera extrinsic parameter simultaneous calibration system based on ground point cloud, which adopts the multi-camera extrinsic parameter simultaneous calibration method as described above, comprising:

[0044] a data acquisition module configured to fix multiple depth cameras on a robot, the robot being positioned at an arbitrary position and initializing parameters, obtaining parameters of each camera from a parameter server of the robot, and obtaining point cloud data collected by the multiple cameras;

[0045] a point cloud segmentation module configured to perform ground segmentation on the point cloud data collected by each of the multiple cameras to obtain a ground point cloud corresponding to each camera;

[0046] A first parameter adjustment module is configured to synchronously adjust the Euler angle parameters of each camera based on the normal vector characteristics of the ground point cloud to obtain the adjusted Euler angle parameters of the camera;

[0047] A second parameter adjustment module is configured to synchronously adjust the height parameter of each camera based on the centroid feature of the ground point cloud to obtain the adjusted height parameter of the camera;

[0048] The calibration module is configured as a calibration program of the robot to update the external parameter data of the multiple cameras based on the Euler angle parameters of the adjusted cameras and the height parameters of the adjusted cameras to obtain the calibrated parameters of the cameras.

[0049] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for simultaneous calibration of multi-camera extrinsic parameters as described above is implemented.

[0050] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for simultaneous calibration of multi-camera extrinsic parameters.

[0051] The present invention provides a method and system for simultaneous calibration of multi-camera extrinsic parameters based on ground point clouds. Through point cloud segmentation and normal vector estimation, the ground point cloud information of multiple cameras is extracted, and the Euler angle parameters of each camera are calculated and adjusted to correct the camera's posture error. By calculating the height difference of the ground point cloud centroid, the camera's height parameters are further adjusted. The extrinsic parameter calibration of multiple cameras is performed simultaneously, which significantly shortens the calibration time, improves the calibration accuracy, and reduces the dependence on the operator's skills, making the calibration work in different environments and configuration conditions simpler and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 This is a flow chart of the method for simultaneous calibration of multi-camera extrinsic parameters provided by the present invention;

[0054] Figure 2Schematic diagram of the Euler angle calibration process for the multi-camera extrinsic parameter simultaneous calibration method provided by the present invention; (a) is a schematic diagram of the initial ground point cloud, (b) is a schematic diagram of the ground point cloud segmentation processing, (c) is a schematic diagram of the comparison of the normal vectors of the ground point cloud and the base coordinate system in the camera coordinate system, and (d) is a schematic diagram of the Euler angle calibration result;

[0055] Figure 3 Schematic diagram of the height calibration process of the multi-camera extrinsic parameter simultaneous calibration method provided by the present invention; (a) is a schematic diagram of the ground point cloud after Euler angle calibration, (b) is the centroid extraction of the ground point cloud, and (c) is the height calibration result;

[0056] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0058] As those skilled in the art can understand, the method of the present invention relies on a robot platform, a main control computer and a depth camera, wherein the robot platform serves as the hardware basis to provide a physical installation location for the depth camera, the main control computer communicates with the robot platform, receives and processes the data collected by the depth camera, and runs the calibration algorithm through the calibration program to obtain accurate camera calibration results.

[0059] like Figure 1 The present invention provides a method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud, comprising the following steps:

[0060] S1. Fix multiple depth cameras on a robot, which is located at an arbitrary position and performs parameter initialization. Obtain configuration parameters of each camera from the robot's parameter server and obtain point cloud data collected by the multiple cameras.

[0061] Among them, the camera parameters include the camera's point cloud topic name, camera coordinate system, camera content coordinate system, etc.

[0062] In one embodiment of the present invention, the robot is provided with N depth cameras, where N is a non-zero natural number greater than 1, and preferably N is set to 4. The calibration method of the present invention can be used with different camera models and numbers of cameras. If a robot needs to calibrate multiple cameras, only the corresponding parameter files need to be modified.

[0063] Specifically, the depth camera and the robot's base coordinate system have the following relationship:

[0064]

[0065] in, Represents the conversion matrix from the camera coordinate system (camera_link) to the base coordinate system (base_link), R represents the rotation matrix, t represents the translation vector, Represents the transformation matrix from the camera content coordinate system (camera_optical_frame) to the camera coordinate system (camera_link), R z (-90°) represents the rotation matrix of -90° counterclockwise rotation around the z axis, R y (90°) represents the rotation matrix of 90° counterclockwise rotation around the y-axis, P base_link Represents any point in the ground point cloud under the base coordinate system (base_link), P camera_optical_frame Represents any point in the ground point cloud in the camera content coordinate system (camera_optical_frame).

[0066] S2. Perform ground segmentation on the point cloud data collected by each of the multiple cameras to obtain a ground point cloud corresponding to each camera.

[0067] Specifically, step S2 includes:

[0068] S21, taking the point cloud data collected by each camera as a set of initial point cloud data, and constructing a KD tree based on each set of point cloud data;

[0069] S22, performing a k-nearest neighbor search for each point in each set of point cloud data based on the KD tree to determine a neighborhood point set for each point in each set of point cloud data;

[0070] S23, calculating the covariance matrix of the neighborhood point set of each point, and performing eigenvalue decomposition on the covariance matrix to obtain the normal vector of each point;

[0071] S24. Use the RANSAC algorithm to iteratively calculate the plane model parameters based on the three-dimensional coordinates of each point in each set of point cloud data and the normal vector of each point, mark the points in each set of point cloud data that conform to the plane model as inliers, and form a ground point cloud corresponding to each camera based on the inliers in each set of point cloud data.

[0072] Furthermore, step S24 specifically includes:

[0073] Step 1: Select any point p in each set of point cloud data i ;

[0074] Step 2: Calculate point p iThe distance to the plane is calculated as follows:

[0075]

[0076] Among them, dis represents the distance from the point to the plane, n represents the normal vector of the plane, and p represents the point p i The coordinate vector, n T represents the transpose of the normal vector n, d represents the offset of the plane on the coordinate axis, ‖·‖ represents the norm of the vector, and |·| represents the absolute value operation;

[0077] Step 3: Point p i The distance to the plane is compared with the preset distance threshold. If point p i If the distance to the plane is less than the preset distance threshold, point p i Marked as inliers;

[0078] Step 4: Repeat steps 1 to 3 until all points in each set of point cloud data are traversed to obtain an inlier set, and form a ground point cloud corresponding to the camera based on the inlier set.

[0079] It is understandable that the preset distance threshold can be set according to actual usage requirements, and the present invention does not impose any specific limitation on this.

[0080] like Figure 2 As shown in the figure, camera_link is the camera coordinate system, base_link is the base coordinate system, and arm_base_link is the robot coordinate system. S3. Synchronously adjust the Euler angle parameters of each camera based on the normal vector features of the ground point cloud to obtain the adjusted Euler angle parameters of the camera.

[0081] Specifically, step S3 includes:

[0082] The point cloud ground normal vector calculated from the ground point cloud camera content coordinate system And calculate the transformation matrix from the camera content coordinate system to the base coordinate system

[0083] Using the transformation matrix The ground normal vector in the base coordinate system Convert to the camera content coordinate system to get the ground normal vector in the camera content coordinate system

[0084] According to the point cloud ground normal vector in the camera content coordinate system Construct the rotation matrix of the camera content coordinate system and the rotation matrix of the base coordinate system;

[0085] Calculate the rotation matrix difference between the rotation matrix of the camera content coordinate system and the rotation matrix of the base coordinate system to form a difference matrix, and extract the Euler angle of the camera based on the difference matrix;

[0086] The Euler angles from the calibration base coordinate system to the camera coordinate system are calibrated according to the rotation matrix difference to obtain the Euler angle parameters of the adjusted camera.

[0087] In one embodiment of the present invention, the transformation matrix from the camera content coordinate system to the base coordinate system is obtained by calling the TF library transformation chain of ROS.

[0088] As you can understand, the TF (Transform) library is a tool for managing coordinate system conversion in ROS. It can maintain the relationship between all coordinate systems in the robot system and provide conversion functions between different coordinate systems.

[0089] In one embodiment of the present invention, the ground normal vector in the base coordinate system Transform the matrix by left multiplication Convert to the camera content coordinate system to get the ground normal vector in the camera content coordinate system in, Represents the component of the ground normal vector in the x-axis direction in the base coordinate system, Represents the component of the ground normal vector in the y-axis direction in the base coordinate system, Represents the component of the ground normal vector in the z-axis direction in the base coordinate system, Represents the component of the ground normal vector in the x-axis direction in the camera content coordinate system, Represents the component of the ground normal vector in the y-axis direction in the camera content coordinate system, Represents the component of the ground normal vector in the z-axis direction in the camera content coordinate system.

[0090] In one embodiment of the present invention, the difference between the ground normal vector in the camera content coordinate system (camera_optical_frame) and the ground normal vector in the base coordinate system (base_link) can be visualized and intuitively observed through rviz.

[0091] As you can understand, rviz (ros visualization) is a three-dimensional visualization tool for ROS (Robot Operating System), which can provide a graphical interface, display the robot status in real time, and visualize sensor data.

[0092] Furthermore, the calculation formula of the Euler angle is:

[0093]

[0094] ry=arctan(-R diff (2,0)

[0095]

[0096] Among them, rx represents the rotation angle of the difference matrix around the x-axis, ry represents the rotation angle of the difference matrix around the y-axis, rz represents the rotation angle of the difference matrix around the z-axis, R diff represents the difference matrix, R diff (2,1) represents the element in the 3rd row and 2nd column of the difference matrix, R diff (2,2) represents the element in the 3rd row and 3rd column of the difference matrix, R diff (2,0) represents the element in the 3rd row and 1st column of the difference matrix, R diff (1,0) represents the element in the 2nd row and 1st column of the difference matrix, R diff (0,0) represents the element in the first row and first column of the difference matrix. Represented by the transpose of the rotation matrix of the camera content coordinate system, R base_link Represents the rotation matrix of the base coordinate system.

[0097] Understandably, R. plane The first column is The normalized value, the second column is the z-axis in the base coordinate system and The third column is the cross product of the first two columns, ensuring that the matrix R plane is an orthogonal matrix. base_link is the rotation matrix of the base coordinate system, which is based on the ground normal vector in the transformed base coordinate system Constructed, the construction method is the same as R plane Similarly, the first column is The normalized value, the second column is the z-axis in the base coordinate system and The third column is the cross product of the first two columns.

[0098] like Figure 3 As shown in FIG, camera_link is the camera coordinate system, base_link is the base coordinate system, and arm_base_link is the robot coordinate system. S4. Synchronously adjust the height parameters of each camera based on the centroid feature of the ground point cloud to obtain the adjusted camera height parameters.

[0099] Specifically, step S4 includes:

[0100] Calculate the average coordinates of all points in the ground point cloud in the camera content coordinate system to obtain the centroid of the ground point cloud;

[0101] Converting the ground point cloud centroid from the camera content coordinate system to the robot base coordinate system to obtain the ground point cloud centroid in the base coordinate system;

[0102] Calculate the difference between the height value of the centroid of the ground point cloud in the base coordinate system and the preset reference height value in the base coordinate system to obtain the height adjustment offset;

[0103] The height parameter of the camera is adjusted according to the height adjustment offset to obtain the adjusted height parameter of the camera.

[0104] Furthermore, the centroid calculation formula of the ground point cloud in the base coordinate system is:

[0105]

[0106] in, Represents the centroid coordinates of the ground point cloud in the base coordinate system, represents the transformation matrix from the camera coordinate system to the base coordinate system, represents the transformation matrix from the camera content coordinate system to the camera coordinate system, Represents the centroid coordinates of the ground point cloud in the camera content coordinate system.

[0107] The present invention automatically completes camera calibration through ground point cloud segmentation and normal vector estimation, eliminating the need for manual adjustment of camera position or attitude, reducing errors caused by human interference. At the same time, through automatic calibration of Euler angles and height, the accuracy and reliability of camera calibration results are ensured, greatly improving the efficiency of camera calibration.

[0108] S5. The robot calibration program simultaneously updates the external parameter data of the multiple cameras based on the adjusted Euler angle parameters of the cameras and the adjusted height parameters of the cameras to obtain calibrated parameters of the cameras.

[0109] The robot's calibration program is initiated via a ROS service call. This program exists as a ROS server, occupies minimal resources, and can be deployed on the robot platform for automatic startup. Upon startup, the program automatically loads the necessary configuration parameters from the parameter server, including the point cloud topic name, camera coordinate system, camera content coordinate system, and the location of the camera TF publish file. The calibration program is adaptive and can adapt to different camera models and numbers, ensuring stable and reliable camera calibration in a variety of application scenarios.

[0110] Specifically, the calibration program runs silently in the background in the form of a ROS server. When calibration is required, a command call is input and the calibration can be repeated. If a robot needs to calibrate multiple cameras, it only needs to modify the corresponding parameter file. The calibration time of each camera does not exceed 1 second. The calibration program is started by calling the command through the ROS service. After the calibration program is started, the Euler angle parameters and height parameters are calibrated. The calibration program supports multiple calibrations. Users can run the calibration multiple times in different environments or after changing the camera position according to actual use needs to ensure the accuracy of external parameter calibration. Each time calibration is performed, the program will automatically overwrite the previous calibration results to ensure that the latest results are applied, which greatly improves the practicality and convenience of camera calibration.

[0111] The present invention first extracts ground point cloud information from multiple cameras through point cloud segmentation and normal vector estimation. The Euler angle parameters for each camera are then calculated and adjusted to correct the camera's attitude error. Next, the camera's height parameters are further adjusted by calculating the height differences between the ground point cloud's centroids. This allows for simultaneous extrinsic calibration of multiple cameras. This significantly shortens calibration time, improves accuracy, and reduces reliance on operator skill, making calibration easier and more efficient in diverse environments and configurations.

[0112] The present invention provides a multi-camera extrinsic parameter simultaneous calibration system based on ground point cloud, which adopts the multi-camera extrinsic parameter simultaneous calibration method as described above, including:

[0113] a data acquisition module configured to fix multiple depth cameras on a robot, the robot being positioned at an arbitrary position and initializing parameters, obtaining parameters of each camera from a parameter server of the robot, and obtaining point cloud data collected by the multiple cameras;

[0114] a point cloud segmentation module configured to perform ground segmentation on the point cloud data collected by each of the multiple cameras to obtain a ground point cloud corresponding to each camera;

[0115] A first parameter adjustment module is configured to synchronously adjust the Euler angle parameters of each camera based on the normal vector characteristics of the ground point cloud to obtain the adjusted Euler angle parameters of the camera;

[0116] A second parameter adjustment module is configured to synchronously adjust the height parameter of each camera based on the centroid feature of the ground point cloud to obtain the adjusted height parameter of the camera;

[0117] The calibration module is configured as a calibration program of the robot to update the external parameter data of the multiple cameras based on the Euler angle parameters of the adjusted cameras and the height parameters of the adjusted cameras to obtain the calibrated parameters of the cameras.

[0118] The multi-camera extrinsic parameter simultaneous calibration device provided by the present invention is described below. The multi-camera extrinsic parameter simultaneous calibration device described below and the multi-camera extrinsic parameter simultaneous calibration method described above can be referenced to each other.

[0119] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4 As shown, the electronic device may include: a processor (processor) 410, a communication interface (Communications Interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute a method for simultaneous calibration of multi-camera extrinsic parameters, which includes: fixing multiple depth cameras on the robot, the robot is located at an arbitrary position and initializing parameters, obtaining parameters of each camera from the robot's parameter server, and obtaining point cloud data collected by multiple cameras; performing ground segmentation on the point cloud data collected by each camera in the multiple cameras to obtain the ground point cloud corresponding to each camera; synchronously adjusting the Euler angle parameters of each camera based on the normal vector features of the ground point cloud to obtain the Euler angle parameters of the adjusted camera; synchronously adjusting the height parameters of each camera based on the center of mass features of the ground point cloud to obtain the height parameters of the adjusted camera; the robot's calibration program updates the extrinsic parameter data of the multiple cameras based on the Euler angle parameters of the adjusted camera and the height parameters of the adjusted camera to obtain the calibrated parameters of the camera.

[0120] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for simultaneous calibration of multi-camera extrinsic parameters provided by the above methods, the method including: fixing multiple depth cameras on a robot, the robot being located at an arbitrary position and performing parameter initialization, obtaining parameters of each camera from the robot's parameter server, and obtaining point cloud data collected by multiple cameras; performing ground segmentation on the point cloud data collected by each camera in the multiple cameras to obtain a ground point cloud corresponding to each camera; synchronously adjusting the Euler angle parameters of each camera based on the normal vector features of the ground point cloud to obtain the Euler angle parameters of the adjusted camera; synchronously adjusting the height parameters of each camera based on the center of mass features of the ground point cloud to obtain the height parameters of the adjusted camera; the robot's calibration program updates the extrinsic parameter data of the multiple cameras based on the Euler angle parameters of the adjusted camera and the height parameters of the adjusted camera to obtain the calibrated parameters of the camera.

[0122] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for simultaneous calibration of multi-camera extrinsic parameters provided by the above-mentioned methods, the method comprising: fixing multiple depth cameras on a robot, the robot being located at an arbitrary position and performing parameter initialization, obtaining parameters of each camera from a parameter server of the robot, and obtaining point cloud data collected by multiple cameras; performing ground segmentation on the point cloud data collected by each of the multiple cameras to obtain a ground point cloud corresponding to each camera; synchronously adjusting the Euler angle parameters of each camera based on the normal vector features of the ground point cloud to obtain the Euler angle parameters of the adjusted camera; synchronously adjusting the height parameters of each camera based on the center of mass features of the ground point cloud to obtain the height parameters of the adjusted camera; the robot's calibration program updates the extrinsic parameter data of the multiple cameras based on the Euler angle parameters of the adjusted camera and the height parameters of the adjusted camera to obtain the calibrated parameters of the camera.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0124] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud, characterized in that: The following steps are involved: S1. A plurality of depth cameras are fixedly mounted on a robot, the robot is located at an arbitrary position and parameters are initialized, configuration parameters of each camera are obtained from a parameter server of the robot, and point cloud data collected by the plurality of cameras are obtained; S2, performing ground segmentation on the point cloud data collected by each of the multiple cameras to obtain a ground point cloud corresponding to each camera; S3, synchronously adjusting the Euler angle parameters of each camera based on the normal vector features of the ground point cloud to obtain the Euler angle parameters of the adjusted camera; S4, synchronously adjusting the height parameter of each camera based on the centroid feature of the ground point cloud to obtain the height parameter of the adjusted camera; S5. The robot calibration program simultaneously updates the external parameter data of the multiple cameras based on the adjusted Euler angle parameters of the cameras and the adjusted height parameters of the cameras to obtain calibrated parameters of the cameras.

2. The method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud according to claim 1, characterized in that: Step S2 specifically includes: S21, taking the point cloud data collected by each camera as a set of initial point cloud data, and constructing a KD tree based on each set of point cloud data; S22, performing a k-nearest neighbor search for each point in each set of point cloud data based on the KD tree to determine a neighborhood point set for each point in each set of point cloud data; S23, calculating the covariance matrix of the neighborhood point set of each point, and performing eigenvalue decomposition on the covariance matrix to obtain the normal vector of each point; S24. Use the RANSAC algorithm to iteratively calculate the plane model parameters based on the three-dimensional coordinates of each point in each set of point cloud data and the normal vector of each point, mark the points in each set of point cloud data that conform to the plane model as internal points, and form a ground point cloud corresponding to each camera based on the internal points in each set of point cloud data.

3. The method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud according to claim 2, characterized in that: Step S24 specifically includes: Step 1: Select any point p in each set of point cloud data i ; Step 2: Calculate point p i The distance to the plane is calculated as follows: Among them, dis represents the distance from the point to the plane, n represents the normal vector of the plane, and p represents the point p i The coordinate vector of T represents the transpose of the normal vector n, d represents the offset of the plane on the coordinate axis, ‖·‖ represents the norm of the vector, and |·| represents the absolute value operation; Step 3: Point p i The distance to the plane is compared with the preset distance threshold. If point p i If the distance to the plane is less than the preset distance threshold, point p i Marked as inliers; Step 4: Repeat steps 1 to 3 until all points in each set of point cloud data are traversed to obtain an inner point set, and form a ground point cloud corresponding to the camera based on the inner point set.

4. The method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud according to claim 1, characterized in that: Step S3 specifically includes: Point cloud ground normal vector calculated from the ground point cloud camera content coordinate system And calculate the transformation matrix from the camera content coordinate system to the base coordinate system using the transformation matrix The ground normal vector in the base coordinate system Convert to the camera content coordinate system to get the ground normal vector in the camera content coordinate system According to the point cloud ground normal vector in the camera content coordinate system Construct the rotation matrix of the camera content coordinate system and the rotation matrix of the base coordinate system; Calculate the rotation matrix difference between the rotation matrix of the camera content coordinate system and the rotation matrix of the base coordinate system to form a difference matrix, and extract the Euler angle of the camera based on the difference matrix; The Euler angles from the calibration base coordinate system to the camera coordinate system are calibrated according to the rotation matrix difference to obtain the Euler angle parameters of the adjusted camera.

5. The method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud according to claim 4, characterized in that: The calculation formula of the Euler angle is: ry=arctan(-R diff (2,0)); Among them, rx represents the rotation angle of the difference matrix around the x-axis, ry represents the rotation angle of the difference matrix around the y-axis, rz represents the rotation angle of the difference matrix around the z-axis, and R diff represents the difference matrix, R diff (2,1) represents the element in the 3rd row and 2nd column of the difference matrix, R diff (2,2) represents the element in the 3rd row and 3rd column of the difference matrix, R diff (2,0) represents the element in the 3rd row and 1st column of the difference matrix, R diff (1,0) represents the element in the 2nd row and 1st column of the difference matrix, R diff (0,0) represents the element in the first row and first column of the difference matrix. Represents the transpose of the rotation matrix of the camera content coordinate system, R base_link Represents the rotation matrix of the base coordinate system.

6. The method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud according to claim 5, characterized in that: Step S4 specifically includes: Calculate the average coordinates of all points in the ground point cloud in the camera content coordinate system to obtain the centroid of the ground point cloud; Convert the centroid of the ground point cloud from the camera content coordinate system to the robot base coordinate system to obtain the centroid of the ground point cloud in the base coordinate system; Calculate the difference between the height value of the centroid of the ground point cloud in the base coordinate system and the preset reference height value in the base coordinate system to obtain the height adjustment offset; The height parameter of the camera is adjusted according to the height adjustment offset to obtain the adjusted height parameter of the camera.

7. The method for simultaneous calibration of multi-camera extrinsic parameters based on ground point cloud according to claim 6, characterized in that: The calculation formula of the centroid of the ground point cloud in the base coordinate system is: in, Represents the centroid coordinates of the ground point cloud in the base coordinate system, represents the transformation matrix from the camera coordinate system to the base coordinate system, represents the transformation matrix from the camera content coordinate system to the camera coordinate system, Represents the centroid coordinates of the ground point cloud in the camera content coordinate system.

8. A multi-camera extrinsic parameter simultaneous calibration system based on ground point cloud, characterized in that: The method for simultaneous calibration of multi-camera extrinsic parameters according to any one of claims 1 to 7 comprises: A data acquisition module is configured to fix a plurality of depth cameras on a robot, the robot is located at an arbitrary position and performs parameter initialization, obtains parameters of each camera from a parameter server of the robot, and obtains point cloud data collected by the plurality of cameras; A point cloud segmentation module, configured to perform ground segmentation on the point cloud data collected by each of the multiple cameras to obtain a ground point cloud corresponding to each camera; A first parameter adjustment module is configured to synchronously adjust the Euler angle parameters of each camera based on the normal vector characteristics of the ground point cloud to obtain the Euler angle parameters of the adjusted camera; A second parameter adjustment module is configured to synchronously adjust the height parameter of each camera based on the centroid feature of the ground point cloud to obtain the height parameter of the adjusted camera; The calibration module is configured as a calibration program of the robot to update the external parameter data of the multiple cameras based on the Euler angle parameters of the adjusted cameras and the height parameters of the adjusted cameras to obtain the calibrated parameters of the cameras.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for simultaneous calibration of multi-camera extrinsic parameters as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for simultaneous calibration of multi-camera extrinsic parameters as described in any one of claims 1 to 7 is implemented.

Images