A multi-camera calibration method, device and related equipment

Abstract

The application discloses a multi-camera calibration method and device and related equipment, wherein the multi-camera calibration method comprises the following steps: acquiring two-dimensional image data and three-dimensional image data of a same target area collected by a plurality of two-dimensional cameras to be calibrated and a preset three-dimensional device; performing three-dimensional reconstruction by using each two-dimensional image data collected by each two-dimensional camera to be calibrated to obtain initial internal and external parameters and initial point cloud data of each two-dimensional camera to be calibrated, and optimizing the initial point cloud data to obtain sparse three-dimensional point cloud data of the target area; generating actual three-dimensional point cloud data of the target area according to the three-dimensional image data acquired by the preset three-dimensional device; registering the sparse three-dimensional point cloud data and the actual three-dimensional point cloud data to obtain a transformation matrix between the point clouds, and obtaining actual internal and external parameters of each two-dimensional camera to be calibrated and relative external parameters between each two-dimensional camera to be calibrated by using the initial internal and external parameters and the transformation matrix. The application is beneficial to multi-camera calibration, and improves the efficiency and accuracy of multi-camera calibration.

Description

Technical Field

[0001] This invention relates to the field of camera parameter calibration technology, and in particular to a multi-camera calibration method, apparatus and related equipment. Background Technology

[0002] With the development of science and technology, image data is increasingly needed in various applications. Currently, it is common practice to acquire images of the same object or region from different angles using multiple cameras, which requires camera calibration. Camera calibration is the process of determining the camera's internal geometric and optical parameters, as well as the three-dimensional position and orientation of the camera coordinate system relative to the world coordinate system. It is crucial for determining the relative relationship between a two-dimensional image and a three-dimensional scene.

[0003] In existing technologies, camera calibration is typically performed directly using calibration boards. However, this approach is problematic when calibrating multiple cameras (more than two cameras) where the optical centers of the cameras are inconsistent. This necessitates pairwise calibration using calibration boards for multi-angle, multi-distance data acquisition and manual intervention, resulting in a complex, inefficient, and inaccurate calibration process. Therefore, existing solutions are not conducive to multi-camera calibration or improving its efficiency and accuracy. Summary of the Invention

[0004] The main objective of this invention is to provide a multi-camera calibration method, apparatus, and related equipment, aiming to solve the problem that the existing scheme of directly calibrating cameras through a calibration plate is not conducive to multi-camera calibration and to improving the efficiency and accuracy of multi-camera calibration.

[0005] To achieve the above objectives, a first aspect of the present invention provides a multi-camera calibration method, wherein the multi-camera calibration method includes:

[0006] Acquire 2D and 3D image data of the same target area from multiple 2D cameras to be calibrated and preset 3D devices;

[0007] Using the two-dimensional image data acquired by the two-dimensional cameras to be calibrated, three-dimensional reconstruction is performed to obtain the initial intrinsic and extrinsic parameters and initial point cloud data of the two-dimensional cameras to be calibrated, and the initial point cloud data is optimized to obtain sparse three-dimensional point cloud data of the target area.

[0008] Based on the three-dimensional image data acquired by the aforementioned preset three-dimensional device, generate the actual three-dimensional point cloud data of the aforementioned target area;

[0009] Register the above sparse 3D point cloud data with the above actual 3D point cloud data, obtain the transformation matrix between the point clouds, and use the above initial intrinsic and extrinsic parameters and the above transformation matrix to obtain the actual intrinsic and extrinsic parameters of each of the above 2D cameras to be calibrated and the relative extrinsic parameters between each of the above 2D cameras to be calibrated.

[0010] A second aspect of the present invention provides a multi-camera calibration apparatus, wherein the multi-camera calibration apparatus comprises:

[0011] The first acquisition module is used to acquire two-dimensional image data and three-dimensional image data of the same target area collected by multiple two-dimensional cameras to be calibrated and preset three-dimensional devices;

[0012] The second acquisition module is used to perform three-dimensional reconstruction using the two-dimensional image data acquired by each of the two-dimensional cameras to be calibrated to obtain the initial intrinsic and extrinsic parameters and initial point cloud data of each of the two-dimensional cameras to be calibrated, and to optimize the initial point cloud data to obtain sparse three-dimensional point cloud data of the target area.

[0013] The third acquisition module is used to generate actual three-dimensional point cloud data of the target area based on the three-dimensional image data acquired by the preset three-dimensional device.

[0014] The calibration module is used to register the sparse 3D point cloud data with the actual 3D point cloud data, obtain the transformation matrix between the point clouds, and use the initial intrinsic and extrinsic parameters and the transformation matrix to obtain the actual intrinsic and extrinsic parameters of each of the 2D cameras to be calibrated and the relative extrinsic parameters between each of the 2D cameras to be calibrated.

[0015] A third aspect of the present invention provides a multi-camera device, comprising a plurality of two-dimensional cameras and the calibration device mentioned in the second aspect above, wherein:

[0016] The aforementioned multiple 2D cameras are used to acquire 2D image data of the target area;

[0017] The aforementioned calibration device is used to calibrate each of the aforementioned two-dimensional cameras using the aforementioned two-dimensional image data and the three-dimensional image data acquired by the preset three-dimensional device, and to obtain the intrinsic and extrinsic parameters of each of the aforementioned two-dimensional cameras and the relative extrinsic parameters between the aforementioned two-dimensional cameras.

[0018] A fourth aspect of the present invention provides a smart terminal, the smart terminal including a memory, a processor, and a multi-camera calibration program stored in the memory and executable on the processor, wherein the multi-camera calibration program, when executed by the processor, implements the steps of any of the above-mentioned multi-camera calibration methods.

[0019] The fifth aspect of the present invention provides a computer-readable storage medium storing a multi-camera calibration program, wherein the multi-camera calibration program, when executed by a processor, implements the steps of any of the above-described multi-camera calibration methods.

[0020] As can be seen from the above, compared with the existing technology that directly uses a calibration board for camera calibration, the present invention eliminates the need for separate calibration using calibration boards for each pair of cameras, and also eliminates the need for manual intervention. Calibration can be achieved simply by obtaining the image data acquired by the 2D camera to be calibrated and the directly acquired 3D information. This simplifies the calibration process and avoids errors caused by manual processes, thereby improving the efficiency and accuracy of multi-camera calibration. Furthermore, it should be noted that existing technologies typically require calibration before the camera can acquire target image data (i.e., the image data to be used subsequently). However, the present invention allows for the direct acquisition of target image data even without calibration, enabling the use of this target image data for calibration and subsequent application of the corresponding target image data, thus enhancing the user experience. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a multi-camera calibration method provided in an embodiment of the present invention;

[0023] Figure 2 This is an embodiment of the present invention. Figure 1 A detailed flowchart of step S200 is shown below;

[0024] Figure 3 This is an embodiment of the present invention. Figure 1 A detailed flowchart of step S400 is shown below;

[0025] Figure 4 This is an embodiment of the present invention. Figure 3 A detailed flowchart of step S402 is shown below;

[0026] Figure 5 This is a schematic diagram illustrating the specific process of a multi-camera calibration method provided in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the structure of a multi-camera calibration device provided in an embodiment of the present invention;

[0028] Figure 7 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0030] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0031] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0032] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] To address at least one problem in the existing technology, this invention provides a multi-camera calibration method that eliminates the need for a calibration board. Instead, it directly utilizes the reconstructed 3D point cloud data acquired by the 2D camera to be calibrated and the actual 3D point cloud data obtained after directly acquiring 3D information from the target area to calibrate multiple 2D cameras.

[0036] Exemplary methods

[0037] like Figure 1 As shown in the figure, this embodiment of the invention provides a multi-camera calibration method. Specifically, the multi-camera calibration method includes the following steps:

[0038] Step S100: Acquire two-dimensional image data and three-dimensional image data of the same target area collected by multiple two-dimensional cameras to be calibrated and preset three-dimensional devices.

[0039] Specifically, multiple 2D cameras to be calibrated capture images of the same target area from different angles, acquiring various 2D image data. A 3D device then acquires 3D image data of the same target area. It should be noted that the aforementioned image data can be used solely for calibration, or it can be the final image data to be acquired by the 2D cameras to be calibrated. This means that the required image data can be acquired directly before calibration is completed, and calibration can be performed accordingly, eliminating the need for prior calibration and simplifying user experience.

[0040] In one embodiment, acquiring two-dimensional image data and three-dimensional image data of the same target area collected by multiple two-dimensional cameras to be calibrated and a preset three-dimensional device includes: continuously capturing images of the same target area by multiple two-dimensional cameras to be calibrated with relatively fixed poses, and acquiring the two-dimensional image data corresponding to each of the two-dimensional cameras to be calibrated; and acquiring three-dimensional image data directly collected by the preset three-dimensional device from the target area, wherein the preset three-dimensional device includes one or more combinations of a three-dimensional scanner, a depth camera, and a lidar.

[0041] In this embodiment, each 2D camera to be calibrated is pre-fixed, and the relative pose between the cameras remains unchanged during the data acquisition process. The optical centers of the cameras can be the same or different, without affecting the subsequent registration effect based on point cloud data. Therefore, there is no need to use a calibration board for optical center calibration, simplifying the calibration process and improving efficiency.

[0042] In one embodiment, the 2D camera to be calibrated is an RGB camera, and the image data is RGB image data. Thus, the obtained image data can be directly applied to subsequent processing steps (such as face recognition, human skeleton reconstruction, etc.) to meet user needs. In actual use, the 2D camera to be calibrated can also be a grayscale camera, an infrared camera, etc., and no specific limitation is made here.

[0043] In one embodiment, multiple 2D cameras to be calibrated can be used to synchronously acquire images of the same target area at the same time, generating image data corresponding to each camera at the same time. Alternatively, multiple cameras can be controlled to continuously acquire images of the target area in the same time sequence, obtaining image data corresponding to each camera. That is, each camera synchronously acquires image data corresponding to multiple time points, preventing the features in the target area obtained during single-frame acquisition from being unclear and improving the robustness of the results. It should be noted that during continuous acquisition, the scene of the target area does not change, and the pose of each camera to be calibrated does not change.

[0044] In one embodiment, the 3D information of the target area is directly acquired using a 3D device, which includes a depth camera and / or LiDAR. The acquired image data is 3D image data, which directly provides depth information of the target area compared to the 2D image data acquired by the 2D camera to be calibrated. It should be noted that in this embodiment, the 3D device includes any one or more combinations of a depth camera, LiDAR, and a 3D scanner. The depth camera can be a camera based on structured light, TOF, or binocular principles. In actual use, other devices capable of acquiring 3D information can also be used, and no specific limitation is made here.

[0045] Step S200: Using the two-dimensional image data collected by each of the two-dimensional cameras to be calibrated, perform three-dimensional reconstruction to obtain the initial intrinsic and extrinsic parameters and initial point cloud data of each of the two-dimensional cameras to be calibrated, and optimize the initial point cloud data to obtain sparse three-dimensional point cloud data of the target area.

[0046] Specifically, after acquiring the image data corresponding to each 2D camera to be calibrated, the target area can be reconstructed in 3D based on a preset 3D reconstruction algorithm to obtain initial point cloud data.

[0047] In one embodiment, such as Figure 2 As shown, step S200 specifically includes the following steps:

[0048] Step S201: Perform three-dimensional reconstruction on the two-dimensional image data acquired by each of the above-mentioned two-dimensional cameras to be calibrated at the same time using the preset SFM (Structure From Motion) algorithm to obtain initial point cloud data and initial intrinsic and extrinsic parameters of each of the above-mentioned two-dimensional cameras to be calibrated.

[0049] In this embodiment, a preset SFM algorithm is used to perform 3D reconstruction on the image data acquired by each of the 2D cameras to be calibrated at the same time to obtain initial point cloud data. Based on this initial point cloud data, the initial intrinsic and extrinsic parameters corresponding to each 2D camera to be calibrated are obtained. It should be noted that the intrinsic parameters are used to reflect its internal parameters, such as focal length and pixel size, while the extrinsic parameters are used to represent the relative pose between the cameras. The SFM algorithm is an offline algorithm for 3D reconstruction based on various collected unordered images. In this embodiment, the SFM algorithm is used, but this is not a specific limitation. In actual use, other 3D reconstruction algorithms can also be used.

[0050] Step S202: Use the two-dimensional image data to perform global optimization on the initial intrinsic and extrinsic parameters to obtain the scale-free intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated.

[0051] In one embodiment, for image data acquired according to a preset time sequence, after obtaining initial intrinsic and extrinsic parameters, the data collected by each 2D camera to be calibrated at the same time are bundled into a set of data. The bundled data sets are then used to perform Bundle Adjustment (BA) optimization on the initial intrinsic and extrinsic parameters to obtain the corresponding intrinsic and extrinsic parameters for each set. The average value is then calculated to obtain accurate scale-free intrinsic and extrinsic parameters, i.e., scale-free intrinsic and extrinsic parameters. It should be noted that "scale-free" in scale-free intrinsic and extrinsic parameters refers to the unknown translation amount in the displacement between two image frames; that is, the specific unit of the translation amount is unknown, and a normalized result is obtained.

[0052] In one embodiment, if each 2D camera to be calibrated only acquires a single frame of image data at the same time, then there is no need to group, bundle, or calculate the average value. The intrinsic and extrinsic parameters optimized by BA can be directly used as the corresponding scale-free intrinsic and extrinsic parameters.

[0053] Step S203: The initial point cloud data is processed using the scale-free intrinsic and extrinsic parameters of each of the above-mentioned two-dimensional cameras to be calibrated, to obtain the above-mentioned sparse three-dimensional point cloud data of the target area.

[0054] More specifically, the initial point cloud data is triangulated and optimized based on the scale-free intrinsic and extrinsic parameters of each 2D camera to be calibrated, resulting in sparse 3D point cloud data of the target region. This sparse 3D point cloud data represents point cloud information of the same target region. The point cloud data is sparse because the points in the data are all feature points extracted from the same target region, and the number of feature points that can be extracted from 2D image data is relatively small. This results in sparse point cloud data that can be formed after 3D reconstruction of 2D image data. This sparse 3D point cloud data also requires less storage and computational resources, which helps improve computational efficiency and thus calibration efficiency.

[0055] Step S300: Based on the three-dimensional image data acquired by the aforementioned preset three-dimensional device, generate the actual three-dimensional point cloud data of the target area.

[0056] In one embodiment, 3D image data of the same scene (the target area captured by the 2D camera to be calibrated) acquired through a preset 3D device is reconstructed according to a preset 3D reconstruction algorithm to obtain the actual 3D point cloud data of the target area. The actual 3D point cloud data of the target area refers to 3D point cloud data obtained through directly acquired 3D information. In this embodiment, the SLAM (Simultaneous Localization and Mapping) algorithm is used, but this is not a specific limitation.

[0057] Step S400: Register the sparse 3D point cloud data with the actual 3D point cloud data, obtain the transformation matrix between the point clouds, and use the initial intrinsic and extrinsic parameters and the transformation matrix to obtain the actual intrinsic and extrinsic parameters of each of the above-mentioned 2D cameras to be calibrated and the relative extrinsic parameters between each of the above-mentioned 2D cameras to be calibrated.

[0058] Both sparse 3D point cloud data and actual 3D point cloud data are 3D representations of the scene in the target area, and the scene does not change. Therefore, they contain the same feature points, and registration can be performed based on the feature points to obtain the intrinsic parameters of each 2D camera to be calibrated and the pose relationship between each camera (i.e., relative extrinsic parameters).

[0059] In this embodiment, as Figure 3 As shown, step S400 specifically includes the following steps:

[0060] Step S401: Using the above-mentioned actual 3D point cloud data as the target point cloud and the above-mentioned sparse 3D point cloud data as the registration point cloud, obtain the transformation matrix between the point clouds.

[0061] Specifically, in this embodiment, the actual 3D point cloud data and sparse 3D point cloud data are registered using the Iterative Closest Point (ICP) algorithm to obtain the final scaled transformation matrices SRT between each 2D camera to be calibrated and the 3D device. The SRT is a 4x4 matrix that indicates the relative pose transformation of the image coordinate system of the 2D camera to be calibrated relative to the depth camera (or LiDAR). The image coordinate system is constructed based on the image data of all the 2D cameras to be calibrated. It should be noted that this embodiment uses the Iterative Closest Point algorithm; other algorithms can be used in actual applications, and no specific limitation is made here.

[0062] Step S402: Based on the above transformation matrix and the scale-free intrinsic and extrinsic parameters of each of the above-mentioned two-dimensional cameras to be calibrated, obtain the target intrinsic and extrinsic parameters of each of the above-mentioned two-dimensional cameras to be calibrated and the relative extrinsic parameters between each of the above-mentioned two-dimensional cameras to be calibrated.

[0063] Furthermore, such as Figure 4 As shown, step S402 specifically includes the following steps:

[0064] Step S4021: Obtain the relative extrinsic parameters between the two-dimensional cameras to be calibrated based on the transformation matrix between the point clouds.

[0065] Step S4022: Optimize the scale-free intrinsic and extrinsic parameters between the two-dimensional cameras to be calibrated using the relative extrinsic parameters between the two-dimensional cameras to be calibrated, and obtain the actual intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated.

[0066] More specifically, the transformation matrix of pose between point clouds is the parameter obtained by transforming sparse 3D point cloud data into the coordinate system of the 3D device. That is, transforming sparse 3D point cloud data into the same coordinate system, obtaining the pose relationship between each 2D camera to be calibrated and the 3D device. Based on the pose relationship, the image data acquired by each 2D camera to be calibrated is mapped to the coordinate system of the 3D device, obtaining the image data corresponding to each 2D camera to be calibrated in the coordinate system of the 3D device, and obtaining the relative pose parameters between each 2D camera to be calibrated in this coordinate system. Furthermore, based on the relative pose parameters, the scale-free intrinsic and extrinsic parameters between each 2D camera to be calibrated are optimized to obtain the scaled intrinsic and extrinsic parameters of each 2D camera to be calibrated, i.e., the actual intrinsic and extrinsic parameters.

[0067] It should be noted that the actual intrinsic and extrinsic parameters and the relative extrinsic parameters mentioned above are the final obtained internal parameters of each 2D camera to be calibrated and the relative poses between each 2D camera to be calibrated, respectively. Using the actual intrinsic and extrinsic parameters and the relative extrinsic parameters, the corresponding coordinates of each pixel in the image acquired by each 2D camera to be calibrated in the world coordinate system and the positional relationships between each 2D camera to be calibrated can be obtained, thus achieving the calibration of all 2D cameras to be calibrated.

[0068] As can be seen from the above, in this embodiment, the calibration of multiple 2D cameras can be quickly and easily achieved by combining the image data acquired by the 2D camera to be calibrated and the 3D information directly acquired from the target area. No calibration board is required, the calculation process is simple, and no manual intervention is needed. Calibration can be achieved simply by obtaining the image data acquired by the 2D camera to be calibrated and the directly acquired 3D information. This simplifies the calibration process, avoids errors caused by manual processes, and improves the efficiency and accuracy of multi-camera calibration.

[0069] Figure 5This is a flowchart illustrating a specific application scenario of a multi-camera calibration method provided in this embodiment of the invention. In this specific application scenario, multiple RGB cameras (i.e., the 2D cameras to be calibrated) are pre-fixed. Throughout the data acquisition and calibration process, each RGB camera remains stationary, and its relative pose does not change. Specifically, multiple RGB cameras acquire continuous RGB data within a target area. Then, the initial intrinsic and extrinsic parameters of each RGB camera are initially obtained using the SFM reconstruction algorithm. Based on the obtained initial intrinsic and extrinsic parameters, the data acquired by multiple RGB cameras at the same time are bundled into a set of data (i.e., one frame of data), and BA optimization is performed to obtain accurate scale-free intrinsic and extrinsic parameters (i.e., scale-free intrinsic and extrinsic parameters). Simultaneously, a sparse point cloud cloud1 (i.e., 3D point cloud data generated based on the bundled data acquired by each RGB camera after SFM algorithm and BA optimization) is also obtained and output.

[0070] It should be noted that when performing BA optimization on multiple sets of data to obtain multiple sets of intrinsic and extrinsic parameters, the average value of the intrinsic parameters and the average value of the extrinsic parameters are used as the corresponding scale-free intrinsic and extrinsic parameters to improve computational accuracy. On the other hand, data of the same scene collected by multiple RGB cameras based on a preset 3D device are acquired. The acquired continuous 3D data is reconstructed using SLAM to obtain the scene's point cloud data cloud2 (i.e., the actual 3D point cloud data) and output. Then, cloud1 and cloud2 are registered. Specifically, sparse point clouds cloud1 and point clouds cloud2 are registered using ICP with scale, with cloud2 as the target point cloud and cloud1 as the registered point cloud, to obtain the final scaled transformation matrix SRT. Based on the relative pose transformation matrix SRT and the scale-free intrinsic and extrinsic parameters, the intrinsic and extrinsic parameters of multiple RGB cameras and the relative extrinsic parameters between each camera are obtained.

[0071] As can be seen, in this invention, calibration is performed using directly acquired 3D information to obtain the internal and external parameters of multiple 2D cameras to be calibrated. This is not limited by site or conditions, does not require complex measurements using calibration boards, and does not require manual intervention, making the operation more convenient and improving calibration efficiency and accuracy.

[0072] Exemplary device

[0073] like Figure 6 As shown, corresponding to the above-described multi-camera calibration method, this embodiment of the invention also provides a multi-camera calibration device, the device comprising:

[0074] The first acquisition module 510 is used to acquire two-dimensional image data and three-dimensional image data of the same target area collected by multiple two-dimensional cameras to be calibrated and preset three-dimensional devices.

[0075] Specifically, multiple 2D cameras to be calibrated capture images of the same target area from different angles, acquiring various 2D image data. A 3D device then acquires 3D image data of the same target area. It should be noted that the aforementioned image data can be used solely for calibration, or it can be the final image data to be acquired by the 2D cameras to be calibrated. This means that the required image data can be acquired directly before calibration is completed, and calibration can be performed accordingly, eliminating the need for prior calibration and simplifying user experience.

[0076] In one embodiment, the 3D information of the target area is directly acquired using a 3D device, which includes a depth camera and / or LiDAR. The acquired image data is 3D image data, which directly provides depth information of the target area compared to the 2D image data acquired by the 2D camera to be calibrated. It should be noted that in this embodiment, the 3D device includes any one or more combinations of a depth camera, LiDAR, and a 3D scanner. The depth camera can be a camera based on structured light, TOF, or binocular principles. In actual use, other devices capable of acquiring 3D information can also be used, which are not specifically limited here.

[0077] The second acquisition module 520 is used to perform three-dimensional reconstruction using the two-dimensional image data acquired by each of the two-dimensional cameras to be calibrated to obtain the initial intrinsic and extrinsic parameters and initial point cloud data of each of the two-dimensional cameras to be calibrated, and to optimize the initial point cloud data to obtain sparse three-dimensional point cloud data of the target area.

[0078] Specifically, after acquiring the image data corresponding to each 2D camera to be calibrated, the target area can be reconstructed in 3D based on a preset 3D reconstruction algorithm to obtain initial point cloud data.

[0079] The third acquisition module 530 is used to generate actual three-dimensional point cloud data of the target area based on the three-dimensional image data acquired by the preset three-dimensional device.

[0080] In one embodiment, 3D image data of the same scene (the target area captured by the 2D camera to be calibrated) acquired through a preset 3D device is reconstructed according to a preset 3D reconstruction algorithm to obtain the actual 3D point cloud data of the target area. The actual 3D point cloud data of the target area refers to 3D point cloud data obtained through directly acquired 3D information. In this embodiment, the SLAM (Simultaneous Localization and Mapping) algorithm is used, but this is not a specific limitation.

[0081] The calibration module 540 is used to register the sparse 3D point cloud data with the actual 3D point cloud data, obtain the transformation matrix between the point clouds, and use the initial intrinsic and extrinsic parameters and the transformation matrix to obtain the actual intrinsic and extrinsic parameters of each of the 2D cameras to be calibrated and the relative extrinsic parameters between each of the 2D cameras to be calibrated.

[0082] Both sparse 3D point cloud data and actual 3D point cloud data are 3D representations of the scene in the target area, and the scene does not change. Therefore, they contain the same feature points, and registration can be performed based on the feature points to obtain the intrinsic parameters of each 2D camera to be calibrated and the pose relationship between each camera (i.e., relative extrinsic parameters).

[0083] Specifically, in this embodiment, the specific functions of the multi-camera calibration device and its various modules can be referred to the corresponding descriptions in the multi-camera calibration method, and will not be repeated here.

[0084] Based on the above embodiments, the present invention also provides a multi-camera device, including a plurality of two-dimensional cameras and the above-described calibration device, wherein:

[0085] The aforementioned multiple 2D cameras are used to acquire 2D image data of the target area;

[0086] The aforementioned calibration device is used to calibrate each of the aforementioned two-dimensional cameras using the aforementioned two-dimensional image data and the three-dimensional image data acquired by the preset three-dimensional device, and to obtain the intrinsic and extrinsic parameters of each of the aforementioned two-dimensional cameras and the relative extrinsic parameters between the aforementioned two-dimensional cameras.

[0087] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 7 As shown. The aforementioned smart terminal includes a processor and a memory. The memory of the smart terminal includes a multi-camera calibration program, and the memory provides an environment for the execution of the multi-camera calibration program. When the multi-camera calibration program is executed by the processor, it implements the steps of any of the aforementioned multi-camera calibration methods. It should be noted that the aforementioned smart terminal may also include other functional modules or units, which are not specifically limited here.

[0088] Those skilled in the art will understand that Figure 7 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. Specifically, the smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0089] In one embodiment, when the multi-camera calibration program is executed by the processor, the following operation instructions are performed:

[0090] Acquire 2D and 3D image data of the same target area from multiple 2D cameras to be calibrated and preset 3D devices;

[0091] Using the two-dimensional image data acquired by the two-dimensional cameras to be calibrated, three-dimensional reconstruction is performed to obtain the initial intrinsic and extrinsic parameters and initial point cloud data of the two-dimensional cameras to be calibrated, and the initial point cloud data is optimized to obtain sparse three-dimensional point cloud data of the target area.

[0092] Based on the three-dimensional image data acquired by the aforementioned preset three-dimensional device, generate the actual three-dimensional point cloud data of the aforementioned target area;

[0093] Register the above sparse 3D point cloud data with the above actual 3D point cloud data, obtain the transformation matrix between the point clouds, and use the above initial intrinsic and extrinsic parameters and the above transformation matrix to obtain the actual intrinsic and extrinsic parameters of each of the above 2D cameras to be calibrated and the relative extrinsic parameters between each of the above 2D cameras to be calibrated.

[0094] This invention also provides a computer-readable storage medium storing a multi-camera calibration program. When the multi-camera calibration program is executed by a processor, it implements the steps of any of the multi-camera calibration methods provided in this invention.

[0095] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0097] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0098] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0099] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of the above modules or units is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0100] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0101] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not mean that the essence of the corresponding technical solutions deviates from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-camera calibration method, characterized in that, include: Acquire 2D and 3D image data of the same target area from multiple 2D cameras to be calibrated and preset 3D devices; The three-dimensional reconstruction is performed on the two-dimensional image data acquired by each of the two-dimensional cameras to be calibrated at the same time using the preset SFM (Structure From Motion) algorithm to obtain the initial point cloud data and the initial intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated. The initial intrinsic and extrinsic parameters are globally optimized using the two-dimensional image data to obtain the scale-free intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated; wherein, the scale-free nature of the scale-free intrinsic and extrinsic parameters means that the translation amount in the displacement between two frames of two-dimensional image data has an unknown unit; Based on the scale-free intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated, the initial point cloud data is triangulated and optimized to obtain sparse three-dimensional point cloud data of the target region. Based on the three-dimensional image data acquired by the preset three-dimensional device, generate the actual three-dimensional point cloud data of the target area; Register the sparse 3D point cloud data with the actual 3D point cloud data, obtain the transformation matrix between the point clouds, and obtain the relative extrinsic parameters between each of the 2D cameras to be calibrated based on the transformation matrix between the point clouds. The scale-free intrinsic and extrinsic parameters between the two-dimensional cameras to be calibrated are optimized by utilizing the relative extrinsic parameters between the two-dimensional cameras to be calibrated, thereby obtaining the actual intrinsic and extrinsic parameters of each two-dimensional camera to be calibrated. Wherein, actual intrinsic and extrinsic parameters represent intrinsic and extrinsic parameters with scale, and the transformation matrix is ​​a matrix used to indicate the relative pose transformation of the image coordinate system of the two-dimensional camera to be calibrated relative to the preset three-dimensional device, and the image coordinate system is a coordinate system constructed based on the two-dimensional image data of all the two-dimensional cameras to be calibrated.

2. The multi-camera calibration method according to claim 1, characterized in that, The acquisition of two-dimensional and three-dimensional image data of the same target area collected by multiple two-dimensional cameras to be calibrated and a preset three-dimensional device includes: Multiple two-dimensional cameras to be calibrated with relatively fixed poses continuously capture images of the same target area to obtain two-dimensional image data corresponding to each of the two-dimensional cameras to be calibrated. Acquire three-dimensional image data of the target area directly acquired by the preset three-dimensional device, wherein the preset three-dimensional device includes one or more combinations of a three-dimensional scanner, a depth camera, and a lidar.

3. The multi-camera calibration method according to claim 1, characterized in that, The process of obtaining the actual intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated and the relative extrinsic parameters between each of the two-dimensional cameras to be calibrated using the initial intrinsic and extrinsic parameters and the transformation matrix includes: The actual intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated and the relative extrinsic parameters between each of the two-dimensional cameras to be calibrated are obtained by using the scale-free intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated after optimization of the initial intrinsic and extrinsic parameters and the transformation matrix.

4. The multi-camera calibration method according to claim 3, characterized in that, The process of registering the sparse 3D point cloud data with the actual 3D point cloud data to obtain the transformation matrix between the point clouds includes: Using the actual 3D point cloud data as the target point cloud and the sparse 3D point cloud data as the registration point cloud, a transformation matrix between the point clouds is obtained.

5. A multi-camera calibration device, characterized in that, include: The first acquisition module is used to acquire two-dimensional image data and three-dimensional image data of the same target area collected by multiple two-dimensional cameras to be calibrated and preset three-dimensional devices; The second acquisition module is used to perform three-dimensional reconstruction on the two-dimensional image data acquired by each of the two-dimensional cameras to be calibrated at the same time using a preset SFM (Structure From Motion) algorithm, to obtain initial point cloud data and initial intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated. The initial intrinsic and extrinsic parameters are globally optimized using the two-dimensional image data to obtain the scale-free intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated; wherein, scale-free in the scale-free intrinsic and extrinsic parameters means that the translation amount in the displacement between two frames of two-dimensional image data has an unknown unit; the initial point cloud data is triangulated and optimized according to the scale-free intrinsic and extrinsic parameters of each of the two-dimensional cameras to be calibrated to obtain sparse three-dimensional point cloud data of the target region. The third acquisition module is used to generate actual three-dimensional point cloud data of the target area based on the three-dimensional image data acquired by the preset three-dimensional device. The calibration module is used to register the sparse 3D point cloud data with the actual 3D point cloud data, obtain the transformation matrix between the point clouds, and obtain the relative extrinsic parameters between each of the 2D cameras to be calibrated based on the transformation matrix between the point clouds; optimize the scale-free intrinsic and extrinsic parameters between each of the 2D cameras to be calibrated using the relative extrinsic parameters between each of the 2D cameras to be calibrated, and obtain the actual intrinsic and extrinsic parameters of each of the 2D cameras to be calibrated. Wherein, actual intrinsic and extrinsic parameters represent intrinsic and extrinsic parameters with scale, and the transformation matrix is ​​a matrix used to indicate the relative pose transformation of the image coordinate system of the two-dimensional camera to be calibrated relative to the preset three-dimensional device, and the image coordinate system is a coordinate system constructed based on the two-dimensional image data of all the two-dimensional cameras to be calibrated.

6. A multi-camera device, characterized in that, It includes multiple two-dimensional cameras and the calibration device as described in claim 5, wherein: The plurality of two-dimensional cameras are used to acquire two-dimensional image data of the target area; The calibration device is used to calibrate each of the two-dimensional cameras using the two-dimensional image data and the three-dimensional image data acquired by the preset three-dimensional device, and to obtain the intrinsic and extrinsic parameters of each of the two-dimensional cameras and the relative extrinsic parameters between each of the two-dimensional cameras.

7. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a multi-camera calibration program stored in the memory and executable on the processor. When the multi-camera calibration program is executed by the processor, it implements the steps of the multi-camera calibration method as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-camera calibration program, which, when executed by a processor, implements the steps of the multi-camera calibration method as described in any one of claims 1-4.

Images