Multi-view camera calibration method and apparatus, device, and storage medium

Abstract

The present application relates to a multi-view camera calibration method and apparatus, a device, and a storage medium. The method comprises: using a two-dimensional calibration board-based marking method to acquire intrinsic parameter data of each camera, and obtaining a camera intrinsic parameter matrix; using a single point calibration object-based marking method to acquire extrinsic parameter data of each camera, and obtaining a camera extrinsic parameter matrix; and on the basis of the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, displaying a camera calibration result. Embodiments of the present application use a marking method combining a two-dimensional calibration board-based marking method and a single point calibration object-based marking method to respectively mark intrinsic parameters and extrinsic parameters of the cameras, such that the intrinsic parameters and extrinsic parameters of completely opposite cameras in a large scene can be quickly acquired at low cost, thereby enabling calibration of cameras to be more convenient and less labor intensive while improving the precision of camera calibration.

Description

Multi-view camera calibration method, device, equipment and storage medium Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a multi-view camera calibration method, apparatus, device, and storage medium. Background Art

[0002] Camera calibration is to estimate the relative pose of the camera (i.e., calculate the internal and external parameters of the camera). In the existing technology, camera calibration methods mainly include three-dimensional standard object calibration, two-dimensional calibration plate calibration, one-dimensional calibration object calibration, and single-point calibration object calibration. Among them, the three-dimensional standard object calibration method has higher accuracy, but it is difficult to make and difficult to make and move in large scenes; the two-dimensional calibration plate calibration method is also accurate enough, but in large scenes, it requires the production of a large flat calibration plate, which is costly and difficult to calibrate horizontally relative cameras; the one-dimensional calibration object calibration method has slightly lower accuracy, low production cost, and is easy to move; the single-point calibration object calibration method is easy to make and easy to move, but can only calibrate external parameters.

[0003] For example, patent application number CN202010570282.4 discloses camera calibration using a freely movable single calibration point and an L-shaped rigid body for indicating the world coordinate system. Patent application number CN201310135850.8 discloses matching the image points of a one-dimensional calibration object using mutual distance as a condition based on the one-dimensional calibration object satisfying a constraint relationship. These calibration methods have the following disadvantages: only one calibration scheme is used, resulting in low accuracy, and both use the BA optimization algorithm, which takes a long time to calibrate and does not display the calibration results in real time.

[0004] Summary of the Invention

[0005] The present application provides a multi-view camera calibration method, apparatus, device and storage medium, which aims to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0006] In order to solve the above problems, this application provides the following technical solutions:

[0007] A multi-view camera calibration method, comprising:

[0008] The two-dimensional calibration plate marking method is used to obtain the intrinsic parameter data of each camera and obtain the camera intrinsic parameter matrix;

[0009] The single-point calibration object marking method is used to obtain the extrinsic parameter data of each camera and obtain the camera extrinsic parameter matrix;

[0010] The camera calibration result is displayed according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix.

[0011] The technical solution adopted in the embodiment of the present application also includes: the use of the two-dimensional calibration plate marking method to obtain the intrinsic parameter data of each camera is specifically:

[0012] Use a single camera to shoot the 2D calibration plate separately to obtain N 2D calibration plate images, where N is the number of cameras;

[0013] The N two-dimensional calibration plate images are converted into data formats, and calculations are performed on the converted images to obtain the intrinsic parameter data of each camera.

[0014] The technical solution adopted in the embodiment of the present application also includes: the single-point calibration object marking method is used to obtain the extrinsic parameter data of each camera, and the camera extrinsic parameter matrix is ​​specifically obtained as follows:

[0015] Fixing N cameras in different directions of a camera fixing frame respectively, and connecting the N cameras to a camera synchronous acquisition system respectively;

[0016] The camera synchronization acquisition system controls N cameras to synchronously shoot the same single-point calibration object to obtain N single-point calibration object images;

[0017] The pixel coordinates in each single-point calibration object image are extracted, the mapping between each camera is calculated based on the pixel coordinates, and the extrinsic parameter matrix between each camera is calculated based on the mapping between the cameras.

[0018] The technical solution adopted in the embodiment of the present application also includes: the single-point calibration object is a continuously moving long rod, and a marker that is distinguished from the background is provided at the end of the long rod.

[0019] The technical solution adopted by the embodiment of the present application further includes: after obtaining the extrinsic parameter data of each camera using the single-point calibration object marking method and obtaining the camera extrinsic parameter matrix, it also includes:

[0020] The camera extrinsic parameter matrix is ​​iteratively optimized using a reprojection error algorithm combined with a three-dimensional point consistency evaluation algorithm to obtain the optimal camera extrinsic parameter matrix.

[0021] The technical solution adopted in the embodiment of the present application also includes: the iterative optimization of the camera extrinsic parameter matrix by using the reprojection error algorithm combined with the three-dimensional point consistency evaluation algorithm is specifically as follows:

[0022] Calculate the average reprojection error of the N single-point calibration object images, reconstruct the three-dimensional points using the camera extrinsic parameter matrix, and calculate the variance of the three-dimensional points;

[0023] Determine whether the average reprojection error and the 3D point variance are both smaller than those in the previous iteration; if so, update the camera parameters; otherwise, do not update;

[0024] The next round of iterative optimization is performed until the number of iterations reaches the set number and the optimal camera extrinsic parameter matrix is ​​obtained.

[0025] The technical solution adopted in the embodiment of the present application further includes: after displaying the camera calibration result according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, it also includes:

[0026] Save the camera intrinsic parameter matrix and camera extrinsic parameter matrix after the iteration.

[0027] Another technical solution adopted in the embodiment of the present application is: a multi-view camera calibration method and device, comprising:

[0028] Intrinsic parameter data acquisition module: used to obtain the intrinsic parameter data of each camera using the two-dimensional calibration plate marking method to obtain the camera intrinsic parameter matrix;

[0029] External parameter data acquisition module: used to obtain the external parameter data of each camera using the single-point calibration object marking method to obtain the camera external parameter matrix;

[0030] Calibration display module: used to display the camera calibration results based on the camera intrinsic parameter matrix and the camera extrinsic parameter matrix.

[0031] Another technical solution adopted by the embodiment of the present application is: a device, the device comprising a processor and a memory coupled to the processor, wherein:

[0032] The memory stores program instructions for implementing the multi-view camera calibration method;

[0033] The processor is used to execute the program instructions stored in the memory to control the multi-view camera calibration method.

[0034] Another technical solution adopted in the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the multi-view camera calibration method.

[0035] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: the multi-view camera calibration method, device, equipment, and storage medium of the embodiments of the present application use a marking method that combines a two-dimensional calibration plate marking method and a single-point calibration object marking method to mark the internal and external parameters of the camera respectively, which can quickly and cheaply obtain the internal and external parameters completely relative to the camera in a large scene, while improving the accuracy of camera calibration, making camera calibration more convenient and labor-saving, and using an external parameter iterative optimization algorithm that combines reprojection error and three-dimensional point consistency evaluation to improve the robustness of camera external parameter optimization. At the same time, the embodiments of the present application can display the camera calibration results and reprojection error in real time, which helps the calibration personnel to more intuitively and quickly determine whether the camera calibration is successful. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a flow chart of a multi-view camera calibration method according to an embodiment of the present application;

[0037] FIG2 is a schematic diagram of a method for obtaining a camera intrinsic parameter matrix in an embodiment of the present application;

[0038] FIG3 is a schematic diagram of a method for obtaining a camera extrinsic parameter matrix according to an embodiment of the present application;

[0039] FIG4 is a schematic structural diagram of a multi-view camera calibration device according to an embodiment of the present application;

[0040] FIG5 is a schematic diagram of the device structure of an embodiment of the present application;

[0041] FIG6 is a schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] The terms "first," "second," and "third" in this application are used only for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of such features. In the description of this application, "multiple" means at least two, for example, two, three, etc., unless otherwise specifically defined. All directional indications in the embodiments of this application (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications also change accordingly. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products, or devices.

[0044] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0045] Please refer to Figure 1, which is a flow chart of a multi-view camera calibration method according to an embodiment of the present application. The multi-view camera calibration method according to an embodiment of the present application comprises the following steps:

[0046] S100: Initialize all cameras and set or read the shooting parameters of each camera;

[0047] In this step, the shooting parameters include but are not limited to the camera shutter speed and frame size.

[0048] S110: Enable the shooting function of all cameras;

[0049] S120: Acquire intrinsic parameter data of each camera using a two-dimensional calibration plate marking method to obtain a camera intrinsic parameter matrix;

[0050] In this step, as shown in Figure 2, there is a schematic diagram of the camera intrinsic parameter matrix acquisition method in the embodiment of the present application. Among them, 101 is a single camera, and the camera does not need to be fixed at this time; 102 is a two-dimensional calibration plate. The camera intrinsic parameter matrix acquisition method is specifically as follows: using the single camera 101 to respectively shoot the two-dimensional calibration plate 102 to obtain N two-dimensional calibration plate images, where N is the number of cameras; performing data format conversion on the N two-dimensional calibration plate images, and then using OpenCV or other toolkits to calculate the converted images to obtain the intrinsic parameter data of each camera. In the embodiment of the present application, the two-dimensional calibration plate 102 is a two-dimensional checkerboard calibration plate. The grid points of the two-dimensional checkerboard calibration plate can be set according to the actual application scenario. In the embodiment of the present application, the checkerboard grid points are set to 3*2. Specifically, other types of two-dimensional calibration plates can also be used.

[0051] S130: Acquire extrinsic parameter data of each camera using a single-point calibration object marking method to obtain a camera extrinsic parameter matrix;

[0052] In this step, please refer to Figure 3, which is a schematic diagram of a camera extrinsic parameter matrix acquisition method according to an embodiment of the present application. Specifically, the camera extrinsic parameter matrix acquisition method includes: fixing N cameras in different directions on a camera fixing frame 201, and connecting each of the N cameras to a camera synchronization acquisition system 202. The camera synchronization acquisition system 202 controls all cameras to synchronously capture a continuously moving single-point calibration object 203, obtaining N single-point calibration object images, extracting pixel coordinates from each single-point calibration object image, calculating the mapping between each camera based on the pixel coordinates, and then calculating the extrinsic parameter matrix between each camera using OpenCV or other toolkits. The camera fixing frame 201 is made of stainless steel or aluminum, providing stable support for the cameras; the single-point calibration object 203 is a long rod with a luminous point or other marker at the end to distinguish it from the background. Specifically, it can also be a marker of other types or shapes. It is understood that the number of cameras N is at least two. In Figure 3, only four cameras are used as an example. The specific number of cameras can be set according to the actual application scenario.

[0053] S140: Iteratively optimize the camera extrinsic parameter matrix using a reprojection error algorithm combined with a three-dimensional point consistency evaluation algorithm to obtain an optimal camera extrinsic parameter matrix;

[0054] In this step, the camera extrinsic parameter matrix is ​​iteratively optimized using the reprojection error algorithm combined with the 3D point consistency evaluation algorithm. Specifically, the following steps are performed: first, the average reprojection error of N single-point calibration object images is calculated, and then the 3D points are reconstructed using the camera extrinsic parameter matrix, and the 3D point variance (i.e., 3D point consistency evaluation) is calculated. It is determined whether the average reprojection error and the 3D point variance are both less than the previous iteration. If so, the camera parameters are updated, otherwise they are not updated. The next round of optimization is iteratively performed until the number of iterations reaches the set number, and the optimal camera extrinsic parameter matrix is ​​obtained. In the embodiment of the present application, the number of iterations set is 3-5 rounds, which can be set according to the actual application scenario.

[0055] S150: After each round of iteration, the camera calibration result is displayed according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix;

[0056] In this step, after each round of iteration, the camera calibration results can be displayed in graphical or digital form, which helps the calibrator to more intuitively and quickly determine whether the camera calibration is successful.

[0057] S160: Save the camera intrinsic parameter matrix and the optimal camera extrinsic parameter matrix after the iteration, and delete the camera intrinsic parameter matrix and the camera extrinsic parameter matrix before the iteration;

[0058] In this step, only the optimal camera intrinsic parameter matrix and camera extrinsic parameter matrix are saved, which is conducive to releasing system resources and reducing system memory usage.

[0059] Based on the above, the multi-view camera calibration method of the second embodiment of the present application uses a marking method that combines the two-dimensional calibration plate marking method and the single-point calibration object marking method to mark the internal and external parameters of the camera respectively. This method can quickly and cheaply obtain the internal and external parameters that are completely relative to the camera in a large scene. While improving the accuracy of camera calibration, it makes camera calibration more convenient and labor-saving. It also uses an external parameter iterative optimization algorithm that combines reprojection error and three-dimensional point consistency evaluation to improve the robustness of camera external parameter optimization. At the same time, the embodiment of the present application can display the camera calibration results and reprojection error in real time, which helps the calibration personnel to more intuitively and quickly determine whether the camera calibration is successful.

[0060] Please refer to FIG4 , which is a schematic diagram of the structure of a multi-view camera calibration method apparatus according to an embodiment of the present application. The multi-view camera calibration method apparatus 40 according to an embodiment of the present application includes:

[0061] Intrinsic parameter data acquisition module 41: used to obtain the intrinsic parameter data of each camera using a two-dimensional calibration plate marking method to obtain a camera intrinsic parameter matrix;

[0062] The extrinsic parameter data acquisition module 42 is used to obtain the extrinsic parameter data of each camera using a single-point calibration object marking method to obtain a camera extrinsic parameter matrix;

[0063] The calibration display module 43 is configured to display the camera calibration result according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix.

[0064] Please refer to Figure 5, which is a schematic diagram of the device structure of an embodiment of the present application. The device 50 includes:

[0065] A memory 51 storing executable program instructions;

[0066] a processor 52 connected to the memory 51;

[0067] The processor 52 is used to call the executable program instructions stored in the memory 51 and execute the following steps: using a two-dimensional calibration plate marking method to obtain the intrinsic parameter data of each camera to obtain a camera intrinsic parameter matrix; using a single-point calibration object marking method to obtain the extrinsic parameter data of each camera to obtain a camera extrinsic parameter matrix; and displaying the camera calibration results based on the camera intrinsic parameter matrix and the camera extrinsic parameter matrix.

[0068] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0069] Please refer to Figure 6, which is a schematic diagram of the structure of the storage medium of an embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 that can implement the following steps: using a two-dimensional calibration plate marking method to obtain the intrinsic parameter data of each camera to obtain a camera intrinsic parameter matrix; using a single-point calibration object marking method to obtain the extrinsic parameter data of each camera to obtain a camera extrinsic parameter matrix; displaying the camera calibration results based on the camera intrinsic parameter matrix and the camera extrinsic parameter matrix. Among them, the program instructions 61 can be stored in the above-mentioned storage medium in the form of a software product, including several instructions for enabling a device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store program instructions, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet. Among them, the server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0071] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-view camera calibration method, characterized in that: include: The two-dimensional calibration plate marking method is used to obtain the intrinsic parameter data of each camera and obtain the camera intrinsic parameter matrix; The single-point calibration object marking method is used to obtain the extrinsic parameter data of each camera and obtain the camera extrinsic parameter matrix; The camera calibration result is displayed according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix.

2. The multi-view camera calibration method according to claim 1, characterized in that: The method of using the two-dimensional calibration plate marking method to obtain the intrinsic parameter data of each camera is specifically as follows: Use a single camera to take pictures of the 2D calibration plate to obtain N 2D calibration plate pictures, where N is the number of cameras. The data formats of the N two-dimensional calibration plate images are converted, and the converted images are calculated to obtain the intrinsic parameter data of each camera.

3. The multi-view camera calibration method according to claim 2, characterized in that: The single-point calibration object marking method is used to obtain the external parameter data of each camera, and the camera external parameter matrix is ​​obtained as follows: Fixing N cameras in different directions of a camera fixing frame respectively, and connecting the N cameras to a camera synchronous acquisition system respectively; The camera synchronous acquisition system controls N cameras to synchronously shoot the same single-point calibration object to obtain N single-point calibration object images; The pixel coordinates in each single-point calibration object image are extracted, the mapping between the cameras is calculated according to the pixel coordinates, and the extrinsic parameter matrix between the cameras is calculated according to the mapping between the cameras.

4. The multi-view camera calibration method according to claim 3, characterized in that: The single-point calibration object is a long rod that moves continuously, and a marker that is distinguished from the background is arranged at the end of the long rod.

5. The multi-view camera calibration method according to any one of claims 1 to 4, characterized in that: After the single-point calibration object marking method is used to obtain the external parameter data of each camera and obtain the camera external parameter matrix, the method further includes: The camera extrinsic parameter matrix is ​​iteratively optimized using a reprojection error algorithm combined with a three-dimensional point consistency evaluation algorithm to obtain the optimal camera extrinsic parameter matrix.

6. The multi-view camera calibration method according to claim 5, characterized in that: The iterative optimization of the camera extrinsic parameter matrix by using the reprojection error algorithm combined with the three-dimensional point consistency evaluation algorithm is specifically as follows: Calculate the average reprojection error of the N single-point calibration object images, reconstruct the three-dimensional points using the camera extrinsic matrix, and calculate the variance of the three-dimensional points; Determine whether the average reprojection error and the 3D point variance are both smaller than those in the previous iteration, if so, update the camera parameters, otherwise do not update; The next round of iterative optimization is performed until the number of iterations reaches the set number and the optimal camera extrinsic parameter matrix is ​​obtained.

7. The multi-view camera calibration method according to claim 6, characterized in that: After displaying the camera calibration result according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, the method further includes: Save the camera intrinsic parameter matrix and camera extrinsic parameter matrix after the iteration.

8. A multi-view camera calibration method and device, characterized in that: include: Intrinsic parameter data acquisition module: used to obtain the intrinsic parameter data of each camera using a two-dimensional calibration plate marking method to obtain the camera intrinsic parameter matrix; External parameter data acquisition module: used to obtain the external parameter data of each camera using the single-point calibration object marking method to obtain the camera external parameter matrix; Calibration display module: used to display the camera calibration result according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix.

9. A device, characterized in that: The device comprises a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the multi-view camera calibration method according to any one of claims 1 to 7; The processor is used to execute the program instructions stored in the memory to control the multi-view camera calibration method.

10. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the multi-view camera calibration method described in any one of claims 1 to 7.

Images