Camera calibration method, equipment, storage medium and device

Abstract

The invention relates to the technical field of computer vision, and discloses a camera calibration method, camera calibration equipment, a storage medium and a camera calibration device. Compared with the prior art that a plurality of plane checkerboard images under various postures are manually shot for camera calibration, the optical center of a camera is controlled to be located at the same spatial position to move and shoot a target; according to the method and the device, automatic acquisition of the target image is realized, so that the camera calibration efficiency is improved, the influence of manual operation on the calibration result is avoided, and the camera calibration is performed on the to-be-calibrated camera in combination with the re-projection error and the structure consistency error, so that the calibration accuracy is improved. Therefore, the defect that the camera calibration error is large only depending on the re-projection error is overcome, and the camera calibration precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a camera calibration method, equipment, storage medium and device. Background Art

[0002] At present, the industry generally uses Zhang’s calibration method to calibrate cameras. Among them, Zhang’s calibration method is also known as Zhang Zhengyou calibration method, which is a technical solution to realize camera intrinsic parameter calibration by manually shooting multiple planar chessboard images in various postures.

[0003] However, the above camera calibration method has the following defects: 1. The acquisition process of multiple planar checkerboard images in various postures is time-consuming and labor-intensive, and the calibration results are greatly affected by manual operations; 2. When there is a difference between the actual imaging model and the theoretical imaging model (especially the distortion model) due to errors in camera design and assembly, the calibration error of Zhang's calibration method is often large. Summary of the invention

[0004] The main purpose of the present invention is to provide a camera calibration method, equipment, storage medium and device, aiming to solve the technical problems that the acquisition process of the existing camera calibration method is time-consuming and labor-intensive, and the calibration result has large errors.

[0005] To achieve the above object, the present invention provides a camera calibration method, which comprises the following steps:

[0006] Controlling the camera to be calibrated to shoot the target in multiple postures to obtain multiple target images, wherein the optical center of the camera to be calibrated in the multiple postures is located at the same spatial position;

[0007] Determine a reprojection error and a structural consistency error of the camera to be calibrated according to the multiple target images, wherein the structural consistency error is a difference in angles formed by a first connecting line and a second connecting line in the multiple target images, and the first connecting line and the second connecting line pass through the optical center;

[0008] The camera to be calibrated is calibrated based on the reprojection error and the structural consistency error.

[0009] Optionally, the performing camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error includes:

[0010] constructing a joint optimization model according to the reprojection error and the structural consistency error, wherein the joint optimization model is used to optimize camera parameters to reduce the reprojection error and the structural consistency error;

[0011] The camera parameters of the camera to be calibrated are iteratively optimized based on the joint optimization target model to perform camera calibration on the camera to be calibrated.

[0012] Optionally, the first connecting line is a connecting line between a first imaging point on the target image and the optical center, the second connecting line is a connecting line between a second imaging point on the target image and the optical center, the first imaging point is an imaging point of a first marking point on the target on the target image, and the second imaging point is an imaging point of a second marking point on the target on the target image; the structural consistency error is determined by the difference in equivalent physical quantities corresponding to the angle in the multiple target images.

[0013] Optionally, the performing camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error includes:

[0014] Constructing a reprojection optimization model according to the reprojection error, wherein the reprojection optimization model is used to optimize camera parameters to reduce the reprojection error;

[0015] Iteratively optimizing the camera parameters of the camera to be calibrated based on the reprojection optimization model;

[0016] Building a structural consistency optimization model according to the structural consistency error, wherein the structural consistency optimization model is used to optimize camera parameters to reduce the structural consistency error;

[0017] The camera parameters of the camera to be calibrated are iteratively optimized again based on the structural consistency optimization model to calibrate the camera to be calibrated.

[0018] Optionally, the performing camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error includes:

[0019] Constructing a reprojection optimization model according to the reprojection error, wherein the reprojection optimization model is used to optimize camera parameters to reduce the reprojection error;

[0020] Iteratively optimizing the camera parameters of the camera to be calibrated based on the reprojection optimization model;

[0021] The camera parameters are modified according to the distribution of the structural consistency error to calibrate the camera to be calibrated.

[0022] Optionally, controlling the camera to be calibrated to photograph the target in multiple postures to obtain multiple target images includes:

[0023] Get the optical center position of the camera to be calibrated;

[0024] The camera to be calibrated is controlled to rotate around the optical center of the camera to be calibrated according to the optical center position, so as to shoot the target in multiple postures to obtain multiple target images.

[0025] Optionally, the performing camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error includes:

[0026] The camera parameters of the camera to be calibrated are optimized based on the reprojection error and the structural consistency error to calibrate the camera to be calibrated, wherein the camera parameters include multiple rotation postures, the same translation variable corresponding to the multiple rotation postures, camera intrinsic parameters, and distortion parameters.

[0027] In addition, to achieve the above-mentioned purpose, the present invention also proposes a camera calibration device, which includes a memory, a processor, and a camera calibration program stored in the memory and executable on the processor, and the camera calibration program is configured to implement the camera calibration method described above.

[0028] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a camera calibration program is stored, and when the camera calibration program is executed by a processor, the camera calibration method as described above is implemented.

[0029] In addition, to achieve the above-mentioned purpose, the present invention also proposes a camera calibration device, which includes: an image acquisition module, an error determination module and a camera calibration module;

[0030] The image acquisition module is used to control the camera to be calibrated to shoot the target in multiple postures to obtain multiple target images, and the optical center of the camera to be calibrated in the multiple postures is located at the same spatial position;

[0031] The error determination module is used to determine the reprojection error and the structural consistency error of the camera to be calibrated according to the multiple target images, wherein the structural consistency error is the difference between the angles formed by the first connecting line and the second connecting line in the multiple target images, and the first connecting line and the second connecting line pass through the optical center;

[0032] The camera calibration module is used to perform camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error.

[0033] The present invention discloses controlling a camera to be calibrated to shoot a target in multiple postures to obtain multiple target images, wherein the optical center of the camera to be calibrated in multiple postures is located at the same spatial position, and the reprojection error and structural consistency error of the camera to be calibrated are determined according to the multiple target images, wherein the structural consistency error is the difference between the angle formed by a first connecting line and a second connecting line in multiple target images, and the first connecting line and the second connecting line pass through the optical center, and the camera to be calibrated is calibrated based on the reprojection error and the structural consistency error. Compared with the prior art in which a camera is calibrated by manually shooting multiple planar chessboard images in multiple postures, the present invention realizes automatic acquisition of target images by controlling the optical center of the camera to be located at the same spatial position to move and shoot the target, thereby improving the camera calibration efficiency, thereby avoiding the influence of manual operation on the calibration result, and since the present invention combines the reprojection error and the structural consistency error to jointly calibrate the camera to be calibrated, thereby overcoming the defect of large error of camera calibration based on the reprojection error alone, thereby improving the camera calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a structural schematic diagram of a camera calibration device in a hardware operating environment involved in an embodiment of the present invention;

[0035] Figure 2 It is a flowchart of a first embodiment of a camera calibration method of the present invention;

[0036] Figure 3 This is a schematic diagram of Zhang's calibration method;

[0037] Figure 4 A schematic diagram of structural consistency error of an embodiment of a camera calibration method of the present invention;

[0038] Figure 5 A flowchart of a specific solution of a camera calibration method according to an embodiment of the camera calibration method of the present invention;

[0039] Figure 6 A schematic diagram of error optimization of an embodiment of a camera calibration method of the present invention;

[0040] Figure 7 A schematic diagram of error optimization of an embodiment of a camera calibration method of the present invention;

[0041] Figure 8 It is a flowchart of a third embodiment of a camera calibration method of the present invention;

[0042] Fig. 9 FIG. 4 is a structural block diagram of a first embodiment of a camera calibration device according to the present invention.

[0043] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0045] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of a camera calibration device in the hardware operating environment involved in the embodiment of the present invention.

[0046] like Figure 1 As shown, the camera calibration device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), and the user interface 1003 may also include a standard wired interface and a wireless interface. The wired interface of the user interface 1003 may be a USB interface in the present invention. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable memory (Non-volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0047] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the camera calibration device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0048] like Figure 1 As shown, the memory 1005 identified as a computer storage medium may include an operating system, a network communication module, a user interface module, and a camera calibration program.

[0049] exist Figure 1 In the camera calibration device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the user device; the camera calibration device calls the camera calibration program stored in the memory 1005 through the processor 1001, and executes the camera calibration method provided in the embodiment of the present invention.

[0050] Based on the above hardware structure, an embodiment of the camera calibration method of the present invention is proposed.

[0051] Reference Figure 2 , Figure 2 The figure is a flow chart of the first embodiment of the camera calibration method of the present invention, and the first embodiment of the camera calibration method of the present invention is proposed.

[0052] It should be understood that, at present, the industry generally adopts Zhang's calibration method to calibrate cameras. Among them, Zhang's calibration method is Zhang Zhengyou calibration method, which is to manually shoot multiple planar chessboard images in various postures to realize the technical solution of camera intrinsic calibration.

[0053] For ease of understanding, refer to Figure 3 To explain, Figure 3 This is a schematic diagram of Zhang's calibration method. In the figure, the calibration process is as follows:

[0054] 1. The camera captures N target images with different postures (the position information of M markers on the target is known), so that the target image can cover the entire camera field of view, thereby obtaining several real target marker points p to image points p i The imaging correspondence is as follows:

[0055] 2. Preset / calculate the initial values ​​of camera internal and external parameters;

[0056] 3. Construct the target equation as follows: i ′ represents the theoretical position of the marker point p after the camera extrinsic parameters, distortion, and intrinsic parameters are transformed and projected into the image. i represents the actual imaging position detected from the image;

[0057]

[0058] 4. Use nonlinear optimization methods such as Gauss-Newton (GN) or Levenberg-Marquardt (LM) to solve the target equation and obtain the camera internal and external parameters corresponding to the minimized reprojection error. Then calibrate the camera using N external parameters (R, t) as well as the camera internal parameters and distortion.

[0059] However, the above camera calibration method has the following defects: 1. The acquisition process of multiple planar checkerboard images in various postures is time-consuming and labor-intensive, and the calibration results are greatly affected by manual operations; 2. When there is a difference between the actual imaging model and the theoretical imaging model (especially the distortion model) due to errors in camera design and assembly, the calibration error of Zhang's calibration method is often large.

[0060] Therefore, in order to overcome the above defects, in this embodiment, 1. the optical center of the camera is controlled to be located at the same spatial position and move to shoot the target, thereby realizing automatic acquisition of the target image, thereby improving the camera calibration efficiency and avoiding the influence of manual operation on the calibration result; 2. the camera to be calibrated is calibrated by combining the reprojection error and the structural consistency error, thereby overcoming the defect of large error in camera calibration based only on the reprojection error, thereby improving the camera calibration accuracy (especially for the case where the imaging model of the camera does not match the theoretical model due to design / assembly errors).

[0061] In a first embodiment, the camera calibration method comprises the following steps:

[0062] Step S10: controlling the camera to be calibrated to shoot the target in multiple postures to obtain multiple target images, wherein the optical center of the camera to be calibrated is located at the same spatial position in the multiple postures.

[0063] It can be understood that the executor of the method of this embodiment can be a camera calibration device with data processing, network communication and program running functions, for example, a terminal device such as a computer, or other electronic devices that can achieve the same or similar functions, and this embodiment does not limit this.

[0064] It should be understood that controlling the camera to be calibrated to shoot the target in multiple postures, and making the optical center of the camera to be calibrated in multiple postures to be located at the same spatial position can be controlling the camera to be calibrated to rotate around the optical center of the camera to be calibrated. Of course, it can also be achieved in other equivalent ways, for example, 1. Controlling the camera to be calibrated to first rotate around a certain spatial position, and then translate, so that the optical center of the camera to be calibrated returns to the original spatial position; 2. Controlling the camera to be calibrated to first translate, and then rotate around a certain spatial position, so that the optical center of the camera to be calibrated returns to the original spatial position; 3. Other forms of translation or rotation combinations to make the optical centers of cameras to be calibrated in different postures at the same spatial position, which is not limited in this embodiment.

[0065] It should be noted that in order to further improve the accuracy of camera calibration, in this embodiment, it can be limited that multiple target images are combined to cover the entire field of view of the camera to be calibrated. Of course, in this embodiment, it is also possible to limit the combination of multiple target images to cover part of the field of view of the camera to be calibrated according to other needs, and this implementation does not impose any restrictions on this.

[0066] Step S20: determining the reprojection error and the structural consistency error of the camera to be calibrated according to the multiple target images, wherein the structural consistency error is the difference between the angles formed by the first connecting line and the second connecting line in the multiple target images, and the first connecting line and the second connecting line pass through the optical center.

[0067] It should be noted that the reprojection error may be the difference between the theoretical projection position and the actual imaging position of the imaging point on the target image, wherein the imaging point may be the imaging point of the marking point on the target on the target image.

[0068] For ease of understanding, refer to Figure 4 This is for illustration only, but does not limit this solution. Figure 4 Figure 1 is a schematic diagram of structural consistency error. In the figure, since the optical center of the camera is located at the same spatial position when the camera is in multiple postures, the angle formed by the first line passing through the optical center (such as the AOA" line in the figure) and the second line passing through the optical center (such as the BOB" line in the figure) remains fixed before and after the camera switches postures. Therefore, the structural consistency error can be defined as the difference between the angles formed by the first line and the second line in multiple target images.

[0069] Furthermore, in order to improve the accuracy of the structural consistency error, in this embodiment, the first line is the line connecting the first imaging point on the target image and the optical center, the second line is the line connecting the second imaging point on the target image and the optical center, the first imaging point is the imaging point of the first marking point on the target on the target image, and the second imaging point is the imaging point of the second marking point on the target on the target image.

[0070] For ease of understanding, refer to Figure 4 This is for illustration only, but does not limit this solution. Figure 4 is a schematic diagram of structural consistency error. In the figure, since the optical center of the camera is located at the same spatial position when the camera is in multiple postures, the angle (such as the angle θ in the figure) formed by the line connecting any pair of marking points on the target (i.e., the first marking point and the second marking point) and the optical center of the camera remains fixed before and after the camera switches postures; according to the camera imaging principle, the angle (such as the angle θ′ in the figure) between the imaging point of the pair of marking points on the target image (i.e., the first imaging point and the second imaging point) and the line connecting the optical center of the camera is also fixed. Therefore, the structural consistency error can also be defined as the difference between the angle θ′ formed by the imaging point on the target image and the line connecting the optical center in multiple target images.

[0071] Of course, the structural consistency error can also be defined as the difference in equivalent physical quantities corresponding to the angle in multiple target images, for example, 1. The difference in the arc length of the arc corresponding to the angle θ′ in multiple target images; 2. The difference in the chord length of the arc corresponding to the angle θ′ in multiple target images; 3. The difference in trigonometric functions corresponding to the angle θ′ in multiple target images; 4. The difference in other equivalent physical quantities corresponding to the angle θ′ in multiple target images, which is not limited in this embodiment.

[0072] Step S30: calibrating the camera to be calibrated based on the reprojection error and the structural consistency error.

[0073] For ease of understanding, refer to Figure 5 This is for illustration only, but does not limit this solution. Figure 5 This is a specific flow chart of the camera calibration method. In the figure, a camera calibration scheme is proposed. High-precision camera intrinsic parameter and distortion calibration can be achieved only by constraining the camera to rotate around the optical center. The specific steps of the calibration scheme are as follows:

[0074] 1. The target (with M marking points on it and the relative position relationship between the marking points is known) is fixed at a specific position in space, and the rotating fixture drives the camera to rotate around the optical center to take N target images so that all target images can cover the entire camera field of view after combination;

[0075] 2. Preset / calculate the initial values ​​of camera internal and external parameters;

[0076] 3. Construct the optimization objective equation, which mainly consists of two parts:

[0077] J=J r +J s

[0078] In the formula, the first part J r is the 3D-2D reprojection error, Part II J s is the structural consistency error.

[0079] 4. Solve the optimization target equation through nonlinear optimization methods such as GN or LM to obtain the camera internal and external parameters corresponding to the minimized reprojection error and the minimized structural consistency error, and calibrate the camera through the camera internal and external parameters.

[0080] The present invention discloses controlling a camera to be calibrated to shoot a target in multiple postures to obtain multiple target images, wherein the optical center of the camera to be calibrated in multiple postures is located at the same spatial position, and the reprojection error and structural consistency error of the camera to be calibrated are determined according to the multiple target images, wherein the structural consistency error is the difference between the angle formed by a first connecting line and a second connecting line in multiple target images, and the first connecting line and the second connecting line pass through the optical center, and the camera to be calibrated is calibrated based on the reprojection error and the structural consistency error. Compared with the prior art in which a camera is calibrated by manually shooting multiple planar chessboard images in multiple postures, the present invention realizes automatic acquisition of target images by controlling the optical center of the camera to be located at the same spatial position to move and shoot the target, thereby improving the camera calibration efficiency, thereby avoiding the influence of manual operation on the calibration result, and since the present invention combines the reprojection error and the structural consistency error to jointly calibrate the camera to be calibrated, thereby overcoming the defect of large error of camera calibration based on the reprojection error alone, thereby improving the camera calibration accuracy.

[0081] Based on the above Figure 2The first embodiment shown provides a second embodiment of the camera calibration method of the present invention.

[0082] In the second embodiment, the step S20 includes:

[0083] constructing a joint optimization model according to the reprojection error and the structural consistency error, wherein the joint optimization model is used to optimize camera parameters to reduce the reprojection error and the structural consistency error;

[0084] The camera parameters of the camera to be calibrated are iteratively optimized based on the joint optimization target model to perform camera calibration on the camera to be calibrated.

[0085] It should be understood that in order to improve the camera calibration accuracy, in this embodiment, the reprojection error and the structural consistency error can be jointly optimized, wherein the joint optimization model can be an optimization objective equation. By solving the optimization objective equation through nonlinear optimization methods such as GN or LM, the camera intrinsic and extrinsic parameters corresponding to the minimized reprojection error and the minimized structural consistency error can be obtained, and the camera calibration is performed through the camera intrinsic and extrinsic parameters.

[0086] In the second embodiment, the step S20 further includes:

[0087] Constructing a reprojection optimization model according to the reprojection error, wherein the reprojection optimization model is used to optimize camera parameters to reduce the reprojection error;

[0088] Iteratively optimizing the camera parameters of the camera to be calibrated based on the reprojection optimization model;

[0089] Building a structural consistency optimization model according to the structural consistency error, wherein the structural consistency optimization model is used to optimize camera parameters to reduce the structural consistency error;

[0090] The camera parameters of the camera to be calibrated are iteratively optimized again based on the structural consistency optimization model to calibrate the camera to be calibrated.

[0091] It is understandable that, based on the optimization principle described above, in this embodiment, the camera calibration can also be performed in other equivalent ways. For example, the reprojection error and the structural consistency error can be optimized separately in a step-by-step iterative manner.

[0092] For ease of understanding, refer to Figure 6 This is for illustration only, but does not limit this solution. Figure 6 This is a schematic diagram of error optimization. In the figure, the reprojection error and structural consistency error are optimized separately through step-by-step iteration.

[0093] In the second embodiment, the step S20 further includes:

[0094] Constructing a reprojection optimization model according to the reprojection error, wherein the reprojection optimization model is used to optimize camera parameters to reduce the reprojection error;

[0095] Iteratively optimizing the camera parameters of the camera to be calibrated based on the reprojection optimization model;

[0096] The camera parameters are modified according to the distribution of the structural consistency error to calibrate the camera to be calibrated.

[0097] It should be understood that based on the optimization principle described above, in this embodiment, the camera calibration can also be performed in other equivalent ways. For example, the optimization iteration can be performed only based on the reprojection error, and then the structural consistency error can be calculated, and the calibration result of the camera intrinsic parameter / distortion can be directly corrected according to the distribution of the structural consistency error, thereby improving the calibration accuracy.

[0098] For ease of understanding, refer to Figure 7 This is for illustration only, but does not limit this solution. Figure 7 This is a schematic diagram of error optimization. In the figure, optimization iteration can be performed only based on the reprojection error, and then the structural consistency error is calculated. According to the distribution of the structural consistency error, the calibration result of the camera intrinsic parameter / distortion can be directly corrected to improve the calibration accuracy.

[0099] Reference Figure 8 , Figure 8 FIG. 3 is a flow chart of a third embodiment of a camera calibration method according to the present invention. Based on the above embodiments, a third embodiment of a camera calibration method according to the present invention is proposed.

[0100] In the third embodiment, the step S10 includes:

[0101] Step S101: Obtain the optical center position of the camera to be calibrated.

[0102] Step S102: controlling the camera to be calibrated to rotate around the optical center of the camera to be calibrated according to the optical center position, so as to photograph the target in multiple postures to obtain multiple target images.

[0103] It should be understood that in order to further simplify the camera calibration process, in this embodiment, the camera to be calibrated can be controlled to rotate around the optical center of the camera to be calibrated so that the camera can shoot the target in multiple postures. The optical center of the camera to be calibrated in multiple postures is located at the same spatial position, and the multiple target images are combined to cover the entire field of view of the camera to be calibrated.

[0104] In the third embodiment, the step S20 includes:

[0105] The camera parameters of the camera to be calibrated are optimized based on the reprojection error and the structural consistency error to calibrate the camera to be calibrated, wherein the camera parameters include multiple rotation postures, the same translation variable corresponding to the multiple rotation postures, camera intrinsic parameters, and distortion parameters.

[0106] It is understandable that since the camera rotates around the optical center during calibration, the relative position between the camera and the target corresponding to each image is fixed, that is, N camera postures can share one translation variable. By using iterative schemes such as GN / LM, minimizing the reprojection error and structural consistency error, optimizing N rotation postures, 1 translation variable, camera intrinsic parameters, and distortion, the number of camera position variables to be optimized can be reduced, which can improve the accuracy and stability of camera calibration.

[0107] In addition, an embodiment of the present invention further provides a storage medium, on which a camera calibration program is stored. When the camera calibration program is executed by a processor, the camera calibration method described above is implemented.

[0108] In addition, refer to Fig. 9 , an embodiment of the present invention also proposes a camera calibration device.

[0109] It should be understood that, at present, the industry generally adopts Zhang's calibration method to calibrate cameras. Among them, Zhang's calibration method is Zhang Zhengyou calibration method, which is to manually shoot multiple planar chessboard images in various postures to realize the technical solution of camera intrinsic calibration.

[0110] For ease of understanding, refer to Figure 3 To explain, Figure 3 This is a schematic diagram of Zhang's calibration method. In the figure, the calibration process is as follows:

[0111] 1. The camera captures N target images with different postures (the position information of M markers on the target is known), so that the target image can cover the entire camera field of view, thereby obtaining several real target marker points p to image points p i The imaging correspondence is as follows:

[0112] 2. Preset / calculate the initial values ​​of camera internal and external parameters;

[0113] 3. Construct the target equation as follows: i ′ represents the theoretical position of the marker point p after the camera extrinsic parameters, distortion, and intrinsic parameters are transformed and projected into the image. i represents the actual imaging position detected from the image;

[0114]

[0115] 4. Use nonlinear optimization methods such as Gauss-Newton (GN) or Levenberg-Marquardt (LM) to solve the target equation and obtain the camera internal and external parameters corresponding to the minimized reprojection error. Then calibrate the camera using N external parameters (R, t) as well as the camera internal parameters and distortion.

[0116] However, the above camera calibration method has the following defects: 1. The acquisition process of multiple planar checkerboard images in various postures is time-consuming and labor-intensive, and the calibration results are greatly affected by manual operations; 2. When there is a difference between the actual imaging model and the theoretical imaging model (especially the distortion model) due to errors in camera design and assembly, the calibration error of Zhang's calibration method is often large.

[0117] Therefore, in order to overcome the above defects, in this embodiment, 1. the optical center of the camera is controlled to be located at the same spatial position and move to shoot the target, thereby realizing automatic acquisition of the target image, thereby improving the camera calibration efficiency and avoiding the influence of manual operation on the calibration result; 2. the camera to be calibrated is calibrated by combining the reprojection error and the structural consistency error, thereby overcoming the defect of large error in camera calibration based only on the reprojection error, thereby improving the camera calibration accuracy (especially for the case where the imaging model of the camera does not match the theoretical model due to design / assembly errors).

[0118] In this embodiment, the camera calibration device includes: an image acquisition module 10, an error determination module 20 and a camera calibration module 30;

[0119] The image acquisition module 10 is used to control the camera to be calibrated to shoot the target in multiple postures to obtain multiple target images, and the optical center of the camera to be calibrated in the multiple postures is located at the same spatial position.

[0120] It should be understood that controlling the camera to be calibrated to shoot the target in multiple postures, and making the optical center of the camera to be calibrated in multiple postures to be located at the same spatial position can be controlling the camera to be calibrated to rotate around the optical center of the camera to be calibrated. Of course, it can also be achieved in other equivalent ways, for example, 1. Controlling the camera to be calibrated to first rotate around a certain spatial position, and then translate, so that the optical center of the camera to be calibrated returns to the original spatial position; 2. Controlling the camera to be calibrated to first translate, and then rotate around a certain spatial position, so that the optical center of the camera to be calibrated returns to the original spatial position; 3. Other forms of translation or rotation combinations to make the optical centers of cameras to be calibrated in different postures at the same spatial position, which is not limited in this embodiment.

[0121] It should be noted that in order to further improve the accuracy of camera calibration, in this embodiment, it can be limited that multiple target images are combined to cover the entire field of view of the camera to be calibrated. Of course, in this embodiment, it is also possible to limit the combination of multiple target images to cover part of the field of view of the camera to be calibrated according to other needs, and this implementation does not impose any restrictions on this.

[0122] The error determination module 20 is used to determine the reprojection error and structural consistency error of the camera to be calibrated based on the multiple target images, wherein the structural consistency error is the difference in the angle formed by the first connecting line and the second connecting line in the multiple target images, and the first connecting line and the second connecting line pass through the optical center.

[0123] It should be noted that the reprojection error may be the difference between the theoretical projection position and the actual imaging position of the imaging point on the target image, wherein the imaging point may be the imaging point of the marking point on the target on the target image.

[0124] For ease of understanding, refer to Figure 4 This is for illustration only, but does not limit this solution. Figure 4 Figure 1 is a schematic diagram of structural consistency error. In the figure, since the optical center of the camera is located at the same spatial position when the camera is in multiple postures, the angle formed by the first line passing through the optical center (such as the AOA" line in the figure) and the second line passing through the optical center (such as the BOB" line in the figure) remains fixed before and after the camera switches postures. Therefore, the structural consistency error can be defined as the difference between the angles formed by the first line and the second line in multiple target images.

[0125] Furthermore, in order to improve the accuracy of the structural consistency error, in this embodiment, the first line is the line connecting the first imaging point on the target image and the optical center, the second line is the line connecting the second imaging point on the target image and the optical center, the first imaging point is the imaging point of the first marking point on the target on the target image, and the second imaging point is the imaging point of the second marking point on the target on the target image.

[0126] For ease of understanding, refer to Figure 4 This is for illustration only, but does not limit this solution. Figure 4is a schematic diagram of structural consistency error. In the figure, since the optical center of the camera is located at the same spatial position when the camera is in multiple postures, the angle (such as the angle θ in the figure) formed by the line connecting any pair of marking points on the target (i.e., the first marking point and the second marking point) and the optical center of the camera remains fixed before and after the camera switches postures; according to the camera imaging principle, the angle (such as the angle θ′ in the figure) between the imaging point of the pair of marking points on the target image (i.e., the first imaging point and the second imaging point) and the line connecting the optical center of the camera is also fixed. Therefore, the structural consistency error can also be defined as the difference between the angle θ′ formed by the imaging point on the target image and the line connecting the optical center in multiple target images.

[0127] Of course, the structural consistency error can also be defined as the difference in equivalent physical quantities corresponding to the angle in multiple target images, for example, 1. The difference in the arc length of the arc corresponding to the angle θ′ in multiple target images; 2. The difference in the chord length of the arc corresponding to the angle θ′ in multiple target images; 3. The difference in trigonometric functions corresponding to the angle θ′ in multiple target images; 4. The difference in other equivalent physical quantities corresponding to the angle θ′ in multiple target images, which is not limited in this embodiment.

[0128] The camera calibration module 30 is used to perform camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error.

[0129] For ease of understanding, refer to Figure 5 This is for illustration only, but does not limit this solution. Figure 5 This is a specific flow chart of the camera calibration method. In the figure, a camera calibration scheme is proposed. High-precision camera intrinsic parameter and distortion calibration can be achieved only by constraining the camera to rotate around the optical center. The specific steps of the calibration scheme are as follows:

[0130] 1. The target (with M marking points on it and the relative position relationship between the marking points is known) is fixed at a specific position in space, and the rotating fixture drives the camera to rotate around the optical center to take N target images so that all target images can cover the entire camera field of view after combination;

[0131] 2. Preset / calculate the initial values ​​of camera internal and external parameters;

[0132] 3. Construct the optimization objective equation, which mainly consists of two parts:

[0133] J=J r +J s

[0134] In the formula, the first part J r is the 3D-2D reprojection error, Part II J s is the structural consistency error.

[0135] 4. Solve the optimization target equation through nonlinear optimization methods such as GN or LM to obtain the camera internal and external parameters corresponding to the minimized reprojection error and the minimized structural consistency error, and calibrate the camera through the camera internal and external parameters.

[0136] The present invention discloses controlling a camera to be calibrated to shoot a target in multiple postures to obtain multiple target images, wherein the optical center of the camera to be calibrated in multiple postures is located at the same spatial position, and the reprojection error and structural consistency error of the camera to be calibrated are determined according to the multiple target images, wherein the structural consistency error is the difference between the angle formed by a first connecting line and a second connecting line in multiple target images, and the first connecting line and the second connecting line pass through the optical center, and the camera to be calibrated is calibrated based on the reprojection error and the structural consistency error. Compared with the prior art in which a camera is calibrated by manually shooting multiple planar chessboard images in multiple postures, the present invention realizes automatic acquisition of target images by controlling the optical center of the camera to be located at the same spatial position to move and shoot the target, thereby improving the camera calibration efficiency, thereby avoiding the influence of manual operation on the calibration result, and since the present invention combines the reprojection error and the structural consistency error to jointly calibrate the camera to be calibrated, thereby overcoming the defect of large error of camera calibration based on the reprojection error alone, thereby improving the camera calibration accuracy.

[0137] Other embodiments or specific implementations of the camera calibration device of the present invention may refer to the above-mentioned method embodiments, which will not be described in detail here.

[0138] It should be noted that in the technical solution of this specification, the operations on the data involved are in compliance with relevant regulations and do not violate public order and good morals. For example, operations on data are performed on the premise of obtaining user authorization. In this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0139] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0141] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A camera calibration method, characterized in that: The camera calibration method comprises: Controlling the camera to be calibrated to shoot the target in multiple postures to obtain multiple target images, wherein the optical center of the camera to be calibrated in the multiple postures is located at the same spatial position; Determine a reprojection error and a structural consistency error of the camera to be calibrated according to the multiple target images, wherein the structural consistency error is a difference in angles formed by a first connecting line and a second connecting line in the multiple target images, and the first connecting line and the second connecting line pass through the optical center; The camera to be calibrated is calibrated based on the reprojection error and the structural consistency error.

2. The camera calibration method according to claim 1, characterized in that: The performing camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error includes: constructing a joint optimization model according to the reprojection error and the structural consistency error, wherein the joint optimization model is used to optimize camera parameters to reduce the reprojection error and the structural consistency error; The camera parameters of the camera to be calibrated are iteratively optimized based on the joint optimization target model to perform camera calibration on the camera to be calibrated.

3. The camera calibration method according to claim 1, wherein: The first connecting line is a connecting line between a first imaging point on the target image and the optical center, the second connecting line is a connecting line between a second imaging point on the target image and the optical center, the first imaging point is an imaging point of a first marking point on the target on the target image, and the second imaging point is an imaging point of a second marking point on the target on the target image; the structural consistency error is determined by the difference in equivalent physical quantities corresponding to the angle in the multiple target images.

4. The camera calibration method according to claim 1, wherein: The performing camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error includes: Constructing a reprojection optimization model according to the reprojection error, wherein the reprojection optimization model is used to optimize camera parameters to reduce the reprojection error; Iteratively optimizing the camera parameters of the camera to be calibrated based on the reprojection optimization model; Building a structural consistency optimization model according to the structural consistency error, wherein the structural consistency optimization model is used to optimize camera parameters to reduce the structural consistency error; The camera parameters of the camera to be calibrated are iteratively optimized again based on the structural consistency optimization model to calibrate the camera to be calibrated.

5. The camera calibration method according to claim 1, wherein: The performing camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error includes: Constructing a reprojection optimization model according to the reprojection error, wherein the reprojection optimization model is used to optimize camera parameters to reduce the reprojection error; Iteratively optimizing the camera parameters of the camera to be calibrated based on the reprojection optimization model; The camera parameters are modified according to the distribution of the structural consistency error to calibrate the camera to be calibrated.

6. The camera calibration method according to any one of claims 1 to 5, characterized in that: The method of controlling the camera to be calibrated to photograph the target in multiple postures to obtain multiple target images includes: Get the optical center position of the camera to be calibrated; The camera to be calibrated is controlled to rotate around the optical center of the camera to be calibrated according to the optical center position, so as to shoot the target in multiple postures to obtain multiple target images.

7. The camera calibration method according to claim 6, wherein: The performing camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error includes: The camera parameters of the camera to be calibrated are optimized based on the reprojection error and the structural consistency error to calibrate the camera to be calibrated, wherein the camera parameters include multiple rotation postures, the same translation variable corresponding to the multiple rotation postures, camera intrinsic parameters, and distortion parameters.

8. A camera calibration device, characterized in that: The camera calibration device comprises: a memory, a processor, and a camera calibration program stored in the memory and executable on the processor, wherein the camera calibration program implements the camera calibration method according to any one of claims 1 to 7 when executed by the processor.

9. A storage medium, characterized in that: The storage medium stores a camera calibration program, and when the camera calibration program is executed by the processor, the camera calibration method according to any one of claims 1 to 7 is implemented.

10. A camera calibration device, characterized in that: The camera calibration device comprises: an image acquisition module, an error determination module and a camera calibration module; The image acquisition module is used to control the camera to be calibrated to shoot the target in multiple postures to obtain multiple target images, and the optical center of the camera to be calibrated in the multiple postures is located at the same spatial position; The error determination module is used to determine the reprojection error and the structural consistency error of the camera to be calibrated according to the multiple target images, wherein the structural consistency error is the difference between the angles formed by the first connecting line and the second connecting line in the multiple target images, and the first connecting line and the second connecting line pass through the optical center; The camera calibration module is used to perform camera calibration on the camera to be calibrated based on the reprojection error and the structural consistency error.

Images