Camera calibration method based on structural sampling of reference points

Abstract

A calibration method according to one embodiment comprises: a first step of detecting reference points of a checkerboard from images obtained by photographing the checkerboard, the detected reference points including calibration reference points and verification reference points; a second step of deriving parameters of known projection models on the basis of the calibration reference points; a third step of deriving accuracy (e calib , e test ) and suitability (S symbol ) on the basis of the verification reference points; and a fourth step of deriving, on the basis of all the reference points detected in the first step with respect to the projection model, parameters of a projection model having maximum suitability.

Description

Camera calibration method based on structural sampling of reference points

[0001] The present invention relates to camera calibration for finding parameters such as the focal length and distortion coefficient of a camera.

[0002]

[0003] Camera calibration (or camera correction technology) is the process of finding camera parameters such as focal length, principal point, and distortion coefficient that represent the projection model of the camera.

[0004] Camera calibration typically uses a checkerboard with a regular pattern, as shown in Figure 1. Because it often resembles a chessboard, it is also called a chessboard. In the case of a chessboard, the intersection points where black and white intersect are used for camera calibration. The regularity of the pattern allows for easy definition of the 3D position (unit: [m]) of reference points on the chessboard, and as shown in Figure 2, by detecting the reference points observed in the camera image, the positions (unit: [pixel]) of the intersection points on the image can be obtained. Therefore, using projection data of a 3D-2D pair, the parameters of the camera projection model, which expresses the relationship between the 3D space and the 2D image, can be estimated.

[0005] Although there are software such as OpenCV and Kalib, there are many choices for camera calibration, which poses many challenges for users.

[0006] First, when acquiring video data for a checkerboard, there's no way to verify whether the amount of acquired data is sufficient. Furthermore, it's difficult to determine whether the camera parameters derived from the acquired video are sufficiently accurate.

[0007] There are various camera projection models, and it is up to the user to choose the complexity (detail) of each projection model. For example, the focal length can be used as a separate parameter for the X-axis and Y-axis of the image (two parameters) or can use the same value (one parameter). Also, the principal point can be used as a camera parameter (two parameters) or fixed as the center point of the image (zero parameters).

[0008] The distortion coefficients offer a much wider range of choices. First, a lens distortion model must be selected. Furthermore, each distortion model also allows for adjustment of complexity (detail). Even for the radial distortion parameter of the most widely used Brown-Conrady distortion model, or ( , ) or ( , , ) allows you to select the complexity (detail) of the distortion representation. However, it is not possible to determine which of these selection options is suitable for my camera or data acquisition situation.

[0009] The problems of the described prior art can be summarized as follows.

[0010] 1) It is difficult to determine whether the camera parameters obtained through camera calibration are sufficiently accurate.

[0011] 2) It is difficult to determine whether the number of checkerboard images taken for camera calibration is sufficient.

[0012] 3) There are various camera projection models depending on the camera projection model and parameter complexity, but it is difficult to determine the optimal model for one's own camera and acquired data.

[0013]

[0014] The present invention was created against this technical background, and divides reference points into calibration reference points and test reference points through a newly proposed structural sampling technique in the present invention, performs camera calibration for various camera projection models and complexities using the calibration reference points and corresponding detection data, and derives the accuracy and suitability of each combination through the verification reference points and corresponding detection data to select the optimal camera projection model and parameter complexity.

[0015]

[0016] In order to solve the above technical problem, a calibration method of one embodiment comprises a first step of detecting reference points of a checkerboard from images of a checkerboard, wherein the detected reference points include a calibration reference point and a verification reference point, a second step of deriving parameters of known projection models based on the calibration reference point, and a second step of deriving accuracy (based on the verification reference point) , ) and goodness of fit ( ) and a fourth step of deriving parameters of the projection model based on all reference points detected in the first step for the projection model having the maximum fitness value.

[0017] In the fourth step, if the accuracy derived in the third step does not satisfy a threshold value set in advance, the process returns to the first step, changes the calibration reference point and verification reference point used in the second step to different ones, and then performs the second step.

[0018] The above accuracy is based on reprojection error.

[0019] The above fitness is judged using the accuracy by the above calibration reference point and the accuracy by the above verification reference point.

[0020] In addition, another embodiment of the present invention discloses a computing device implementing the above-described method and a recording medium recording a computer-readable program.

[0021]

[0022] The present invention can solve the problems of the prior art as follows.

[0023] The present invention can determine the accuracy of the camera calibration result as well as the suitability considering overfitting through verification data.

[0024] According to the present invention, since it is possible to determine the accuracy and suitability of the calibration result, it is possible to determine whether the number of currently acquired camera images is sufficient for the specified camera projection model and complexity.

[0025] According to the present invention, since it is possible to determine the accuracy and suitability of the calibration result, it is possible to select an optimal combination among various camera projection models and parameter complexities.

[0026]

[0027] Figure 1 shows an example of a checker board.

[0028] Figure 2 illustrates reference points observed in camera images.

[0029] FIG. 3 is a flowchart illustrating a camera calibration method according to one embodiment of the present invention.

[0030] Figure 4 illustrates structural sampling that can be used to examine extrapolation performance.

[0031] Figure 5 illustrates structural sampling that can be used to examine the performance of my shop.

[0032] Figure 6 is a schematic diagram explaining the reprojection error.

[0033] Fig. 7 is a block diagram illustrating a computing device (800) in which the calibration method of the above-described embodiment is executed.

[0034] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, detailed descriptions of well-known functions or components that may obscure the gist of the present invention will be omitted in the following description and the attached drawings. Additionally, throughout the specification, the term "including" a component does not exclude other components, unless specifically stated otherwise, but rather implies the inclusion of other components.

[0035] Additionally, while terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms may be used to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0036] The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning within the context of the relevant technology, and shall not be construed in an idealized or overly formal sense unless explicitly defined herein.

[0038]

[0039] The present invention uses a proposed structural sampling technique to divide a given reference point into a calibration reference point and a test reference point.

[0040] Camera calibration techniques proposed to date perform calibration for a single camera projection model and complexity using all reference points. That is, from the perspective of the present invention, existing methods use all reference points as calibration reference points and do not use verification reference points.

[0041] FIG. 3 is a flowchart illustrating a camera calibration method according to one embodiment of the present invention.

[0042] As illustrated in FIG. 3, a camera calibration method of one embodiment includes a first step (S10) of detecting reference points of a checkerboard from images taken of a checkerboard, and distinguishing the detected reference points from calibration reference points and verification reference points, a second step (S20) of deriving parameters of known projection models based on the calibration reference points, and an accuracy ( , ) and goodness of fit ( ) and a fourth step (S40) of deriving parameters of the projection model based on all reference points detected in the first step for the projection model that maximizes the fitness derived in the third step.

[0043] In one embodiment, if the accuracy derived from the third step does not satisfy the threshold value in the fourth step, the process returns to the first step and changes the calibration reference point and verification reference point used in the second step to different ones, and then the second step is performed.

[0044]

[0045] The calibration method described above, as an example, is similar to the process of dividing given data into training data and test data to avoid overfitting and select an appropriate model during the machine learning or deep learning process. In machine learning, training data is used for learning, and test data is used to measure the actual model's performance.

[0046] However, in machine learning (or deep learning), it is common to randomly sample without any special rules when dividing training data and test data.

[0047] In contrast, the present invention differs from random sampling in that it uses structured sampling, as described below, which is also a characteristic of the present invention. Structured sampling enables more efficient selection of an appropriate model from a pattern of reference points with regularity (compared to random sampling).

[0048] The present invention aims to move beyond the simple problem of conventional camera calibration and instead solve the problem of simultaneously selecting a camera projection model and performing camera calibration. Therefore, the present invention applies a method similar to machine learning's model selection and model parameter learning, dividing data into separate training and validation tasks.

[0049] The reference point data of the checkerboard has the characteristic of being regular, so it is more effective to use structured sampling that considers regularity than random sampling used in existing machine learning.

[0050] According to a calibration method of one embodiment, in the first step (S10), reference points of the checkerboard are detected from images taken of the checkerboard, and the detected reference points are divided into calibration reference points and verification reference points.

[0051] Here, the reference points of the checkerboard obtained from the image are sampled as calibration reference points and verification reference points, and this sampling is based on structural sampling as described above.

[0052] A detailed explanation of this structural sampling is as follows:

[0053] Most of the reference points on the checkerboard are arranged regularly, and by selecting test reference points considering this structural regularity, the extrapolation and interpolation performance of the camera calibration results can be conveniently examined.

[0054] First, a structural sampling technique that can examine extrapolation performance is to select reference points on the outer edge of the reference point pattern as verification reference points, as illustrated in Figure 4. Verification reference points can be selected from any of the upper, lower, left, or right edges of the checkerboard, or a combination thereof.

[0055] Using the same principle, interpolation performance can be examined by selecting points corresponding to the internal portion of the reference point pattern based on structural sampling, as illustrated in Figure 5. Similarly, reference points can be selected using up, down, left, or right, or a combination thereof.

[0056] In Figures 4 and 5, green represents the verification reference point and red represents the calibration reference point.

[0057] Meanwhile, in this embodiment, parameters of known projection models are derived based on calibration reference points (second step), and the accuracy of the projection models is obtained through verification reference points (third step).

[0058] A widely used camera projection model in this field is the Brown-Conrady model. The Brown-Conrady model expresses lens distortion as a polynomial function for each distance r from the camera center, as shown in Equation 1 below. The distorted rays are projected onto the image through a pinhole camera.

[0059]

[0060] In mathematical expression 1, ( represents the position on a normalized plane where the distance of the light incident on the camera is 1, represents the distortion of the corresponding position by a distortion model expressed as a term function. r is the straight-line distance from the center of the plane to the ray in the normalized plane. It is calculated as follows.

[0061] Another example of a camera projection model is the Kannala-Brandt model, which is widely used in cameras using fisheye lenses. The Kannala-Brandt model expresses lens distortion as a polynomial function of the incident angle θ of the light incident on the camera, as shown in Equation 2 below.

[0062]

[0063] In mathematical expression 2 is the angle of incidence of the light entering the camera. It is calculated as follows.

[0064] Camera calibration is the coefficient of the distortion model in Equations 1 and 2. , , , It is the process of finding parameters such as focal length and principal point location required for pinhole camera projection along with back.

[0065] And in one example, the calculation of accuracy in the third step can utilize a method widely used in the art called reprojection error.

[0066] Figure 6 is a schematic diagram illustrating reprojection errors. In Figure 6, blue represents detected reference points, red represents projected reference points, and green represents reprojection errors.

[0067] As illustrated, the reprojection error can be expressed as the Euclidian distance between the position (in pixels) of a reference point detected in an image and the position (in pixels) obtained by projecting the corresponding 3D reference point onto the image. A smaller reprojection error value is considered more accurate.

[0068] When M images are captured for N 3D reference points, the average reprojection error, which is an accuracy indicator, is as shown in the mathematical expression 3 below.

[0069]

[0070] In mathematical expression 3, is the location (unit: [pixel]) of the ith reference point detected in the jth image, is the 3D position of the ith reference point (unit: [meter]), and represents the rotation and translation of the i-th image, i.e., the camera pose at that time. c represents the parameters of the camera model, such as the focal length and distortion coefficient of the camera, The function represents a camera model that projects a given 3D point onto the corresponding image using camera model parameters and camera pose.

[0071]

[0072] In addition, the fitness in the third step can be judged by comparing the accuracy by the calibration reference point and the accuracy by the verification reference point. In general, the reprojection error by the calibration reference point ( ) is low, and the reprojection error by the verification reference point ( ) is relatively high. However, if the camera projection model generalizes well without overfitting, the validation reprojection error ( ) is the calibration reprojection error ( ) has a value of similar size. Therefore, the ratio of reprojection errors by two data How much projection error is due to the verification reference point ( ) is the projection error due to the calibration reference point ( ) can be judged to be close to the target. The values ​​of the two reprojection errors should be low, and the ratio of the two reprojection errors should be high to avoid overfitting. By synthesizing this, the fitness of the camera projection model can be expressed numerically as in the following mathematical expression 4.

[0073]

[0074] In mathematical formula 4 and represent weights representing the influence of the calibration reprojection error and the verification reprojection error, respectively.

[0075] As described above, the camera model and parameters that maximize the camera projection model fitness index can be judged as the appropriate camera projection model and parameters.

[0076] In one embodiment, the goodness-of-fit metric can determine not only whether the camera projection model is selected, but also whether the camera calibration results are sufficiently accurate and whether the number of given images is sufficient.

[0077] In other words, if the fitness index does not satisfy the desired performance, it can be judged as overfitting, and it can be seen that the complexity of the camera projection model should be reduced or the number of images capturing the checkerboard should be increased.

[0078]

[0079] Figure 7 is a block diagram illustrating a computing device (800) that executes the calibration method of the above-described embodiment, reconstructing a series of processing steps according to the above-described identification method from the perspective of hardware configuration. Therefore, to avoid redundancy in explanation, only an outline focusing on the functions and operations of each component will be provided.

[0080] The computing device (800) includes a memory (810) that stores a program for calibrating a camera, a processor (820) that calculates a projection model suitable for the camera and parameters of the projection model according to the program, and the processor (820) detects reference points of the checkerboard from images taken of the checkerboard, and the detected reference points are distinguished from calibration reference points and verification reference points. The processor (820) performs a first process of deriving parameters of known projection models based on the calibration reference points, and a second process of deriving parameters of known projection models based on the verification reference points. The processor (820) calculates the accuracy ( , ) and goodness of fit ( ) and a fourth process for deriving parameters of the projection model based on all reference points detected in the first step for the projection model that maximizes the fitness.

[0081] The processor (820) controls the operation to return to the first process and change the calibration reference point and verification reference point used in the second process to something else and proceed with the second process if the accuracy derived from the third process does not satisfy the threshold value or less in the fourth process.

[0082] The above processor (820) calculates accuracy based on a reprojection error, and compares the accuracy by the calibration reference point and the accuracy by the verification reference point.

[0083]

[0084] Meanwhile, the calibration method of the present invention described above can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes any type of recording device that stores data readable by a computer system.

[0085] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. Furthermore, computer-readable recording media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the present invention can be readily inferred by programmers in the technical field to which the present invention pertains.

[0086] The present invention has been described above, focusing on various embodiments thereof. Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiments should be considered illustrative rather than limiting. The scope of the present invention is set forth in the claims, not the foregoing description, and all differences within the scope equivalent thereto should be construed as being encompassed by the present invention.

Claims

1. A first step of detecting reference points of a checkerboard from images taken of a checkerboard, wherein the detected reference points include calibration reference points and verification reference points; A second step of deriving parameters of known projection models based on the above calibration reference points; Accuracy based on the above verification reference points ( , ) and goodness of fit ( ) in the third step; and A fourth step of deriving parameters of the projection model based on all reference points detected in the first step for the projection model that maximizes the fitness; A camera calibration method comprising:

2. In paragraph 1, A camera calibration method, wherein if the accuracy derived from the third step does not satisfy the threshold value in the fourth step, the method returns to the first step and changes the calibration reference point and verification reference point used in the second step to different ones and performs the second step.

3. In paragraph 1, A camera calibration method where the above accuracy is based on reprojection error.

4. In paragraph 1, A camera calibration method, wherein the above suitability is determined by comparing the accuracy by the above calibration reference point and the accuracy by the above verification reference point.

Images