Camera calibration device and method thereof

Abstract

An apparatus for camera calibration includes: a memory for storing code for camera calibration; and a processor for executing the code to perform operations for camera calibration.

Description

Technical Field

[0001] This disclosure relates to camera calibration apparatus and methods thereof, and more specifically, to camera calibration apparatus and methods thereof configured to obtain camera intrinsic parameters using a plurality of feature points in an acquired image and to perform calibration using those parameters. Background Technology

[0002] Camera calibration refers to the process of adjusting a camera so that it can correctly capture and interpret images. This process involves correcting lens distortion, color distortion, and geometric distortion to improve image accuracy. The main goal is to accurately reproduce objects in the real world, which plays a crucial role in various fields such as computer vision, robotics, augmented reality, virtual reality, and autonomous vehicles.

[0003] Figure 1 A pinhole camera model is shown to illustrate camera calibration.

[0004] refer to Figure 1 The diagram shows a total of four coordinate systems: camera coordinate system, normalized coordinate system, pixel coordinate system, and world coordinate system, each shown starting from the bottom right.

[0005] The world coordinate system and camera coordinate system are 3D coordinate systems, while the normalized coordinate system and pixel coordinate system are 2D coordinate systems. The world coordinate system is a reference system used to represent the position of an object, while the camera coordinate system is a camera-based reference system. The pixel coordinate system, also known as the image coordinate system, is the coordinate system of the image that is actually visible to the eye. The pixel coordinate system corresponds to the image plane with its origin at the top left corner and extending to the right and downwards.

[0006] A point P = (X,Y,Z) in 3D space is projected onto a point pimg = (x,y) on the image plane through the focal point of a camera or lens.

[0007] The normalized coordinate system corresponds to the image coordinate system in which the influence of the camera's intrinsic parameters is removed. The normalized coordinate system is a unitless (normalized) coordinate system and defines a fictitious image plane at a distance of 1 from the camera's focus. A point in the normalized coordinate system is defined as P' = (u, v). Therefore, if the camera's intrinsic parameters are known, the transformation between image (pixel) coordinates and normalized coordinates becomes possible.

[0008] [Mathematical Expression 1]

[0009] In mathematical formula 1, fx is the focal length in the horizontal direction, fy is the focal length in the vertical direction, and cx and cy are principal points, which represent the image (pixel) coordinates where the optical axis intersects the image (pixel) plane.

[0010] The rotation and translation transformation matrix ([R|t]) used to transform the world coordinate system to the camera coordinate system can be modeled as follows.

[0011] [Mathematical Expression 2]

[0012] In mathematics, camera calibration is the process of finding the transformation relationship between 3D spatial coordinates and 2D image coordinates, or the parameters that describe the transformation relationship.

[0013] The matrix in mathematical formula 1 is called the intrinsic parameter, while the matrix [R|t] in mathematical formula 2 is called the extrinsic parameter.

[0014] When using a wide-angle or ultra-wide-angle lens with a large field of view, a wider range can be captured, but this results in relatively more significant image distortion.

[0015] Such image distortion is not only a visual problem, but also becomes particularly problematic when precise numerical calculations are required through image analysis. For example, if image coordinates are transformed into physical coordinates to determine the actual location of an object detected from the image, significant errors may occur depending on the degree of image distortion.

[0016] Camera distortion (i.e., lens distortion) includes radial distortion and tangential distortion.

[0017] If there is no lens distortion, the points (Xc,Yc,Zc) in 3D space are projected onto the points (x_n_u,y_n_u) on the normalized image plane through central projection.

[0018] [Mathematical Expression 3]

[0019] If the normalized coordinates reflecting lens distortion are represented as (x_n_d, y_n_d), then the lens distortion model can be represented as follows.

[0020] [Mathematical Expression 4]

[0021] In mathematical formula 4,

[0022] In mathematical formula 4, the first term on the right-hand side represents radial distortion, and the second term represents tangential distortion. K1, K2, and K3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, and r_u is the distance (radius) from the center (principal point) when there is no distortion.

[0023] Using the intrinsic parameters from Formula 1, the normalized coordinates reflecting lens distortion can be represented as image pixel coordinates (x_p_d, y_p_d).

[0024] [Mathematical Expression 5]

[0025] In mathematical formula 5, skew_cf x It is the asymmetric coefficient, which is usually set to 0 (i.e., omitted).

[0026] In other words, coordinates on the normalized image plane can be transformed into coordinates on the image plane that reflect distortion.

[0027] This relationship allows for the correction of image distortion or distortion in image coordinates. The correction of distortion in image coordinates will be explained below.

[0028] In practical image processing, obtaining the corrected image coordinates may be far more important than obtaining the corrected image itself.

[0029] However, obtaining corrected image coordinates from distorted image coordinates without a closed solution is a challenging problem, and therefore, an approximate solution is usually required.

[0030] One method to find the corrected image coordinates p_u from the distorted image coordinates p_d is to assume p_u = p_d and distort p_u using Equation 4. Then, the error between the distorted point and the actual p_d is calculated, and this error is applied in reverse to p_u. For example, if the distorted point from p_u is δ larger than p_d, then p_u is reduced by δ. This process is repeated until the error is less than a threshold.

[0031] In summary, camera calibration or camera distortion correction uses pairs of image (pixel) coordinates and their corresponding 3D coordinates (i.e., world coordinates) to calculate intrinsic or extrinsic parameters. Based on these parameters, distortions in the image coordinates can be corrected.

[0032] Although camera calibration and camera distortion correction have been briefly described, much material has been published prior to the submission of this application and can be consulted.

[0033] Traditional camera calibration uses images captured by a checkerboard or 3D fixture structure placed at close range, along with the known 3D coordinates of the intersections on the checkerboard and their corresponding image coordinates.

[0034] This disclosure aims to propose a method for calibrating the intrinsic parameters of a camera without using known information such as checkerboard patterns. Summary of the Invention

[0035] Therefore, this disclosure relates to camera calibration equipment and methods that substantially eliminate one or more problems caused by the limitations and disadvantages of related technologies.

[0036] The purpose of this disclosure is to provide a method for calibrating camera intrinsic parameters.

[0037] Those skilled in the art will understand that the purposes that can be achieved by this disclosure are not limited to those specifically described above, and that the above and other purposes that can be achieved by this disclosure will become clearer from the following detailed description.

[0038] To achieve these and other advantages and in accordance with the purposes of this disclosure, as embodied and broadly described herein, a camera calibration apparatus is provided. The apparatus may include: a memory configured to store code for camera calibration; and a processor configured to execute the code to perform operations for camera calibration.

[0039] Additionally or alternatively, operations for camera calibration may include obtaining two-dimensional position estimates of at least a pair of points of an object from the acquired image.

[0040] Additionally or alternatively, operations for camera calibration may include obtaining a three-dimensional position estimate of the pair of points.

[0041] Additionally or alternatively, operations for camera calibration may include obtaining two-dimensional transformation information of the acquired three-dimensional position estimate.

[0042] Additionally or alternatively, operations for camera calibration may include calculating the error between two-dimensional transformation information and two-dimensional position estimation.

[0043] Additionally or alternatively, the operation for camera calibration may include adjusting the parameters used for camera calibration based on errors or determining the parameters used for camera calibration as the final parameters.

[0044] Alternatively or additionally, a three-dimensional position estimate can be obtained based on a two-dimensional position estimate and parameters used for camera calibration.

[0045] Additionally or alternatively, initial values ​​for parameters used for camera calibration may be used, and these initial values ​​may include a focal length corresponding to the product of the width of the acquired image and a predetermined value, and / or a principal point corresponding to the center of the acquired image.

[0046] Additionally or alternatively, operations for camera calibration may include adjusting parameters for camera calibration based on an error exceeding a threshold.

[0047] Additionally or alternatively, the operation for camera calibration may include determining the parameters used for camera calibration as the final parameters based on the error being less than a threshold.

[0048] Additionally or alternatively, operations for camera calibration may include calculating errors by comparing the distance between points of the pair of points or the position of the pair of points with two-dimensional transformation information and two-dimensional position estimation.

[0049] Additionally or alternatively, operations for camera calibration may include a user interface on a display that outputs the pose of the object for guidance, based on a two-dimensional position estimate of the pair of points that is unavailable or difficult to obtain.

[0050] Additionally or alternatively, the operations for camera calibration may include re-executing the code for camera calibration based on the detection of a connection to a new camera.

[0051] Additionally or alternatively, operations for camera calibration may include executing code for camera calibration at multiple discrete zoom levels within a zoom level range supported by the camera.

[0052] Additionally or alternatively, operations for camera calibration may include determining the final parameters by executing code for camera calibration on a predetermined number of images.

[0053] In another aspect of this disclosure, a method for camera calibration is provided. This method can be performed by a camera calibration device.

[0054] Additionally or alternatively, the method may include obtaining a two-dimensional position estimate of at least one pair of points of the object from the acquired image.

[0055] Additionally or alternatively, the method may include obtaining a three-dimensional position estimate of the pair of points.

[0056] Additionally or alternatively, the method may include obtaining two-dimensional transformation information of the obtained three-dimensional position estimate.

[0057] Additionally or alternatively, the method may include calculating the error between the two-dimensional transformation information and the two-dimensional position estimate.

[0058] Additionally or alternatively, the method may include: adjusting the parameters used for camera calibration based on the error, or determining the parameters used for camera calibration as the final parameters.

[0059] In another aspect of this disclosure, a computer-readable medium is provided for storing code configured to be executed by a computer or processor performing the methods described above for camera calibration.

[0060] In another aspect of this disclosure, a computer-readable medium storing code configured to be executed by a computer or processor performing operations for camera calibration is provided. These operations may include: outputting a user interface on a display to guide a user's pose to obtain a user image; obtaining at least one pair of points corresponding to the user from the obtained user image; determining parameters for camera calibration based on the at least one pair of points; and outputting a three-dimensional position estimation result or two-dimensional transformation information based on the three-dimensional position estimation result, based on the pair of points and the determined parameters.

[0061] Those skilled in the art will understand that the above aspects of this disclosure are merely a part of the various embodiments of this disclosure, and various modifications and alternatives can be developed from the following technical features of this disclosure.

[0062] This disclosure has the following effects.

[0063] According to this disclosure, camera parameters can be calibrated or updated to values ​​close to actual values. Furthermore, the pose or position of an object can be accurately estimated based on the calibrated or updated camera parameters.

[0064] According to this disclosure, the obtained parameters can be used to perform camera parameter acquisition and object pose or position estimation without depending on objects other than the target object.

[0065] The effects of this disclosure are not limited to those specifically described above, and any other effects not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains from the following detailed description. Attached Figure Description

[0066] The accompanying drawings are included to provide a further understanding of this disclosure and are incorporated in and constitute a part of this application. The drawings illustrate embodiments of the disclosure and, together with the description, serve to illustrate the principles of the disclosure. In the drawings:

[0067] Figure 1 A pinhole camera model is shown to illustrate camera calibration;

[0068] Figure 2 A flowchart of a method for camera calibration according to this disclosure is shown;

[0069] Figure 3 (a)- Figure 3 (c) shows the results of projecting the joint position estimate of the object onto the captured image based on the camera intrinsic parameters obtained through camera calibration;

[0070] Figure 4 (a)- Figure 4 (b) shows an environment with camera calibration in the background and an environment without camera calibration in the background;

[0071] Figure 5 and Figure 6 (a)- Figure 6 (b) An example of a user scenario for camera calibration according to this disclosure is shown;

[0072] Figure 7 (a)- Figure 7 (b) illustrates a second user scenario example for camera calibration according to this disclosure;

[0073] Figure 8 (a)- Figure 8 (c) An example of a third user scenario for camera calibration according to this disclosure is shown;

[0074] Figure 9 An example of a fourth user scenario for camera calibration according to this disclosure is shown;

[0075] Figure 10 (a)- Figure 10 (b) shows a fifth user scenario example for camera calibration according to this disclosure;

[0076] Figure 11 (a)- Figure 11 (c) A sixth user scenario example for camera calibration according to this disclosure is shown;

[0077] Figure 12 (a)- Figure 12 (b) shows an example of a seventh user scenario for camera calibration according to this disclosure;

[0078] Figure 13 (a)- Figure 13 (b) Shows an image of an object (user) in a specific pose and the estimated three-dimensional joint positions of the object (user) derived from the image;

[0079] Figure 14 (a)- Figure 14 (b) is a graph used to compare the three-dimensional estimation results obtained based on the initial camera intrinsics with the three-dimensional estimation results obtained based on the camera intrinsics adjusted by camera calibration according to this disclosure;

[0080] Figure 15 (a)- Figure 15 (d) is a graph used to compare the 3D estimation results obtained based on actual camera intrinsics and initial camera intrinsics with the 3D estimation results obtained based on camera intrinsics adjusted by camera calibration according to this disclosure; and

[0081] Figure 16 A block diagram of a camera calibration device according to this disclosure is shown. Detailed Implementation

[0082] In the following description, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification, identical or equivalent components will be provided with the same reference numerals and their descriptions will not be repeated. The suffixes “module” and “unit” used for components in the following description are assigned or used for ease of description and do not inherently have different meanings or functions. The use of suffixes is merely for ease of reference and should not be considered as conveying a unique distinction in meaning or function. Detailed descriptions of related technologies will be omitted if it is believed that they obscure the gist of the embodiments disclosed in this specification. It should be understood that the accompanying drawings are merely to provide a better understanding of the embodiments disclosed herein, and the technical concepts of this disclosure are not limited to the drawings. Therefore, this disclosure should be construed as covering all modifications, equivalents, and substitutions within the scope of the concepts and technologies disclosed herein.

[0083] While terms such as "first" and "second" can be used to describe various components, the components mentioned above are not limited by these terms. These terms are only used to distinguish one component from another.

[0084] When it is said that a component is "connected" or "coupled" to another component, it can be directly connected or coupled to the other component, but it should be understood that other components may also be present in between. On the other hand, when it is said that a component is "directly connected" or "directly coupled" to another component, it should be understood that no other components are present in between.

[0085] Unless otherwise explicitly stated in the context, the singular expression covers the plural expression.

[0086] In this disclosure, terms such as “comprising” or “including” are intended to indicate the presence of features, numbers, steps, operations, components, parts or combinations thereof as specified in the specification, rather than excluding the presence or possibility of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0087] Camera intrinsics represent the relationship between two-dimensional coordinates of an image and three-dimensional coordinates, and camera calibration refers to the process of estimating camera intrinsics.

[0088] Camera intrinsic parameters include focal length, principal point, and a value representing distortion.

[0089] The focal length is expressed in pixels as the distance between the lens and the image sensor, and in this disclosure, it is assumed that there is no distortion between the lens and the image sensor, and that the principal point distance is zero.

[0090] The aforementioned method using a checkerboard pattern suffers from the inconvenience of having to capture and prepare more than 20 images of the checkerboard. To address this, methods have been proposed using vanishing points identified in the images instead of the checkerboard. However, this presents a constraint requiring the identification of vanishing points around the user, and it may not be guaranteed that the vanishing points exist in the actual shooting space. Therefore, this disclosure proposes a method for estimating camera intrinsics that is convenient for the user and applicable to various environments. According to this disclosure, from the user's perspective, it is sufficient to ensure that the camera captures the user's entire body to estimate the camera intrinsics.

[0091] Figure 2 A flowchart of a method for camera calibration according to this disclosure is shown. Figure 2 The method for camera calibration described herein can be executed by a camera calibration device or its processor, or the method can be implemented by code configured to perform operations according to the method, executed by the device or processor. In the following, the operations are simply described as being performed by a camera calibration device (10) (hereinafter referred to as device (10)), and reference will be made to… Figure 16 Further details are provided regarding the camera calibration device (10).

[0092] The device (10) can be configured to acquire an image obtained by capturing an object (S210). Images can be captured by a camera.

[0093] The device (10) can be configured to estimate the two-dimensional location of an object from the acquired image (S220). The estimated two-dimensional location of the object may include estimating the two-dimensional location of at least one pair of points of the object.

[0094] As an example, when using two pairs of points, the position of an object's joint can be set to a single point. A first pair of points can be configured consisting of two points corresponding to the joints being connected, and a second pair of points can be configured consisting of two points corresponding to other joints being connected. Preferably, one pair of points is arranged along the vertical axis of the image, and the other pair is arranged along the horizontal axis of the image. Since the focal length in the camera intrinsics exists separately for the horizontal direction (x-axis) and the vertical direction (y-axis), the focal length can be measured or calibrated in both the horizontal (x-axis) and vertical (y-axis) directions.

[0095] The device (10) can be configured to obtain two-dimensional coordinate values ​​of at least one pair of points.

[0096] Estimating the two-dimensional position of a specific part of an object from an image is important in computer vision, and this estimation can be used in applications such as behavior analysis, augmented reality / virtual reality, and robot control. A general overview of estimating the two-dimensional position of a specific part of an object from an image is briefly described below. This disclosure uses, but is not limited to, this method.

[0097] There are two main approaches for accurately estimating the two-dimensional coordinates of specific parts of an object from an image: feature-based methods and deep learning-based methods. Feature-based methods detect and analyze features that can distinguish specific parts of an object in an image. Feature detection includes algorithms such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), and Oriented FAST and Rotated BRIEF (ORB) to find specific patterns. Alternatively, feature detection may include comparing the image to a predefined template to estimate a specific part or using the structural relationships of the object to accurately estimate that part.

[0098] Deep learning automatically extracts specific parts and predicts their coordinates through data and training. For deep learning, keypoint detection networks can be used to directly predict the coordinates of regions of interest in an image using CNNs (Convolutional Neural Networks). Heatmap-based estimation can be used, where the network generates a heatmap (probability distribution) for each part of the image and converts the location of the heatmap's maximum value into the coordinates of the specific part. Alternatively, methods can be used that detect objects (e.g., YOLO, Faster R-CNN) and then find keypoints.

[0099] Two-dimensional positions can be estimated and predicted through the processes of data preparation, preprocessing, model training, and inference.

[0100] Furthermore, the process of estimating or identifying the position of joints of an object from an image is called pose estimation, which is also widely used in computer vision. Joint position estimation uses almost the same methods as those previously described for estimating the two-dimensional position of specific parts of an object.

[0101] The device (10) can be configured to estimate the three-dimensional position of at least one pair of points, which are objects with obtained two-dimensional positions (S230). To estimate the three-dimensional position, camera intrinsics and previously obtained two-dimensional positions can be used.

[0102] The following provides a general overview of three-dimensional position estimation based on two-dimensional position estimation from images. This disclosure may be used with, but is not limited to, this method.

[0103] The process of estimating three-dimensional position based on two-dimensional position estimation uses two-dimensional information from the image, as well as additional information (such as camera characteristics or prior knowledge), to obtain three-dimensional position information.

[0104] Estimating a 3D position from a 2D position is performed as follows. This can be done broadly using methods that utilize a single image or methods that utilize multiple images.

[0105] Without additional information, depth information may be uncertain. Therefore, estimating 3D position information (i.e., coordinates) from a single image can be inaccurate. To address this issue, depth can be estimated based on prior knowledge of the actual size or shape of a particular object. For example, a checkerboard pattern can be used for camera calibration. Alternatively, 3D position information can be estimated based on data learned through a deep learning model.

[0106] Alternatively, a multi-image approach uses images from two or more cameras to perform 3D position estimation. Based on images of the same object obtained from two cameras, 2D position information about the object is matched. Depth information can be obtained through triangulation to estimate the final 3D position information.

[0107] As an alternative approach, a single image and depth sensor can also be used for measurement. By combining an RGB camera with a depth sensor to directly measure depth information, a three-dimensional position estimate can be obtained from a two-dimensional position estimate.

[0108] For initial 3D position estimation, initial values ​​of camera intrinsics can be used. These initial values ​​may include a focal length value corresponding to the product of the width of the acquired image and a predetermined value, and / or a principal point corresponding to the center of the acquired image.

[0109] The device (10) can be configured to obtain two-dimensional transformation information from the obtained three-dimensional position estimate (S240). This is to obtain information for comparison with the two-dimensional estimation information obtained in S220. The comparison between the obtained two-dimensional estimation information and the obtained two-dimensional transformation information is for the purpose of ultimately evaluating the appropriateness of the camera intrinsics used for the three-dimensional position estimate or the degree of matching between the camera intrinsics and the actual camera intrinsics.

[0110] To obtain two-dimensional transformation information from the acquired two-dimensional position estimate, a transformation relation (e.g., a transformation matrix) can be used. The transformation relation or transformation matrix can have an inverse relationship with the transformation relation (e.g., the transformation matrix) used to obtain the three-dimensional position estimate in S230.

[0111] Alternatively, the two-dimensional transformation information can be represented by simply projecting the obtained three-dimensional position estimate onto a two-dimensional plane. However, this disclosure is not limited thereto. For example, projecting onto a two-dimensional plane involves setting the value on the z-axis to a specific value, such as 0, when representing the obtained three-dimensional position estimate in a three-dimensional coordinate system (such as (x1, y1, z1)).

[0112] The device (10) can be configured to calculate or obtain the error between the obtained two-dimensional estimation information and the obtained two-dimensional transformation information (S250). The error can be calculated or obtained based on the distance between at least one pair of points forming the basis of the two-dimensional estimation information or based on the position of at least one pair of points.

[0113] For example, when the obtained two-dimensional estimation information is A and B (where A and B represent points), and when the obtained two-dimensional transformation information is A' and B' (where A' and B' represent points), the following differences can be calculated or obtained: the difference between the distance between A and B and the distance between A' and B'; the difference between the positions of A and A' (i.e., the difference between two-dimensional coordinate values); or the difference between the positions of B and B' (i.e., the difference between two-dimensional coordinate values).

[0114] The device (10) can be configured to determine whether two pieces of information are within the error range (S260). The error range can be pre-configured.

[0115] Therefore, if the difference calculated or obtained in S250 is less than or equal to the error, the device (10) can be configured to determine the camera intrinsics used for three-dimensional position estimation in S230 as the final camera intrinsics (S270).

[0116] If the difference calculated or obtained in S250 is greater than the error, the device (10) can be configured to adjust (or update) the camera intrinsics used for 3D position estimation in S230 (S280). Therefore, when performing 3D position estimation (S230), the device (10) can be configured to use the adjusted (or updated) camera intrinsics. (See later...) Figure 3 (a)- Figure 3 (c) Explain the adjustment or update of camera intrinsic parameters.

[0117] although Figure 2 An embodiment is shown in which S230 is re-executed using adjusted camera intrinsics, but it can also be modified to re-execute the process of obtaining an image of the object (S210).

[0118] In S220, it may be difficult or impossible to obtain a two-dimensional position estimate of at least one pair of points of the object. That is, the pose or appearance of the object in the acquired image may be in a state where it is difficult to estimate or obtain the position of a pair of points. Since the focal length exists separately in the camera intrinsics for the horizontal direction (x-axis) and the vertical direction (y-axis), at least one pair of points is used to measure or calibrate at least one of the focal lengths in the horizontal direction (x-axis) and the vertical direction (y-axis). Therefore, in cases where it is difficult to estimate or obtain the position of at least one pair of points due to the pose or appearance of the object, the device (10) can be configured to output a graphical interface for object pose guidance to a human-machine interface, such as a display. Figure 7Provide its details.

[0119] refer to Figure 2 The described method for camera calibration is an operation for estimating camera characteristics (i.e., intrinsic parameters), therefore, the method needs to be executed individually for each camera. For example, in a scenario where cameras (e.g., webcams) are connected to and mounted on a display device such as a TV, the method can be executed for the first camera and used according to... Figure 2 Camera calibration. However, if the first camera is removed and a second camera is connected and installed, the calibration process needs to be repeated according to... Figure 2 Camera calibration. Therefore, the device (10) can be configured to perform camera calibration again when a connection to a new camera is detected.

[0120] Additionally, refer to Figure 2 The described method for camera calibration is an operation for estimating camera characteristics (i.e., intrinsic parameters), therefore, the method needs to be performed individually for each zoom level of the camera. For example, if the camera supports optical or digital zoom in the range of 1.0x to 3.0x, the device (10) can be configured to perform the calibration based on the zoom level range supported by the camera. Figure 2 Camera calibration. In other words, the device (10) can be configured to perform camera calibration at each of a plurality of discrete zoom levels within a zoom level range supported by the camera.

[0121] Additionally, camera calibration can be performed on multiple images. That is, according to... Figure 2 Camera calibration can be repeated multiple times for a single camera. Specifically, camera calibration can be repeated a pre-configured number of times. Furthermore, camera calibration can be performed multiple times for different images. For example, a pre-configured number of images can be acquired, either sequentially or non-sequentially (S210), and the calibration can be performed sequentially for those images. Figure 2 Camera calibration.

[0122] Figure 3 (a)- Figure 3 (c) shows the result of projecting the joint position estimate of the object onto the captured image based on the camera intrinsic parameters obtained through camera calibration.

[0123] In 3(a)- Figure 3 In (c), 310, 320, and 330 represent the results of projecting the estimated 3D positions of objects 31, 32, and 33 onto the 2D image. The closer the 2D projection values ​​of the estimated 3D positions match the objects in the image, the more accurate the estimation result. That is, Figure 3 (a) indicates the state where the estimation result is most accurate.

[0124] Figure 3(b) shows the estimation results when the focal length used for estimating the camera intrinsics differs from the focal length of the actual camera intrinsics. In this case, it can be observed that the distances between points (i.e., joints) differ from the actual distances.

[0125] Figure 3 (c) shows the estimation results when the principal point in the estimated camera intrinsics differs from the principal point in the actual camera intrinsics. In this case, it can be observed that the positions of the feature points in the estimation results differ from the actual joint positions of the object.

[0126] When projecting 3D position information onto a 2D plane (or 2D image), if the distance between two points is greater than or less than the distance in the 2D position estimation result (or the distance of objects in the image), the focal length needs to be adjusted. If the distance between two points represents a translation on the 2D plane compared to the 2D position estimation result (or the distance of objects in the image), the principal point position needs to be adjusted.

[0127] Therefore, the device (10) can continue to perform the following process to adjust the focal length and principal point.

[0128] 1. The device (10) obtains the distance between two points (e.g., shoulder length, distance from foot to pelvis, or other joint distance) from the transformation information obtained by projecting three-dimensional position information onto a two-dimensional plane. Then, the device (10) adjusts the focal length by comparing the obtained distance with the distance between two points in the two-dimensional position estimation result (or the object in the image).

[0129] 2. The device (10) obtains the positions of two points from the transformation information obtained by projecting the three-dimensional position information onto the two-dimensional plane. Then, the device (10) calculates the position difference (along the horizontal or vertical axis) between the corresponding two points in the two-dimensional position estimation result (or objects in the image) and adjusts the principal point accordingly.

[0130] The device (10) can use the distance or position difference between two points, which can be obtained from i) two-dimensional position estimation and ii) information obtained by transforming the three-dimensional position estimation into two dimensions, to obtain information about the principal point and focal length in the camera intrinsics.

[0131] When the focal length changes, the size (proportion) of the object appears different when the three-dimensional position estimate is projected onto a two-dimensional plane. Based on this characteristic, the device (10) can obtain the ratio of the distance between two points from the two-dimensional position estimate and the information obtained by transforming the three-dimensional position estimate into two dimensions, and use this ratio to adjust the focal length.

[0132] More specifically, when the device (10) adjusts or acquires focus in the horizontal direction (x-axis), the device (10) uses joints that appear to be aligned along the x-axis (e.g., left and right pelvic joints, or left and right shoulder joints). In the same manner, in order to adjust or acquire focus in the vertical direction (y-axis), the device (10) uses joints that appear to be aligned along the y-axis (e.g., center of the pelvis to the head, or pelvis to the ankle).

[0133] Therefore, the device (10) can be configured to adjust the focal length of the camera used for three-dimensional position estimation such that the distance between two points in the information obtained by transforming the three-dimensional position estimation into two dimensions converges (or is adjusted within the error range) to the distance between two points in the two-dimensional position estimation.

[0134] Furthermore, the position of the object appears different when the principal point changes and when the three-dimensional position estimate is projected onto the two-dimensional plane. Based on this feature, the device (10) can obtain information about the positions of the two points from the two-dimensional position estimate and the information obtained by transforming the three-dimensional position estimate into two dimensions, and use the obtained information to adjust the principal point.

[0135] More specifically, in order to obtain the x-axis principal point and the y-axis principal point, for the x-axis principal point, the device (10) uses the coordinates of the joint that is aligned along the y-axis, and for the y-axis principal point, the device (10) uses the coordinates of the joint that is aligned along the x-axis.

[0136] Therefore, the device (10) can be configured to adjust the principal point of the camera used for three-dimensional position estimation such that the information about the positions (i.e., coordinates) of the two points in the information obtained by transforming the three-dimensional position estimation into two dimensions converges (or is adjusted within the error range) to the information about the positions of the two points in the two-dimensional position estimation.

[0137] The combination of points used for adjustment (i.e., a pair of points, such as joints) can be variable, and there is no restriction on using a specific pair of points. Additionally, the overall positional difference between a pair of points (e.g., the distance between pelvic joints or the average difference in the positions of multiple joints) can be used.

[0138] The device (10) again uses the adjusted camera intrinsics to perform three-dimensional position estimation. The device (10) can be configured to repeat the above process until i) the difference or error between the two-dimensional position estimation and ii) the information obtained by transforming the three-dimensional position estimation into two dimensions becomes less than a threshold. Thus, the device (10) can be configured to obtain the final camera intrinsics.

[0139] The final camera intrinsics are stored in the memory of the device (10), and can be reused when the same camera is used again. The device (10) can perform the above process (i.e., calibration) again when the camera is changed or when the camera is powered on.

[0140] The process of adjusting the focal length and principal point in the camera intrinsic parameters has been described above. However, the device (10) according to this disclosure can perform the camera calibration process by adjusting the focal length or principal point. Furthermore, it has been stated that the focal length in the camera intrinsic parameters is adjusted for both the horizontal and vertical focal lengths. However, the device (10) according to this disclosure can perform the camera calibration process by adjusting either the horizontal or vertical focal length.

[0141] Figure 4 (a)- Figure 4 (b) shows the camera-calibrated environment with objects in the background and the camera-calibrated environment without objects in the background.

[0142] Figure 4 (a) illustrates a scenario where objects exist around the user (30), and Figure 4 (b) shows a scenario where there are no objects around the user (30).

[0143] In the prior art, camera calibration has been performed by capturing an image of an object with a known size or length (such as a checkerboard) and using that image.

[0144] As referenced above Figure 2 As described, camera calibration is performed using at least one pair of points of the object. Therefore, when... Figure 4 (b) shows that when there are no objects around the user (30), camera parameters can be adjusted or estimated based on the image of the user (30).

[0145] In other words, according to this disclosure, camera parameters for capturing corresponding images can be adjusted and estimated using only images related to the user (30). Therefore, according to Figure 2 Camera calibration methods can be used not only in situations such as Figure 4 (a) is executed when there are objects around the user (30) or object shown, and can be performed as follows Figure 4 (b) is executed when there are no objects around the user (30) or object shown in (b).

[0146] When there are no objects around, such as Figure 4 As shown in (b), the device (10) or the device's processor can perform the following operations. The processor can be configured to:

[0147] - Output the user interface for attitude guidance to the display to obtain the user's image.

[0148] - Obtain at least one pair of points corresponding to the user from the obtained user image.

[0149] - Use at least one pair of points obtained to determine the parameters used for camera calibration, and

[0150] - Based on at least one pair of points obtained and the determined camera calibration parameters, output the three-dimensional position estimation result or the two-dimensional transformation information based on the three-dimensional position estimation result.

[0151] Figure 5 and Figure 6 An example of a user scenario for camera calibration according to this disclosure is shown.

[0152] Calibration can be performed according to this disclosure by using a camera (20) to capture a user (3) and utilizing the captured images.

[0153] Therefore, as Figure 5 As shown, the display device (1) can output guidance content to the user (3) and guide the user (3) to follow the actions of the coach (40). When the user (3) is following the actions of the coach (40), a reference can be executed. Figure 2 The camera calibration method described herein. In this case, the display device (1) may include or be integrated with the camera calibration device (10).

[0154] Once the camera calibration method according to this disclosure is completed, the display device (1) can be configured to output a user interface (50) on the display to notify that the camera calibration or adjustment of the camera intrinsic parameters is complete.

[0155] according to Figure 5 The output of guided content on the display device (1) may include the output of warm-up exercise content from a fitness application or service. In other words, while outputting warm-up exercise content, the display device (1) can be configured to acquire an image of the user (3) and perform actions such as... Figure 2 The camera calibration method shown.

[0156] Figure 6 (a)- Figure 6 (b) shows the output Figure 5 The warm-up exercise video shown is followed by an example of a scene that the display device (1) is capable of outputting.

[0157] Figure 6 (a) shows a list of various fitness guidance content for user (3), Figure 6(b) shows recommended fitness guidance content for the user (3). The display device (1) or camera calibration device (10) can be configured to output fitness guidance content based on any of the weather, time, or the user's (3) condition status.

[0158] like Figure 6 (a)- Figure 6 As shown in (b), the display device (1) or camera calibration device (10) can be configured to output a user interface (50) on the display to notify that the camera calibration or camera intrinsic parameter adjustment is complete.

[0159] Figure 7 (a)- Figure 7 (b) shows a second user scenario example for camera calibration according to this disclosure.

[0160] Calibration can be performed according to this disclosure by capturing an image of the user with a camera (not shown) and using the captured image.

[0161] Therefore, as Figure 7 As shown in (a), the display device (1) can output guidance content for the user and prompt the user to follow the actions of the coach (40) in the guidance content. When the user follows the actions of the coach (40), a reference can be executed. Figure 2 The camera calibration method described in this disclosure. In this case, the display device (1) may include the camera calibration device (10), or may be integrated with the camera calibration device (10).

[0162] according to Figure 2 The camera calibration method, if the difference between the two-dimensional position estimate in S260 and the information obtained by transforming the three-dimensional position estimate into two dimensions is within the error range, then the final camera intrinsic parameters are confirmed, and the process is completed. However, if the difference between the two-dimensional position estimate in S260 and the information obtained by transforming the three-dimensional position estimate into two dimensions is outside the error range, then the calibration can be repeated according to... Figure 2 The process. Even if the camera intrinsic parameters (S280) are adjusted, the difference between the two pieces of information may not fall within the error range.

[0163] Although it has been based on Figure 2 The camera calibration method is described as simply determining the difference between two pieces of information, but modified embodiments can be considered where camera calibration is performed for multiple parts of an object or user or different regions of an image (e.g., center, right, left, etc.).

[0164] Therefore, in order to fully implement the law Figure 2 The camera calibration method, the display device (1) or the camera calibration device (10) can be configured to output Figure 7(a) shows the guiding content, then outputs... Figure 7 (b) shows the guiding content, which is consistent with... Figure 7 The content shown in (a) is different. It is different from... Figure 7 Compared to the content in (a), Figure 7 The guidance content in (b) is configured to make more body parts of the coach (40) visible. Therefore, more body parts of the user can also be captured via images. This allows for more efficient camera calibration.

[0165] like Figure 6 (a)- Figure 6 As shown in (b), the display device (1) or camera calibration device (10) can be configured to output a user interface (50) on the display to notify that the camera calibration or camera intrinsic parameter adjustment is complete.

[0166] Please refer to later Figure 11 (a)- Figure 11 (c) Describe camera calibration for multiple regions of the image.

[0167] Figure 8 (a)- Figure 8 (c) shows an example of a third user scenario for camera calibration according to this disclosure.

[0168] refer to Figure 8 (a)- Figure 8 (c) It can be observed that the images obtained by capturing the user (30) are displayed on the display device (1) at different sizes. This can be explained by distinguishing between camera calibration based on the camera's zoom level or camera calibration based on the distance between the user and the camera.

[0169] First, camera calibration based on camera zoom level will be described. When a camera provides zoom capability and is able to capture images at various zoom levels, camera intrinsic parameters can vary depending on the zoom level. Therefore, calibration based on camera zoom level can be performed. Figure 2 A camera calibration method. In one embodiment, according to... Figure 2 The camera calibration method can be configured to perform across the entire range of zoom levels supported by the camera. In another embodiment, according to Figure 2 The camera calibration method can be configured to perform at multiple discrete zoom levels across the full range of zoom levels supported by the camera.

[0170] Next, camera calibration based on the distance between the user and the camera will be described. Camera distortion can vary between the center of the image and the surrounding area. Additionally, camera calibration can vary depending on the distance from the camera to the object. Therefore, the display device (1) or camera calibration device (10) can output a user interface (not shown) on the display to guide the user (30) to position themselves closer to or further away from the camera. The display device (1) or camera calibration device (10) can adjust the size or position of the user (30) within the acquired image. In other words, the display device (1) or camera calibration device (10) can be configured to perform calibration based on the acquired image. Figure 2 Camera calibration method.

[0171] Figure 9 An example of a fourth user scenario for camera calibration according to this disclosure is shown.

[0172] The display device (1) or camera calibration device (10) can be configured to output a user interface (90) on the display to guide the position or orientation of the user (30). The display device (1) or camera calibration device (10) can be configured to perform actions based on the acquired image. Figure 2 Camera calibration method.

[0173] Figure 10 (a)- Figure 10 (b) shows a fifth user scenario example for camera calibration according to this disclosure.

[0174] Figure 10 (a)- Figure 10 (b) Cases involving capturing multiple objects or users in a single image.

[0175] Figure 10 (a) shows an image in which multiple users (34, 35) are captured, with their bodies overlapping each other. In this case, the display device (1) or camera calibration device (10) may not be able to perform camera calibration properly. To solve this problem, the display device (1) or camera calibration device (10) can be configured to output a user interface (90) on the display to guide the position or posture of the users (34, 35).

[0176] Figure 10 (b) shows images in which multiple users (34, 35) are captured, with their bodies not overlapping. Specifically, Figure 10 (b) shows a model based on, for example Figure 10(a) The user interface (90) described above, which guides the user (34, 35) to a position or posture, obtains an image after the user (34, 35) changes position or posture. The display device (1) or camera calibration device (10) can be configured to perform actions based on the obtained image. Figure 2 Camera calibration method.

[0177] Figure 11 (a)- Figure 11 (c) shows a sixth user scene example for camera calibration according to this disclosure.

[0178] refer to Figure 11 (a)- Figure 11 (c) It can be observed that the position of the user (30) is different in the images captured and displayed on the display device (1). This is to perform camera calibration on specific areas of the images captured by the camera.

[0179] Although not shown, a user interface that guides the user (30)'s movement can be displayed on the screen.

[0180] The display device (1) or camera calibration device (10) can be configured to perform calibration for each image according to... Figure 2 Camera calibration, including, for example Figure 11 (a)- Figure 11 (c) shows the different positions of the user or object (30).

[0181] When according to Figure 2 When errors occur in specific areas of an image during camera calibration, the following can be used: Figure 11 (a)- Figure 11 (c) Implementation example. For example, the implementation example can be used when the difference between the two-dimensional location information and the information obtained by transforming the three-dimensional location information into two dimensions fails to converge within the error range described above. For example, if for Figure 11 The image shown in (a) is based on Figure 2 Errors occurring during camera calibration can be addressed by... Figure 11 (b) or Figure 11 The image shown in (c) is executed according to... Figure 2 Camera calibration. In this case, not only can the user's (30) movement be output on the display, but also the user interface guidance error in camera calibration can be output on the display.

[0182] Figure 12 (a)- Figure 12 (b) shows a seventh user scene example for camera calibration according to this disclosure.

[0183] Figure 12 (a)- Figure 12(b) shows the results of transforming the three-dimensional position estimate calibrated by the camera into two dimensions on the acquired image (121). In this case, a user interface (120) showing accuracy feedback and guiding the readjustment of camera intrinsics can be output on the display.

[0184] Figure 12 (b) shows the results of transforming the three-dimensional position estimate into two dimensions on the acquired image through camera calibration (122). Specifically, Figure 12 (b) indicates the result after performing camera calibration again.

[0185] Figure 13 (a)- Figure 13 (b) shows an image of an object (user) in a specific pose and the estimated three-dimensional joint positions of the object (user) derived from the image.

[0186] Figure 13 (a) shows an image of the user (36). This image can be output on a display device (1).

[0187] Figure 13 (b) shows the basis Figure 13 (a) Results of the 3D position estimation for the user obtained 360°.

[0188] The display device (1) or camera calibration device (10) can be configured to output a three-dimensional position estimate by adjusting and estimating camera intrinsic parameters in real time, such as Figure 13 As shown in (b).

[0189] Real-time adjustment or estimation based on camera intrinsics to display 3D position estimates can provide a user interface for comparing the accuracy of movement between the user's pose and the coach's pose in the guided content, or simultaneously displaying both the user's pose and the coach's pose.

[0190] Figure 14 (a)- Figure 14 (b) is a graph used to compare the three-dimensional estimation results obtained based on the initial camera intrinsics with the three-dimensional estimation results obtained based on the camera intrinsics adjusted by camera calibration according to this disclosure.

[0191] Figure 14 (a) shows an image and information (141) obtained by transforming a three-dimensional position estimate of an object into two dimensions based on initial camera intrinsics, wherein information 141 is superimposed on the image. Figure 14 (b) shows an image and information (142) obtained by transforming a three-dimensional position estimate of an object into two dimensions based on camera intrinsics obtained by camera calibration according to the present disclosure, wherein the information (142) is superimposed on the image.

[0192] The 3D position estimate (142) of the object is an estimate of the position of the object's joints. The better the 3D position estimate (142) is aligned with the object (30) in the image, the closer the adjusted or estimated camera intrinsics are to the actual camera intrinsics.

[0193] Figure 15 (a)- Figure 15 (d) is a graph used to compare the three-dimensional estimation results obtained based on the actual camera intrinsics and the initial camera intrinsics with the three-dimensional estimation results obtained based on the camera intrinsics adjusted by camera calibration according to this disclosure.

[0194] Figure 15 (a) shows an image of an object (user) in a specific pose. Figure 15 (b) shows the 3D position estimation results of the object based on the actual camera intrinsic parameters 361. Figure 15 (c) shows the 3D position estimation results of the object based on the initial camera intrinsics (362), and Figure 15 (d) shows the result of three-dimensional position estimation of the object based on camera intrinsics obtained by camera calibration according to this disclosure (363).

[0195] Figure 16 A block diagram of a camera calibration device according to this disclosure is shown.

[0196] The camera calibration device (10) may include a memory (100) configured to store code for camera calibration.

[0197] The camera calibration device (10) may include a processor (101) configured to execute code for performing operations for camera calibration.

[0198] Operations used for camera calibration may include obtaining two-dimensional position estimates of at least one pair of points of an object from the acquired image.

[0199] Additionally, operations for camera calibration may include obtaining a three-dimensional position estimate of the pair of points.

[0200] Additionally, operations for camera calibration may include obtaining two-dimensional transformation information from the acquired three-dimensional position estimate.

[0201] Additionally, operations used for camera calibration may include calculating the error between two-dimensional transformation information and two-dimensional position estimation.

[0202] Additionally, operations for camera calibration may include adjusting camera calibration parameters based on errors or determining the camera calibration parameters as the final parameters.

[0203] In addition, three-dimensional position estimates can be obtained based on two-dimensional position estimates and camera calibration parameters.

[0204] Use initial values ​​for camera calibration parameters, and these initial values ​​may include the focal length corresponding to the product of the width of the acquired image and a predetermined value, and / or the principal point corresponding to the center of the acquired image.

[0205] Additionally, operations for camera calibration may include adjusting camera calibration parameters when the error exceeds a threshold.

[0206] Additionally, operations for camera calibration may include determining the camera calibration parameters as the final parameters when the error is less than a threshold.

[0207] Additionally, operations for camera calibration may include calculating errors by comparing the distance between the pair of points or the position of the pair of points with the two-dimensional transformation information and the two-dimensional position estimate.

[0208] Additionally, operations for camera calibration may include: when it is impossible or difficult to obtain a two-dimensional position estimate of the pair of points, outputting a user interface on the display to guide the pose of the object.

[0209] Additionally, operations for camera calibration may include re-executing the code for camera calibration when a connection to a new camera is detected.

[0210] Additionally, operations for camera calibration may include executing code for camera calibration at multiple discrete zoom levels within the zoom level range supported by the camera.

[0211] Additionally, operations for camera calibration may include determining the final parameters by executing camera calibration code on a predetermined number of images.

[0212] Additionally or alternatively, the camera calibration device (10) may include a transmit / receive interface (102). The transmit / receive interface (102) may be configured to receive images acquired from the camera. The transmit / receive interface (102) may be configured to transmit camera parameters obtained through camera calibration or two-dimensional or three-dimensional position estimates of objects in images obtained based on the parameters.

[0213] Additionally, the camera calibration device (10) can be configured to include a camera. The camera can capture an object or user and generate an image including the object or user.

[0214] The above text has already referenced Figure 16 A camera calibration device (10) according to this disclosure is described. Even without reference... Figure 16 As described above, the camera calibration device (10) according to this disclosure can also perform the above-described functions. Figures 2 to 15The operation.

[0215] In another aspect of this disclosure, the aforementioned proposed or inventive operations may also be provided as code that can be implemented, executed, or performed by a "computer" (i.e., a general concept including a system-on-a-chip (SoC) or a processor (or microprocessor), a computer-readable storage medium including the aforementioned code, or a computer program product. The scope of this disclosure can be extended to code, computer-readable storage media including code, or computer program products.

[0216] Exemplary embodiments of this disclosure have been provided to enable those skilled in the art to implement and practice this disclosure. Although the foregoing description has been provided with reference to exemplary embodiments of this disclosure, those skilled in the art will understand that the disclosure as set forth in the following claims can be modified and altered in various ways. Therefore, this disclosure is intended to provide the widest scope consistent with the principles and novel features disclosed herein, and not to be limited to the embodiments disclosed herein.

Claims

1. A device for camera calibration, the device comprising: The memory is configured to store code used for the camera calibration; as well as The processor is configured to execute the code to perform operations for the camera calibration. The operation includes: Obtain at least one set of two-dimensional position estimates of points of the object from the acquired image; Obtain the three-dimensional position estimate of the paired points; Obtain the two-dimensional transformation information of the obtained three-dimensional position estimate; Determine the error between the two-dimensional transformation information and the two-dimensional position estimate; and Based on the error, adjust the parameters used for camera calibration or determine the parameters used for camera calibration as the final parameters.

2. The apparatus of claim 1, wherein, The three-dimensional position estimate is obtained based on the two-dimensional position estimate and the parameters used for camera calibration.

3. The device according to claim 2, wherein, The parameters for camera calibration that are adjusted or determined as the final parameters include initial values ​​for camera calibration, and The initial value includes each of the following: Focal length corresponds to the product of the width of the acquired image and a predetermined value, or The principal point corresponds to the center of the obtained image.

4. The device according to claim 1, wherein, The operation includes adjusting the parameters used for camera calibration based on the error being greater than a threshold.

5. The device according to claim 1, wherein, The operation includes determining the parameters used for camera calibration as the final parameters based on the error being less than a threshold.

6. The device according to claim 1, wherein, The operation includes determining the error by comparing the two-dimensional transformation information with the distance between the pairs of points or the positions of the pairs of points estimated by the two-dimensional position.

7. The device according to claim 1, wherein, The operation includes outputting a user interface on the display to guide the pose of the object based on the two-dimensional position estimate of the paired points, which is either unavailable or difficult to obtain.

8. The device according to claim 1, wherein, The operation includes re-executing the code used for camera calibration based on the detection of a connection to the new camera.

9. The device according to claim 1, wherein, The operation includes executing the code for camera calibration at multiple discrete zoom levels within a zoom level range supported by the camera.

10. The device according to claim 1, wherein, The operation includes determining the final parameters by executing the code used for camera calibration on a predetermined number of images.

Images