CAMERA CALIBRATION METHOD AND DEVICE

Abstract

A camera calibration method includes receiving an image captured by a camera of a vehicle, detecting a vehicle body portion of the vehicle from the image, and estimating a pose of the camera based on the vehicle body portion.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a camera calibration technology, and more particularly, to a camera calibration method and apparatus capable of estimating a pose of a camera based on a vehicle shape or a vehicle body portion of a vehicle detected while traveling. BACKGROUND

[0002] Various image recognition technologies are required to operate an autonomous vehicle. For example, it is important to identify lanes and vanishing points while a vehicle is moving.

[0003] The vanishing point refers to a point where, when parallel straight lines in a 3-dimensional (3D) space are extended infinitely and projected onto a 2-dimensional (2D) plane, the straight lines meet on the plane. As an example of vanishing point detection, a building can be reinterpreted by analyzing an architectural structure by determining vanishing points and vanishing lines in three orthogonal directions. In the 3D transformation of a 2D image including architectural structures, detecting a vanishing point can create a depth map. The reason for this is that an estimation of relative depth is possible because an area where the vanishing point is located generally corresponds to the farthest area in the image when a 3D space is transformed into a 2D image.

[0004] Furthermore, vanishing point information can serve as a reference for lane detection in an autonomous vehicle or an important basis for analyzing location information in an autonomous driving system such as a robot. This is because a road can be recognized by connecting major edges that are connected from the vanishing point.

[0005] The vehicle estimates vanishing points with line segments in an image while driving and estimates a vanishing line based on the estimated vanishing points to estimate extrinsic parameters of a camera and a road surface, thereby estimating a pose of the camera.

[0006] However, there is a problem that the accuracy of the vanishing points is reduced due to the image distortion during the vehicle's driving, and it is difficult to accurately estimate the vanishing line due to the limitation of the vanishing points according to the behavior of the vehicle during the driving. SUMMARY

[0007] The following summary is a simplified summary of certain features. The summary is not a comprehensive overview and is not intended to identify important or critical elements.

[0008] Systems, devices, and methods for calibrating a camera of a vehicle are described. A method may include receiving an image of a vehicle captured by a camera; detecting a vehicle body portion of the vehicle in the image; estimating, based on a slope of an edge associated with the vehicle body portion, a pose of the camera; and performing image recognition on a calibrated image, wherein the calibrated image is based on a second image captured by the camera that was calibrated based on the estimated pose of the camera.

[0009] Also or alternatively, a device may include a receiver configured to receive an image captured by a camera of a vehicle; a detector configured to detect a vehicle body portion of the vehicle in the image; and an estimator configured to estimate a pose of the camera based on an inclination of an edge associated with the vehicle body portion. The device may be configured to perform image recognition on a calibrated image, wherein the calibrated image is based on a second image captured by the camera that was calibrated based on the estimated pose of the camera.

[0010] These and other features and benefits are described in more detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings: Fig. 1 shows an operational flow diagram for a camera calibration method of the present disclosure; Fig. 2 shows an operational flowchart for an example of a process for detecting a vehicle body portion of a vehicle; Fig. 3 shows an operational flow diagram of an example of a camera roll estimation process; Fig. Figure 4 shows an example diagram describing a camera roll estimation process; Fig. 5 is an operational flowchart of an example of a camera pan estimation method; Fig. Figure 6 shows an example diagram to describe the camera pan estimation process; Fig. 7 shows an example diagram for describing the relationship between an image region and a Gaussian sphere region; Fig. 8 shows an operational flowchart of an example of a method for estimating a camera tilt; Fig. 9 shows an example diagram for describing a process for estimating a camera tilt; Fig. 10 shows an example diagram for describing a method for estimating a camera pose in an example that violates the Manhattan World Assumption (MWA); Fig. 11 shows an example diagram for explaining a method for estimating a camera pose in another example that violates the Manhattan World Assumption (MWA); Fig. 12 shows the configuration of a camera calibration device according to another example of the present disclosure; and Fig. 13 is a block diagram of a computer system for performing a camera calibration method according to an example of the present disclosure. DETAILED DESCRIPTION

[0012] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice it. However, the present disclosure may take various forms and is not limited to the examples described herein.

[0013] When describing the present disclosure, it is determined that a detailed description of a known configuration or function might obscure the gist of the present disclosure, a detailed description thereof will be omitted. Furthermore, in the drawings, parts not related to the description of the present disclosure are omitted, and similar parts are denoted by similar reference numerals.

[0014] It is understood that an element described as "connected," "coupled," or "attached" to another element may be directly connected or coupled to the other element, or there may be intervening elements. Unless expressly described otherwise, the word "comprise" or "include" is to be understood as implying the inclusion of the mentioned elements, but not the exclusion of others.

[0015] In the present disclosure, terms such as "first" and "second" are used only to distinguish one element from other elements and do not limit the order or meaning of the elements unless explicitly stated. Therefore, throughout the present disclosure, a first element in one example may be referred to as a second element in another example, and similarly, the second element in one example may be referred to as the first element in another example.

[0016] In the present disclosure, different elements are used only to uniquely describe their features and do not necessarily imply that the elements are separate. That is, a plurality of elements may be integrated to form a single hardware or software unit, or an element may be distributed to form a plurality of hardware or software units. Accordingly, such integrated or distributed examples are also within the scope of the present disclosure, even if not explicitly mentioned.

[0017] In the present disclosure, the elements described in various examples do not necessarily represent essential components, and some may be optional elements. Accordingly, examples consisting of a subset of the elements described in an example are also within the scope of the present disclosure. Furthermore, examples that include further elements in addition to the elements described in the various examples are also within the scope of the present disclosure.

[0018] In the present disclosure, expressions for positional relationships used in the description, such as top, bottom, left, or right, are described for convenience, and when the drawings shown in the description are viewed inversely, the positional relationships described in the description can also be interpreted in reverse.

[0019] In the present disclosure, terms such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", "at least one of A, B or C" may include any of the elements listed together in the corresponding term or any possible combination thereof.

[0020] To solve the problem of the difficulty of estimating an accurate vanishing line while a vehicle (e.g., a vehicle having and / or connected to a camera) is moving, the present disclosure provides systems, apparatus, and methods for detecting a vehicle shape and / or a vehicle body portion of the vehicle captured by the camera. The vehicle shape and / or vehicle body portion may include part or all of a handle, hood, and / or side of the vehicle. The methods disclosed herein may improve the estimation accuracy of camera pose, e.g., the three-axis angles (roll, pan, pitch) of the camera, by using the detected vehicle shape and / or vehicle body portion as a reference instead of, or in addition to, a vanishing line and / or vanishing point.

[0021] The present disclosure may estimate a camera pose based on the Manhattan World Assumption (MWA) for the transformation between an image domain and a Gaussian sphere domain. In particular, the present disclosure may estimate a camera pan based on the included angle between the camera's optical axis and a straight line in the vehicle body domain using the MWA. The present disclosure may estimate a camera tilt based on the included angle between the camera's optical axis and a feature point vector, such as a vector corresponding to a feature point associated with (e.g., from) a handle.

[0022] The present disclosure may use the analyzed (e.g., determined) vehicle body area as a substitute for and / or as a supplement to a vanishing point and / or a vanishing line. The vehicle body area of ​​the vehicle captured in an image may be analyzed and thus used to estimate the camera pose.

[0023] In the following, a method and an apparatus according to the present disclosure will be described with reference to the Fig. 1 to 12 described.

[0024] Fig. 1 is an operational flow diagram illustrating a camera calibration method according to an example of the present disclosure, which may be performed by a camera calibration device. The camera calibration device may be provided in the vehicle and / or may be remote from the vehicle (e.g., in communication with the vehicle and / or capable of receiving information from the vehicle and / or transmitting information to the vehicle). For example, the camera calibration device may be capable of communicating with a camera of the vehicle.

[0025] With reference to Fig. 1, a camera calibration method according to an example of the present disclosure may include receiving (e.g., in real time or near real time) an image captured by an image capturing device, e.g., a camera, while a vehicle is traveling, and detecting a vehicle body portion (e.g., a portion of the image corresponding to an image of a part of the vehicle) of the vehicle from the received image (S110 and S120).

[0026] At S110, an image captured by at least one camera (e.g., at least one of a front camera, a rear camera, a front corner camera, and / or a rear corner camera) may be received. In the examples discussed herein, an image captured by the rear corner camera may be received.

[0027] In S120, a vehicle body region may be detected in the image. For example, one or more edge points may be detected in a series of camera images, and it may be determined that the one or more edge points do not change significantly over time (e.g., while other regions of the image or images appear to change and / or move over time). Other edge detection methods may also or alternatively be used to detect an edge and / or one or more edge points of the vehicle body region. An edge of the detected vehicle body region in the image may be determined based on the position(s) of the detected edge points.

[0028] Based on the vehicle body area captured in S120, an inclination of the edge of the captured vehicle body area can be estimated, feature points of the captured vehicle body area can be extracted, and / or the camera pose can be estimated. For example, the roll, pan, and tilt angles of the three axes can be estimated (S130, S140, and S150).

[0029] In S150, the roll of the camera may be estimated based on the estimated pitch, the pan of the camera may be estimated based on the position of the edge of the vehicle body portion, and the tilt of the camera may be estimated based on the feature point of the vehicle body portion, thereby estimating the pose of the camera.

[0030] A method according to an example of the present disclosure will be described with reference to the Fig. 2 to 9 are described in detail.

[0031] Fig. 2 is an operation flowchart of an example of a method for detecting a vehicle body portion of a vehicle, and is an operation flowchart of an example of S120 in Fig. 1.

[0032] Referring to Fig. 2, detecting the vehicle body area of ​​the vehicle (S120) may include retrieving or receiving image data captured by a camera while the vehicle is traveling at a certain speed or higher, correcting image distortion, and then detecting edge points in the image in which the distortion is corrected (S210, S220, and S230).

[0033] Since it is difficult to identify time-invariant edge points (e.g., those that do not change with time between consecutive images) when the vehicle is in a stationary state, edge points can be detected from an image based on the vehicle's speed being greater than or equal to a predetermined value.

[0034] When the edge points in the image are detected in S230, a first straight line for the edge points can be extracted to determine the number of outliers by applying a certain logic, e.g., AND logic, to the edge points detected over time, and applying RANSAC (Random Sample Consensus) to the edge points to which the AND logic was applied (S240).

[0035] Applying AND logic to the edge points in S240 can be used to easily detect the vehicle body area because the positions of the edge points of the vehicle body are maintained even when the vehicle is moving or traveling, but the edge points detected outside the vehicle body are moving.

[0036] RANSAC is an algorithm for removing noise from a dataset and predicting a model. It has the property of completely ignoring data above a certain threshold, making it robust against outliers. For example, RANSAC can extract an ideal model that has no noise and whose maximum data is identical (e.g., a first-order straight line). RANSAC is well known to those skilled in the art, so a detailed description is omitted.

[0037] When a first straight line is extracted for the edge points contained in an area of ​​the image at S240, the number of outliers can be measured using a result of RANSAC. If the number of outliers is less than a predetermined certain value, the corresponding edge points can be determined as the vehicle body area. By this method, the vehicle body area can be determined (S250, S260, and S270). For example, since an outlier exceeding the certain value, that is, a large number of outliers, means that an edge point has been detected in an area other than the vehicle, the edge point in the corresponding area can be determined as an edge point of the vehicle body area if the outlier is less than or equal to the certain value.

[0038] For example, if an image taken by the camera is as shown in Fig. 4A, an area 400 of the vehicle can be identified by the method in Fig. 2 as the vehicle body area. In this case, the area 400 of the vehicle may include a grip area 410 corresponding to a feature point, as in Fig. 4A shown.

[0039] Fig. 3 is an operational flow diagram of a method for estimating camera roll, for example, based on a vehicle body area obtained by the method in Fig. 2 is recorded.

[0040] With reference to Fig. 3, a method for estimating camera roll may estimate the camera roll by retrieving a first-order straight line that is a result of RANSAC for the detected edge points of the vehicle body region and calculating a slope of the first-order straight line for the vehicle body region with respect to a vertical line (S130, S320 and S330).

[0041] In S330, the first-order straight line slope for the vehicle body area can be estimated as camera roll, since the side of the vehicle is assumed to be perpendicular to the road surface in a 3D space and the slope of the vehicle body area is assumed to be camera roll. For example, as in Fig. 4B, an inclination of the straight line 420 of the edge of the vehicle body portion, that is, the inclination of the straight line 420 of the edge of the vehicle body portion with respect to the vertical line, can be estimated as the camera roll.

[0042] In S330, the slope of the first straight line with respect to the vertical line may be determined (e.g., calculated), a histogram of a gradient of the image may be generated to output a slope corresponding to the peak of the histogram, and the slope may then be rotated by 90 degrees, thereby estimating the camera roll.

[0043] In S330, the camera roll may be estimated using feature points (e.g., and / or feature point vectors and / or feature point lines) that are included (e.g., detected) in the vehicle body area. As in Fig. As shown in Figure 4C, camera roll can be estimated using the grip region 410, which is a feature included in the vehicle body region. For example, since the grip region 410, which is a feature of the vehicle body region, forms a right angle with the edge of the vehicle body region, an included angle between a horizontal line and the grip region 410 can be estimated as camera roll.

[0044] In S330, the camera roll can be estimated by generating a 3D circle of the vertical line, generating a 3D circle of the detected edge of the vehicle, and then determining (e.g., calculating) an included angle between the two 3D circles. Since the vertical line is perpendicular to the roadway, it can be assumed that the included angle between the vertical line and the edge of the vehicle body is equal to the camera roll. This may be because the vertical line is rotated by the camera roll. The method described above is a method using MWA, which was developed with reference to Fig. 7 is described in more detail.

[0045] Fig. Figure 5 is an operational flow diagram of an example of a method for estimating camera pan. The camera pan can be estimated using an estimated camera roll.

[0046] Referring to Fig. 5, a method for estimating a camera pan may include retrieving an edge point detected as a vehicle body region and compensating the edge point of the vehicle body region with the estimated camera roll (S510 and S520).

[0047] In S520, when the edge point of the vehicle body area is compensated by the estimated camera roll, a 3D circle for the edge point for which the camera roll is compensated may be generated, a 3D circle of the optical axis of the camera may be generated, and an included angle β1 between the two 3D circles may be determined (e.g., calculated) (S530, S540, and S550).

[0048] Based on (e.g., using as input) a preset design value for the camera, e.g., a camera position in a corresponding vehicle and the edge point compensated with camera roll, an included angle α1 between a line between the camera and the edge point and the side of the vehicle can be determined (e.g., calculated), and / or a camera pan can be estimated using the difference (β1-α1) between the two included angles (S560 and S570).

[0049] That is, the method for estimating the camera pan as described in Fig. 6, may include determining (e.g., calculating) the included angle β1 between an optical axis 610 of the camera and a line between the camera and an edge point 620, the design value of the camera, for example, the included angle α1 between the line between the camera position and the edge point 620 and a side of the vehicle. The camera pan may be estimated based on a difference between the two included angles (β1-α1).

[0050] Here, the included angle β1 between the optical axis of the camera and the edge point can be determined (e.g. calculated) based on the Manhattan World Assumption (MWA), as in Fig. 7A. That is, based on the MWA, a linear component in an image domain can be expressed as a circle in a Gaussian sphere. The linear component in the image domain and the circle in the Gaussian sphere can be transformed into each other. That is, as shown in Fig. As shown in Figure 7B, the linear component corresponding to the optical axis of the camera 610 can be transformed or expressed into a circle 710 in the Gaussian sphere domain, and the line 620 between the camera and the edge point can also be transformed or expressed into a circle 720. Therefore, the included angle β1 between the optical axis of the camera and the edge point of the vehicle body region can be determined (e.g., calculated) using the two circles 710 and 720.

[0051] MWA is briefly described as follows. Each plane can be represented by a vector perpendicular to the plane. The six faces of a hexahedron can be represented by six vectors—without considering directionality, they can be represented by three vectors. A virtual space formed only by the faces belonging to the three vectors is called MWA, and each linear component passes through one of the three faces corresponding to one of the three vectors. When the camera pose is changed, the hexahedron also rotates, and the three vectors also rotate. Therefore, the camera pose can be estimated by knowing how much the three vectors are rotated. The camera pose can be estimated by finding three vectors of the vehicle in the MWA, assuming the vehicle is a hexahedron. Furthermore, as shown in Fig. 7A, a method of expressing a linear component of an image area and a circle of a Gaussian sphere area is obvious to those skilled in the art, so a description thereof is omitted.

[0052] Fig. 8 is an operation flowchart of an example of a method for estimating a camera tilt, which is an operation flowchart of a method for estimating a camera tilt using a camera roll and a camera pan estimated by the methods described above.

[0053] Referring to Fig. 8, a method for estimating a camera tilt may include retrieving an edge point recognized as a vehicle body area and a feature point, for example, a feature point for a handle, and compensating the edge point and the feature point with the camera roll and pan estimated in the above method (S810 and S820).

[0054] Based on the edge point and the feature point compensated by the camera roll and pan, a vector perpendicular to the vehicle body area edge, i.e., a feature point vector, can be determined (e.g., calculated), a 3D circle for the feature point vector and a 3D circle for the camera optical axis can be generated, and an included angle β2 between the two circles can be determined (e.g., calculated) (S830, S840, S850, and S860).

[0055] Furthermore, using a preset design value for the camera, e.g., a camera position in the vehicle and a feature point vector compensated by camera roll and camera pan, an included angle α2 between a line between the camera and the feature point vector and the top of the vehicle can be determined (e.g., calculated), and a camera tilt can be estimated using the difference between the two included angles (β2-α2) (S870 and S880).

[0056] That is, the method for estimating the camera tilt as in Fig. 9, may include determining (e.g., calculating) the included angle β2 between an optical axis of the camera 910 and the line between the camera and a feature point vector 920, the design value of the camera, e.g., the included angle α2 between the feature point vector 920, which is a line between the camera position and the feature point, and the top of the vehicle, and estimating the tilt of the camera based on the difference between the two included angles.

[0057] As described above, the camera calibration method according to an example of the present disclosure can detect a vehicle body portion in an image captured from a moving vehicle and estimate the camera pose based on MWA using the inclination for the vehicle body portion, the feature point, the camera optical axis, and the design value. Furthermore, the camera calibration method according to an example of the present disclosure can use any features of a vehicle perpendicular or horizontal to the road surface, e.g., the side of the vehicle, the front bumper of the vehicle, the rear bumper of the vehicle, the connection between the glass and the top of the vehicle, or the like, as a vehicle body portion captured by a camera.

[0058] The above description has been made for the case where the MWA is satisfied, and can also be applied to a case where a curve or straight line violating the MWA is detected, which can be explained with reference to the Fig. 10 to 11.

[0059] Fig. 10 is an exemplary diagram for describing a method of estimating a camera pose in an example that violates the Manhattan World Assumption (MWA), showing an exemplary diagram for describing a method of estimating a camera pose using a curve when a curve that violates the MWA is detected.

[0060] As in Fig. 10A, a curve 1010 may be detected in a vehicle body region detected in an image captured by a camera, and curves that are not straight lines violate the MWA. In one example of the present disclosure, a camera pose may be estimated by transforming a curve into a straight line suitable for the MWA. For example, a straight line suitable for the MWA that is closest to the detected curve may be generated from a drawing of the corresponding vehicle. That is, the relationship between the detected curve 1010 and a straight line 1020 may be known using the drawing of the vehicle, and thereby the detected curve 1010 may be transformed into the straight line 1020 by generating a virtual line 1020 corresponding to the detected curve 1010 for the detected curve 1010. For example, as shown in Fig. As shown in Figure 10B, because a front hood is symmetrical on the left and right sides like a decalcomania, when the vehicle body portion captured in the image is the front hood, a straight line 1030 is found that makes the detected curve 1010 symmetrical, and a straight line 1020 perpendicular to the straight line 1030 and tangent to the detected curve 1010 is generated. In this case, the generated straight line 1020 can satisfy the MWA. Accordingly, the method according to an example of the present disclosure can estimate the pose of the camera based on a straight line obtained by transforming a curve.

[0061] Fig. 11 is an exemplary diagram for describing a method of estimating a camera pose in another example that violates the Manhattan World Assumption (MWA), showing an exemplary diagram for describing a method of estimating a camera pose using a straight line calibration when a straight line that does not satisfy the MWA is detected.

[0062] As in Fig. As shown in Figure 11A, a straight line rather than a curve may be detected in a vehicle body region captured in an image taken by a camera. However, a plane formed by the detected straight line 1120 may violate the MWA. A camera pose can be estimated while maintaining the MWA by calibrating a straight line that violates the MWA against a drawing (e.g., a representation / sketch, e.g., a known shape of the vehicle) of a corresponding vehicle.

[0063] For example, if, as in Fig. 11A, when the straight line 1120 is detected on the passenger seat window side in the vehicle body area captured in the image, an included angle α3 with a vertical line 1110 can be identified by referring to the drawing of a corresponding vehicle. Therefore, after estimating a roll value with the straight line 1120, the camera pose can be estimated using the drawing of the vehicle by calibrating the roll value by the included angle α3, even if a straight line violating the MWA is detected.

[0064] As described above, it is possible to estimate the camera pose using all curves and straight lines detectable by the vehicle (e.g., in the vehicle body area of ​​a camera image) with the methods described above, even if a curve or straight line violating the MWA is detected.

[0065] That is, as in Fig. As shown in Figure 11B, the camera pose can be estimated using all curves and straight lines detectable (e.g., in an image from the camera) of the vehicle by fitting any vehicle segment to a straight line 1130 suitable for the MWA.

[0066] As described above, in the camera calibration method according to an example of the present disclosure, when the vehicle body portion of a vehicle is detected in an image taken while the vehicle is traveling, and one of the horizontal direction information and the vertical direction information is obtained based on the vehicle body portion, the camera pose can be estimated using a straight line obtained by either the horizontal direction information or the vertical direction information and a vanishing point that can be obtained while the vehicle is traveling.

[0067] The camera calibration method according to an example of the present disclosure can estimate the camera pose using only directional information from the captured vehicle body area without detecting a vanishing point. For example, horizontal directional information can be obtained from the vehicle body area captured by the camera, such as in an image of the handle or hood, and vertical directional information can be obtained from the body area, such as in an image of the side of the vehicle.

[0068] The camera calibration method according to an example of the present disclosure may transform a curve or straight line that violates the MWA into a straight line suitable for the MWA and / or generate a straight line suitable for the MWA based on a detected curve or straight line. The transformed and / or generated line suitable for the MWA may be used to detect the camera pose using any (e.g., all) curves and / or straight lines detectable in the vehicle body area.

[0069] Furthermore, the camera calibration method according to an example of the present disclosure may use an artificial intelligence network, such as a deep learning-based network, as a method for recognizing a vehicle body area in an image captured by a camera. For example, segmentation may be used to recognize a vehicle body area using an artificial intelligence network.

[0070] Furthermore, the camera calibration method according to an example of the present disclosure can detect a straight line corresponding to a vanishing line based on the vehicle body area. The camera pose can be estimated using the detected straight line in the vehicle body area and a vanishing point detected in an image captured by the camera while driving.

[0071] Furthermore, the camera calibration method according to an example of the present disclosure may estimate the camera pose based on the horizontal direction information and the vertical direction information (e.g., based on the horizontal direction information and the vertical direction information acquired in the vehicle body area). In this case, the method of the present disclosure may estimate the camera pose using only the horizontal direction information and the vertical direction information (e.g., without the need to acquire and / or use a vanishing line or vanishing point).

[0072] Fig. 12 shows a configuration of a camera calibration apparatus according to another example of the present disclosure, showing a configuration of an apparatus embodying the methods of Fig. 1 to 11.

[0073] With reference to Fig. 12, a camera calibration apparatus 1200 according to another example of the present disclosure may include a receiver 1210, a detector 1220, an estimator 1230, and a memory 1240.

[0074] Memory 1240 may be a component for storing data related to the technology of the present disclosure. Memory 1240 may store data, such as one or more images captured by a camera, algorithms related to the technology of the present disclosure, e.g., RANSAC, edge points, RANSAC results and vehicle drawings, vehicle data, or camera design values, and / or instructions that, when executed, cause camera calibration device 1200 to perform one or more of the methods disclosed herein.

[0075] The receiver 1210 may receive an image captured by a camera of a vehicle.

[0076] The receiver 1210 may receive an image captured by a camera when (e.g., based on) the vehicle is traveling at a certain speed or higher.

[0077] The detector 1220 can detect a body area of ​​the host vehicle in the image received from the receiver 1210.

[0078] The detector 1220 may detect one or more edge points for one or more regions of each of a plurality of regions of the image, measure a number of outliers of the one or more edge points of each region of the one or more regions (e.g., using RANSAC), and determine (e.g., detect) a region of the image in which the number of outliers is equal to or less than a preset certain value (e.g., a threshold) as the vehicle body region.

[0079] The estimator 1230 may estimate a pose of the camera based on the vehicle body area detected by the detector 1220.

[0080] The estimator 1230 may estimate the inclination of the vehicle body area based on the vehicle body area, extract a feature point of the vehicle body area, and estimate the pose of the camera using the inclination and the feature point.

[0081] The estimator 1230 may estimate a slope for an edge of the vehicle body area and estimate a camera roll based on the estimated slope of the edge.

[0082] The estimator 1230 may compensate the edge point of the vehicle body area with the estimated roll, generate a 3D circle of the roll-compensated edge point and a 3D circle of the camera optical axis, determine (e.g., calculate) a first included angle between the two 3D circles, determine (e.g., calculate) a second included angle between a preset design value of the camera and the roll-compensated edge point, and estimate a camera pan using the first included angle and the second included angle.

[0083] The estimator 1230 may compensate the edge point and the feature point of the vehicle body region with the estimated roll and pan, determine (e.g., calculate) a feature point vector perpendicular to the edge using the edge and feature point of the vehicle body region compensated with the roll and pan, determine (e.g., calculate) a third included angle between a 3D circle of the feature point vector and a 3D circle of the optical axis of the camera, determine (e.g., calculate) a fourth included angle between a design value of the camera and the feature point compensated with the roll and pan, and estimate a tilt of the camera using the third and fourth included angles.

[0084] The estimator 1230 may detect a straight line corresponding to a vanishing line based on the vehicle body area, and estimate the pose of the camera using the detected straight line and the vanishing point detected by an image of the vehicle while traveling.

[0085] When a curve through the vehicle body area is detected, the estimator 1230 may generate a virtual line corresponding to a straight line by using the curve and curve information for the vehicle body area, for example, drawing information of the vehicle, and estimate the camera pose using the virtual line and the vanishing point.

[0086] The estimator 1230 may estimate the camera pose based on horizontal direction information and vertical direction information when the horizontal direction information and the vertical direction information of the vehicle body area are detected based on the vehicle body area.

[0087] Although a description is omitted in the apparatus according to another example of the present disclosure, the apparatus according to another example of the present disclosure can implement all of the methods described in the Fig. 1 to 11, which is obvious to a person skilled in the art.

[0088] Fig. 13 is a block diagram of a computer system for carrying out a camera calibration method of the present disclosure.

[0089] With reference to Fig.13, a method for estimating a vanishing point according to another example of the present disclosure described above may be implemented by a computer system. A computer system 2000 may include at least a processor 2100, a memory 2300, a user interface input device 2400, a user interface output device 2500, a memory 2600, and a network interface 2700 interconnected via a system bus 2200.

[0090] Processor 2100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in memory 2300 and / or storage 2600. Memory 2300 and storage 2600 may include various types of volatile or non-volatile storage media. For example, memory 2300 may include a ROM (Read Only Memory) 2310 and a RAM (Random Access Memory) 2320.

[0091] As a result, the operations of the method or algorithm described in connection with the examples disclosed herein may be embodied directly in hardware or a software module executed by processor 2100, or a combination thereof. The software module may be located on a storage medium (i.e., memory 2300 and / or memory 2600), such as RAM, flash memory, ROM, EPROM, EEPROM, register, hard disk, removable disk, and CD-ROM. The example storage medium may be coupled to processor 2100, and processor 2100 may read information from and store information on the storage medium. Alternatively, the storage medium may be integrated with processor 2100. Processor 2100 and the storage medium may be encapsulated in an application-specific integrated circuit (ASIC).The ASIC may be housed in a user terminal. Alternatively, the processor 2100 and the storage medium may be housed as separate components in the user terminal.

[0092] The present disclosure has been made to solve the above-mentioned problems encountered in the prior art while maintaining the advantages achieved by the prior art.

[0093] One aspect of the present disclosure provides a camera calibration method and apparatus capable of estimating the pose of a camera based on the vehicle shape or vehicle body area of ​​a vehicle captured while the vehicle is traveling.

[0094] One aspect of the present disclosure provides a camera calibration method and apparatus capable of estimating the pose of a camera using information about the vehicle body portion of a vehicle acquired by the camera, for example, at least one of horizontal direction information and vertical direction information and a vanishing point obtainable while the vehicle is traveling.

[0095] One aspect of the present disclosure provides a camera calibration method and apparatus capable of estimating the pose of a camera using only two pieces of directional information without detecting a vanishing point when there are horizontal directional information for the vehicle body portion captured by the camera, such as handle or hood information, and vertical directional information, such as side of the vehicle.

[0096] One aspect of the present disclosure provides a camera calibration method and apparatus capable of estimating (e.g., configured to estimate) the pose of a camera using all curves and straight lines detectable in a vehicle by performing, generating, or calibrating a transformation to a straight line suitable for the Manhattan World Assumption (MWA) when a curve and straight line violating the MWA is detected.

[0097] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art to whom the present disclosure is directed from the following description.

[0098] According to one aspect of the present disclosure, a camera calibration method includes receiving an image captured by a camera of a vehicle, detecting a vehicle body portion of the vehicle from the image, and estimating a pose of the camera based on the vehicle body portion.

[0099] Detecting the vehicle body region may include detecting one or more edge points for one or more regions of a plurality of regions of the image, measuring a number of outliers of the edge points in each of the one or more regions using RANSAC (RANdom Sample Consensus), and detecting (e.g., determining) a region of the one or more regions of the image as a vehicle body region in which a number of outliers is less than or equal to a preset value (e.g., a threshold).

[0100] Estimating the pose of the camera may include estimating a tilt of the vehicle body portion based on the vehicle body portion; extracting a feature point of the vehicle body portion; and estimating the pose of the camera using the tilt and the feature point.

[0101] Estimating the pose of the camera may include estimating a tilt of an edge of the vehicle body area and estimating a roll of the camera based on the estimated tilt of the edge.

[0102] Estimating the pose of the camera may include compensating an edge point of the vehicle body area with the estimated roll, generating a 3D circle of the roll-compensated edge point and a 3D circle of an optical axis of the camera, determining (e.g., calculating) a first included angle between the two 3D circles, calculating a second included angle between a preset design value of the camera and the roll-compensated edge point, and estimating (e.g., determining and / or calculating an estimate) a pan of the camera using the first included angle and the second included angle.

[0103] Estimating the pose of the camera may include compensating the edge point and the feature point of the vehicle body portion with the estimated roll and the estimated pan, calculating a feature point vector perpendicular to the edge using the edge and the feature point of the vehicle body portion compensated with the roll and pan, calculating a third included angle between a 3D circle of the feature point vector and a 3D circle of the optical axis of the camera, calculating a fourth included angle between the design value of the camera and the feature point compensated with the roll and pan, and estimating a tilt of the camera using the third and fourth included angles.

[0104] Estimating the pose of the camera may include detecting a straight line corresponding to a vanishing line based on the vehicle body area, and estimating the pose of the camera using the detected straight line and a vanishing point detected by the image while the vehicle is traveling.

[0105] Estimating the pose of the camera may include generating a virtual line corresponding to the straight line using a curve and curve information for the vehicle body portion, and estimating the pose of the camera using the virtual line and the vanishing point when the curve is detected by the vehicle body portion.

[0106] Estimating the pose of the camera may include estimating the pose of the camera based on horizontal direction information and vertical direction information when the horizontal direction information and the vertical direction information for the vehicle body area are acquired based on the vehicle body area.

[0107] According to the present disclosure, a camera calibration device may include a receiver configured to receive an image captured by a camera of a vehicle, a detector configured to detect a vehicle body portion of the vehicle in the image, and an estimator configured to estimate a pose of the camera based on the detected vehicle body portion.

[0108] The detector may detect an edge point in one or more of a plurality of regions in the image; determine (e.g., measure) a number of outlier edge points for each of the one or more regions (e.g., using RANSAC (RANdom SAmple Consensus)), and detect (e.g., determine) a region (e.g., cumulative of the one or more regions) in which the number of outliers is less than or equal to a preset value, as the vehicle body region (e.g., a portion of the vehicle body region).

[0109] The estimator can estimate a tilt of the vehicle body area based on the vehicle body area, extract a feature point of the vehicle body area, and estimate the pose of the camera using the tilt and the feature point.

[0110] The estimator can estimate a tilt of an edge of the vehicle body area and estimate camera roll based on the estimated tilt of the edge.

[0111] The estimator may compensate an edge point of the vehicle body area with the estimated roll, generate a 3D circle of the roll-compensated edge point and a 3D circle of an optical axis of the camera, determine (e.g., calculate) a first included angle between the two 3D circles, determine (e.g., calculate) a second included angle between a preset design value of the camera and the roll-compensated edge point, and estimate a pan of the camera using the first included angle and the second included angle.

[0112] The estimator may compensate the edge point and the feature point of the vehicle body region with the estimated roll and the estimated pan, determine (e.g., calculate) a feature point vector perpendicular to the edge using the edge and the feature point of the vehicle body region compensated with the roll and pan, determine (e.g., calculate) a third included angle between a 3D circle of the feature point vector and a 3D circle of the optical axis of the camera, determine (e.g., calculate) a fourth included angle between the design value of the camera and the feature point compensated with the roll and pan, and estimate a tilt of the camera using the third and fourth included angles.

[0113] The estimator can detect a straight line corresponding to a vanishing line based on the vehicle body area and estimate the pose of the camera using the detected straight line and a vanishing point detected by the image while the vehicle is traveling.

[0114] The estimator can generate a virtual line corresponding to the straight line by using a curve and curve information for the vehicle body area, and estimate the pose of the camera using the virtual line and the vanishing point when the curve is detected by the vehicle body area.

[0115] The estimator can estimate the pose of the camera based on horizontal direction information and vertical direction information when the horizontal direction information and the vertical direction information for the vehicle body area are acquired based on the vehicle body area.

[0116] The features of the present disclosure briefly summarized above are merely exemplary aspects of the detailed description of the present disclosure described below and do not limit the scope of the present disclosure.

[0117] The above description is merely illustrative of the technical idea of ​​the present disclosure, and various modifications and variations can be made by those skilled in the art without departing from the essential features of the present disclosure. Accordingly, the examples disclosed in the present disclosure are not intended to limit the technical idea of ​​the present disclosure but to describe the present disclosure, and the scope of the technical idea of ​​the present disclosure is not limited by the example. The scope of the present disclosure should be interpreted by the following claims, and all technical ideas within the scope of the equivalent range thereto should be deemed to be included in the scope of the present disclosure.

[0118] According to the present disclosure, it is possible to provide a camera calibration method and apparatus capable of estimating a camera pose based on the vehicle shape or vehicle body portion of a vehicle detected while traveling.

[0119] According to the present disclosure, it is possible to estimate the posture using information about the vehicle body portion of a vehicle captured by the camera, for example, at least one of horizontal direction information and vertical direction information and a vanishing point, which are obtainable during the travel of the vehicle.

[0120] According to the present disclosure, it is possible to estimate the camera pose using only two pieces of direction information without detecting a vanishing point if horizontal direction information for the vehicle body portion captured by the camera, such as information about the handle or the hood, and vertical direction information, such as information about the side of the vehicle, can be obtained.

[0121] According to the present disclosure, it is possible to transform, generate, or calibrate a curve or straight line that violates the MWA into a straight line suitable for the MWA when the curve or straight line is detected, thereby estimating the camera pose using all curves and straight lines that are detectable in the vehicle.

[0122] The effects obtainable in the present disclosure are not limited to the above-mentioned effects, and any other effects not mentioned herein will be clearly understood by those skilled in the art to whom the present disclosure is directed from the following description.

[0123] Although the present disclosure has been described with reference to examples and the accompanying drawings, the present disclosure is not limited thereto, but may be variously modified and changed by those skilled in the art without departing from the spirit and scope of the present disclosure as claimed in the following claims.

Claims

[1] Method comprising: Receiving an image of a vehicle captured by a camera; detecting a vehicle body portion of the vehicle in the image; Estimating, based on a slope of an edge connected to the vehicle body area, a position of the camera; and Performing image recognition on a calibrated image, wherein the calibrated image is based on a second image captured by the camera that was calibrated based on the estimated pose of the camera. [2] The method of claim 1, wherein detecting the vehicle body area comprises: detecting an edge point in one or more of a plurality of regions in the image; Determining, using RANSAC (RANdom SAmple Consensus), a number of outlier edge points in each region of the one or more regions; and Detecting a region of the one or more regions as a vehicle body region having a corresponding number of outlier edge points less than or equal to a preset value. [3] The method of claim 1, wherein estimating the pose of the camera comprises: Estimating the slope of the edge connected to the vehicle body area; Extracting a feature point of the vehicle body area; and Estimate the camera pose based on the tilt and feature point. [4] The method of claim 3, wherein the edge connected to the vehicle body portion is an edge of the vehicle body portion, and wherein estimating the pose of the camera comprises: Estimate camera roll based on the estimated edge tilt. [5] The method of claim 4, wherein estimating the position of the camera comprises: Compensating, based on the estimated roll, an edge point of the vehicle body area; Generate: a three-dimensional (3D) circle based on the compensated edge point, and a 3D circle of an optical axis of the camera; Determining a first included angle between the two 3D circles; Determining a second included angle between a preset design value of the camera and the compensated edge point; and Estimate, based on the first included angle and the second included angle, a pan of the camera. [6] The method of claim 5, wherein compensating the edge point further comprises compensating the edge point based on the estimated pan, and wherein estimating the pose of the camera further comprises: Compensating the feature point of the vehicle body area based on the estimated roll and pan; Determining, based on the compensated edge point and the compensated feature point, a feature point vector perpendicular to the edge of the vehicle body area, Determining a third included angle between a 3D circle corresponding to the feature point vector and a 3D circle corresponding to the optical axis of the camera; Determining a fourth included angle between the design value of the camera and the compensated feature point; and Estimate a camera tilt based on the third and fourth included angles. [7] The method of claim 1, wherein estimating the pose of the camera comprises: Detecting a straight line corresponding to a vanishing line determined based on the vehicle body area; and Estimate, based on the detected straight line and a vanishing point based on the vanishing line, the position of the camera while the vehicle is being driven. [8] The method of claim 7, wherein estimating the position of the camera comprises: Generating a virtual line corresponding to the straight line based on a curve detected in the vehicle body area; and Estimate the camera position based on the virtual line and vanishing point. [9] The method of claim 1, wherein estimating the pose of the camera comprises: Estimating the pose of the camera based on horizontal direction information and vertical direction information, wherein the horizontal direction information and the vertical direction information are acquired based on the vehicle body area. [10] Device comprising: a receiver configured to receive an image captured by a camera of a vehicle; a detector configured to detect a vehicle body portion of the vehicle in the image; and an estimator configured to estimate a pose of the camera based on an inclination of an edge connected to the vehicle body area, wherein the apparatus is configured to perform image recognition on a calibrated image, the calibrated image being based on a second image captured by the camera that was calibrated based on the estimated pose of the camera. [11] The apparatus of claim 10, wherein the detector is configured to: detect an edge point in one or more of a plurality of regions in the image; using RANSAC (RANdom Sample Consensus), to determine a number of outlier edge points in each region of the one or more regions; and a region of the one or more regions as the vehicle body region having a corresponding number of outlier edge points that is less than or equal to a preset value. [12] The apparatus of claim 10, wherein the estimator is configured to: to estimate the slope of the edge connected to the vehicle body area; Extracting a feature point of the vehicle body area; and to estimate the position of the camera based on the inclination and the feature point. [13] The apparatus of claim 12, wherein the edge connected to the vehicle body portion is an edge of the vehicle body portion, and wherein the estimator is configured to: to estimate camera roll based on the estimated slope of the edge. [14] The apparatus of claim 13, wherein the estimator is configured to: to compensate for an edge point of the vehicle body area based on the estimated roll; to generate: a 3D circle of the compensated edge point, and a 3D circle of an optical axis of the camera; to determine a first included angle between the two 3D circles; to determine a second included angle between a preset design value of the camera and the compensated edge point; and to estimate a pan of the camera based on the first included angle and the second included angle. [15] The apparatus of claim 14, wherein the estimator is configured to: to compensate the edge point and the feature point of the vehicle body area based on the estimated roll and pan; based on the compensated edge and the compensated feature point, determine a feature point vector perpendicular to the edge of the vehicle body area; determine a third included angle between a 3D circle corresponding to the feature point vector and a 3D circle of the optical axis of the camera; to determine a fourth included angle between the design value of the camera and the compensated feature point; and to estimate a tilt of the camera based on the third and fourth included angles. [16] The apparatus of claim 10, wherein the estimator is configured to: to capture a straight line corresponding to a vanishing line determined based on the vehicle body area; and to estimate the position of the camera while the vehicle is being driven based on the detected straight line and a vanishing point determined based on the vanishing line. [17] The apparatus of claim 16, wherein the estimator is configured to: to generate a virtual line corresponding to the straight line based on a curve detected in the vehicle body area; and to estimate the position of the camera based on the virtual line and the vanishing point. [18] The apparatus of claim 10, wherein the estimator is configured to: to estimate the pose of the camera based on horizontal direction information and vertical direction information, wherein the horizontal direction information and the vertical direction information are acquired based on the vehicle body area.

Images