Camera calibration method and device

Abstract

The invention relates to a camera calibration method and equipment. The camera calibration method includes: receiving an image captured by a camera of a vehicle; detecting a vehicle body area of the vehicle from the image; and estimating a posture of the camera based on the vehicle body region.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from Korean Patent Application No. 10-2023-0094095 filed in the Korean Intellectual Property Office on July 19, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a camera calibration technology, and more particularly, to a camera calibration method and apparatus capable of estimating a pose of a camera based on a vehicle shape or a body area of ​​a vehicle captured while the vehicle is traveling. Background Art

[0004] Various image recognition technologies are needed to operate autonomous vehicles. For example, it is important to recognize lanes and identify vanishing points while the vehicle is driving.

[0005] A vanishing point is a point where parallel straight lines in a 3-dimensional (3D) space meet on a 2-dimensional (2D) plane if the lines are infinitely extended and projected onto the plane. As an example of using the detection of vanishing points, by analyzing the architectural structure by obtaining the vanishing points and vanishing lines in three orthogonal directions, the building can be reinterpreted. In the 3D transformation of a 2D image containing the architectural structure, a depth map can be generated by detecting the vanishing points. The reason for this is that the estimation of relative depth is possible because as the 3D space is transformed into a 2D image, the portion where the vanishing point is located generally corresponds to the farthest portion in the image.

[0006] In addition, vanishing point information can be a reference for lane detection in autonomous vehicles or an important basis for analyzing position information in autonomous driving systems such as robots. This is because roads can be detected by connecting the main edges connected from the vanishing points.

[0007] When the vehicle is traveling, the vanishing point is estimated using line segments in the image, and a vanishing line is estimated based on the estimated vanishing point to estimate the external parameters of the camera and the road surface, thereby estimating the camera's pose.

[0008] However, there are problems in that the accuracy of the vanishing point decreases due to image distortion while the vehicle is traveling, and since the boundary of the vanishing point is defined according to the behavior of the vehicle while traveling, it is difficult to accurately estimate the vanishing line. Summary of the invention

[0009] The following summary presents a simplified overview of certain features. This summary is not an exhaustive overview and is not intended to identify key or critical elements.

[0010] Systems, devices, and methods for calibrating a camera of a vehicle are described. A method may include: receiving an image captured by a camera of the vehicle; detecting a body region of the vehicle in the image; estimating a pose of the camera based on a slope of an edge associated with the body region; performing image recognition on a calibrated image; the calibrated image is based on a second image acquired by the camera, the second image being calibrated based on the estimated pose of the camera.

[0011] Additionally or alternatively, a device may include: a receiver configured to receive an image captured by a camera of a vehicle; a detector configured to detect a body region of the vehicle in the image; and an estimator configured to estimate a pose of the camera based on a slope of an edge associated with the body region. The device may be configured to perform image recognition on a calibrated image, wherein the calibrated image is based on a second image acquired by the camera, the second image being calibrated based on the estimated pose of the camera.

[0012] These and other features and advantages are described in greater detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings:

[0014] Figure 1 An operation flow chart of the camera calibration method of the present disclosure is shown;

[0015] Figure 2 An operational flow chart illustrating an example of a process of detecting a body area of ​​a vehicle;

[0016] Figure 3 An operational flow diagram illustrating an example of a process for estimating camera roll;

[0017] Figure 4 An example diagram for describing a process of estimating camera roll is shown;

[0018] Figure 5 An operational flow diagram illustrating an example of a process for estimating camera pan;

[0019] Figure 6 An example diagram for describing a process of estimating camera pan is shown;

[0020] Figure 7 An example diagram for describing the relationship between an image domain and a Gaussian sphere domain is shown;

[0021] Figure 8 An operational flow chart illustrating an example of a process for estimating camera tilt;

[0022] Fig. 9 An example diagram for describing a process of estimating camera tilt is shown;

[0023] Fig.10 An example diagram is shown for describing a method for estimating camera pose in an example where the Manhattan World Assumption (MWA) is violated;

[0024] Fig.11 An example diagram illustrating a method of estimating a camera pose in another example where the Manhattan World Assumption (MWA) is violated is shown;

[0025] Fig.12 shows a configuration of a camera calibration device according to another example of the present disclosure; and

[0026] Fig.13 is a block diagram of a computing system for performing a camera calibration method according to examples of the present disclosure. DETAILED DESCRIPTION

[0027] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may include several different forms and is not limited to the examples described herein.

[0028] When describing the present disclosure, if it is determined that the detailed description of a known configuration or function may obscure the gist of the present disclosure, its detailed description will be omitted. In addition, in the accompanying drawings, components irrelevant to the description of the present disclosure are omitted, and similar reference numerals are attached to similar components.

[0029] It will be understood that if an element is referred to as being "connected," "coupled," or "fixed" to another element, it may be directly connected or coupled to the other element, or intervening elements may be present. Furthermore, unless explicitly described to the contrary, the words "comprise" or "include" will be understood to imply the inclusion of the stated elements but not the exclusion of any other elements.

[0030] In the present disclosure, terms such as "first" and "second" are used only for the purpose of distinguishing one element from other elements, and do not limit the order or importance of the elements unless otherwise specified. Therefore, within the scope of the present invention, a first element in one example may be referred to as a second element in another example, and similarly, a second element in one example may be referred to as a first element in another example.

[0031] In the present disclosure, different elements are used only to clearly describe their features and do not necessarily mean that the elements are separate. That is, multiple elements can be integrated to form one hardware or software unit, or one element can be distributed to form multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed examples are also included in the scope of the present disclosure.

[0032] In the present disclosure, the elements described in various examples do not necessarily mean essential components, and some may be optional elements. Therefore, examples consisting of a subset of the elements described in an example are also included in the scope of the present disclosure. In addition, examples including other elements other than the elements described in various examples are also included in the scope of the present disclosure.

[0033] In the present disclosure, for the convenience of description, the expressions describing the positional relationship used in the specification, such as top, bottom, left or right, are used, and if the drawings shown in the specification are viewed in reverse, the positional relationship described in the specification can also be interpreted in the opposite way.

[0034] In the present disclosure, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", "at least one of A, B or C" may include any one of the items listed together in the corresponding phrase, or any possible combination thereof.

[0035] In order to solve the problem that it is difficult to accurately estimate vanishing lines when a vehicle (e.g., a vehicle including and / or connected to a camera) is driving, the present disclosure provides systems, devices, and methods for detecting a vehicle shape and / or body area of ​​a vehicle captured by a camera. The vehicle shape and / or body area may include some or all of a handle, a hood, and / or a side of the vehicle. The method disclosed herein can improve the estimation accuracy of the camera's pose (e.g., the camera's three-axis angles (roll, pan, tilt)) by using the detected vehicle shape and / or body area as a reference instead of, or in addition to, the vanishing lines and / or vanishing points.

[0036] The present disclosure may estimate the camera pose based on the Manhattan World Assumption (MWA) of the transformation between the image domain and the Gaussian sphere domain. Specifically, the present disclosure may use the MWA to estimate the camera pan based on the angle between the camera optical axis and a straight line in the vehicle body region. The present disclosure may estimate the camera tilt based on the angle between the camera optical axis and a feature point vector, such as a vector corresponding to a feature point associated with a handle (e.g., of the handle).

[0037] The present disclosure may use the analyzed (eg, determined) body area as an alternative and / or supplement to vanishing points and / or vanishing lines.The body area of ​​the vehicle detected in the image may be analyzed and thus used to estimate the pose of the camera.

[0038] In the following, reference will be made to Figures 1 to 12 Methods and apparatus according to the present disclosure are described.

[0039] Figure 1 is an operational flow chart illustrating an example camera calibration method according to the present disclosure, which may be performed by a camera calibration device. The camera calibration device may be disposed in a vehicle and / or may be remote from the vehicle (e.g., in communication with the vehicle and / or capable of receiving information from the vehicle and / or sending information to the vehicle). For example, the camera calibration device may be capable of communicating with a camera of the vehicle.

[0040] refer to Figure 1 According to an example camera calibration method of the present disclosure, the camera calibration method may include receiving (e.g., in real time or near real time) an image captured by an image capture device (e.g., a camera) while a vehicle is traveling, and detecting a body area of ​​the vehicle (e.g., an image area corresponding to an image of a portion of the vehicle) from the received image (S110 and S120).

[0041] At S110, an image captured by at least one camera (eg, at least one of a front camera, a rear camera, a front corner camera, and / or a rear corner camera) may be received. In the example discussed herein, an image captured by a rear corner camera may be received.

[0042] At S120, a body region of the vehicle may be detected in the image. For example, one or more edge points may be detected in a series of images from a camera, and it may be determined that the one or more edge points do not substantially change over time (e.g., while other regions of the image may appear to change and / or move over time). Other edge detection methods may also or alternatively be used to detect edges and / or edge points of the body region. The edges of the body region detected in the image may be determined based on the positions of the detected edge points.

[0043] Based on the body area detected in S120, the slope of the edge of the detected body area can be estimated, feature points of the detected body area can be extracted, and / or the pose of the camera can be estimated. For example, the three-axis angles of roll, pan and tilt can be estimated (S130, S140 and S150).

[0044] At S150 , the roll of the camera may be estimated based on the estimated slope, the pan of the camera may be estimated based on the position of the edge of the body area, and the tilt of the camera may be estimated based on the feature points of the body area, thereby estimating the pose of the camera.

[0045] Reference Figures 2 to 9 Methods according to examples of the present disclosure are described in detail.

[0046] Figure 2 is an operational flowchart of an example of a process of detecting a body area of ​​a vehicle, and is Figure 1 An example operation flow chart of S120 is shown.

[0047] refer to Figure 2 , detecting the body area of ​​the vehicle (S120) may include calling or receiving image data captured by a camera when the vehicle is traveling at a certain speed or higher, correcting image distortion, and then detecting edge points in the distortion-corrected image (S210, S220, and S230).

[0048] Because it is difficult to identify time-invariant edge points (eg, edge points that do not change over time between consecutive images) if the vehicle is stationary, edge points may be detected from the image based on the vehicle's speed being greater than or equal to a predetermined value.

[0049] If edge points in the image are detected in S230, by applying specific logic (e.g., AND logic) to the edge points detected over time and applying RANSAC (Random Sampling Consensus) to the edge points to which the AND logic has been applied, a first-order straight line of the edge points can be extracted to determine the number of outliers (S240).

[0050] Applying AND logic to the edge points in S240 may be used to easily detect the body region because the positions of the edge points of the body region remain unchanged even if the vehicle is moving or driving, but edge points detected outside the body region may move.

[0051] Here, RANSAC is an algorithm for removing noise from a data set and predicting a model, and has the characteristic of completely ignoring data above a certain threshold, making it robust to outliers. For example, RANSAC can extract an ideal model (e.g., a first-order straight line) where there is no noise and the maximum data is the same. RANSAC is known to those skilled in the art, and therefore a detailed description thereof will be omitted.

[0052] For the area of ​​the image, if the first-order straight line of the edge point therein is extracted in S240, the number of outliers can be measured using the result of RANSAC. If the number of outliers is less than a preset specific value, the corresponding edge point can be determined as the body area of ​​the vehicle. Through this process, the body area of ​​the vehicle can be detected (S250, S260, and S270). For example, since the number of outliers exceeds a specific value, that is, a large number of outliers, it means that edge points have been detected in areas other than the vehicle, so if the number of outliers is less than or equal to a specific value, the edge points in the corresponding area can be determined as edge points of the body area.

[0053] For example, if the image captured by the camera is Figure 4 As shown in (a), it can be Figure 2 The processing detects the area of ​​the vehicle as the body area. In this case, the area of ​​the vehicle may include a handle area 410 corresponding to the feature point, such as Figure 4 As shown in (a).

[0054] Figure 3 For example, based on Figure 2 An operational flow chart of a process of processing detected vehicle body regions to estimate camera roll.

[0055] refer to Figure 3 , the process of estimating the camera roll can estimate the camera roll by calling the first-order straight line (which is the result of RANSAC for the detected edge points of the vehicle body area) and calculating the slope of the first-order straight line of the vehicle body area relative to the vertical line (S310, S320 and S330).

[0056] In S330, since it is assumed that the side of the vehicle is perpendicular to the road surface in the 3D space and the slope of the body area is assumed to be the camera roll, the slope of the first-order straight line of the body area can be estimated as the camera roll. Figure 4 As shown in (b), the slope of the straight line 420 at the edge of the vehicle body region (ie, the slope of the straight line 420 at the edge of the vehicle body region relative to the vertical line) can be estimated as the camera roll.

[0057] In S330 , the slope of the first-order straight line relative to the vertical line may be determined (eg, calculated), a histogram of the gradient of the image may be generated to output a slope corresponding to a peak of the histogram, and then the slope may be rotated 90 degrees to estimate the camera roll.

[0058] In S330, the camera roll may be estimated using feature points (eg, and / or feature point vectors and / or feature point lines) included in (eg, detected in) the vehicle body region. Figure 4As shown in (c), the camera roll may be estimated using the handle region 410, which is a feature included in the vehicle body region. For example, since the handle region 410 (which is a feature included in the vehicle body region) forms a right angle with the edge of the vehicle body region, the angle between the horizontal line and the handle region 410 may be estimated as the camera roll.

[0059] In S330, the roll of the camera may be estimated by generating a 3D circle of the vertical line, generating a 3D circle of the detected edge of the vehicle, and then determining (e.g., calculating) the angle between the two 3D circles. Here, since the vertical line is perpendicular to the road surface, it may be assumed that the angle between the vertical line and the edge of the vehicle body area is equivalent to the roll of the camera. The reason for this assumption may be that the vertical line is rotated by the camera roll. The above method is a method using MWA, which refers to Figure 7 Described in further detail.

[0060] Figure 5 is an operational flow chart of an example of a process of estimating camera pan. Camera pan may be estimated using estimated camera roll.

[0061] refer to Figure 5 , the process of estimating the camera pan may include calling edge points detected as the vehicle body region and compensating the edge points of the vehicle body region using the estimated camera roll ( S510 and S520 ).

[0062] In S520, if the edge points of the vehicle body area are compensated by the estimated camera roll, a 3D circle can be generated for the edge points compensated for the camera roll, a 3D circle of the optical axis of the camera can be generated, and the angle β1 between the two 3D circles can be determined (e.g., calculated) (S530, S540 and S550).

[0063] Based on preset design values ​​of the camera (e.g., used as input) and edge points compensated for camera roll, the preset design values ​​corresponding to, for example, the camera position in the vehicle, the angle α1 between the line between the camera and the edge point and the side of the vehicle can be determined (e.g., calculated), and / or the difference between the two angles (β1-α1) can be used to estimate the camera pan (S560 and S570).

[0064] That is, Figure 6 As shown in , the process of estimating camera pan may include determining (e.g., calculating) an angle β1 between a line between the camera and edge point 620 and the camera optical axis 610 and a design value of the camera, for example, an angle α1 between a line between the camera position and edge point 620 and the side of the vehicle. The camera pan may be estimated based on the difference between the two angles (β1-α1).

[0065] Here, if Figure 7As shown in (a), the angle β1 between the camera optical axis and the edge point can be determined (e.g., calculated) based on the Manhattan World Assumption (MWA). That is, based on MWA, the linear component in the image domain can be expressed as a circle in the Gaussian sphere domain. The linear component in the image domain and the circle in the Gaussian sphere domain can be transformed into each other. That is, as Figure 7 As shown in (b), the linear component corresponding to the camera optical axis 610 can be transformed into or represented as a circle 710 in the Gaussian sphere domain, and the line 620 between the camera and the edge point can also be transformed into or represented as a circle 720. Therefore, the two circles 710 and 720 can be used to determine (e.g., calculate) the angle β1 between the camera optical axis and the edge point of the vehicle body area.

[0066] MWA is briefly described as follows. Any plane can be represented by a vector perpendicular to that plane. The six faces of a hexahedron can be represented by six vectors - ignoring directionality, they can be represented by three vectors. The virtual space formed only by the faces belonging to these three vectors is called MWA, and any linear component passes through one of the three faces corresponding to one of the three vectors. If the camera pose changes, the hexahedron also rotates, and the three vectors also rotate. Therefore, the camera pose can be estimated by knowing how much the three vectors have rotated. Assuming that the vehicle is a hexahedron, the camera pose can be estimated by finding the three vectors of the vehicle in MWA. In addition, as Figure 7 As shown in (a), a method of representing the linear component of the image domain and the circle of the Gaussian sphere domain is obvious to those skilled in the art, and thus a description thereof is omitted.

[0067] Figure 8 : is an operational flowchart of an example of a process of estimating a camera tilt, which is an operational flowchart of a process of estimating a camera tilt using the camera roll and the camera pan estimated by the above-described process.

[0068] refer to Figure 8 , the process of estimating the camera tilt may include calling edge points and feature points detected as the vehicle body area (e.g., feature points for the handle), and compensating the edge points and feature points using the camera roll and camera pan estimated in the above process (S810 and S820).

[0069] Based on the edge points and feature points compensated by camera roll and camera pan, the vector perpendicular to the edge of the vehicle body area (i.e., the feature point vector) can be determined (e.g., calculated), a 3D circle of the feature point vector and a 3D circle of the camera optical axis can be generated, and the angle β2 between the two circles can be determined (e.g., calculated) (S830, S840, S850, and S860).

[0070] In addition, using the preset design values ​​of the camera (e.g., the camera position in the vehicle) and the feature point vector compensated by camera roll and camera pan, the angle α2 between the line between the camera and the feature point vector and the top surface of the vehicle can be determined (e.g., calculated), and the difference between the two angles (β2-α2) can be used to estimate the camera tilt (S870 and S880).

[0071] That is, Fig. 9 As shown, the process of estimating the camera tilt can include determining (e.g., calculating) the angle β2 between the line between the camera and the feature point vector 920 and the camera optical axis 910, the design value of the camera, for example, the angle α2 between the feature point vector 920 (which is the line between the camera position and the feature point) and the top of the vehicle, and estimating the tilt of the camera based on the difference between the two angles.

[0072] As described above, the camera calibration method according to the example of the present disclosure can detect the body area in the image captured by the moving vehicle, and estimate the camera pose based on MWA using the slope, feature points, camera optical axis and design value of the body area. In addition, the camera calibration method according to the example of the present disclosure can utilize all the features of the vehicle that are perpendicular or horizontal to the road surface (e.g., the side of the vehicle, the front bumper of the vehicle, the rear bumper of the vehicle, the connection between the glass and the top of the vehicle, etc.) as the body area captured by the camera.

[0073] The above description is made for the case where MWA is satisfied, and it can also be applied to the case where a curve or straight line that violates MWA is detected, which will be referred to in Figure 10 to Figure 11 Give a description.

[0074] Fig.10 is an exemplary diagram for describing a method of estimating a camera pose in an example in which the Manhattan World Assumption (MWA) is violated, and shows an exemplary diagram for describing a method of estimating a camera pose using a curve if a curve violating the MWA is detected.

[0075] like Fig.10 As shown in (a), a curve 1010 may be detected in a vehicle body area detected in an image captured by a camera, and the curve, which is not a straight line, violates the MWA. In an example of the present disclosure, the camera pose can be estimated by transforming the curve into a straight line suitable for the MWA. For example, a straight line suitable for the MWA and closest to the detected curve can be generated by a drawing of the corresponding vehicle. That is, the relationship between the detected curve 1010 and the straight line 1020 can be known using the drawing of the vehicle, and through this, the detected curve 1010 can be transformed into the straight line 1020 by generating a virtual line 1020 that matches the detected curve 1010 for the detected curve 1010. For example, as Fig.10As shown in (b), if the body area detected in the image is the front hood, since the front hood is symmetrical left and right, like a decal pattern, a straight line 1030 that makes the detected curve 1010 symmetrical is found, and a straight line 1020 that is perpendicular to the straight line 1030 and tangent to the detected curve 1010 is generated. In this case, the generated straight line 1020 can satisfy the MWA. Therefore, the method according to the example of the present disclosure can estimate the pose of the camera based on the straight line obtained by transforming the curve.

[0076] Fig.11 is an exemplary diagram for describing a method of estimating a camera pose in another example in which the Manhattan World Assumption (MWA) is violated, which shows an exemplary diagram for describing a method of estimating a camera pose using calibration of a straight line that dissatisfies the MWA if the straight line is detected.

[0077] like Fig.11 As shown in (a), a straight line may be detected instead of a curve in the body area detected in the image captured by the camera. However, the plane formed by the detected straight line 1120 may violate the MWA. The camera pose can be estimated while satisfying the MWA by calibrating the straight line that violates the MWA using a drawing of the corresponding vehicle (e.g., a representation / schematic diagram of a known shape of the vehicle).

[0078] For example, Fig.11 As shown in (a) of FIG. 1 , if a straight line 1120 on the passenger seat window side is detected in the vehicle body area detected in the image, the angle α3 with the vertical line 1110 can be identified by referring to the drawing of the corresponding vehicle. Therefore, even if a straight line that violates the MWA is detected, after estimating the roll value with the straight line 1120, the roll value can be calibrated by the angle α3, and the camera pose can be estimated using the drawing of the vehicle.

[0079] As described above, even if a curve or straight line that violates the MWA is detected, the camera pose can be estimated using the above method using all detectable curves and straight lines of the vehicle (eg, in the body area of ​​the image from the camera).

[0080] That is, Fig.11 As shown in (b), by fitting any vehicle line segment to a straight line 1130 suitable for MWA, all detectable curves and straight lines of the vehicle (e.g., in an image from a camera) can be used to estimate the camera pose.

[0081] As described above, in the camera calibration method according to the example of the present disclosure, if a body area of ​​a vehicle is detected in an image captured when the vehicle is traveling and either horizontal direction information or vertical direction information is obtained based on the body area, the camera posture can be estimated using a straight line obtained by either the horizontal direction information or the vertical direction information and a vanishing point obtainable when the vehicle is traveling.

[0082] The camera calibration method according to the example of the present disclosure can estimate the camera pose using only the direction information from the detected body area without detecting the vanishing point. For example, the horizontal direction information can be obtained from the body area of ​​the vehicle captured by the camera (such as in the image of the handle or the hood), and the vertical direction information can be obtained from the body area (such as in the image of the side of the vehicle).

[0083] The camera calibration method according to the example of the present disclosure can transform a curve or straight line that violates the MWA into a straight line suitable for the MWA and / or generate a straight line suitable for the MWA based on the detected curve or straight line. The transformed and / or generated line suitable for the MWA can be used to detect the camera pose using any (e.g., all) curves and / or straight lines that can be detected in the vehicle body area.

[0084] In addition, the camera calibration method according to the example of the present disclosure can use an artificial intelligence network (e.g., a deep learning-based network) as a method for detecting the body area in the image captured by the camera. For example, a segmentation technique can be used to detect the body area using an artificial intelligence network.

[0085] In addition, the camera calibration method according to the example of the present invention can detect straight lines corresponding to vanishing lines based on the vehicle body area. When the vehicle is traveling, the camera pose can be estimated using the straight lines detected in the vehicle body area and the vanishing points detected in the image captured by the camera.

[0086] In addition, the camera calibration method according to the example of the present disclosure may estimate the camera pose based on the horizontal direction information and the vertical direction information (e.g., based on the horizontal direction information and the vertical direction information detected in the vehicle body area). In this case, the method of the present invention may estimate the camera pose using only the horizontal direction information and the vertical direction information (e.g., without detecting and / or using vanishing lines or vanishing points).

[0087] Fig.12 The configuration of a camera calibration device according to another example of the present disclosure is shown. Figures 1 to 11 The method of device configuration.

[0088] refer to Fig.12, a camera calibration apparatus 1200 according to another example of the present disclosure may include a receiver 1210 , a detector 1220 , an estimator 1230 , and a storage device 1240 .

[0089] The storage device 1240 may be a component for storing data related to the technology of the present disclosure. The storage device 1240 may store data such as one or more images captured by a camera, algorithms related to the technology of the present disclosure (e.g., RANSAC, edge points, RANSAC results), and vehicle drawings, vehicle data or camera design values ​​and / or instructions that, when executed, cause the camera calibration device 1200 to perform one or more methods disclosed herein.

[0090] Receiver 1210 may receive images captured by a camera of the vehicle.

[0091] Receiver 1210 may receive images captured by the camera if (eg, based on) the vehicle is traveling at a particular speed or higher.

[0092] The detector 1220 may detect a body area of ​​the host vehicle in the image received by the receiver 1210 .

[0093] Detector 1220 can detect one or more edge points for one or more regions of multiple regions of an image, measure the number of outliers in the one or more edge points of each of the one or more regions (e.g., using RANSAC), and determine (e.g., detect) a region in the image where the number of outliers is equal to or less than a preset specific value (e.g., a threshold) as a vehicle body region.

[0094] The estimator 1230 may estimate the pose of the camera based on the vehicle body region detected by the detector 1220 .

[0095] The estimator 1230 may estimate a slope of the vehicle body region based on the vehicle body region, extract feature points of the vehicle body region, and estimate a pose of the camera using the slope and the feature points.

[0096] The estimator 1230 may estimate a slope of an edge of the vehicle body region and estimate a camera roll based on the estimated slope of the edge.

[0097] The estimator 1230 can use the estimated roll to compensate for the edge points of the vehicle body area, generate a 3D circle of the edge points that have been compensated for the roll and a 3D circle of the optical axis of the camera, determine (e.g., calculate) a first angle between the two 3D circles, determine (e.g., calculate) a second angle between a preset design value of the camera and the edge points that have been compensated for the roll, and estimate the panning of the camera using the first angle and the second angle.

[0098] The estimator 1230 can compensate for edge points and feature points of the vehicle body area with the estimated roll and pan, determine (e.g., calculate) a feature point vector perpendicular to the edge using the edge points and feature points of the vehicle body area compensated for with the roll and pan, determine (e.g., calculate) a third angle between a 3D circle of the feature point vector and a 3D circle of the optical axis of the camera, determine (e.g., calculate) a fourth angle between a design value of the camera and the feature points compensated for with the roll and pan, and estimate the tilt of the camera using the third angle and the fourth angle.

[0099] The estimator 1230 may detect straight lines corresponding to vanishing lines based on the vehicle body region while traveling, and estimate the pose of the camera using the detected straight lines and vanishing points detected through an image of the vehicle.

[0100] If a curve is detected through the vehicle body region, the estimator 1230 may generate a virtual line corresponding to the straight line using the curve and curve information of the vehicle body region (eg, drawing information of the vehicle), and estimate the camera pose using the virtual line and the vanishing point.

[0101] If horizontal direction information and vertical direction information of the vehicle body region are detected based on the vehicle body region, the estimator 1230 may estimate the camera pose based on the horizontal direction information and the vertical direction information.

[0102] Even if description is omitted in a device according to another example of the present disclosure, the device according to another example of the present disclosure may include Figures 1 to 11 This is obvious to those skilled in the art.

[0103] Fig.13 is a block diagram of a computing system for executing the camera calibration method of the present disclosure.

[0104] refer to Fig.13 The method for estimating a vanishing point according to another example of the present disclosure may be implemented by a computing system. The computing system 2000 may include at least one processor 2100, a memory 2300, a user interface input device 2400, a user interface output device 2500, a storage device 2600, and a network interface 2700 connected to each other via a system bus 2200.

[0105] The processor 2100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 2300 and / or the storage device 2600. The memory 2300 and the storage device 2600 may include various types of volatile or non-volatile storage media. For example, the memory 2300 may include a ROM (Read Only Memory) 2310 and a RAM (Random Access Memory) 2320.

[0106] Therefore, the operation of the method or algorithm described in conjunction with the examples disclosed herein may be directly embodied in a hardware or software module executed by the processor 2100, or in a combination thereof. The software module may reside on a storage medium (i.e., memory 2300 and / or storage device 2600) such as RAM, flash memory, ROM, EPROM, EEPROM, register, hard disk, removable disk, and CD-ROM. An exemplary storage medium may be coupled to the processor 2100, and the processor 2100 may read information from the storage medium and may record information in the storage medium. Alternatively, the storage medium may be integrated with the processor 2100. The processor 2100 and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a user terminal. In another case, the processor 2100 and the storage medium may reside in a user terminal as separate components.

[0107] The present disclosure has been made to solve the above-mentioned problems occurring in the prior art while keeping the advantages achieved by the prior art intact.

[0108] An aspect of the present disclosure provides a camera calibration method and apparatus capable of estimating a pose of a camera based on a vehicle shape or a body area of ​​a vehicle captured while the vehicle is traveling.

[0109] One aspect of the present invention provides a camera calibration method and apparatus capable of estimating the pose of a camera using information about a vehicle's body area captured by a camera (e.g., at least one of horizontal direction information and vertical direction information and a vanishing point that can be obtained when the vehicle is traveling).

[0110] One aspect of the present invention provides a camera calibration method and apparatus capable of estimating the posture of a camera using only two pieces of direction information, namely, horizontal direction information (e.g., information about a handle or a hood) and vertical direction information (e.g., information about the side of the vehicle) of a body area of ​​a vehicle captured by a camera, without detecting a vanishing point.

[0111] One aspect of the present disclosure provides a camera calibration method and apparatus that is capable of estimating (e.g., configured to estimate) a camera's pose by utilizing all curves and straight lines detectable in a vehicle by performing a transformation, generating, or performing calibration on straight lines suitable for a Manhattan World Assumption (MWA) if curves and straight lines violating the MWA are detected.

[0112] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art to which the present disclosure belongs from the following description.

[0113] According to an aspect of the present disclosure, a camera calibration method includes: receiving an image captured by a camera of a vehicle; detecting a body region of the vehicle from the image; and estimating a pose of the camera based on the body region.

[0114] Detecting a vehicle body area may include detecting one or more edge points in one or more regions of a plurality of regions of an image, measuring the number of outliers in the edge points in each of the one or more regions using RANSAC (Random Sampling Consensus), and detecting (e.g., determining) an area in which the number of outliers in the one or more regions of the image is less than or equal to a preset value (e.g., a threshold) as a vehicle body area.

[0115] The step of estimating the pose of the camera may include: estimating a slope of the vehicle body region based on the vehicle body region; extracting feature points of the vehicle body region; and estimating the pose of the camera using the slope and the feature points.

[0116] Estimating the pose of the camera may include estimating a slope of an edge of the vehicle body region and estimating a roll of the camera based on the estimated slope of the edge.

[0117] Estimating the camera pose may include: compensating edge points of a vehicle body area using estimated roll; generating a 3D circle of the edge points that have been compensated for roll and a 3D circle of the optical axis of the camera; determining (e.g., calculating) a first angle between the two 3D circles, calculating a second angle between a preset camera design value and the edge points after roll compensation, and estimating (e.g., determining and / or calculating an estimate) the panning of the camera using the first angle and the second angle.

[0118] The step of estimating the posture of the camera may include: compensating edge points and feature points of the vehicle body area using estimated roll and estimated pan; calculating feature point vectors perpendicular to the edges using edge points and feature points of the vehicle body area compensated for roll and pan, calculating a third angle between a 3D circle of the feature point vector and a 3D circle of the optical axis of the camera, calculating a fourth angle between a design value of the camera and the feature points compensated for roll and pan, and estimating the tilt of the camera using the third angle and the fourth angle.

[0119] The step of estimating the pose of the camera may include detecting a straight line corresponding to a vanishing line based on the vehicle body region; and estimating the pose of the camera using the detected straight line and a vanishing point detected through an image while the vehicle is traveling.

[0120] The estimating the pose of the camera may include: if a curve is detected through the vehicle body region, generating a virtual line corresponding to the straight line using the curve and curve information of the vehicle body region; and estimating the pose of the camera using the virtual line and the vanishing point.

[0121] The estimating the pose of the camera may include estimating the pose of the camera based on the horizontal direction information and the vertical direction information if horizontal direction information and vertical direction information of the vehicle body region are detected based on the vehicle body region.

[0122] According to the present disclosure, a camera calibration device may include: a receiver configured to receive an image captured by a camera of a vehicle; a detector configured to detect a body area of ​​the vehicle in the image; and an estimator configured to estimate a pose of the camera based on the detected body area.

[0123] The detector can detect edge points in one or more regions among multiple regions in an image; for each of the one or more regions, determine (e.g., measure) the number of outliers in the edge points in each of the one or more regions (e.g., using RANSAC (Random Sampling Consensus)), and detect (e.g., determine) a region where the number of outliers is less than or equal to a preset value (e.g., the accumulation of one or more regions) as a body region (e.g., a part of the body region).

[0124] The estimator may estimate a slope of the vehicle body region based on the vehicle body region, extract feature points of the vehicle body region, and estimate a pose of the camera using the slope and the feature points.

[0125] The estimator may estimate a slope of an edge of the vehicle body region and estimate a roll of the camera based on the estimated slope of the edge.

[0126] The estimator can use the estimated roll-compensated edge points of the vehicle body area to generate a 3D circle of the edge points that have been roll-compensated and a 3D circle of the optical axis of the camera, determine (e.g., calculate) a first angle between the two 3D circles, determine (e.g., calculate) a second angle between a preset design value of the camera and the edge points that have been roll-compensated, and estimate the panning of the camera using the first angle and the second angle.

[0127] The estimator can use the estimated roll and the estimated pan to compensate for the edge points and feature points of the vehicle body area, determine (e.g., calculate) a feature point vector perpendicular to the edge using the edge points and feature points of the vehicle body area compensated for the roll and pan, determine (e.g., calculate) a third angle between a 3D circle of the feature point vector and a 3D circle of the optical axis of the camera, determine (e.g., calculate) a fourth angle between a design value of the camera and the feature points compensated for the roll and pan, and estimate the tilt of the camera using the third angle and the fourth angle.

[0128] The estimator may detect straight lines corresponding to vanishing lines based on the vehicle body region, and estimate the pose of the camera using the detected straight lines and vanishing points detected through images while the vehicle is traveling.

[0129] If a curve is detected through the vehicle body region, the estimator may generate a virtual line corresponding to the straight line using the curve and curve information of the vehicle body region, and estimate the pose of the camera using the virtual line and the vanishing point.

[0130] If horizontal direction information and vertical direction information of the vehicle body region are detected based on the vehicle body region, the estimator may estimate the pose of the camera based on the horizontal direction information and the vertical direction information.

[0131] The features briefly summarized above for the present disclosure are merely exemplary aspects of the detailed description of the present disclosure to be described below, and do not limit the scope of the present disclosure.

[0132] The above description is only an illustration of the technical concept of the present disclosure, and a person skilled in the art to which the present disclosure belongs may make various modifications and changes without departing from the essential features of the present disclosure. Therefore, the examples disclosed in the present disclosure are not intended to limit the technical concept of the present disclosure, but are used to describe the present disclosure, and the scope of the technical concept of the present disclosure is not limited by the examples. The scope of protection of the present disclosure shall be interpreted by the attached claims, and all technical concepts within the scope equivalent thereto shall be interpreted as included within the scope of the present disclosure.

[0133] According to the present disclosure, a camera calibration method and apparatus capable of estimating a camera pose based on a vehicle shape or a body area of ​​a vehicle captured while the vehicle is traveling can be provided.

[0134] According to the present disclosure, a pose may be estimated using information (eg, at least one of horizontal direction information and vertical direction information) about a vehicle body region captured by a camera and a vanishing point available while the vehicle is traveling.

[0135] According to the present disclosure, if horizontal direction information (such as information about a handle or a hood) and vertical direction information (such as information about the side of the vehicle) of a body area of ​​a vehicle captured by a camera are available, the camera pose can be estimated using only these two pieces of direction information without detecting a vanishing point.

[0136] According to the present disclosure, if a curve or straight line that violates the MWA is detected, the curve or straight line can be transformed, generated, or calibrated into a straight line suitable for the MWA, thereby estimating the camera pose using all curves and straight lines that can be detected in the vehicle.

[0137] Effects obtainable in the present disclosure are not limited to the above-mentioned effects, and any other effects not mentioned herein will be clearly understood from the following description by those skilled in the art to which the present disclosure pertains.

[0138] Although the present disclosure has been described above with reference to the examples and drawings, the present disclosure is not limited thereto but various modifications and changes may be made by those skilled in the art without departing from the spirit and scope of the present disclosure as claimed in the appended claims.

Claims

1. A camera calibration method, comprising: receiving an image captured by a camera of the vehicle; detecting a body region of the vehicle in the image; estimating a pose of the camera based on a slope of an edge associated with the vehicle body region; as well as Image recognition is performed on a calibrated image, wherein the calibrated image is based on a second image acquired by the camera, the second image being calibrated based on the estimated pose of the camera.

2. The camera calibration method according to claim 1, wherein: Detecting the vehicle body area includes: Detecting edge points in one or more regions of a plurality of regions in the image; determining a number of outliers in the edge points in each of the one or more regions using random sampling consistency; and A region in the one or more regions where the number of corresponding abnormal values ​​in the edge points is less than or equal to a preset value is detected as the vehicle body region.

3. The camera calibration method according to claim 1, wherein: Estimating the pose of the camera includes: estimating a slope of the edge associated with the body region; extracting feature points of the vehicle body area; and A pose of the camera is estimated based on the slope and the feature points.

4. The camera calibration method according to claim 3, wherein: The edge associated with the body region is an edge of the body region, and wherein estimating the pose of the camera comprises: The roll of the camera is estimated based on the estimated slope of the edge.

5. The camera calibration method according to claim 4, wherein: Estimating the pose of the camera includes: compensating edge points of the vehicle body region based on the estimated roll; generate: The 3D circle of compensated edge points, and A three-dimensional circle of the optical axis of the camera; determining a first angle between two three-dimensional circles; Determining a second angle between a preset design value of the camera and the compensated edge point; and A pan of the camera is estimated based on the first angle and the second angle.

6. The camera calibration method according to claim 5, wherein: Compensating the edge points further comprises compensating the edge points based on an estimated pan, and wherein estimating a pose of the camera further comprises: compensating the feature points of the vehicle body region based on the estimated roll and the estimated pan; Based on the compensated edge points and the compensated feature points, determining a feature point vector perpendicular to the edge of the vehicle body area, Determine a third angle between the three-dimensional circle corresponding to the feature point vector and the three-dimensional circle corresponding to the optical axis of the camera; determining a fourth angle between the design value of the camera and the compensated feature point; and The tilt of the camera is estimated based on the third angle and the fourth angle.

7. The camera calibration method according to claim 1, wherein: Estimating the pose of the camera includes: detecting a straight line corresponding to a vanishing line determined based on the vehicle body region; and The posture of the camera when the vehicle is traveling is estimated based on the detected straight lines and the vanishing points based on the vanishing lines.

8. The camera calibration method according to claim 7, wherein: Estimating the pose of the camera includes: generating a virtual line corresponding to the straight line based on a curve detected in the vehicle body region; and A pose of the camera is estimated based on the virtual line and the vanishing point.

9. The camera calibration method according to claim 1, wherein: Estimating the pose of the camera includes: The posture of the camera is estimated based on horizontal direction information and vertical direction information, wherein the horizontal direction information and the vertical direction information are detected based on the vehicle body region.

10. A camera calibration device, comprising: a receiver configured to receive an image captured by a camera of the vehicle; a detector configured to detect a body region of the vehicle in the image; as well as an estimator configured to estimate a pose of the camera based on a slope of an edge associated with the vehicle body region, The camera calibration device is configured to perform image recognition on a calibrated image, wherein the calibrated image is based on a second image acquired by the camera, and the second image is calibrated based on the estimated pose of the camera.

11. The camera calibration device according to claim 10, wherein: The detector is configured as: Detecting edge points in one or more regions of a plurality of regions in the image; determining a number of outliers in the edge points in each of the one or more regions using random sampling consistency; as well as A region in the one or more regions where the number of corresponding abnormal values ​​in the edge points is less than or equal to a preset value is detected as the vehicle body region.

12. The camera calibration device according to claim 10, wherein: The estimator is configured as: estimating a slope of the edge associated with the body region; Extracting feature points of the vehicle body area; as well as A pose of the camera is estimated based on the slope and the feature points.

13. The camera calibration device according to claim 12, wherein: The edge associated with the body region is an edge of the body region, and wherein the estimator is configured to: The roll of the camera is estimated based on the estimated slope of the edge.

14. The camera calibration device according to claim 13, wherein: The estimator is configured as: compensating edge points of the vehicle body region based on the estimated roll; generate: The 3D circle of compensated edge points, and A three-dimensional circle of the optical axis of the camera; determining a first angle between two three-dimensional circles; Determining a second angle between a preset design value of the camera and the compensated edge point; as well as A pan of the camera is estimated based on the first angle and the second angle.

15. The camera calibration device according to claim 14, wherein: The estimator is configured as: compensating the edge points and the feature points of the vehicle body region based on the estimated roll and the estimated pan; Determining a feature point vector perpendicular to the edge of the vehicle body region based on the compensated edge points and the compensated feature points; Determine a third angle between the three-dimensional circle corresponding to the feature point vector and the three-dimensional circle corresponding to the optical axis of the camera; determining a fourth angle between the design value of the camera and the compensated feature point; and The tilt of the camera is estimated based on the third angle and the fourth angle.

16. The camera calibration device according to claim 10, wherein: The estimator is configured as: detecting a straight line corresponding to a vanishing line determined based on the vehicle body region; as well as The pose of the camera when the vehicle is traveling is estimated based on the detected straight lines and the vanishing points determined based on the vanishing lines.

17. The camera calibration device according to claim 16, wherein: The estimator is configured as: generating a virtual line corresponding to the straight line based on a curve detected in the vehicle body region; as well as A pose of the camera is estimated based on the virtual line and the vanishing point.

18. The camera calibration device according to claim 10, wherein: The estimator is configured as: The posture of the camera is estimated based on the horizontal direction information and the vertical direction information, wherein: The horizontal direction information and the vertical direction information are detected based on the vehicle body region.

Images