Method and device for estimating posture of camera

Abstract

The invention relates to a method and device for estimating the posture of a camera. The method comprises the following steps: dividing an input image received from a camera into a plurality of areas; dividing the plurality of areas into a first detection area for detecting vanishing points or overturning and a second detection area for detecting overturning; extracting representative features from features of the input image by using a filter pre-generated for each of the plurality of regions based on a camera design value; calculating a feature error of each of the first detection region and the second detection region based on the representative feature; and estimating the attitude of the camera based on the feature error.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from Korean Patent Application No. 10-2023-0114009 filed on August 29, 2023, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a technology for estimating a camera pose, and more particularly, to a method and apparatus for estimating a camera pose, which can improve the estimation accuracy of the camera pose based on a vanishing point guide filter. Background Art

[0004] Various image recognition technologies are required to operate autonomous vehicles. For example, recognizing lanes and identifying vanishing points while driving are also important factors.

[0005] A vanishing point is a point where parallel straight lines that extend infinitely in a three-dimensional space and are projected onto a two-dimensional plane intersect at one point on the plane. As an example of using vanishing point detection, buildings can be reinterpreted by analyzing artificial structures by finding vanishing points and vanishing lines in three mutually orthogonal directions. In the 3D conversion of a 2D image containing an artificial structure, a depth map can be generated by detecting vanishing points. This is because as the 3D space is changed to a 2D image, the part where the vanishing point is located usually corresponds to the farthest point in the image, so the relative depth can be estimated.

[0006] In addition, vanishing point information is an important basis for lane detection in autonomous vehicles or position information analysis in autonomous driving systems such as robots. This is because the road can be detected by connecting the main edges extending from the vanishing points.

[0007] While driving, the vehicle may estimate the external parameters of the camera and the road surface by estimating the vanishing point using line segments in the image and estimating a vanishing line based on the estimated vanishing point, thereby estimating the pose of the camera.

[0008] However, when the vehicle is moving, the accuracy of the vanishing point decreases due to image distortion, and it is difficult to estimate the exact vanishing line due to the vanishing point boundary phenomenon caused by the vehicle behavior while moving. Summary of the invention

[0009] The present disclosure is dedicated to solving the above-mentioned problems arising in the prior art and improving camera posture estimation.

[0010] One aspect of the present disclosure provides a method for estimating a camera pose based on a vanishing point guided filter. Another aspect of the present disclosure relates to a device capable of estimating a camera pose based on a vanishing point guided filter.

[0011] Another aspect of the present disclosure provides a method and apparatus capable of estimating a camera pose from an input image by utilizing a vanishing point-guided anchor and at least one pre-trained network.

[0012] Another aspect of the present disclosure provides a method and apparatus capable of extracting representative features of each divided region of an input image based on a vanishing point guided filter. After extracting the representative features, the method can estimate the camera pose by utilizing the extracted representative features, thereby improving the accuracy of camera pose estimation even when driving. In addition, the apparatus can estimate the camera pose by utilizing the extracted representative features.

[0013] Another aspect of the present disclosure provides a method and apparatus capable of generating a vanishing point guided filter based on a design value of a camera mounted on a vehicle. Then, the method and apparatus can estimate the camera pose of the corresponding vehicle, so that even if the design value of the camera attached to the vehicle is different, the camera pose of the vehicle can be clearly estimated.

[0014] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art to which the present disclosure belongs from the following description.

[0015] A method for estimating a camera posture may include: based on receiving an image from a camera, dividing the image into multiple areas; dividing the multiple areas into a first detection area and a second detection area, wherein the first detection area is associated with a vanishing point and the second detection area is associated with a roll; extracting representative features of the image based on detecting one or more features of the image and using one or more filters in each of the multiple areas, wherein the one or more filters are based on design values ​​of the camera; determining a first feature error in the first detection area and a second feature error in the second detection area based on the representative features; and estimating the camera posture based on the first feature error and the second feature error.

[0016] Dividing the multiple areas may include dividing the multiple areas into a first detection area and a second detection area based on a vanishing point, a vanishing line, and a preset vanishing line threshold, wherein the vanishing point is based on a design value of the camera.

[0017] Extracting representative features may include extracting representative features from one or more features of the image by applying one or more of: a first filter generated based on a vanishing point, wherein the vanishing point is associated with a design value of the camera; and a second filter generated based on a horizontal line in each of a plurality of regions.

[0018] Extracting the representative features may include: detecting one or more features based on edges previously detected from the image and one or more filters; and extracting the representative features based on the one or more features and a design value of the camera.

[0019] Extracting representative features may include: extracting the following features as representative features from one or more features: a feature whose distance from a vanishing point is less than or equal to a first value, wherein the vanishing point is associated with a design value of the camera; or a feature whose angle with the flip is less than or equal to a second value, wherein the flip is associated with the design value of the camera.

[0020] Determining the first characteristic error may include: determining the distance between the representative feature and the vanishing point as the first characteristic error, wherein the vanishing point is associated with the design value of the camera, and determining the second characteristic error includes: determining the angle between the representative feature and the flip as the second characteristic error, wherein the flip is associated with the design value of the camera.

[0021] Estimating the posture of the camera includes: estimating a position where a first feature error is minimized; and estimating an angle where a second feature error is minimized.

[0022] Estimating the posture of the camera includes: estimating a position where a first feature error is minimized in the first detection area as a vanishing point; and estimating an angle where a second feature error is minimized in the second detection area as a flip.

[0023] A method for estimating a camera pose may include: based on receiving an image from a camera, dividing the image into multiple regions; extracting representative edges of the image from the edges of the image based on a first training network and an anchor filter, wherein the anchor filter is pre-generated based on a design value of the camera; determining a first feature error for estimating a vanishing point and a second feature error for estimating a flip based on a second training network using the representative edge as input; and generating the pose of the camera based on the first feature error and the second feature error.

[0024] The second training network may include a regression network.

[0025] The second training network can automatically generate the posture of the camera based on the first feature error and the second feature error.

[0026] The method may further include: dividing the plurality of regions into a first detection zone and a second detection zone by the first training network; and determining the edge of the image by the first training network, wherein the edge of the image includes the edges of the plurality of regions located in the first detection zone and the second detection zone.

[0027] An apparatus for estimating a camera pose may include: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform one or more operations described herein.

[0028] A non-transitory computer-readable medium may store instructions that, when executed, cause performance of one or more operations described herein.

[0029] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of specific embodiments of the present disclosure described below and do not limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other objects, features and advantages of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings:

[0031] Figure 1 is a flow chart illustrating a method for estimating camera pose according to aspects of the present disclosure;

[0032] Figure 2 is a diagram showing a scheme for dividing an input image;

[0033] Figure 3 is a diagram showing a scheme for dividing a detection area;

[0034] Figure 4 It is shown Figure 1 A flowchart of an example of operation S140;

[0035] Figure 5 is a diagram showing an example of a filter generated for each region;

[0036] Figure 6 is a diagram illustrating an operation of extracting representative features from an input image;

[0037] Figure 7 is a diagram showing a scheme for estimating a vanishing point of a camera;

[0038] Figure 8 is a flow chart illustrating a method for estimating a camera pose according to another aspect of the present disclosure;

[0039] Fig. 9 is a diagram illustrating a method of using a network;

[0040] Fig.10is a block diagram showing an apparatus for estimating a camera pose according to yet another aspect of the present disclosure; and

[0041] Fig.11 is a block diagram illustrating a computing system for executing a method for estimating a camera pose according to aspects of the present disclosure. DETAILED DESCRIPTION

[0042] Hereinafter, various examples of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure is not limited to the examples set forth herein and can be variously modified in many different forms.

[0043] It will be understood that when an element is referred to as being "connected" or "coupled" to another element, the element may be directly connected to the other element or indirectly connected to the other element. In addition, when some parts "include" or "have" certain elements, unless explicitly described to the contrary, this means that other elements may be further included without excluding them.

[0044] Expressions such as "first" or "second" may refer to an element regardless of its priority or importance, and may be used to distinguish one element from another, but are not limited to these components. Thus, without departing from the scope of the present disclosure, a first component of one example may be referred to as a second component of another example. Similarly, a second component of one example may be referred to as a first component of another example.

[0045] In the present disclosure, components may be distinguished from each other to clearly describe features, and it does not mean that the components must be separated. That is, multiple components may be integrated to form a single hardware or software unit, or a single component may be dispersed to form multiple hardware or software units. Therefore, even if not separately mentioned, such integrated or dispersed examples are also included in the scope of the present disclosure.

[0046] In the present disclosure, the components described in various examples are not necessarily meant to be essential components, and some may be optional components. Therefore, examples consisting of a subset of the components described in the examples are also included in the scope of the present disclosure. In addition, examples that also include other components in addition to the components described in the various examples are also included in the scope of the present disclosure.

[0047] In the present disclosure, for the convenience of description, expressions such as up, down, left, right, etc., used in this specification are described. When the drawings shown in this specification are viewed in reverse, the positional relationships described in the specification can be interpreted in the opposite manner.

[0048] As used herein, each of phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any or all possible combinations of items listed together in the corresponding one of these phrases.

[0049] Aspects of the present disclosure estimate camera poses, such as vanishing points and rolls of a camera, based on design values ​​of the camera and a vanishing point guided filter.

[0050] According to aspects of the present disclosure, a vanishing point guided filter may be generated based on design values ​​of a camera, and since the vanishing point guided filter is generated based on the design values, the estimation accuracy of the posture of a camera installed in a vehicle may be improved.

[0051] According to aspects of the present disclosure, by implementing an artificial intelligence network for extracting representative features from an input image based on a vanishing point guided filter and an artificial intelligence network for estimating a camera pose, such as a vanishing point and a flip of a camera, by using the representative features as input, at least one network can be used to estimate a vanishing point and a flip from an input image. Of course, according to aspects of the present disclosure, it is not necessary to use two artificial intelligence networks, and an artificial intelligence network for extracting representative features or an artificial intelligence network for estimating a camera pose by using representative features can be applied.

[0052] In the following, reference will be made to Figures 1 to 10 Methods and apparatus according to aspects of the present disclosure are described.

[0053] Figure 1 Detailed description is a flow chart showing a method for estimating a camera pose according to an aspect of the present disclosure, which shows an operation flow of an apparatus for estimating a camera pose provided in a vehicle.

[0054] Reference Figure 1 In S110 and S120, the method for estimating camera pose according to aspects of the present disclosure receives an image captured by a camera disposed in a vehicle, such as a moving vehicle, and divides the received image, ie, the input image, into a plurality of regions.

[0055] In this case, in S120, if Figure 2As shown, the input image can be divided into a plurality of grids, and the number of divided grids can be determined according to the number of filters generated based on the design value of the camera. The setting factor of the number of divided grids is not limited or constrained by the number of filters. The design value of the camera may include external parameters and internal parameters of the camera. The external parameters may indicate the relative position and direction between the camera and the photographed object. The internal parameters may include the focal length, optical center, radial distortion coefficient, etc. of the lens system.

[0056] After the input image is divided into a plurality of grids in operation S120, the plurality of grids may be divided into a first detection area for estimating a vanishing point or a flip of a camera and a second detection area for estimating a flip of the camera in operation S130.

[0057] According to an example, in operation S130, the plurality of grids may be divided into a first detection area and a second detection area based on a vanishing point calculated using a design value of the camera, a vanishing line calculated using a design value of the camera, and a vanishing line threshold preset based on the vanishing line. The vanishing point may be determined using internal and external parameters based on the geometry of the camera.

[0058] For example, Figure 3 As shown, in operation S130, the grid between two vanishing line thresholds 330 can be divided into the second detection area by the vanishing line 320 calculated by the design value of the camera through the vanishing point 310 calculated by the design value of the camera, the vanishing line 340 rotated 90 degrees, and the vanishing line threshold 330, and the remaining grids can be divided into the first detection area. Figure 3 , the grids in the upper region among the grids between the vanishing line 340 rotated 90 degrees and the threshold value may be excluded from the second detection region for estimating flipping. This may be because there is a flow vector found at the vanishing point in the sky region of the image, and there is no information for finding flipping.

[0059] After dividing the first detection area for estimating a vanishing point or a flip and the second detection area for estimating a flip in operation S130, representative features of the input image may be extracted by using a filter generated in advance for each divided area, ie, each grid, in S140.

[0060] According to an example, in operation S140, features corresponding to filters, such as edges (or straight lines), can be detected by utilizing edges detected for an input image and filters for each grid, and representative features for estimating vanishing points and flips of a camera can be extracted based on the detected features and design values ​​of the camera, such as vanishing points and flips calculated using the design values ​​of the camera.

[0061] like Figure 5As shown, the filter of each mesh applied in operation S140 may be one of a filter generated based on a vanishing point calculated using a design value of a camera and a filter generated based on a horizontal line. Figure 5 , the filter 510 generated based on the horizontal line can be applied to each of the horizontal grids and the vertical grids in a certain area based on the vanishing point 310 calculated using the design value, and the filter 520 generated based on the vanishing point can be applied to the remaining area. Figure 5 The filter 520 generated based on the vanishing point includes a filter including a plurality of straight lines, but this is for the purpose of illustrating that it may be a filter including a straight line passing through the vanishing point. In fact, the vanishing point-based filter may include a straight line passing through the center point of each grid.

[0062] In other words, in operation S140, among the edges existing in each mesh, an edge corresponding to a filter of the mesh may be detected as a feature corresponding to the filter, and a representative feature may be extracted from the detected features.

[0063] Specifically, Figure 4 As shown, operation S140 may include an operation of extracting representative features for vanishing point VP detection or estimation and an operation of extracting representative features for flip detection or estimation. Each operation will be described in turn below.

[0064] Describing the operation of extracting representative features for estimating vanishing points, in S410 and S420, edges detected in the input image, such as road signs, roadside lines, buildings, etc., may be called, and among the called edges, straight lines (edges), i.e., features, corresponding to the applied filter may be detected. For example, Figure 6 As shown, the straight line for vanishing point VP detection can be detected by detecting the edge corresponding to the filter of each grid among the edges called for vanishing point detection in the input image. In this case, Figure 6 The straight lines used for vanishing point detection in can be the features detected by each filter.

[0065] In addition, in S430 and S440, the distance between each detected straight line and the vanishing point VP calculated using the design value may be calculated, and if the calculated distance is less than or equal to a preset first value, the corresponding feature may be used or extracted as a representative feature. In this case, the first value may be determined by a person or operator who provides the technology disclosed herein, and may be determined by considering the estimation accuracy of the vanishing point, etc.

[0066] Similarly, the operation of extracting representative features for estimating flipping is described. In S410 and S420, edges detected for the input image, such as edges of vehicles, buildings, etc., may be called, and among the called edges, straight lines (edges), i.e., features, corresponding to the applied filter may be detected. For example, Figure 6 As shown, the straight lines for flip detection can be detected by detecting the edges corresponding to the filters of each grid in the edges called for flipping of the input image. In this case, Figure 6 The straight lines in for flip detection can be the features detected by each filter.

[0067] In S430 and S440, the angle between each detected straight line and the flip calculated using the design value is calculated, and if the calculated angle is less than or equal to a preset second value, the corresponding feature can be used or extracted as a representative feature. In this case, the second value can be determined by a person or operator who provides the technology disclosed herein, and can be determined by considering the estimated accuracy of the flip, etc.

[0068] In other words, the scheme according to aspects of the present disclosure may not use all the edges detected by the filter, but may use the vanishing points and flips calculated using the design values ​​of the camera to extract features (representative features) for estimating the vanishing points and flips, and through such operations, the computational complexity of estimating the camera's posture may be reduced.

[0069] Refer again Figure 1 , after the straight lines (or edges) corresponding to the representative features for vanishing point detection and the representative features for flip detection are detected through the above operation S140, in S150 and S160, the feature errors of each of the first detection area and the second detection area can be calculated based on the detected representative features, and the posture of the camera, such as the vanishing point and flip, can be estimated based on the feature errors of each detection area.

[0070] In this case, the calculation of the feature error in each of the first detection area and the second detection area may refer to the calculation of the feature error in each detection area based on the representative features detected in the corresponding detection area and the design value of the camera, and refers to the calculation of the feature error for vanishing point estimation and the feature error for flip estimation by utilizing all representative features extracted from each detection area.

[0071] According to the example, in operation S150, the distance between each detected representative feature (or straight line) and the vanishing point calculated using the design value of the camera can be calculated as a first feature error, and the angle between each detected representative feature and the flip calculated using the design value of the camera can be calculated as a second feature error.

[0072] In this case, the first characteristic error and the second characteristic error can be expressed as the following Equation 1 and Equation 2.

[0073] [Equation 1]

[0074] VP=(x VP ,y VP )

[0075]

[0076] L n =(a,b)←s(ax+b+1)=0

[0077]

[0078] Where VP represents the vanishing point coordinates in the image, represents the vanishing point coordinates converted to the homogeneous coordinate system, L n represents the detected straight line equation containing two parameters a and b, Represents the equation of the line converted to a homogeneous coordinate system, errVP n represents the first characteristic error.

[0079] [Equation 2]

[0080] θ=roll

[0081] R n =slope n

[0082] errθ n =|θ-R n |

[0083] Where θ represents the flip calculated using the design value, R n represents the inclination angle of the line detected as a representative feature, errθ n Represents the second characteristic error.

[0084] According to aspects, in operation S160, if the first feature error and the second feature error of each straight line detected as a representative feature are calculated according to Equations 1 and 2, the position where the first feature error is minimized can be estimated as the vanishing point of the camera, and the angle at which the second feature error is minimized can be estimated as the flip of the camera.

[0085] In this case, the vanishing point and flip of the camera can be estimated by the following Equation 3 and Equation 4.

[0086] [Equation 3]

[0087]

[0088] [Equation 4]

[0089]

[0090] For example, refer to Figure 7 Describing the process of estimating the vanishing point of the camera, the first feature error errVP may be calculated based on the distance between the vanishing point 310 of the camera calculated using the design value and each of the straight lines 701, 702, and 703 detected for vanishing point detection. n , and the position 710 of the vanishing point where the first feature error is minimized can be estimated as the vanishing point of the camera. Figure 7 There are shown three straight lines 701, 702, and 703 for vanishing point detection, but the number of straight lines for vanishing point detection is not limited to or restricted to three, and may include straight lines through Figure 4 Therefore, we can use Figure 4 All the straight lines detected in the image are used to estimate the position of the vanishing point that minimizes the first feature error as the vanishing point of the camera. Similarly, by estimating the camera flip solution, Figure 7 Similarly, the angle at which the angle between the flip of the camera calculated based on the design value and the tilt angle of the straight line detected for flip detection is minimized can be estimated as the flip of the camera.

[0091] In addition, the method for estimating the camera posture according to aspects of the present disclosure may be implemented by an artificial intelligence network. Figure 8 and Fig. 9 Describes a method for estimating camera pose using an artificial intelligence network.

[0092] Figure 8 is a flowchart illustrating a method for estimating a camera pose according to another aspect of the present disclosure, which illustrates an operation flow of estimating a camera pose using a first network and a second network.

[0093] Reference Figure 8 In S810 and S820, a method for estimating a camera posture according to another example of the present disclosure receives an image captured by a camera disposed in a moving vehicle, and divides the received image, ie, the input image, into a plurality of regions.

[0094] After the input image is divided into a plurality of grids in operation S820, in S830 and S840, the divided regions may be divided into a first detection region for estimating a vanishing point or flip and a second detection region for estimating a flip of a camera by using a pre-trained first network, such as an artificial intelligence network for edge detection, and representative edges may be extracted from the edges of the input image by using a pre-generated vanishing point-based anchor point (or anchor point filter). In this case, the representative edge may correspond to Figures 1 to 7 Representative features described in .

[0095] According to an example, the first network may be a network that detects edges of an input image based on an anchor filter and an image as input, detects edges corresponding to the anchor filter through the detected edges and the anchor filter, and then extracts representative edges from the detected edges. In this case, the anchor filter may be an anchor filter based on a vanishing point calculated based on a design value of a camera.

[0096] After extracting representative edges of the input image through the first network, in S850 and S860, feature errors of each of the first detection area and the second detection area can be calculated based on the representative edges by utilizing a pre-trained second network, such as an artificial intelligence network for estimating the camera pose, and the pose of the camera, such as vanishing point and flip, can be estimated based on the feature errors of each detection area.

[0097] Although the reference Figure 8 The operations of the first network and the second network are described in detail, but the method according to another example of the present disclosure can clearly extract the pose of the camera by using the two networks, because it automatically extracts representative features of the input image from the first network by using the divided multiple regions and anchor filters as input, and automatically extracts the pose of the camera from the second network by using the automatically extracted representative features as input. Of course, each of the first network and the second network can be pre-trained based on the training data used to train each network.

[0098] In addition, although Figure 8 The operation of dividing the detection area that can be performed by the first network is shown, but the operation of dividing the detection area can be performed by image analysis, etc., and the operation of extracting representative edges using information about the input image and the detection area and an anchor filter can be performed in the first network.

[0099] In addition, the second network can automatically extract the vanishing point and flip of the camera by using the design value of the camera and information about the representative edge of the input image which can be the output of the first network as input. In this case, the second network can include a regression network, such as a general recurrent neural network (GRNN).

[0100] In addition, if Fig. 9 As shown, the first network used in the solution according to another example of the present disclosure may be an edge detector 910 for detecting edges associated with vanishing points in an input image, and the second network may be a regression network 920, such as a GRNN.

[0101] In this case, if Fig. 9 As shown, the edge detector 910 can detect edges related to the vanishing point based on the input image, information 940 about the edges detected in the input image, and the anchor point filter 930 pre-generated based on the vanishing point, and provide the detected edges to the second network 920. In addition, the regression network 920 can estimate the vanishing point of the camera by using the output of the first network 910, that is, the edge corresponding to the vanishing point and the design value of the camera (if necessary) as input to estimate the pitch and yaw of the camera and the flip of the camera.

[0102] As described above, the method according to aspects of the present disclosure can estimate the pose of the camera based on a vanishing point guided filter, or estimate the pose of the camera from an input image by utilizing a vanishing point guided anchor point (or anchor point filter) and at least one pre-trained network.

[0103] In addition, the method according to the example of the present disclosure can extract representative features (or straight lines and edges) of each divided area of ​​the input image based on a vanishing point guided filter, and use the extracted representative features to estimate the camera pose, thereby improving the camera pose estimation accuracy even when driving.

[0104] In addition, according to the example method of the present disclosure, a vanishing point guided filter can be generated based on the design value of the camera installed on the vehicle and the camera pose of the corresponding vehicle can be estimated, so that even if the design value of the camera attached to the vehicle is different, the camera pose of the corresponding vehicle can be clearly estimated.

[0105] Fig.10 is a block diagram showing an apparatus for estimating a camera posture according to another aspect of the present disclosure, and shows a method for executing Figures 1 to 9 The apparatus of the scheme shown in FIG.

[0106] Reference Fig.10 According to another aspect of the present disclosure, the device 1000 for estimating the camera posture may include a receiving device 1010, a dividing device 1020, a region dividing device 1030, an extracting device 1040, a calculating device 1050, an estimating device 1060 and a storing device 1070.

[0107] The storage device 1070 can be a configuration element for storing all data related to the device of the present disclosure, and can store all data related to the technology of the present disclosure, such as camera design values, filter information, received input images, algorithms for providing detection areas, schemes for calculating the distance between a straight line and a vanishing point, schemes for calculating the angle between the slope of a straight line and a flip, edge information extracted from an image, and estimated camera posture information.

[0108] The receiving device 1010 may receive an image captured by a vehicle camera. That is, the receiving device may receive an input image from the camera for estimating the posture of the camera.

[0109] In this case, the receiving device 1010 may pre-process the input image and then receive the pre-processed input image.

[0110] The dividing device 1020 may divide the input image received by the receiving device 1010 into a plurality of regions or grids.

[0111] The region dividing means 1030 may divide the region divided by the dividing means 1020 into a first detection region for detecting a vanishing point or a reversal and a second detection region for detecting a reversal.

[0112] According to an example, the area dividing device 1030 may divide the divided area into a first detection area and a second detection area based on a vanishing point calculated using a design value of a camera and a preset vanishing line threshold.

[0113] The extracting device 1040 may extract representative features from the features of the input image by using a filter pre-generated for each region based on a design value of the camera.

[0114] According to an example, the extracting device 1040 may apply one of a vanishing point-based filter or a horizon-based filter calculated using a design value of a camera to each of the divided regions to extract representative features from features of the input image.

[0115] According to an example, the extracting device 1040 may detect features corresponding to the filter by using edges detected for the input image and the filter, and may extract representative features for estimating the vanishing point and flipping of the camera based on the detected features and the design values ​​of the camera.

[0116] According to the example, the extraction device 1040 can extract features whose distance from the vanishing point calculated using the design value of the camera is less than or equal to a first value or whose flip angle calculated using the design value of the camera is less than or equal to a second value from the features as representative features.

[0117] The calculation device 1050 can calculate the feature error in each of the first detection area and the second detection area based on the representative features of the input image extracted by the extraction device 1040.

[0118] According to the example, the computing device 1050 can calculate the distance between each representative feature extracted by the extraction device 1040 and the vanishing point that can be calculated using the design value of the camera as a first feature error, and can calculate the angle between each representative feature and the flip calculated using the design value of the camera as a second feature error.

[0119] The estimation device 1060 can estimate the posture of the camera based on the feature errors calculated by the calculation device 1050, such as the first feature error used for vanishing point estimation and the second feature error used for flip estimation.

[0120] According to an example, the estimation device 1060 may estimate the position at which the first feature error is minimized as the vanishing point of the camera, and may estimate the angle at which the second feature error is minimized as the flip of the camera.

[0121] According to an example, the estimation device 1060 may estimate the position where the feature error is minimized in the first detection area as the vanishing point of the camera, and may estimate the angle where the feature error is minimized in the second detection area as the flip of the camera.

[0122] Another example device according to the present disclosure may include Figures 1 to 10 All the content described in the scheme.

[0123] According to aspects of the present disclosure, a method for estimating a camera posture includes: dividing an input image received from a camera into multiple areas; dividing the multiple areas into a first detection area for detecting a vanishing point or a flip and a second detection area for detecting a flip; extracting representative features from features of the input image by using a filter pre-generated for each of the multiple areas based on a design value of the camera; calculating a feature error of each of the first detection area and the second detection area based on the representative features; and estimating the posture of the camera based on the feature error.

[0124] According to an example, dividing the multiple areas may include: dividing the multiple areas into a first detection area and a second detection area based on a vanishing point, a vanishing line, and a preset vanishing line threshold calculated from a design value of the camera.

[0125] According to an example, extracting the representative feature may include extracting the representative feature from features of the input image by applying one of a filter generated based on a vanishing point calculated using a design value of a camera and a filter generated based on a horizontal line of each of the plurality of regions.

[0126] According to an example, extracting the representative feature may include detecting the feature by using edges detected with respect to the input image and a filter, and extracting the representative feature based on the feature and a design value of the camera.

[0127] According to an example, extracting representative features may include: extracting features from the features whose distance from a vanishing point calculated by using a design value of the camera is less than or equal to a first value or whose angle of flip calculated based on the design value of the camera is less than or equal to a second value as representative features.

[0128] According to an example, calculating the feature error may include calculating the distance between each representative feature and a vanishing point calculated based on a design value of the camera as a first feature error, and calculating the angle between each representative feature and a flip calculated based on a design value of the camera as a second feature error.

[0129] According to an example, estimating the pose of the camera may include: estimating a position where the first feature error is minimized as a vanishing point of the camera, and estimating an angle where the second feature error is minimized as a flip of the camera.

[0130] According to an example, estimating the pose of the camera may include: estimating a position where a feature error in the first detection area is minimized as a vanishing point of the camera, and estimating an angle where a feature error in the second detection area is minimized as a flip of the camera.

[0131] According to another aspect of the present disclosure, a method for estimating a camera posture includes: dividing an input image received from a camera into multiple regions; extracting representative edges from the edges of the input image by utilizing a pre-learned first network and an anchor filter pre-generated based on a design value of the camera; calculating a first feature error for estimating a vanishing point and a second feature error for estimating a flip by utilizing a pre-trained second network of the input representative edges; and estimating the posture of the camera based on the first feature error and the second feature error.

[0132] According to aspects of the present disclosure, the second network may include a regression network.

[0133] According to another aspect of the present disclosure, a device for estimating a camera posture includes: an image division device, configured to divide an input image received from a camera into multiple regions; a region division device, configured to divide the multiple regions into a first detection region for detecting a vanishing point or a flip and a second detection region for detecting a flip; an extraction device, configured to extract representative features from features of the input image by using a filter pre-generated for each of the multiple regions based on a design value of the camera; a calculation device, configured to calculate a feature error of each of the first detection region and the second detection region based on the representative features; and an estimation device, configured to estimate the posture of the camera based on the feature error.

[0134] According to aspects of the present disclosure, the area division device may divide a plurality of areas into a first detection area and a second detection area based on vanishing points, vanishing lines, and a preset vanishing line threshold calculated from design values ​​of the camera.

[0135] According to aspects of the present disclosure, the extracting device may extract representative features from features of an input image by applying one of a filter generated based on a vanishing point calculated using a design value of a camera and a filter generated based on a horizontal line of each of a plurality of regions.

[0136] According to aspects of the present disclosure, the extracting device may detect features by using edges detected with respect to an input image and filters, and extract representative features based on the features and design values ​​of a camera.

[0137] According to aspects of the present disclosure, the extraction device can extract features from the features whose distance from the vanishing point calculated by using the design value of the camera is less than or equal to a first value or whose angle of flip calculated based on the design value of the camera is less than or equal to a second value as representative features.

[0138] According to aspects of the present disclosure, the computing device may calculate the distance between each representative feature and the vanishing point calculated based on the design value of the camera as a first feature error, and calculate the angle between each representative feature and the flip calculated based on the design value of the camera as a second feature error.

[0139] According to aspects of the present disclosure, the estimation device may estimate the position where the first feature error is minimized as the vanishing point of the camera, and estimate the angle where the second feature error is minimized as the flip of the camera.

[0140] According to aspects of the present disclosure, the estimation device may estimate the position where the feature error is minimized in the first detection area as the vanishing point of the camera, and estimate the angle where the feature error is minimized in the second detection area as the flip of the camera.

[0141] Fig.11 is a block diagram illustrating a computing system for executing a method for estimating a camera pose according to aspects of the present disclosure.

[0142] Reference Fig.11 As described above, the method for estimating the camera posture according to the example of the present disclosure can be implemented by a computing system. The computing system 2000 may include at least one processor 2100, a memory 2300, a user interface input device 2400, a user interface output device 2500, a storage device 2600, and a network interface 2700 connected through a system bus 2200.

[0143] The processor 2100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 2300 and / or the storage device 2600. The memory 2300 and the storage device 2600 may include various volatile or non-volatile storage media. For example, the memory 2300 may include a read-only memory (ROM) 2310 and a random access memory (RAM) 2320.

[0144] Therefore, the process of the method or algorithm described in the example of the present disclosure can be directly implemented by the hardware, software module or combination thereof run by the processor 2100. The software module can reside in a storage medium (i.e., memory 2300 and / or storage device 2600) such as the following: RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk or CD-ROM. An exemplary storage medium can be connected to the processor 2100, and the processor 2100 can read information from the storage medium and can write information to the storage medium. In another method, the storage medium can be integrated with the processor 2100. The processor 2100 and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a user terminal. In another method, the processor 2100 and the storage medium can reside in a user terminal as a separate component.

[0145] According to the present disclosure, the pose of a camera may be estimated based on a vanishing point guided filter, or the pose of a camera may be estimated from an input image by using a vanishing point guided anchor point and at least one pre-trained network.

[0146] According to the present disclosure, representative features of each divided area of ​​an input image can be extracted based on a vanishing point guided filter, and the camera pose can be estimated using the extracted representative features, thereby improving the camera pose estimation accuracy even when driving.

[0147] According to the present disclosure, a vanishing point guided filter can be generated based on the design values ​​of a camera mounted on a vehicle and the camera pose of the corresponding vehicle can be estimated, thereby clearly estimating the camera pose of the corresponding vehicle even if the design values ​​of the camera attached to the vehicle may be different.

[0148] Effects obtained by various examples of the present disclosure may not be limited to the above-mentioned effects, and those skilled in the art will clearly understand other effects based on the disclosure.

[0149] Although examples of the present disclosure are described for illustrative purposes, it will be understood by those skilled in the art that various modifications, additions and substitutions may be made without departing from the scope and spirit of the present disclosure. Therefore, the examples disclosed in the present disclosure are provided for descriptive purposes only and do not limit the technical concepts of the present disclosure, and it should be understood that these examples are not intended to limit the scope of the technical concepts of the present disclosure. The scope of protection of the present disclosure should be understood through the attached claims, and all technical concepts within the equivalent scope should be interpreted as being within the scope of the rights of the present disclosure.

Claims

1. A method for estimating a camera posture, comprising: Based on an image received from a camera, dividing the image into a plurality of regions; dividing the plurality of regions into a first detection zone and a second detection zone, wherein the first detection zone is associated with a vanishing point and the second detection zone is associated with a rollover; extracting representative features of the image based on detecting one or more features of the image and using one or more filters for each of the plurality of regions, wherein the one or more filters are based on design values ​​of the camera; determining a first characteristic error in the first detection zone and a second characteristic error in the second detection zone based on the representative characteristic; as well as The posture of the camera is estimated based on the first feature error and the second feature error.

2. The method according to claim 1, wherein: Dividing the plurality of regions comprises: The plurality of areas are divided into the first detection area and the second detection area based on the vanishing point, the vanishing line, and a preset vanishing line threshold, wherein the vanishing point is based on a design value of the camera.

3. The method according to claim 1, wherein: Extracting the representative features includes: The representative feature is extracted from one or more features of the image by applying one or more of the following: a first filter generated based on the vanishing point, wherein the vanishing point is associated with a design value of the camera; and A second filter is generated based on the horizontal line of each of the plurality of regions.

4. The method according to claim 1, wherein: Extracting the representative features includes: detecting the one or more features based on previously detected edges from the image and the one or more filters; and The representative feature is extracted based on the one or more features and a design value of the camera.

5. The method according to claim 4, wherein: Extracting the representative features includes: The following features are extracted from the one or more features as the representative features: A feature having a distance from the vanishing point that is less than or equal to a first value, wherein the vanishing point is associated with a design value of the camera; or A feature in which the angle between the flip and the camera is less than or equal to a second value, wherein the flip is associated with a design value of the camera.

6. The method according to claim 4, wherein: Determining the first characteristic error includes: determining a distance between the representative feature and the vanishing point as the first feature error, wherein the vanishing point is associated with a design value of the camera, and Determining the second characteristic error includes: An angle between the representative feature and the flip is determined as the second feature error, wherein the flip is associated with a design value of the camera.

7. The method according to claim 6, wherein: Estimating the posture of the camera includes: estimating a position at which the first characteristic error is minimized; and An angle is estimated at which the second characteristic error is minimized.

8. The method according to claim 1, wherein: Estimating the posture of the camera includes: estimating a position in the first detection area where the first characteristic error is minimized as the vanishing point; and The angle that minimizes the second characteristic error in the second detection area is estimated as the flip.

9. A method for estimating a camera posture, comprising: Based on an image received from a camera, dividing the image into a plurality of regions; extracting a representative edge of the image from the edge of the image based on a first training network and an anchor filter, wherein the anchor filter is pre-generated based on a design value of the camera; determining a first characteristic error for estimating a vanishing point and a second characteristic error for estimating a flip based on a second training network using the representative edge as an input; as well as The posture of the camera is generated based on the first feature error and the second feature error.

10. The method according to claim 9, wherein: The second training network includes a regression network.

11. The method according to claim 9, wherein: The second training network automatically generates the posture of the camera based on the first feature error and the second feature error.

12. The method according to claim 9, further comprising: The first training network divides the multiple regions into a first detection area and a second detection area; as well as The first training network determines edges of the image, wherein the edges of the image include edges of the plurality of regions located in the first detection zone and the second detection zone.

13. A device for estimating a camera posture, comprising: one or more processors; as well as a memory storing instructions which, when executed by the one or more processors, cause the apparatus to: Dividing the image received from the camera into a plurality of regions; dividing the plurality of regions into a first detection zone associated with a vanishing point and a second detection zone associated with a rollover; extracting representative features of the image based on detecting one or more features of the image and using one or more filters for each of the plurality of regions, wherein the one or more filters are based on design values ​​of the camera; determining a first characteristic error in the first detection zone and a second characteristic error in the second detection zone based on the representative characteristic; and The posture of the camera is estimated based on the first feature error and the second feature error.

14. The device according to claim 13, wherein: When the instructions are executed by the one or more processors, the device: The plurality of areas are divided into the first detection area and the second detection area based on the vanishing point, the vanishing line, and a preset vanishing line threshold, wherein the vanishing point is based on a design value of the camera.

15. The device according to claim 13, wherein: When the instructions are executed by the one or more processors, the device: The representative feature is extracted from one or more features of the image by applying one or more of the following: a first filter generated based on the vanishing point, wherein the vanishing point is associated with a design value of the camera; or A second filter is generated based on the horizontal line of each of the plurality of regions.

16. The device according to claim 13, wherein: When the instructions are executed by the one or more processors, the device: detecting the one or more features based on edges previously detected from the image and the one or more filters; and The representative feature is extracted based on the one or more features and a design value of the camera.

17. The device according to claim 16, wherein: When the instructions are executed by the one or more processors, the device: The following features are extracted from the one or more features as the representative features: A feature having a distance from the vanishing point that is less than or equal to a first value, wherein the vanishing point is associated with a design value of the camera; or A feature in which the angle between the flip and the camera is less than or equal to a second value, wherein the flip is associated with a design value of the camera.

18. The device according to claim 16, wherein: When the instructions are executed by the one or more processors, the device: determining a distance between the representative feature and the vanishing point as the first feature error, wherein the vanishing point is associated with a design value of the camera; and An angle between the representative feature and the flip is determined as the second feature error, wherein the flip is associated with a design value of the camera.

19. The device according to claim 18, wherein: When the instructions are executed by the one or more processors, the device: estimating a position at which the first characteristic error is minimized; and An angle is estimated at which the second characteristic error is minimized.

20. The device according to claim 13, wherein: When the instructions are executed by the one or more processors, the device: estimating a position in the first detection area where the first characteristic error is minimized as the vanishing point; and The angle that minimizes the second characteristic error in the second detection area is estimated as the flip.

Images