Method for generating ground live data and method and apparatus for estimating virtual points using same

Abstract

The invention relates to a method for generating ground live data and a method and apparatus for estimating a virtual point using the same. A method and an apparatus for estimating a vanishing point are provided. The method comprises: receiving an input image; and estimating a vanishing point of the input image using an artificial intelligence network pre-trained by vanishing point ground live (GT) data generated based on the actual image data. Estimating the vanishing point includes estimating a depth map or optical flow map of the input image, estimating a gradient map of the depth map or optical flow map, and estimating the vanishing point of the input image based on the gradient map and a predetermined reference gradient map.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority from Korean Patent Application No. 10-2023-0099152 filed in the Korean Intellectual Property Office on July 28, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to techniques for generating vanishing point ground truth (GT) and estimating vanishing points using the vanishing point GT. More specifically, the present disclosure relates to methods and devices for generating vanishing point GT of actual data (or actual image data) and training an artificial intelligence network using the vanishing point GT to improve the accuracy of estimating the vanishing points of an input image. Background Art

[0004] Several image recognition technologies are needed for operating an autonomous vehicle. As an example, recognizing lanes while driving and identifying vanishing points are among the important elements.

[0005] The vanishing point indicates a point where when parallel straight lines are infinitely extended in a three-dimensional (3D) space and projected onto a 2D plane, the projected straight lines meet at a point on a two-dimensional (2D) plane. As an example, using the detection of the vanishing point, vanishing points and vanishing lines in three orthogonal directions can be obtained to analyze an architectural structure and again analyze a building. The vanishing point can be detected in the 3D conversion of a 2D image including an architectural structure to generate a depth map. This is because relative depth estimation can be performed because the portion where the vanishing point is located when converting a 3D space into a 2D image corresponds to the farthest position in the image.

[0006] Vanishing point information is a criterion for lane detection in autonomous vehicles or an important basis for analyzing position information in autonomous driving systems such as robots. This is because it can connect important edges connected from vanishing points and detect roads.

[0007] A conventional technique for detecting vanishing points and vanishing lines is to use computer vision (CV) techniques to detect vanishing points and vanishing lines. However, since straight lines need to be detected when using CV techniques, there are constraints that are affected by the driving environment. Summary of the invention

[0008] The present disclosure has been made to solve the above-mentioned problems occurring in the prior art while keeping the advantages achieved by the prior art intact.

[0009] Aspects of the present disclosure provide a method and apparatus for generating vanishing point ground truth (GT) of real data and training an artificial intelligence network using the vanishing point GT to improve the accuracy of estimating vanishing points of an input image.

[0010] Another aspect of the present disclosure provides a technique for generating a vanishing point GT of actual data.

[0011] Another aspect of the present disclosure provides a technology for training artificial intelligence (eg, CenterNet) using vanishing point GTs of actual data and estimating vanishing points of an input image using CenterNet.

[0012] The technical problems to be solved by the present disclosure are not limited to the above problems. A person skilled in the art in the field to which the present disclosure belongs should more clearly understand any other technical problems not mentioned in this article from the following description.

[0013] According to one aspect of the present disclosure, a method for estimating a vanishing point may include: receiving an input image; and estimating a vanishing point of the input image using an artificial intelligence network pre-trained by vanishing point ground truth (GT) data generated based on actual image data. Estimating the vanishing point may include estimating a depth map or an optical flow map of the input image. Estimating the vanishing point may also include estimating a gradient map of the depth map or the optical flow map. Estimating the vanishing point may also include estimating the vanishing point of the input image based on the gradient map and a predetermined reference gradient map.

[0014] According to an embodiment, estimating the vanishing point may include estimating a heat map of the static object based on the gradient map and a predetermined reference gradient map and estimating the vanishing point based on the heat map.

[0015] According to an embodiment, estimating the vanishing point may include performing a two-dimensional (2D) Gaussian fitting of the heat map of the static object, and estimating a center point of the 2D Gaussian as the vanishing point.

[0016] According to an embodiment, the artificial intelligence network may include a network for estimating key points corresponding to vanishing points in the heat map.

[0017] According to an embodiment, estimating the depth map or the optical flow map may include: when estimating the depth map, detecting at least one predetermined dynamic object from an input image; and estimating a depth map of static objects excluding a region of the detected at least one predetermined dynamic object.

[0018] According to an embodiment, the artificial intelligence network may include CenterNet.

[0019] In addition, the method may further include: generating a predetermined reference gradient map. Generating the reference gradient map may include generating a reference space in which only static objects and reference vanishing points exist. Generating the reference gradient map may also include generating a reference depth map or a reference optical flow map based on the reference space and the reference vanishing point. Generating the reference gradient map may also include generating a gradient map for the reference depth map or the reference optical flow map to generate a predetermined reference gradient map.

[0020] According to an embodiment, when a depth map or an optical flow map of an image is generated, when a gradient map of a depth map or an optical flow map is generated, when a heat map of a gradient map is generated by Gaussian fitting of the gradient map and key points are generated using Gaussian center points by Gaussian fitting, and when the coordinates of the Gaussian center points are the same as the coordinates of a vanishing point detected by at least one predetermined vanishing point detection technique, vanishing point GT data may be generated based on the heat map and the key points.

[0021] According to another aspect of the present disclosure, a device for estimating a vanishing point may include: a receiver that receives an input image; and an estimation device that estimates a vanishing point of the input image using an artificial intelligence network pre-trained by vanishing point GT data generated based on actual image data. The estimation device may estimate a depth map or an optical flow map of the input image, may estimate a gradient map of the depth map or the optical flow map, and may estimate a vanishing point of the input image based on the gradient map and a predetermined reference gradient map.

[0022] According to an embodiment, the estimating device may estimate a heat map of the static object based on the gradient map and a predetermined reference gradient map, and may estimate a vanishing point based on the heat map.

[0023] According to an embodiment, the estimation device may perform 2D Gaussian fitting of the heat map of the static object, and may estimate a central point of the 2D Gaussian as a vanishing point.

[0024] According to an embodiment, the artificial intelligence network may include a network for estimating key points corresponding to vanishing points in the heat map.

[0025] According to an embodiment, when estimating the depth map, the estimating device may detect at least one predetermined dynamic object from the input image, and may estimate the depth map of the static object except for the region of the detected at least one predetermined dynamic object.

[0026] According to an embodiment, the artificial intelligence network may include CenterNet.

[0027] In addition, the device may further include a generator for generating a predetermined reference gradient map. The generator may generate a reference space in which only static objects and reference vanishing points exist. The generator may also generate a reference depth map or a reference optical flow map based on the reference space and the reference vanishing point. The generator may also generate a gradient map for the reference depth map or the reference gradient map to generate a reference gradient map.

[0028] According to an embodiment, when a depth map or an optical flow map of an image is generated, when a gradient map of a depth map or an optical flow map is generated, when a heat map of a gradient map is generated by Gaussian fitting of the gradient map and key points are generated using Gaussian center points by Gaussian fitting, and when the coordinates of the Gaussian center points are the same as the coordinates of a vanishing point detected by at least one predetermined vanishing point detection technique, vanishing point GT data may be generated based on the heat map and the key points.

[0029] According to another aspect of the present disclosure, a method for generating a vanishing point ground truth (GT) may include generating a depth map or an optical flow map for an input image. The method may also include generating a gradient map of the depth map or the optical flow map. The method may also include generating a heat map of the gradient map by Gaussian fitting of the gradient map and generating key points using Gaussian center points by Gaussian fitting. The method may also include generating a vanishing point GT based on the heat map and the key points when the coordinates of the Gaussian center point are the same as the coordinates of the vanishing point detected by at least one predetermined vanishing point detection technique.

[0030] According to an embodiment, the input image may include actual image data.

[0031] The features briefly summarized above with respect to the present disclosure are merely exemplary aspects of the detailed description of the present disclosure described below and do not limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings:

[0033] Figure 1A and Figure 1B An example for describing CenterNet is shown;

[0034] Figure 2 An operational flow chart of a method for generating a vanishing point ground truth (GT) according to an embodiment of the present disclosure is shown;

[0035] Figure 3 An operational flow chart of a method for estimating a vanishing point according to another embodiment of the present disclosure is shown;

[0036] Figure 4 An operational flow diagram illustrating an embodiment of a process for generating a reference gradient map;

[0037] Figure 5A and Figure 5B An example is shown for describing a scheme for estimating vanishing points in a depth map;

[0038] Fig. 6A and Figure 6B An example is shown for describing a scheme for generating a depth map of a static object;

[0039] Fig. 7A and Figure 7B An example is shown for describing a scheme for estimating vanishing points in an optical flow map;

[0040] Figure 8 A block diagram showing an apparatus for estimating a vanishing point according to another embodiment of the present disclosure; and

[0041] Fig. 9 A block diagram of a computing system for executing a method for estimating a vanishing point according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0042] Hereinafter, embodiments of the present disclosure will be described more fully with reference to the accompanying drawings, which can be easily understood by those skilled in the art. However, the present disclosure can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0043] When describing the embodiments of the present disclosure, it has been determined that the detailed description of well-known configurations or functions may obscure the main points of the present disclosure, so the detailed description thereof is omitted. Parts not related to the description of the present disclosure are omitted in the drawings, and similar parts are represented by similar reference numerals throughout the specification.

[0044] In the present disclosure, when a component is referred to as being "connected" or "coupled" to another component, it includes not only the case where the component is directly connected to another component, but also the case where these components are indirectly connected to another component and there are other devices therebetween. In addition, when a component is referred to as "including", "comprising" or "having" another component, it means that the component may also include other components without excluding other components, as long as there is no contrary description.

[0045] In the present disclosure, terms such as "first" and "second" are used only for the purpose of distinguishing one component from another, and do not limit the order, importance, etc. of the components unless otherwise specified. Therefore, within the scope of the present disclosure, a first component in an embodiment may be referred to as a second component in another embodiment. Similarly, a second component in an embodiment may be referred to as a first component in another embodiment.

[0046] In the present disclosure, components that are distinguished from each other are only used to clearly explain each feature and do not necessarily mean that the components are separated. In other words, multiple components can be integrated to form a single hardware or software unit, or a single component can be distributed to form multiple hardware or software units. Therefore, even if not specifically mentioned, integrated or separate implementations are also included in the scope of the present disclosure.

[0047] In the present disclosure, the components described in various embodiments may not necessarily refer to essential components, and some components may be optional components. Therefore, an embodiment consisting of a subset of the components described in the embodiments is also included in the scope of the present disclosure. Therefore, an embodiment that also includes another component of the components described in various embodiments is also included in the scope of the present disclosure.

[0048] In the present disclosure, for the convenience of description, the expressions of the positional relationship used in the present specification are described, for example, the top, the bottom, the left and the right. When the drawings shown in the present disclosure are viewed in the opposite manner, the positional relationship described in the present disclosure can be interpreted in the opposite manner. When the parts, devices, elements, etc. of the present disclosure are described as having a purpose or performing an operation, function, etc., the parts, devices, or elements should be considered to be "configured to" meet the purpose or perform the operation or function in this article.

[0049] In the present disclosure, each of phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any or all possible combinations of items enumerated together in the corresponding one phrase.

[0050] Figure 1A and Figure 1B An example for describing CenterNet is shown. Embodiments of the present disclosure may generate vanishing point ground truth (GT) data based on actual image data, and may use the vanishing point GT data to train an artificial intelligence network (e.g., CenterNet, which is a key point estimation network). Embodiments of the present disclosure may use CenterNet to estimate the vanishing points of an input image to improve the accuracy of estimating the vanishing points of the input image.

[0051] Embodiments of the present disclosure may train a keypoint estimation network for estimating keypoints using vanishing point GT data generated based on actual image data, and thus improve the accuracy of estimating vanishing points of an image in an actual field.

[0052] The vanishing point estimation technology in the present disclosure may use a key point estimation network trained by vanishing point GT data generated based on actual image data, and is described as CenterNet for the convenience of description in the detailed description below. However, the key point estimation network is not limited to or restricted to CenterNet, and may include all types of key point estimation networks capable of estimating key points.

[0053] First, CenterNet is briefly described below. CenterNet can be a network that uses three pairs of feature points to improve the shortcomings of CornerNet for predicting bounding boxes using two pairs of feature points. CenterNet can predict bounding boxes using the upper left end, lower right end, and center point information. For example, Figure 1A As shown, CenterNet may consist of a bounding box 110 and a center point 120 .

[0054] like Figure 1BAs shown, the bounding box 110 and the center point 120 may be represented as a Gaussian kernel. For example, the bounding box 110 may be represented as a Gaussian kernel, and the center point 120 may be represented as a peak point of the Gaussian kernel.

[0055] Therefore, the bounding box 110 should be changed to a 2D Gaussian to train CenterNet. Since the vanishing point does not have the concept of a bounding box, the embodiment of the present disclosure can generate a 2D Gaussian (where the vanishing point is the center point), and CenterNet can be trained using the 2D Gaussian to estimate the vanishing point using CenterNet.

[0056] Because training data (ie, vanishing point GT data) for training a network is generated using simulated data in the related art, it is difficult to actually apply the vanishing point GT data to an actual field.

[0057] The technology according to the embodiment of the present disclosure can generate vanishing point GT data based on actual image data, and can train CenterNet by using the vanishing point GT data based on the actual image data as training data. Thus, the technology can apply the vanishing point GT data to the actual scene to estimate the vanishing point.

[0058] In other words, CenterNet in an embodiment of the present disclosure may train a 2D Gaussian of vanishing point GT data to estimate the vanishing points of an input image.

[0059] The following reference Figure 2 A description is given of a process of generating vanishing point GT data based on actual image data.

[0060] Figure 2 FIG. 4 is a flowchart showing an operation of a method for generating a vanishing point GT according to an embodiment of the present disclosure.

[0061] refer to Figure 2 In S210 and S220, the method for generating a vanishing point GT according to an embodiment of the present disclosure may receive an input image (which is actual image data), and may generate a depth map or a depth image for the received input image.

[0062] Here, in S220, the method for generating the vanishing point GT may generate a depth map in various ways for converting an input image into a depth map. Since the process of generating the depth map is obvious to those of ordinary skill in the art, a detailed description thereof is omitted.

[0063] According to an embodiment, in S220, the method for generating the vanishing point GT may detect a dynamic object among the objects included in the input image, and may generate a depth map using the remaining area (i.e., only the area of ​​the static object) except the area of ​​the detected dynamic object. At this time, the dynamic object may be detected by image analysis or a dynamic object detection network for detecting a predetermined dynamic object.

[0064] When a depth map for an input image is generated in S220 , a gradient map for the depth map may be generated in S230 .

[0065] According to an embodiment, in S230, the method for generating the vanishing points GT may divide the depth map into grids, and may generate a gradient for each grid, thereby generating a gradient map of the depth map.

[0066] In the present disclosure, since the amount of pixel-by-pixel calculation is large in calculation using depth, a gradient map for a depth map is generated to reduce the amount of calculation.

[0067] Here, the process of generating a gradient map in a depth map may be performed by various schemes. Since such schemes are obvious to those of ordinary skill in the art, detailed descriptions thereof are omitted.

[0068] When the gradient map is generated in S230 , in S240 , a heat map and key points may be generated by Gaussian fitting.

[0069] Here, in S240 , the method for generating the vanishing point GT may generate a central point of a 2D Gaussian as a key point corresponding to the vanishing point.

[0070] In S250 , the vanishing points of the input image may be detected by using different predetermined or predefined vanishing point detection techniques or different vanishing point tracking techniques.

[0071] According to an embodiment, in S250 , the method for generating the vanishing point GT may detect the vanishing point of the input image through each of a template matching (TM)-VDC technique, a line inference (LI)-VDC technique, and an optical flow technique.

[0072] The TM-VDC technique can use the vanishing point tracking technique to crop the area where the vanishing point exists, and can find the same area as the cropped area in the next frame. Here, it can be assumed that the TM-VDC technique uses TM to find the same area and moves the vanishing point by the distance where the cropped area moves.

[0073] The LI-VDC technique may be a technique in which two parallel straight lines intersect each other on an image and the intersection point is assumed to be a vanishing point. The LI-VDC technique may detect lines, may fit lines to the straight lines, and may detect a point at which the straight lines intersect each other as a vanishing point.

[0074] The optical flow technique may be a technique for estimating a rotation matrix and a transformation matrix (RT) between a previous frame and a next frame and assuming that the vanishing point is moved by the rotation matrix (R). The optical flow technique may detect the optical flow and estimate RT using epipolar geometry, and may use R to calculate the amount by which the vanishing point is moved.

[0075] Although it is shown that S250 is performed after S240 , S250 may be performed in parallel after receiving the input image.

[0076] When the vanishing point is detected by the Gaussian center point and each technique in S240 and S250, it is determined in S260 whether the coordinates of the Gaussian center point and the coordinates of the vanishing point detected by each technique are identical to each other. When it is determined that the two coordinates are identical to each other ("Yes" in S260), in S270, the vanishing point GT data may be generated using the heat map and key points generated in S240.

[0077] The above-described processing is performed on each piece of actual image data. As a result, when vanishing point GT data based on actual images is collected, the collected vanishing point GT data can be used as training data for training CenterNet.

[0078] Although the use of Figure 2 The depth map of the input image in the image is used to generate the vanishing point GT data based on the actual image, but the use of the depth map of the input image to generate the vanishing point GT data is not constrained or limited. The vanishing point GT data can be generated by generating an optical flow map of the input image and generating a gradient map of the optical flow map.

[0079] In the following, reference Figures 3 to 8 , describes the use of CenterNet trained via 2D Gaussian learning and exploiting Figure 2 A method for estimating the vanishing points of an image by processing the vanishing point GT data of the actual image generated.

[0080] Figure 3 The flowchart of the method for estimating a vanishing point according to another embodiment of the present disclosure is shown. Figure 4 An operational flow diagram of an embodiment of a process for generating a reference gradient map is shown. Figure 5A and Figure 5B An example for describing a scheme for estimating a vanishing point in a depth map is shown. Fig. 6A and Figure 6B An example is shown for describing a scheme for generating a depth map of a static object. Fig. 7A and Figure 7B An example for describing a scheme for estimating vanishing points in an optical flow map is shown.

[0081] Reference Figure 3According to another embodiment of the present disclosure, a method for estimating a vanishing point may include a process of receiving an input image (which is actual image data) (S310), and includes a process of estimating a vanishing point of the input image using a CenterNet trained by vanishing point GT data based on the actual image (S320, S330, S340, and S350).

[0082] According to an embodiment, in S310 , the method for estimating a vanishing point may receive an image captured in real time by a camera provided in a vehicle (eg, an autonomous vehicle, a vehicle equipped with an advanced driver assistance system (ADAS)), etc.) as an input image.

[0083] The process of estimating the vanishing point of the input image using CenterNet ( S320 and S330 ) may estimate the depth map of the input image received in S310 , and may estimate a gradient map of the estimated depth map.

[0084] like Figure 5A As shown, when an input image is received, in S320, the method for estimating a vanishing point may estimate a depth map ( Figure 5B ). When the road surface and the wall area (arrow) are assumed to be 2D Gaussians in the depth map in the technology of the present disclosure, the center point thereof may be the vanishing point X. Thus, the depth map may be estimated to estimate the vanishing point by the 2D Gaussians.

[0085] According to an embodiment, in S320, the method for estimating the vanishing point may detect a dynamic object from the input image, and may estimate a depth map of a remaining area except for the detected dynamic object area to reduce an error in the estimation of the vanishing point, which error may be generated by the dynamic object. The method for estimating the vanishing point may improve the accuracy of estimating the vanishing point when estimating the depth map. For example, in order to exclude a dynamic object area from the depth map, the dynamic object may be detected from the input image to exclude the dynamic object area. Fig. 6A The detected dynamic object area 610 is shown. Figure 6B As shown in , a depth map of static objects in which only dynamic object areas are excluded can be estimated. Here, the dynamic objects can be detected by an object detection network for detecting dynamic objects, and can be detected based on a segmentation mask for dynamic objects. Of course, the way of detecting dynamic objects is not limited to the above-mentioned way.

[0086] When the gradient map of the input image is estimated in S330, in S340 and S350, a heat map of a static object of the input image may be estimated based on the gradient map estimated within CenterNet, and a predetermined or predefined reference gradient map and vanishing point of the input image may be estimated using the estimated heat map.

[0087] According to an embodiment, in S350 , the method for estimating a vanishing point may perform 2D Gaussian fitting of a heat map of a static object, and may estimate a Gaussian center point as a vanishing point through the Gaussian fitting.

[0088] In addition, the reference gradient map compared in S340 may be preset and generated in various schemes and may be used in CenterNet. Figure 4 , the process of generating a reference gradient map, such as an anchor gradient map, is described.

[0089] The process of generating a reference gradient map for a depth map may generate a reference space in which only static objects (such as a rectangular tunnel) exist, and an end-of-line (EOL) camera calibration may be performed to generate reference vanishing points, such as Figure 4 as shown in .

[0090] A reference depth map may be generated based on the reference space and the reference vanishing point generated through the above process, and a reference gradient map for the reference depth map may be generated to generate a depth map-based reference gradient map (or a depth map-based anchor point).

[0091] As such, the method for estimating a vanishing point according to an embodiment of the present disclosure may generate a vanishing point GT of actual image data. The method for estimating a vanishing point may train an artificial intelligence network using the vanishing point GT, such as CenterNet, which is a key point estimation network. The method for estimating a vanishing point may estimate a vanishing point of an input image using CenterNet. Therefore, the accuracy of estimating a vanishing point of an input image may be improved, and constraints may be resolved according to a driving environment with improved accuracy.

[0092] In addition, the method for estimating a vanishing point according to an embodiment of the present disclosure can accurately estimate the vanishing point of an input image. Therefore, it is possible to improve the accuracy of estimating the posture of a camera set in a vehicle.

[0093] In addition, the method for estimating vanishing points according to the embodiment of the present disclosure can generate vanishing points GT of actual image data, and can train a key point estimation network such as CenterNet. Thus, the trained key point estimation network can be applied to estimate the vanishing points of images in the actual scene.

[0094] In addition, the method for estimating the vanishing point according to the embodiment of the present disclosure is not limited to the following. Fig. 7A and Figure 7B The optical flow map based on the input image shown ( Figure 7B ) to estimate the input image ( Fig. 7A ) is the depth map of the vanishing point. At this time, in the optical flow map used for the input image, as Figure 7BAs shown, the vector direction of the optical flow can be represented as color, and the intensity of the optical flow can be represented as concentration. CenterNet can be used to estimate the vanishing point based on this optical flow map similar to the depth map. Of course, in order to estimate the vanishing point based on the optical flow map, it is desired to generate vanishing point GT data based on the optical flow map of the actual image data, and use the generated vanishing point GT data to train CenterNet.

[0095] In addition, in order to estimate the vanishing point based on the optical flow map in CenterNet, it is desirable to use an anchor point based on the optical flow map instead of using an anchor point based on the depth map to generate a reference gradient map. In other words, when estimating the optical flow map of the input image using CenterNet and estimating the gradient map based on the optical flow map, the reference gradient map can be an anchor point based on the optical flow map. At this time, the process of generating an anchor point based on the optical flow map can generate a reference optical flow map based on the reference space and the reference vanishing point, and can generate a reference gradient map for the reference optical flow map. Thus, an anchor point based on the optical flow map can be generated.

[0096] As described above, the method according to the embodiment of the present disclosure can estimate the vanishing point of the input image based on the optical flow map, and estimate the vanishing point of the input image based on the depth map. In other words, the method of the present disclosure can be applied to images based on depth maps and can also be applied to images based on optical flow maps.

[0097] Figure 8 A block diagram of an apparatus for estimating a vanishing point according to another embodiment of the present disclosure is provided, which illustrates a method for executing Figures 3 to 7B A block diagram of an apparatus for performing a vanishing point method.

[0098] refer to Figure 8 According to another embodiment of the present disclosure, a device 800 for estimating a vanishing point may include a receiver 810 , an estimating device 820 , a generator 830 , and a storage device 840 .

[0099] The storage device 840 may be a configuration device for storing all data fragments associated with the technology of the present disclosure, and the configuration device may store CenterNet or data such as a gradient map.

[0100] The receiver 810 may receive an input image to estimate a vanishing point.

[0101] According to one embodiment, the receiver 810 may receive an image captured in real time by a camera provided in a vehicle (eg, an autonomous vehicle, a vehicle equipped with a driver assistance system (ADAS)), as an input image.

[0102] The estimation device 820 may estimate the vanishing point of the input image using CenterNet trained by vanishing point GT data based on the actual image.

[0103] According to an embodiment, the estimation device 820 may estimate a depth map of the input image received in the receiver 810 using CenterNet. The estimation device 820 may estimate a gradient map of the estimated depth map. The estimation device 820 may estimate a heat map of a static object of the input image based on the estimated gradient map and a predetermined reference gradient map. The estimation device 820 may estimate a vanishing point of the input image using the estimated heat map.

[0104] According to an embodiment, the estimation device 820 may detect a dynamic object from an input image, and may estimate a depth map of a remaining area except for the detected dynamic object area to reduce an error in estimating a vanishing point generated by the dynamic object. The accuracy of estimating the vanishing point may be improved.

[0105] The generator 830 may be a configuration device for generating a reference gradient map, which may generate a reference space in which only static objects exist, such as a rectangular tunnel. The generator 830 may generate a reference vanishing point to generate a reference depth map or a reference optical flow map based on the reference space and the reference vanishing point. The generator 830 may generate a reference gradient map for a reference depth map or a reference optical flow map. Thus, a reference gradient map based on a depth map or a reference gradient map based on an optical flow map may be generated.

[0106] Here, since the generator 830 is a configuration device for generating a reference gradient map used in CenterNet, the reference gradient map has already been generated and stored or applied to CenterNet, and the configuration device may be omitted or removed.

[0107] Although the description of the apparatus according to another embodiment of the present disclosure is omitted, the apparatus according to another embodiment of the present disclosure may include Figures 3 to 7B This is obvious to those skilled in the art.

[0108] Likewise, the device for generating vanishing points is used to implement Figure 2 The method for generating vanishing point GT shown in .

[0109] Fig. 9 A block diagram of a computing system for executing a method for estimating a vanishing point according to another embodiment of the present disclosure is illustrated.

[0110] refer to Fig. 9 The method for estimating a vanishing point according to another embodiment of the present disclosure described above may be implemented by a computing system. The computing system 1000 may include at least a processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage device 1600, and a network interface 1700 connected to each other via a system bus 1200.

[0111] The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and / or the storage device 1600. The memory 1300 and the storage device 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a read-only memory (ROM) 1310 and a random access memory (RAM) 1320. Of course, the method for generating the vanishing point GT according to the embodiment of the present disclosure is also implemented by Fig. 9 computing system to achieve this.

[0112] Thus, the operation of the method or algorithm described in conjunction with the embodiments disclosed in this specification can be directly implemented with a hardware module, a software module, or a combination of a hardware module and a software module, which is executed by the processor 1100. The software module may reside on a storage medium (i.e., memory 1300 and / or storage device 1600) such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, and CD-ROM. The storage medium may be coupled to the processor 1100. The processor 1100 may read information from the storage medium and may write information to the storage medium. Alternatively, the storage medium may be integrated with the processor 1100. The processor 110 and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a user terminal. In another case, the processor 1100 and the storage medium may reside in a user terminal as separate components.

[0113] According to the present disclosure, the device for estimating the vanishing point can generate the vanishing point GT of the actual data, and can use the vanishing point GT to train an artificial intelligence network (e.g., CenterNet, which is a key point estimation network) to estimate the vanishing point of the input image. Therefore, the accuracy of estimating the vanishing point of the input image can be improved, and constraints can be resolved according to the driving environment.

[0114] According to the present disclosure, an apparatus for estimating a vanishing point may accurately estimate a vanishing point of an input image, and thus may improve accuracy in estimating a pose of a camera provided in a vehicle.

[0115] According to the present disclosure, the device for estimating the vanishing point can generate the vanishing point GT of the actual data and can train the network. Thus, the vanishing point GT can be applied to estimate the vanishing point in the actual field.

[0116] The effects achieved by the present disclosure may not be limited to the above effects. A person skilled in the art to which the present disclosure belongs may more clearly understand other advantages not described above from the following detailed description.

[0117] In the above, although the present disclosure has been described with reference to the embodiments and the accompanying drawings, the present disclosure is not limited thereto. Without departing from the spirit and scope of the present disclosure claimed in the attached claims, a person of ordinary skill in the art to which the present disclosure belongs may make various modifications and changes to the present disclosure. Therefore, the embodiments disclosed in the present disclosure are not intended to limit the technical spirit of the present disclosure. The scope of the technical spirit of the present disclosure is not limited to such embodiments. The scope of the present disclosure should be interpreted based on the attached claims, and all technical concepts within the scope equivalent to the claims should be included in the scope of the present disclosure.

Claims

1. A method for estimating a vanishing point, the method comprising: receiving an input image; as well as using an artificial intelligence network pre-trained with vanishing point ground truth GT data generated based on actual image data to estimate the vanishing points of the input image, The estimation of the vanishing point includes: estimating a depth map or an optical flow map of the input image, estimating a gradient map of the depth map or the optical flow map, and The vanishing point of the input image is estimated based on the gradient map and a predetermined reference gradient map.

2. The method according to claim 1, wherein: Estimating the vanishing point includes estimating a heat map of the static object based on the gradient map and the predetermined reference gradient map and estimating the vanishing point based on the heat map.

3. The method according to claim 2, wherein: Estimating the vanishing point includes performing a two-dimensional 2D Gaussian fit of the heat map of the static object and estimating a center point of the 2D Gaussian as the vanishing point.

4. The method according to claim 2, wherein: The artificial intelligence network includes a network for estimating key points corresponding to the vanishing points in the heat map.

5. The method according to claim 1, wherein: Estimating the depth map or the optical flow map comprises: when estimating the depth map, detecting at least one predetermined dynamic object from the input image; and A depth map of static objects excluding the detected area of ​​at least one predetermined dynamic object is estimated.

6. The method according to claim 1, wherein: The artificial intelligence network includes CenterNet.

7. The method according to claim 1, further comprising: generating the predetermined reference gradient map, Wherein, generating the predetermined reference gradient map comprises: Generate a reference space with only static objects and reference vanishing points, generating a reference depth map or a reference optical flow map based on the reference space and the reference vanishing point, and A gradient map of the reference depth map or the reference optical flow map is generated to generate the predetermined reference gradient map.

8. The method according to claim 1, wherein: When a depth map or an optical flow map of an image is generated, when a gradient map of the depth map or the optical flow map is generated, when a heat map of the gradient map is generated by Gaussian fitting of the gradient map and key points are generated using Gaussian center points by the Gaussian fitting, and when the coordinates of the Gaussian center points are the same as the coordinates of the vanishing points detected by at least one predetermined vanishing point detection technique, vanishing point GT data is generated based on the heat map and the key points.

9. A device for estimating a vanishing point, the device comprising: a receiver configured to receive an input image; as well as an estimation device configured to estimate the vanishing points of the input image using an artificial intelligence network pre-trained by vanishing point GT data generated based on actual image data, The estimation device is configured to estimate a depth map or an optical flow map of the input image, estimate a gradient map of the depth map or the optical flow map, and estimate the vanishing point of the input image based on the gradient map and a predetermined reference gradient map.

10. The device according to claim 9, wherein: The estimating device is further configured to estimate a heat map of the static object based on the gradient map and the predetermined reference gradient map and to estimate the vanishing point based on the heat map.

11. The device according to claim 10, wherein: The estimation device is further configured to perform a 2D Gaussian fitting of the heat map of the static object and estimate a center point of the 2D Gaussian as the vanishing point.

12. The device according to claim 10, wherein: The artificial intelligence network includes a network for estimating key points corresponding to the vanishing points in the heat map.

13. The device according to claim 9, wherein: The estimating device is further configured to, when estimating the depth map, detect at least one predetermined dynamic object from the input image, and estimate a depth map of static objects excluding a region of the detected at least one predetermined dynamic object.

14. The device according to claim 9, wherein: The artificial intelligence network includes CenterNet.

15. The apparatus according to claim 9, further comprising a generator, wherein the generator is configured to: generating the predetermined reference gradient map; Generate a reference space with only static objects and reference vanishing points; generating a reference depth map or a reference optical flow map based on the reference space and the reference vanishing point; and A gradient map of the reference depth map or the reference optical flow map is generated to generate the predetermined reference gradient map.

16. The device according to claim 9, wherein: The vanishing point GT data is generated based on the heat map and the key points when a depth map or an optical flow map of an image is generated, when a gradient map of the depth map or the optical flow map is generated, when a heat map of the gradient map is generated by Gaussian fitting of the gradient map and key points are generated using Gaussian center points by the Gaussian fitting, and when the coordinates of the Gaussian center points are the same as the coordinates of the vanishing points detected by at least one predetermined vanishing point detection technique.

17. A method for generating a vanishing point GT, the method comprising: Generate a depth map or optical flow map of the input image; Generating a gradient map of the depth map or the optical flow map; Generating a heat map of the gradient map by Gaussian fitting of the gradient map and generating key points by using Gaussian center points by the Gaussian fitting; as well as When the coordinates of the Gaussian center point are the same as the coordinates of the vanishing point detected by at least one predetermined vanishing point detection technique, the vanishing point GT is generated based on the heat map and the key points.

18. The method according to claim 17, wherein: The input image includes actual image data.

Images