Method and apparatus for displaying three-dimensional objects

Abstract

A method and apparatus having a three-dimensional object display are provided. A method of displaying a three-dimensional (3D) object includes acquiring a driving image from a single camera, classifying line segments including a road surface into one or more groups based on a change in curvature of the road surface, estimating a pitch angle corresponding to a tilt angle between the road surface and the single camera for each of the one or more groups, generating 3D information of the road surface, and displaying the 3D object visually superimposed on the road surface based on the 3D information of the road surface.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of Korean Patent Application No. 10-2019-0148843, filed on November 19, 2019, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0003] The following description relates to methods and apparatus for displaying three-dimensional (3D) objects. Background Technology

[0004] Three-dimensional (3D) information about the road surface is used to generate information for autonomous driving and advanced driver assistance systems (ADAS). For example, 3D information about the road surface can be obtained by calculating disparity through comparison between left and right images acquired by a stereo camera, and then estimating depth information for all areas, including the road surface, based on the disparity. However, when using a single camera, it is difficult to estimate depth information by calculating disparity. Therefore, it may be challenging to acquire 3D information about the road surface using a single camera. Summary of the Invention

[0005] The summary portion of this invention is provided to introduce, in a simplified form, some concepts further described below in the detailed description. This summary portion is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter.

[0006] In one general aspect, a method for displaying three-dimensional (3D) objects includes: acquiring a driving image including a road surface from a single camera; classifying line segment markings on the road surface into one or more groups based on the degree of change in the curvature of the road surface; estimating, for each of the one or more groups, a pitch angle of the road surface corresponding to the tilt angle between the road surface and the single camera; generating 3D information of the road surface using the pitch angle for each of the one or more groups; and displaying a 3D object visually superimposed on the road surface based on the 3D information of the road surface.

[0007] Classifying line segment markers into one or more groups can include grouping line segment markers based on the degree of change in the curvature of the line segment.

[0008] Classifying line segment markers into one or more groups may include: comparing the degree of change in the curvature of the line segment with a reference change; and, based on the comparison results, classifying the line segment markers into multiple groups based on points where the degree of change in curvature is greater than the reference change.

[0009] Each of the one or more groups can form corresponding planes that are different from each other on the road surface.

[0010] Estimating the pitch angle can include converting partial images of the travel image for each of the one or more groups to bird's eye view (BEV) images for each of the one or more groups, and estimating the pitch angle of the road surface based on the BEV images for each of the one or more groups.

[0011] Estimating the pitch angle can include estimating a pitch angle that satisfies at least one of a first condition that line segment marks included in the BEV images for each of the one or more groups are parallel to each other and a second condition that the line segment marks have the same length as the pitch angle of the road surface.

[0012] Estimating the pitch angle can include calculating positions of the line segment marks included in the BEV images for each of the one or more groups based on an initial pitch angle, repeatedly adjusting the pitch angle until at least one of the first condition and the second condition is satisfied, and determining the pitch angle that satisfies at least one of the first condition and the second condition as the pitch angle of the road surface.

[0013] The travel image can include a sequence of images over time. Generating the 3D information of the road surface can include smoothing the pitch angle of the road surface over time for each of the one or more groups, and generating the 3D information of the road surface included in the travel image using the smoothed pitch angle.

[0014] Smoothing the pitch angle can include smoothing the pitch angle of the road surface over time for each of the one or more groups using a temporal filter.

[0015] Displaying the 3D object can include rendering a 3D road model reflecting the 3D information of the road surface, and displaying the 3D object on the 3D road model.

[0016] The method can further include extracting line segments of the road surface corresponding to the travel lane from the travel image, and calculating curvatures of the extracted line segments.

[0017] Extracting the line segments of the road surface can include detecting line segments having a length greater than or equal to a predetermined length included in the travel image by searching for edges from the travel image, and estimating the line segments of the road surface corresponding to the travel lane by grouping the detected line segments based on a travel direction.

[0018] The method can further include extracting a left line and a right line of the travel lane from the travel image, and calculating curvatures of the left line and the right line of the travel lane.

[0019] The method can further include outputting the 3D information of the road surface.

[0020] The travel image can include at least one of an RGB image and a grayscale image.

[0021] In another general aspect, an apparatus for displaying a 3D object includes a communication interface configured to acquire a travel image from a single camera; a processor configured to classify line segment markers included in the travel image into one or more groups based on a degree of change in curvature of the line segments, estimate, for each of the one or more groups, a pitch angle of a road surface corresponding to a tilt angle between the road surface and the single camera, generate, for each of the one or more groups, 3D information of the road surface included in the travel image using the pitch angle of the road surface, and represent the 3D object on the road surface based on the 3D information of the road surface; and a display configured to display a 3D road model representing the 3D object.

[0022] The processor can be configured to group the line segment markers based on the degree of change in curvature of the line segments.

[0023] The processor can be configured to compare the degree of change in curvature of the line segments to a reference change, and based on a result of the comparison, classify the line segment markers into a plurality of groups based on points where the degree of change in curvature is greater than the reference change.

[0024] Each of the one or more groups can form a plane distinct from each other on the road surface.

[0025] The processor can be configured to convert a partial image of the travel image for each of the one or more groups into a bird's eye view (BEV) image for each of the one or more groups, and estimate the pitch angle of the road surface based on the BEV image for each of the one or more groups.

[0026] The processor can be configured to estimate, as the pitch angle of the road surface, a pitch angle that satisfies at least one of a first condition that line segment markers included in the BEV image for each of the one or more groups are parallel to each other and a second condition that the line segment markers have the same length.

[0027] The processor can be configured to calculate, based on an initial pitch angle, positions of the line segment markers included in the BEV image for each of the one or more groups, repeatedly adjust the pitch angle until at least one of the first condition and the second condition is satisfied, and determine, as the pitch angle of the road surface, the pitch angle that satisfies at least one of the first condition and the second condition.

[0028] The travel image can include a sequence of images over time. The processor can be configured to smooth the pitch angle of the road surface over time for each of the one or more groups, and generate the 3D information of the road surface included in the travel image using the smoothed pitch angle.

[0029] The processor can be configured to smooth the pitch angle of the road surface over time for each of the one or more groups using a temporal filter.

[0030] The processor can be configured to render a 3D road model reflecting the 3D information of the road surface and to represent the 3D object on the 3D road model.

[0031] The processor can be configured to extract line segments of the road surface corresponding to the travel lane from the travel image and to calculate curvatures of the extracted line segments.

[0032] The processor can be configured to detect line segments included in the travel image and having a length greater than or equal to a predetermined length by searching for edges from the travel image and to estimate line segments of the road surface corresponding to the travel lane by grouping the detected line segments based on a travel direction.

[0033] The processor can be configured to extract left and right lines of the travel lane from the travel image and to calculate curvatures of the left and right lines of the travel lane.

[0034] The communication interface can be configured to output the 3D information of the road surface.

[0035] In another general aspect, an apparatus for displaying a three-dimensional (3D) object includes a single camera sensor, a processor, and a display. The single camera sensor is configured to acquire a travel image including a road surface. The processor is configured to classify line segment markers of the road surface into one or more groups based on a degree of change in curvatures of the line segment markers, to estimate, for each of the one or more groups, a pitch angle of the road surface corresponding to an inclination angle between the road surface and the single camera sensor, to generate, for each of the one or more groups, 3D information of the road surface using the pitch angle. The display is configured to output, based on the 3D information, the 3D object visually superimposed on the road surface.

[0036] Other features and aspects will become apparent from the following detailed description, the drawings and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart illustrating an example method of displaying a three-dimensional (3D) object.

[0038] Figure 2 An example method of extracting line segment markers of a road surface from a travel image is illustrated.

[0039] Figure 3 An example method of classifying line segment markers into at least one group is illustrated.

[0040] Figure 4 An example method of estimating a pitch angle is shown.

[0041] Figure 5 An example method of estimating a pitch angle is shown.

[0042] Figure 6 An example method of smoothing a pitch angle is shown.

[0043] Figure 7 An example 3D object shown on a road surface is shown.

[0044] Figure 8 and 9 An example of an apparatus for displaying a 3D object is shown.

[0045] Throughout the drawings and detailed description, unless otherwise described or specified, like reference characters designate like elements, features, and structures. The drawings can not be to scale, and the relative dimensions, proportions, and depiction of elements in the drawings can be exaggerated for purpose of clarity, illustration, and convenience. DETAILED DESCRIPTION

[0046] The following detailed description is provided to help the reader understand the methods, apparatuses, and / or systems described herein. However, various changes, modifications, and equivalents can become apparent to the reader familiar with the disclosure of this application. For example, the order of the operations described herein can be changed, except where otherwise specified or required, and not all operations are necessarily performed, in order to implement the methods described herein. In addition, descriptions of features known to those of ordinary skill in the art can be omitted so as not to obscure the description of the features that are most directly related to the methods, apparatuses, and / or systems described herein.

[0047] The features described herein can be implemented in different forms and are not to be construed as limited to the examples described herein. Rather, the examples described herein have been provided so that this disclosure will be thorough, and will fully convey the scope of the methods, apparatuses, and / or systems to those skilled in the art.

[0048] The following detailed description of examples disclosed by this disclosure is merely intended to describe and does not limit the examples in that the examples can be implemented in various forms. The examples are not intended to limit but are intended to cover various modifications, equivalents, and alternatives within the scope of the claims.

[0049] Although terms such as "first" or "second" are used to explain various components, the components are not limited to the terms. The terms should be used only to distinguish one component from another component. For example, a "first" component can be referred to as a "second" component, or similarly, a "second" component can be referred to as a "first" component, within the scope of the concept according to the present disclosure.

[0050] It should be understood that when a component is referred to as being "connected" to another component, the component can be directly connected or coupled to the other component, or there can be intervening components present.

[0051] Unless the context clearly indicates otherwise, as used herein, the singular also includes the plural. It is also to be understood that the term "including" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0052] Unless otherwise defined herein, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs in view of the disclosure provided herein. Unless otherwise defined herein, terms defined in commonly used dictionaries are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure provided herein, and are not to be interpreted in an idealized or overly formal sense.

[0053] Hereinafter, examples will be described in detail with reference to the accompanying drawings, and the same reference numerals in the drawings always denote the same elements.

[0054] The following examples can include or apply to line segment display in an augmented reality (AR) navigation system of a smart vehicle, or include generation of visual information, and / or assist in steering of an autonomous vehicle. In addition, the examples can be used to display visual information including three-dimensional (3D) objects in example embodiments including a smart system, such as a head-up display (HUD) installed for driving assistance or full autonomous driving in a vehicle, or other displays, in order to assist in safe and comfortable driving. Some examples include or apply to, for example, an autonomous vehicle, a smart vehicle, a smart phone, an augmented reality (AR) HUD, or a mobile device.

[0055] Figure 1 is a flowchart illustrating an example method of displaying a 3D object. Referring to Figure 1 In operation 110, an apparatus for displaying a 3D object (hereinafter, referred to as a "display apparatus") acquires a driving image from a single camera. The driving image can include a road surface image and / or a road image including a vehicle, a lane, a curb, a sidewalk, and / or a surrounding environment, as inFigure 2 The driving image 210 shown in the non-limiting example. The driving image can include, for example, an RGB image or a grayscale image. The driving image can be a single frame image or a series of frame images collected over time. In this document, it should be noted that the use of the term "may" (e.g., with respect to what an example or embodiment can include or implement) means that there is at least one example or embodiment that includes or implements the feature, but not all examples and embodiments are limited to this.

[0056] The display device can acquire one or more driving images for each frame using a single camera installed at the front of the driving vehicle. The calibration information of the single camera can be predetermined.

[0057] In operation 120, the display device classifies the line segment markers into one or more groups based on the determined curvature variation of the line segments of the road surface included in the driving image. In the following description, the "line segment marker" corresponds to, for example, a part or all of the marker used to mark the line segment. For example, the display device can group the line segment markers based on the degree of curvature variation of the line segments.

[0058] Before operation 120, the display device can extract the line segments of the road surface corresponding to the driving lane from the driving image. For example, the display device can extract the left and right lines of the driving lane from the driving image. The display device can extract the line segments by detecting the line segment markers, or extract the line segments based on the line segment probability of each pixel of the driving image according to the lane probability map. The method by which the display device extracts the line segments of the road surface will be further described below with reference to Figure 2 The method by which the display device extracts the line segments of the road surface will be further described below with reference to

[0059] In operation 120, the display device can calculate the curvatures of the extracted line segments. For example, the display device can calculate the curvatures of the left and right lines that distinguish the driving lane from other lanes, and / or the curvatures of the left and right lines of other lanes other than the driving lane. The display device can calculate the degree of variation, for example, the change, of each of the curvatures of the line segments. The display device can classify the line segment markers that are considered straight lines based on the degree of variation of each of the curvatures of the line segments into one group. The example method by which the display device classifies the line segment markers into one or more groups will be further described below with reference to Figure 3 The example method by which the display device classifies the line segment markers into one or more groups will be further described below with reference to

[0060] In operation 130, the display device estimates a pitch angle of the road surface corresponding to a tilt angle between the single camera and the road surface, for each of the one or more groups. For example, since the local area of the road surface gradually changes, the traveling direction on the road can be expressed as an angle change of the road surface. Accordingly, the 3D information of the road surface can be generated based on the pitch angle of the road surface. For example, the display device can convert the partial image in the travel image for each of the one or more groups into a bird's eye view (BEV) image for each of the one or more groups. The display device can estimate the pitch angle of the road surface based on the BEV image for each of the one or more groups. An example method in which the display device estimates the pitch angle will be further described below with reference to Figure 4 and Figure 5 An example method in which the display device estimates the pitch angle will be further described below with reference to

[0061] In operation 140, the display device generates 3D information of the road surface included in the travel image based on the pitch angle of the road surface, for each of the one or more groups. For example, when the travel image includes a sequence of images over time, the display device can smooth the pitch angle of the road surface over time, for each of the one or more groups. A method in which the display device smooths the pitch angle will be further described below with reference to Figure 6 A method in which the display device generates 3D information of the road surface included in the travel image based on the smoothed pitch angle will be further described below with reference to

[0062] Since it is assumed that each group of line segments is a plane, it is assumed that a roll angle of the 3D information of the road surface is zero. In addition, a yaw angle can be expected from the BEV image, and the pitch angle is estimated in operation 130. The display device can generate the 3D information of the road surface based on the roll angle, the yaw angle, and the pitch angle.

[0063] In operation 150, the display device displays a 3D object on the road surface based on the 3D information of the road surface. An example method in which the display device displays the 3D object on the road surface will be further described below with reference to Figure 7 An example method in which the display device displays the 3D object on the road surface will be further described below with reference to

[0064] In the following description, the term "road" refers to a path on which a vehicle travels, and includes various types of roads, as non-limiting examples, such as a highway, a national road, a local road, an expressway, or a road. In an example, the road includes one or more lanes.

[0065] The term "lane" refers to a road space distinguished by line segments marked on a surface of a road. The term "traveling lane" refers to a lane in which a traveling vehicle is traveling among a plurality of lanes, i.e., a lane space occupied and used by the traveling vehicle. A single lane is distinguished from other lanes by its left and right lines.

[0066] The term "line segment" is understood as various types of line segment markings, for example, solid or dashed lines marked in color. The color on the road surface may, for example, include white, yellow, or light blue. In addition to the line segments for distinguishing lanes, the various types of line segments belonging to line segment markings can also include, for example, zigzag lines, bus lane lines, or pedestrian separation lines. In the following description, the left and right lines among the line segments for distinguishing individual lanes are referred to as "lane boundary lines" to distinguish the left and right lines from other line segments.

[0067] In addition, the term "traveling vehicle" refers to the user's vehicle among the vehicles traveling on the road, and corresponds to, for example, an autonomous vehicle, or a smart vehicle or a wise vehicle equipped with an advanced driving assistance system (ADAS).

[0068] Figure 2 An example method of extracting line segment markings of a road surface from a travel image is shown. In Figure 2 In operation 220, the display device searches for edges in the travel image 210. For example, as a non-limiting example, the display device can search for edges in the travel image 210 using a Canny edge detection scheme or a Hough transform scheme.

[0069] In operation 230, the display device detects line segments having a length greater than or equal to a predetermined length from the edges found in operation 220. The predetermined length can correspond to, for example, the length of a lane dashed line on the road surface or a length similar to the length of the lane dashed line.

[0070] In operation 240, the display device selects a predetermined line segment from the line segments detected in operation 230. The predetermined line segment can correspond to, for example, the left and right lane boundary lines of the travel lane. For example, the display device can group line segments (e.g., line segment markings) having a length greater than or equal to a predetermined length based on the travel direction, and can select a predetermined line segment from a predetermined area of line segment features or edges among the grouped line segment markings. In an example, when a predetermined line segment is not selected from the predetermined area, the display device can predict a candidate line segment having the highest possibility of becoming a line segment, and can select the predetermined line segment.

[0071] In operation 250, the display device performs line segment fitting on the predetermined line segment selected in operation 240. The line segment markings of the travel road can be solid lines and / or dashed lines. For example, when the line segment markings are dashed lines or partial dashed lines, the display device can construct a single straight line or a single curved line by connecting the markings of the dashed lines or partial dashed lines. The display device can perform line segment fitting on the predetermined line segment, i.e., the lane boundary line.

[0072] For example, in an example in which a portion of the line segment marking is blocked by an obstacle and / or is not visible, the display device can predict the line segment based on known information (e.g., the width of the lane, the color of the lane other than the line segment, the portion of the portion line segment that is seen). The display device can estimate a confidence value based on the likelihood of the predicted line segment, and can use the confidence value to generate 3D information of the road surface.

[0073] In operation 260, the display device separates the line segment on the road surface by the predetermined line segment for which the line segment fitting is performed in operation 250.

[0074] Through the above-described process, the display device can separate the line segment 275 on the road surface as shown in the image 270 depicted in FIG. 27. Figure 2

[0075] According to an example, the display device can extract various line segments or line segment markings from the travel image using, for example, a convolutional neural network (CNN), a deep neural network (DNN), or a support vector machine (SVM) that is pre-trained to recognize the line segment marking of the road surface. The CNN can be pre-trained to recognize various line segment markings of the road surface image, and can be, for example, a region-based CNN. For example, the CNN can be trained to recognize a bounding box of the line segment marking and the line segment to be detected from the travel image.

[0076] Figure 3 An example method of classifying the line segment marking into one or more groups is shown. In Figure 3 In FIG. 31, the line segment markings 311 and 313 are classified into a first group 310, and the line segment markings 331 and 333 are classified into a second group 330.

[0077] The line segment separated through the above-described process can correspond to a lane boundary line for distinguishing a travel lane. The travel lane can be, for example, a straight section or a curved section. The display device can distinguish the straight section and the curved section based on, for example, the curvatures of the left line and the right line of the travel lane in which the travel vehicle is traveling, such that both the straight section and the curved section can be represented with a straight marking.

[0078] In one or more examples, a single camera is included to capture a front view image in a direction in which the vehicle travels. In this example, among the line segment markings of the left line and the right line of the travel lane in the captured travel image, the line segment markings of the same height are predicted to have the same depth. Accordingly, the line segment markings of the same height of the left line and the right line are classified into a group. The height can refer to a distance between a lower end of a group and an upper end of the same group. The display device can group the line segment markings based on both the curvature change of the left line and the curvature change of the right line.

[0079] ​The display device can compare the reference change with a degree of change in curvature of each line segment in the left and right lines that are divided into a group based on the same height. The display device can classify the line segment markers into groups based on points where the degree of change in curvature of the left and right lines is greater than the reference change.

[0080] For example, Figure 3 Points 340 and 350 are depicted as points where the degree of change in curvature of each line segment in the left and right lines is greater than the reference change. The display device classifies line segment markers 311 and 313 into a first group 310 and classifies line segment markers 331 and 333 into a second group 330 based on points 340 and 350. In this example, the road surface corresponding to the first group 310 forms a relatively flat plane having a parallelogram shape. In addition, the road surface corresponding to the second group 330 forms a relatively flat plane having a parallelogram shape that is distinguishable from the first group 310.

[0081] In an example, when only one group is formed in the travel image, the entire road surface in the travel image can form a single plane. In another example, when more than one group (e.g., five groups) is formed in the travel image, the road surface in the travel image can be divided into five sections, and the curvatures corresponding to at least adjacent ones of the five sections in the road surface can be predicted to be relatively different from each other.

[0082] Figure 4 An example method of estimating a pitch angle is shown. Figure 4 A partial image 410 showing line segment markers 415, a bird's eye view (BEV) image 420 obtained by converting the partial image 410, and an image 430 in which positions of the line segment markers 415 are adjusted according to a pitch angle estimated from the BEV image 420 are shown.

[0083] The display device converts a partial image 410 for each group (e.g., one group in this example) in the travel image into a BEV image 420. In this example, the BEV image 420 is obtained by converting a partial image 410 corresponding to a group formed by classifying line segment markers 415 in the travel image, rather than converting the entire travel image.

[0084] The line segment marks 415 in the partial image 41() are displayed to converge toward a vanishing point. When the partial image 410 is converted into the BEV image 420, the line segment marks appearing in the BEV image 420 can be parallel to each other when the partial image 410 is viewed from above. In an example in which a road surface in the partial image 410 has a tilt angle, for example, the line segment marks appearing in the BEV image 420 can diverge or converge at a predetermined angle instead of being parallel to each other. In an example, an angle that makes the line segment marks appearing in the BEV image 420 parallel to each other can be determined, and the angle can correspond to a pitch angle.

[0085] For example, the display device can convert the partial image 410 into the BEV image 420 using inverse perspective mapping (IPM). The IPM removes a distance effect from the partial image 410 having the distance effect and converts position information of a plane of the partial image 410 into position information of a world coordinate system. Through the IPM, the position information of the plane of the partial image 410 is converted into the position information of the world coordinate system.

[0086] For example, assume that there is no roll value because the traveling vehicle does not move in a roll angle direction. In this example, a yaw angle yaw(ψ) and a pitch angle pitch(θ) on a plane of the partial image 410 can be expressed as and

[0087] In addition, world coordinates (x world , y world , z world ) corresponding to the partial image 410 are expressed, for example, as shown in Equation 1 below.

[0088] Equation 1:

[0089]

[0090] The display device can remove a distance effect from the world coordinates (x world , y world , z world ) of the partial image 410 and convert position information of a plane of the partial image 410 into position information of a world coordinate system, for example, as shown in Equation 2 below.

[0091] Equation 2:

[0092]

[0093] In Equation 2,

[0094] The display device can acquire the BEV image 420 by applying IPM based on a central portion of the bounding box corresponding to the partial image 410. In one or more examples, the display device can apply IPM to the detected line segment markers 415, excluding a background portion of the partial image 410, thereby reducing a computation amount.

[0095] In one or more examples, the display device can estimate the BEV image 420 corresponding to the partial image 410 by performing IPM based on pre-acquired calibration information of a single camera. The calibration information can include, for example, extrinsic parameters.

[0096] The display device can estimate a pitch angle of a road surface based on the BEV image 420. A reference position of the BEV image 420 can be changed based on a gradient θ of the road surface. For example, when the partial image 410 is converted into the BEV image 420, the display device can estimate an angle satisfying at least one of a first condition that the line segment markers 415 in the partial image 410, i.e., the left and right line segment markers 415 included in the same group, are parallel to each other, and a second condition that the line segment markers 415 have the same length, as the pitch angle of the road surface.

[0097] The display device calculates positions of the line segment markers included in the BEV image 420 for each group based on the initial pitch angle. The display device can repeatedly adjust the pitch angle until the line segment markers included in the BEV image 420 satisfy either or both of the first and second conditions. The display device determines the pitch angle satisfying either or both of the first and second conditions as the pitch angle of the road surface.

[0098] When the pitch angle that makes the line segment markers 415 satisfy the above conditions is reflected in the BEV image 420, the positions of the line segment markers included in the BEV image 420 can be adjusted such that the line segment markers can be parallel to each other, as shown in the image 430.

[0099] Figure 5 An example method of estimating a pitch angle is shown. Figure 5 A travel image 510 and a BEV image 530 are shown.

[0100] For example, line segment markers 511, 513, and 515 appearing in the travel image 510 of a travel road having a predetermined gradient are not parallel to each other. The display device converts the travel image 510 using the above-described IPM, optimizes the pitch angle to satisfy the above conditions, and generates the BEV image 530. In the BEV image 530, line segment markers 531, 533, and 535 are parallel to each other.

[0101] In this example, even if the yaw angle is not individually corrected, the parallel condition of the line segment markers 531, 533, and 535 extracted by the optimized pitch angle is not affected.

[0102] In Equation 2, The coordinate system of the reference plane having a flat road surface is represented by Z. In addition, Z corresponds to the height of a single camera that captures the driving image 510, for example.

[0103] When Z is reflected, an optimized pitch angle can be calculated, which will be described below.

[0104] For example, the slope of each of the line segments 531, 533, and 535 of the road surface included in the BEV image 530 is defined as In this example, the display device can estimate the pitch angle by performing optimization to satisfy, for example, Equation 3 shown below.

[0105] Equation 3:

[0106]

[0107] The display device can estimate the pitch angle in the 3D information of the road surface converted into the BEV image 530 through the above optimization.

[0108] Figure 6 An example method of smoothing the pitch angle is shown. Figure 6 A plot 610 showing the pitch angle corresponding to the driving image that changes over time, and BEV images 620, 630, 640, and 650 corresponding to the changing pitch angle in the driving image are shown.

[0109] As shown in the BEV images 620, 630, 640, and 650 of Figure 6 the pitch angle estimated over time changes in value. Accordingly, the pitch angle of the road surface for each of one or more groups can be smoothed over time. For example, the display device can smooth the pitch angle of the road surface for each group over time using a temporal filter (e.g., a Kalman filter). The display device can generate 3D information of the road surface included in the driving image based on the smoothed pitch angle.

[0110] Figure 7 An example 3D object displayed and superimposed on the road surface is shown. Figure 7 A driving image 710 is shown, which includes a speed limit display object 720, a vehicle speed display object 730, and a road slope display object 740 appearing in the driving image 710.

[0111] In the example of Figure 7 each group from the driving image 710 is assumed to be a plane, and thus the roll angle of the 3D information of the road surface is considered to be non-existent. In addition, the yaw angle is obtained from the BEV image of the driving image 710, and the pitch angle is estimated through the above Figures 1 to 6 processing.

[0112] The display device renders a 3D road model reflecting 3D information of the road surface based on the 3D information associated with the pitch angle, the roll angle, and the yaw angle of the road surface acquired in the above-described process, and displays the 3D object on the 3D road model.

[0113] The display device displays a 3D object on the road surface of the driving image 710, the 3D object including, for example, a speed limit display object 720, a vehicle speed display object 730, and a road slope display object 740. The 3D object displayed on the road surface of the driving image 710 by the display device is not necessarily limited thereto, and various 3D objects, for example, a pedestrian object, a vehicle object, or a pet object, can be displayed. The 3D object can be a 3D virtual object, for example, an AR object, which can be visually superimposed in a real-world environment, for example, a road surface.

[0114] The display device can superimpose a 3D virtual object on the road surface, or allow interaction between the road surface and the 3D virtual object.

[0115] Figure 8 An example display device 800 for displaying a 3D object is shown. In Figure 8 The display device 800 includes a single camera 810, a camera processor 820, a processor 830, a graphics processor 840, a display 850, and a memory 860. The camera processor 820, the processor 830, the graphics processor 840, and the memory 860 communicate with each other via a communication bus 870.

[0116] The display device 800 receives a driving image captured by the single camera 810, and outputs 3D information of a road surface included in the driving image and / or a 3D road model.

[0117] The single camera 810 captures a driving image.

[0118] The camera processor 820 converts a partial image for each of one or more groups in the driving image to a BEV image for each of the one or more groups by performing IPM based on calibration information (for example, extrinsic parameters) of the single camera 810.

[0119] The processor 830 classifies line segments of the road surface in the driving image into one or more groups based on a curvature variation of the line segments. The processor 830 estimates a pitch angle of the road surface based on the BEV image for each of the one or more groups. The processor 830 generates 3D information of the road surface included in the driving image based on the pitch angle of the road surface for each of the one or more groups.

[0120] The graphics processor 840 renders a 3D road model reflecting 3D information of a road surface and represents a 3D object on the 3D road model.

[0121] The display 850 displays the 3D road model representing the 3D object.

[0122] The memory 860 stores data and final results generated in the processing processes of the camera processor 820, the processor 830, and / or the graphics processor 840, in addition to the driving image captured by the single camera 810.

[0123] Figure 9 is a block diagram illustrating an example display device 900 for displaying a 3D object. In Figure 9 The display device 900 includes a single camera 910, a communication interface 920, a processor 930, and a display 940. The display device 900 further includes a memory 950. The single camera 910, the communication interface 920, the processor 930, the display 940, and the memory 950 communicate with each other via a communication bus 905.

[0124] The display device 900 can include, for example, an autonomous vehicle, a smart vehicle, a smartphone, or a mobile device.

[0125] The single camera 910 captures a driving image.

[0126] The communication interface 920 acquires the driving image from the single camera 910. The communication interface 920 outputs 3D information of a road surface generated by the processor 930.

[0127] The processor 930 classifies line segments of a road surface included in the driving image into one or more groups based on a change in curvature of the line segments. For each of the one or more groups, the processor 930 estimates a pitch angle of the road surface corresponding to a tilt angle between the single camera 910 and the road surface. For each of the one or more groups, the processor 930 generates 3D information of the road surface included in the driving image based on the pitch angle. The processor 930 represents a 3D object on the road surface based on the 3D information of the road surface.

[0128] The display 940 displays a 3D road model representing the 3D object. The display 940 can include, for example, an AR HUD.

[0129] The memory 950 stores the driving image and / or the 3D road model representing the 3D object.

[0130] Further, the processor 930 performs one or more processes described herein with reference to Figures 1 to 8One or more processes described are configured by execution of instructions to perform one or more or all of the processes described herein. The processor 930 is a hardware implementation of a data processing apparatus having circuitry physically structured to perform desired operations. For example, the desired operations include code or such instructions, for example included in a program. Hardware implementation of a data processing apparatus includes, for example, a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a multi-processor, an application-specific integrated circuit (ASIC), and a field-programmable gate array (FPGA).

[0131] The processor 930 executes a program and controls the display device 900. Code of the program executed by the processor 930 is stored in the memory 950.

[0132] The memory 950 stores various information generated in the above-described processes of the processor 930. In addition, the memory 950 stores various data and programs. For example, the memory 950 includes a volatile memory or a non-volatile memory. The memory 950 includes a high-capacity storage medium such as a hard disk to store various data.

[0133] The display device 800, single camera 810, single camera 910, camera processor 820, processor 830, 930, display 850, 940, graphics processor 840, memory 860, 950, communication interface 920, display device, processor, device, unit, module, apparatus, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform operations described in this application, where appropriate, include controllers; sensors; generators; drivers; memories; comparators; arithmetic logic; adders; subtractors; multipliers; dividers; integrators; and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components that perform operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements, such as logic gates arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and perform instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes or is connected to one or more memories that store instructions or software for execution by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For the sake of brevity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples a plurality of processors or computers can be used, or a processor or computer can include a plurality of processing elements, or a plurality of types of processing elements, or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors or another processor and another controller. The one or more processors or a processor and a controller can implement a single hardware component, or two or more hardware components. The hardware components can have any one or more of a variety of processing configurations, examples of which include a single-processor system, one or more multi-processor systems, a multiprocessor system, a multi-core processor system, and a single processor system.

[0134] The hardware components that perform the operations described in this applicationFigures 1 to 9 The methods are executed by computing hardware, such as one or more processors or computers implemented as described above, which execute instructions or software to perform the operations described in this application performed by these methods. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors or a processor and a controller, and one or more other operations may be performed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may perform a single operation or two or more operations.

[0135] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code generated by a compiler. In another example, the instructions or software include higher-level code that is executed by the one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification (which disclose algorithms for performing the operations performed by the hardware components and methods described above).

[0136] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the methods as described above, as well as any related data, data files, and data structures, can be recorded, stored, or fixed in one or more non-transitory computer-readable storage media or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random- access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, a hard disk drive (HDD), a solid-state drive (SSD), a card-type memory such as a multimedia card or a micro card (e.g., a secure digital (SD) or extreme digital (XD)), a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured in a non-transitory manner to store instructions or software and any related data, data files, and data structures and to provide the instructions or software and any related data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any related data, data files, and data structures are distributed over a networked computer system so that the one or more processors or computers store, access, and execute the instructions and software and any related data, data files, and data structures in a distributed manner.

[0137] While the present disclosure includes specific examples, it will be apparent to those skilled in the art after understanding the disclosure provided herein that various changes in form and details can be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects in each example should be considered as being applicable to similar features or aspects in other examples. Suitable results can be achieved if the described techniques are performed in a different order, and / or if the components of the described systems, architectures, devices, or circuits are combined in a different manner, and / or if the described systems, architectures, devices, or circuits are replaced with other components or their equivalents. Thus, the scope of the disclosure should not be limited by the specific examples described herein, but should be given the broadest interpretation of the claims and their equivalents, along with all modifications that can fall within the interpretations.

Claims

1. A processor-implemented method of displaying a three-dimensional object, the method comprising: acquiring a travel image including a road surface from a single camera; classifying line segment markers of the road surface into one or more groups by comparing a degree of change in curvature of the line segments of the road surface to a reference change, and based on a result of the comparison, classifying the line segment markers into a plurality of groups based on points at which the degree of change in curvature is greater than the reference change; for each of the one or more groups, estimating a pitch angle of the road surface corresponding to an inclination angle between the road surface and the single camera; for each of the one or more groups, generating three-dimensional (3D) information of the road surface using the pitch angle of the road surface; and based on the 3D information, displaying the 3D object visually superimposed on the road surface.

2. The method of claim 1, wherein, Each of the one or more groups forms a corresponding plane on the road surface that is different from the others.

3. The method of claim 1, wherein, Estimating the pitch angle of the road surface for each of the one or more groups includes: converting a partial image of the travel image for each of the one or more groups into a bird’s eye view (BEV) image for each of the one or more groups; and based on the BEV image for each of the one or more groups, estimating the pitch angle of the road surface for each of the one or more groups.

4. The method of claim 3, wherein, Estimating the pitch angle of the road surface for each of the one or more groups includes: estimating a pitch angle that satisfies either or both of a first condition that line segment markers included in the BEV image for each of the one or more groups are parallel to each other and a second condition that the line segment markers have a same length as the pitch angle of the road surface.

5. The method of claim 4, wherein, Estimating the pitch angle of the road surface for each of the one or more groups includes: calculating a position of the line segment markers included in the BEV image for each of the one or more groups based on an initial pitch angle; repeatedly adjusting the pitch angle until either or both of the first condition and the second condition are satisfied; and determining a pitch angle that satisfies either or both of the first condition and the second condition as the pitch angle of the road surface.

6. The method of claim 1, wherein, the travel image includes a sequence of images over time, and generating 3D information of the road surface includes: for each of the one or more groups, smoothing the pitch angle of the road surface over time; and generating 3D information of the road surface included in the travel image using the smoothed pitch angle.

7. The method of claim 6, wherein, Smoothing the pitch angle of the road surface includes smoothing the pitch angle of the road surface over time for each of the one or more groups using a temporal filter.

8. The method of claim 1, wherein, Displaying the 3D object includes: rendering a 3D road model reflecting the 3D information of the road surface; and displaying the 3D object visually superimposed on the 3D road model.

9. The method of claim 1, further comprising: extracting, from the travel image, a line segment of the road surface corresponding to a travel lane; and calculating a curvature of the extracted line segment.

10. The method of claim 9, wherein, The extracting the line segment of the road surface includes: detecting a line segment included in the travel image and having a length greater than or equal to a predetermined length by searching for an edge in the travel image; and estimating a line segment of the road surface corresponding to the travel lane by grouping the detected line segments based on a travel direction. 11.The method of claim 9, further comprising: extracting a left line and a right line of the travel lane from the travel image; and calculating a curvature of the left line and the right line of the travel lane. 12.The method of claim 1, further comprising: outputting 3D information of the road surface.

13. The method of claim 1, wherein, The travel image includes any one or both of an RGB image and a grayscale image. 14.A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method of claim 1. 15.An apparatus for displaying a three-dimensional object, the apparatus comprising: a communication interface configured to acquire, from a single camera, a travel image including a road surface; a processor configured to compare a degree of change in curvature of a line segment of the road surface with a reference change, classify line segment markers of the line segment of the road surface into one or more groups based on a result of the comparison based on points at which the degree of change in curvature is greater than the reference change, estimate, for each of the one or more groups, a pitch angle of the road surface corresponding to an inclination angle between the road surface and the single camera, generate, for each of the one or more groups, three-dimensional (3D) information of the road surface using the pitch angle of the road surface, and represent a 3D object visually superimposed on the road surface based on the 3D information; and a display configured to display a 3D road model representing the 3D object. Each of the one or more groups forms a corresponding plane different from each other on the road surface.

16. The apparatus of claim 15, wherein, The processor is further configured to convert a partial image of the travel image for each of the one or more groups into a bird's eye view (BEV) image for each of the one or more groups, and estimate the pitch angle of the road surface based on the BEV image for each of the one or more groups.

17. The apparatus of claim 15, wherein, The processor is further configured to estimate a pitch angle satisfying any one or both of a first condition and a second condition as the pitch angle of the road surface, the first condition being that line segment markers included in the BEV image for each of the one or more groups are parallel to each other, and the second condition being that the line segment markers have the same length.

18. The apparatus of claim 17, wherein, ​ 19. The apparatus of claim 18, wherein, The processor is further configured to calculate a position of a line segment marker included in the BEV image for each of the one or more groups based on an initial pitch angle, repeatedly adjust the pitch angle until either or both of the first condition and the second condition are satisfied, and determine a pitch angle at which either or both of the first condition and the second condition are satisfied as the pitch angle of the road surface.

20. The apparatus according to claim 15, wherein, the travel image includes a sequence of images over time, and the processor is further configured to smooth the pitch angle of the road surface over time for each of the one or more groups and generate 3D information of the road surface included in the travel image using the smoothed pitch angle.

21. The apparatus of claim 20, wherein, the processor is further configured to smooth the pitch angle of the road surface over time for each of the one or more groups using a temporal filter.

22. The apparatus of claim 15, wherein, the processor is further configured to render a 3D road model reflecting the 3D information of the road surface and represent the 3D object on the 3D road model.

23. The apparatus of claim 15, wherein, the processor is further configured to extract a line segment of the road surface corresponding to a travel lane from the travel image and calculate a curvature of the extracted line segment.

24. The apparatus of claim 23, wherein, the processor is further configured to detect a line segment having a length greater than or equal to a predetermined length included in the travel image by searching for an edge from the travel image and estimate a line segment of the road surface corresponding to the travel lane by grouping the detected line segments based on a travel direction.

25. The apparatus of claim 23, wherein, the processor is further configured to extract a left line and a right line of the travel lane from the travel image and calculate a curvature of the left line and the right line of the travel lane.

26. The apparatus of claim 15, wherein, the communication interface is configured to output the 3D information of the road surface.

27. An apparatus for displaying a three-dimensional object, comprising: a single camera sensor configured to acquire a travel image including a road surface; and a processor configured to: classify line segment markers of the road surface into one or more groups by comparing a degree of change in curvature of the line segment of the road surface to a reference change and, based on a result of the comparison, based on points at which the degree of change in curvature is greater than the reference change, classify the line segment markers into a plurality of groups, estimate a pitch angle of the road surface corresponding to an inclination angle between the road surface and the single camera sensor for each of the one or more groups, and generate 3D information of the road surface using the pitch angle for each of the one or more groups.

28. The apparatus of claim 27, further comprising: a display configured to output a 3D object visually superimposed on the road surface based on the 3D information.

Images