Content visualization method and device

Abstract

A content visualization method and device are provided. The content visualization device generates an object arrangement space based on a road surface and obstacles, and determines a display position of a graphic representation corresponding to route guidance content in the object arrangement space.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2018-0122382 filed on October 15, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0003] The following description relates to techniques for visualizing content. Background Art

[0004] In order to assist driving, augmented reality (AR) representing various visual information is provided through a display or navigation system installed on a vehicle or other transportation means. For example, efforts have been made to display driving information through an AR-based head-up display (HUD). Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a content visualization method is provided, comprising: generating a three-dimensional (3D) object layout space based on a road area ahead; determining a display position of route guidance content based on an indication point of the route guidance content and the 3D object layout space; and three-dimensionally visualizing a graphic object corresponding to the route guidance content at the determined display position.

[0007] The indication point may include a point at which an operation corresponding to the route guidance content is performed.

[0008] The generating may include generating a 3D object arrangement space based on a field of view (FOV) of a user and a distance from a viewpoint of the user to an obstacle.

[0009] The three-dimensionally visualizing may include generating a first graphical representation corresponding to a left eye of the user and a second graphical representation corresponding to a right eye of the user, and providing a parallax between the first graphical representation and the second graphical representation.

[0010] The providing may include tracking the position of a left eye and a right eye of a user.

[0011] The determining may include, in response to the indication point being outside the 3D object arrangement space, determining a position in the 3D object arrangement space as the display position.

[0012] The determining may include: in response to the height of the indication point being lower than the bottom of the 3D object arrangement space, adapting the height of the display position to the bottom of the 3D object arrangement space.

[0013] The determining may include adapting a height of the display position to a bottom of the 3D object arrangement space in response to the road surface and the space corresponding to the user's line of sight not intersecting.

[0014] The determining may include updating the display position to be located within the 3D object arrangement space in response to a distance from the user to the display position being greater than a maximum distance of the 3D object arrangement space.

[0015] The generating may include generating a 3D object arrangement space based on a field of view (FOV) of a user and a body of a vehicle in which the content visualization device is installed.

[0016] The three-dimensionally visualizing may include determining a content movement trajectory based on a field of view (FOV) of a user and a body of a vehicle, and visualizing the graphic object by adjusting a height of the graphic object based on the content movement trajectory in response to a content visualization device approaching an indication point.

[0017] The bottom of the 3D object arrangement space may be adapted to the surface of the road area estimated in the vicinity of the vehicle.

[0018] The generating of the 3D object arrangement space may include adapting a bottom of the 3D object arrangement space to a road surface up to a maximum depth of the 3D object arrangement space.

[0019] The three-dimensionally visualizing may include visualizing a shadow object in an area below the graphical object in a 3D object arrangement space.

[0020] The visualization of the shadow object may include maintaining brightness of the shadow area relative to an area surrounding the shadow area below the graphic object, and visualizing the graphic object with brightness higher than brightness of the shadow area.

[0021] The content visualization method may include determining a subsequent display position of subsequent route guidance content in response to the content visualization apparatus reaching a display point of the route guidance content, and visualizing another graphic object corresponding to the subsequent route guidance content at the subsequent display position.

[0022] The three-dimensionally visualizing may include determining a transparency of the graphic object based on a distance from a maximum distance of the 3D object arrangement space to the graphic object.

[0023] The route guidance content may include a plurality of point objects arranged along a line, wherein the plurality of point objects may be spaced apart from each other at unit intervals.

[0024] The determining may include: in response to the indication point being different from the display position, obtaining temporary guide content corresponding to the display position, wherein the three-dimensionally visualizing may include: visualizing a graphic object corresponding to the temporary guide content at the display position until the 3D object arrangement space includes the indication point.

[0025] The content visualization method may include visualizing a graphic object corresponding to information about a lane in which a content visualization apparatus may be traveling, and a graphic object corresponding to a dangerous object on a road in which the content visualization apparatus may be located.

[0026] The three-dimensionally visualizing may include visualizing the graphic object while gradually moving the graphic object from a previous display position to a new display position in response to a display position being changed while the graphic object can be visualized.

[0027] In another general aspect, a content visualization device is provided, comprising: a processor configured to generate a three-dimensional (3D) object layout space based on a road area ahead, and determine a display position based on an indication point of route guidance content and the 3D object layout space; and a display configured to three-dimensionally visualize a graphic object corresponding to the route guidance content at the determined display position.

[0028] In another general aspect, a content visualization apparatus is provided, comprising: a sensor disposed on a device and configured to sense information related to a state of the device and a distance from obstacles near the device; a processor configured to generate a three-dimensional (3D) object layout space based on a front road area, and determine a display position of route guidance content based on an indication point of the route guidance content and the 3D object layout space; and a head-up display (HUD) configured to visualize a 3D graphic object corresponding to the route guidance content at the determined display position.

[0029] The content visualization apparatus may include a memory configured to store route guidance content, information related to a state of a device, a map database, and instructions that, when executed, configure a processor to determine any one or any combination of a 3D object arrangement space and a display position.

[0030] The processor may be configured to determine the indication point based on road information in a map database.

[0031] The device may include a vehicle, and the information related to the status of the device may include any one or any combination of the following: location information of the vehicle, current geographic coordinates of the vehicle, road information corresponding to the location of the vehicle, a lane in which the vehicle may be traveling, a width of a lane in which the vehicle may be traveling, a width of a road on which the vehicle may be traveling, the number of lanes of the road, a center line of a lane, a turning point on the road, a traffic signal on the road, a speed of the vehicle, fuel of the vehicle, a fuel level of the vehicle, and maintenance information of the vehicle.

[0032] The sensor can be configured to track the position of the user's left eye and right eye in response to the vehicle being in motion, and the processor can be configured to generate a first graphical representation corresponding to the position of the user's left eye, generate a second graphical representation corresponding to the position of the user's right eye, and present a 3D graphical object on the HUD based on a parallax between the first graphical representation and the second graphical representation.

[0033] The object arrangement space may include a rectangular cone-shaped area extending from the user's eyes toward a road area in front of the device.

[0034] In another general aspect, a content visualization method is provided, comprising: generating a three-dimensional (3D) object layout space based on a user's field of view (FOV) and a distance from obstacles near the user; determining a display position of route guidance content based on a point at which an operation corresponding to the route guidance content is to be performed and the 3D object layout space; and visualizing a 3D graphic object corresponding to the route guidance content at the determined display position.

[0035] The determining of the display position may include: in response to the point being located outside the object arrangement space, determining a position corresponding to the point in the object arrangement space as the display position.

[0036] The position corresponding to the point may be located substantially close to a limiting surface of the object arrangement space and on an axis from a center of an eye box of the user toward the point.

[0037] The content visualization method may include: in response to the 3D object arrangement space intersecting with a road surface, limiting the 3D object arrangement space.

[0038] The visualization of the 3D graphic object may include: visualizing a shadow object below the 3D graphic object in the object arrangement space, and increasing brightness of a portion of the object arrangement space that may be arranged around the 3D graphic object.

[0039] The content visualization method may include increasing transparency of the route guide content adjacent to a boundary of the 3D object arrangement space.

[0040] Other features and aspects will become apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 An example of visualization content is shown.

[0042] Figure 2 An example of a configuration of a content visualization apparatus is shown.

[0043] Figure 3 An example of a configuration of a head-up display (HUD) of a content visualization device is shown.

[0044] Figure 4 and Figure 5 An example of a content visualization method is shown.

[0045] Figure 6 An example of obtaining route guidance content is shown.

[0046] Figure 7 An example of determining a display position and visualizing a graphical object is shown.

[0047] Figures 8 to 10 An example of a graphic object having a display position determined to be in the object arrangement space is shown.

[0048] Fig.11 and Fig.12 An example of determining the display position of a graphic object corresponding to route guidance content when the indication point is blocked is shown.

[0049] Fig.13 and Fig.14 An example of visualizing route guidance content based on a content movement trajectory is shown.

[0050] Fig.15 An example of determining the display position of the route guidance content when the content visualization device enters and exits an uphill road is shown.

[0051] Fig.16 An example of a content visualization method is shown.

[0052] Fig.17 An example is shown in which the content visualization device changes the position of the route guidance content and deforms the graphic object in response to the sight line being ensured.

[0053] Fig.18 An example of a graphic object corresponding to a path indication line is shown.

[0054] Fig.19 An example of visualizing lane information is shown.

[0055] Fig. 20 An example of dynamic visualization of route guidance content is shown.

[0056] Fig.21 An example of a static visualization of route guidance content is shown.

[0057] Fig. 22 An example of warning of a dangerous object is shown.

[0058] Fig.23 An example of providing speed limit information is shown.

[0059] Fig.24 An example of visualizing a shadow object corresponding to route guidance content is shown.

[0060] Throughout the drawings and detailed description, unless otherwise described or specified, the same reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0061] The following detailed description is provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent. For example, the order of operations described herein is merely an example and is not limited to those order of operations set forth herein, but may be significantly changed after understanding the disclosure of the present application, except for operations that must be performed in a certain order. In addition, for greater clarity and brevity, descriptions of features well known in the art may be omitted.

[0062] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be apparent after understanding the disclosure of the present application.

[0063] The terms used herein are for the purpose of describing specific examples only, not for limiting the examples. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms "include", "have", and / or "comprise" when used herein indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0064] When a component is connected to another component, it includes not only the situation of directly connecting the component, but also the situation of connecting the component through another component between them. In addition, when a component includes a constituent element, unless otherwise specifically stated, other elements may also be included in the component without excluding other elements. Although various members, components, regions, layers or parts may be described herein using terms such as "first", "second", "third", "A", "B", (a) and (b), these members, components, regions, layers or parts are not limited by these terms. On the contrary, these terms are only used to distinguish a member, component, region, layer or part from another member, component, region, layer or part. Therefore, without departing from the teaching of the example, the first member, component, region, layer or part mentioned in the example described herein may also be referred to as the second member, component, region, layer or part.

[0065] If the specification states that one component is “connected,” “coupled,” or “engaged” to a second component, the first component may be directly “connected,” “coupled,” or “engaged” to the second component, or a third component may be “connected,” “coupled,” or “engaged” between the first and second components. However, if the specification states that the first component is “directly connected” or “directly engaged” to the second component, the third component may not be “connected” or “engaged” between the first and second components. Similar expressions such as “between” and “directly between” and “adjacent to” and “immediately adjacent to” should also be interpreted in this manner.

[0066] In this document, use of the term "may" with respect to examples or embodiments (e.g., with respect to what an example or embodiment may include or implement) means that there is at least one example or embodiment in which such features are included or implemented, and all examples and embodiments are not limited thereto.

[0067] When describing an example with reference to the accompanying drawings, the same reference numerals represent the same constituent elements, and repeated descriptions related thereto will be omitted. In the description of the example, when it is considered that a detailed description of a well-known related structure or function will lead to an obscure interpretation of the present disclosure, such description will be omitted.

[0068] Figure 1 An example of visualization content is shown.

[0069] The content visualization device visualizes the graphic objects corresponding to the route guidance content 121. The content visualization device arranges the graphic objects at the positions in the object arrangement space 110 and visualizes them. In the example, the content visualization device provides the graphic objects stereoscopically by presenting the left image and the right image to the left eye and the right eye of the user respectively. In the example, the left image and the right image include graphic objects spaced a certain distance from each other, and the distance corresponds to the parallax of the depth along the horizontal axis. Therefore, the user recognizes the depth of the stereoscopically rendered graphic objects.

[0070] In an example, the object arrangement space 110 is a three-dimensional (3D) space in which the graphic objects are to be arranged. In an example, the object arrangement space 110 is a space in which the graphic objects having depth are stereoscopically visualized. The coordinates of each point belonging to the object arrangement space 110 are mapped at a scale that is the same as or similar to the geographic coordinates of the physical world. In an example, the boundary of the object arrangement space 110 is determined based on the structure of a head-up display (HUD). The content visualization device visualizes the graphic objects in a space between a minimum depth (e.g., a minimum distance) and a maximum depth (e.g., a maximum distance), wherein the minimum depth and the maximum depth are determined based on the structure of the HUD. Figure 1 In the example of , the minimum distance is 5 meters (m) and the maximum distance is 70 m. However, the examples are not limited thereto, and other distances may be used without departing from the spirit and scope of the described illustrative examples.

[0071] In the example, the object arrangement space 110 is determined based on a space corresponding to the user's line of sight. The shape and size of the object arrangement space 110 are determined based on the field of view (FOV) provided by the HUD and the size of the eye box 190. The object arrangement space 110 is a space arranged in a rectangular cone shape extending from the eye box 190 to correspond to the FOV. The eye box 190 is an area arranged so that both eyes of the user can be positioned in the eye box 190.

[0072] In an example, the content visualization device is implemented to visualize various types of graphic representations at display locations in the object arrangement space 110 above the windshield 180 , and to visualize different types of graphic representations in different object arrangement spaces 110 .

[0073] For example, the types of graphical representations to be displayed in the object arrangement space 110 include route guidance content 121, path indication lines 122, warning content 123, and driving-related content 124. The types of graphical representations to be displayed in the object arrangement space 110 may be modified, and other types of graphical representations, such as roadblocks, road conditions, may be used without departing from the spirit and scope of the illustrative examples described.

[0074] The user recognizes the corresponding graphical representation as existing at the physical position corresponding to the display position in the object arrangement space 110. As described above, in the example, each coordinate in the object arrangement space 110 is mapped to a single physical coordinate.

[0075] Herein, the display position refers to a point where a graphical object is to be displayed, for example, a point in a 3D space. The display position has an absolute 3D coordinate or a 3D coordinate relative to a vehicle. The display position is fixed or moved according to the operation of the content visualization device.

[0076] Vehicles as described herein refer to any means of transportation, delivery, or transmission, such as a car, truck, tractor, scooter, motorcycle, bicycle, amphibious vehicle, snowmobile, boat, public transportation vehicle, bus, monorail, train, tram, autonomous or self-driving vehicle, smart vehicle, self-driving vehicle, unmanned aerial vehicle, electric vehicle (EV), hybrid vehicle, smart mobile device, or drone. In an example, smart mobile device includes mobile devices such as electric wheels, electric skateboards, and electric bicycles. In an example, vehicles include motor vehicles and non-motor vehicles, such as vehicles with a power engine (e.g., a cultivator or motorcycle), bicycles, or carts.

[0077] In addition to the vehicle described herein, the methods and apparatus described herein may also be included in various other devices, such as smart phones, walking assistance devices, wearable devices, safety devices, robots, mobile terminals, and various Internet of Things (IoT) devices.

[0078] The term "road" is an improved passage, route or connection between two locations that allows travel by foot or some form of transport (e.g., vehicle). Road refers to a path on which vehicles travel, and includes various types of roads, such as highways, national roads, local roads, expressways, farm roads, district roads, high-speed national roads, and expressways.

[0079] The term "lane" refers to a road space distinguished by lane boundaries marked on the road surface. The term "current driving lane" refers to the lane in which the vehicle is currently traveling, and indicates the lane space currently occupied and used by the vehicle. The current driving lane is also called the "ego lane". The term "lane boundary" refers to a solid or dotted line marked on the road surface to distinguish lanes. In the present disclosure, the term "lane boundary" can be used interchangeably with the term "lane marking".

[0080] These methods and apparatus are used for navigation equipment (e.g., augmented reality head-up display (AR 3D HUD)) of vehicles and road guidance information in autonomous vehicles. In an example, the methods and apparatus described herein can be used to generate information to support drivers or control autonomous vehicles. In an example, the examples described herein can also be used to interpret visual information for intelligent systems installed in vehicles for fully autonomous driving or driving assistance, and to assist safe and comfortable driving. The examples described herein are applicable to vehicles and vehicle management systems, such as autonomous vehicles, automatic or autonomous driving systems, smart vehicles, advanced driver assistance systems (ADAS), navigation systems that assist vehicles to safely remain in the lane in which the vehicle is traveling, smart phones or mobile devices. Examples related to displaying road guidance information for vehicles are provided only as examples, and other examples are considered to be fully within the scope of the present disclosure, such as training, games, medical applications, public safety, tourism and marketing.

[0081] Route guidance information is information that guides the user to travel based on the route. In the example, the route is set by the user. Route guidance information includes the distance the user needs to go straight and the turns to be made at the intersection. The route is the path that the user needs to travel from the departure point to the destination. Route guidance content 121 is content in which the route guidance information to be provided for traveling along the route to the destination is visualized, and includes, for example, numbers and letters indicating the distance the user needs to go straight, and arrows indicating turns to be made at the intersection (e.g., left turn and right turn). The path indication line 122 is a line indicating the path to be traveled to the destination, and is visualized as route guidance information in a form different from that of the route guidance content 121. Warning content 123 includes warning messages that will be provided to the user as needed in the current driving environment.

[0082] The driving-related content 124 is content related to driving. For example, the driving-related content 124 is information related to the speed, acceleration, position, fuel, and maintenance of the vehicle. The fuel-related information represents the remaining gasoline amount of a gasoline vehicle, the remaining diesel amount of a diesel vehicle, the remaining hydrogen amount of a hydrogen vehicle, or the charge amount of an electric vehicle. The maintenance-related information represents information that will be used to determine whether the vehicle needs maintenance, such as the condition of the engine oil, the condition of the transmission oil, the accumulated driving distance, and the tire pressure. However, the examples are not limited thereto, and other types and forms of driving-related content are considered to be fully within the scope of the present disclosure.

[0083] The content visualization device visualizes the route guidance content 121 and the path indication line 122 at a position spaced apart from the user. For reference, when the content visualization device is mounted on a vehicle, the user is the driver of the vehicle.

[0084] A driver driving a vehicle equipped with a content visualization device may mainly focus on a location far from the vehicle on the road 170. The content visualization device visualizes content such as route guidance content 121 or path indication line 122 at a location far from the vehicle, thereby reducing visual fatigue of the user. That is, the content visualization device reduces interference caused by depth confusion of the user by visualizing a graphic object indicating a route at a distance corresponding to a depth similar to the depth at which the user is focused.

[0085] The content visualization device estimates the position of the vehicle using sensors such as camera sensors, global positioning systems (GPS), and light detection and ranging (LIDAR). Taking into account the error distance from the position of the vehicle to the position of the driver's eyes, the content visualization device visualizes the graphic objects corresponding to the route guidance content 121 as matching the real road. The device for visualizing the content can visualize the graphic objects corresponding to the route guidance content 121 on the actual road by considering the error interval as much as the driver's eye position from the position of the vehicle. When using high-definition (HD) map data, the content visualization device matches the route guidance content 121 with a more accurate position. Through this matching, the content visualization device improves the driver's psychological stability.

[0086] Driving-related content 124 associated with general information to be provided to the driver in a normal driving environment is visualized at a first position (e.g., a relatively close position) spaced a certain distance from the user. When it is determined that a warning needs to be provided to the driver, the content visualization device provides a warning to the user by visualizing a portion or all of the driving-related content 124 at a position farther than the first position. The vehicle speed is visualized at a position of about 50 m ahead in the form of warning content 123. When the situation requiring a warning is resolved, the content visualization device moves a portion or all of the driving-related content 124 back to the first position.

[0087] Unlike the route guidance content 121 that the user frequently looks at while driving, the user looks at the driving-related content 124 only when the user needs to look at it. Since the user looks at the driving-related content 124 at a relatively low frequency, the probability that the driving-related content 124 causes interference is relatively low even if the driving-related content 124 is visualized at a relatively close position. In order to ensure the driving view, the driving-related content 124 is visualized at a corner (e.g., the lower end) of the object arrangement space 110. Figure 1 In the example of , the relatively close position is within the range of 5 m to 17 m, and the relatively far position is within the range of 17 m to 70 m. However, the example is not limited thereto.

[0088] For reference, Figure 1As shown, the road 170 includes at least one lane. The lane line 150 is a boundary line that separates one lane from another lane. However, the example is not limited to the case where the lane line 150 exists on the road 170 or the lane. The center line 160 refers to a boundary line indicating that vehicles are prohibited from crossing the line.

[0089] Figure 2 An example of a configuration of a content visualization device is shown. Figure 2 , the content visualization apparatus 200 includes a sensor 210 , a display 220 , a processor 230 , and a memory 240 .

[0090] The sensor 210 senses information for visualizing the content. The sensor 210 measures the distance to an object located near the user, and includes sensors such as LIDAR and radio detection and ranging (RADAR). In the example, the sensor 210 senses information related to the state of the device on which the content visualization device 200 is installed. When the content visualization device 200 is installed on a vehicle, the sensor 210 senses information related to the vehicle, for example, the location information of the vehicle, the road information corresponding to the location of the vehicle, and the driving related information of the vehicle. In the example, the driving related information is information such as information related to the driving of the vehicle, and includes, for example, information related to the speed, acceleration, location, fuel, and maintenance of the vehicle. The sensor 210 includes an internal sensor that captures the interior of the vehicle. The internal sensor includes a camera sensor, such as a color camera, an infrared sensor, a depth sensor, and a thermal imaging camera. However, the example is not limited thereto, and other sensors 210 may be used without departing from the spirit and scope of the described illustrative examples.

[0091] In the example, the position information of the vehicle indicates the current coordinates of the vehicle and the lane in which the vehicle is currently traveling. In the example, the sensor 210 obtains the two-dimensional (2D) coordinates of the vehicle through GPS. In the example, the sensor 210 obtains a front view image of the front view of the vehicle, and the processor 230 determines the lane in which the vehicle is currently traveling among multiple lanes of the road, for example, the self lane, from the front view image. However, the example is not limited to this. The processor 230 estimates the current position of the vehicle based on the image collected by the sensor 210.

[0092] The term "processor" as used herein is a hardware-implemented data processing device having circuits that are physically constructed to perform desired operations. For example, the desired operations include codes or instructions included in a program. Hardware-implemented data processing devices include, but are not limited to, for example, microprocessors, central processing units (CPUs), processor cores, multi-core processors, multiprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). Processor 230 executes programs and controls content visualization apparatus 200. The code of the program executed by processor 230 is stored in memory 240. A further description of processor 230 is provided below.

[0093] The road information includes any one or any combination of the following: the width of the road, the number of lanes of the road, the width of each lane, the center line, turning points, traffic signals, and other traffic related information.

[0094] The display 220 visualizes content and provides content to the user. In an example, the display 220 is a physical structure including one or more hardware components that provide the ability to present a user interface and / or receive user input. In an example, a head-up display (HUD) device is used to display the content on the windshield or a separate screen of the vehicle, or the content is displayed on an augmented reality head-up display (AR HUD). In an example, the HUD forms a projection surface in front of the user, and provides content to the user through the projection surface. The display 220 provides a left image to the user's left eye and a right image to the user's right eye. The processor 230 generates a left image and a right image, the left image including a first graphic representation corresponding to the driver's left eye, and the right image including a second graphic representation corresponding to the driver's right eye. In an example, the display 220 provides a left image and a right image to form a parallax between them. The display 220 visualizes content with depth as a stereoscopic graphic object, and provides content to the user by spacing the content visualization graphic object of the left image and the content visualization graphic object of the right image from each other based on binocular parallax. The display 220 visualizes a graphic object corresponding to the route guidance content at the display position determined by the processor 230. In the example, the position of the user's eyes is measured by the sensor 210 (e.g., an internal sensor) and provided to the processor 230. The position of the user's eyes is always tracked while the vehicle is traveling, so that even when the driver moves his head up, down, right, and left or adjusts the seat height, the content can be delivered stereoscopically.

[0095] The processor 230 generates or creates content to be visualized through the display 220, deforms and adjusts the content to be visualized through the display 220. The processor 230 generates an object arrangement space based on the road area estimated in front of the vehicle. The processor 230 determines the display position based on the indication point of the route guidance content and the object arrangement space. The processor 230 fixes the display position or dynamically adjusts the display position.

[0096] The indication point of the route guidance content is a point at which the user needs to perform an operation corresponding to the route guidance content. When the route guidance content is content indicating a turn (e.g., left turn) at an intersection, the indication point is an entrance point of the intersection at which the user needs to turn.

[0097] In an example, the processor 230 determines the indication point based on the road information included in the map database. The processor 230 determines the turning area to be turned based on the route guidance information provided to the vehicle, and determines the entry point of the turning area as the indication point. The turning area is an area including a turning point.

[0098] However, the operation of the processor 230 is not limited to the above example. The processor 230 may also perform the above operation together with the operation described below. Figures 3 to 24 Any one or any combination of the operations described. Figures 3 to 24 The operation of processor 230 is further described.

[0099] The memory 240 temporarily or permanently stores information for visualizing the content. The memory 240 stores instructions to be executed by the processor 230 to perform reference Figures 3 to 24 The operation described. The memory 240 stores route guidance information, driving related content, and a map database. The map database is a database that stores map data. The map database stores an HD map database. For example, the HD map database includes information related to fine data, such as the number of lanes, the width of each lane, and the position of the center line. A further description of the memory 240 is provided below.

[0100] In an example, the content visualization device 200 is implemented as a 3D HUD for a vehicle, or a navigation system that provides a route for the vehicle. In an example, the content visualization device 200 sends the content to the electronic control unit (ECU) or vehicle control unit (VCU) of the vehicle. The ECU or VCU displays the content on the display 220 of the vehicle. However, the display of content is not limited to the above examples, and any other instrument cluster, vehicle infotainment system, screen in the vehicle, or display panel in the vehicle can perform the display function. Without departing from the spirit and scope of the described illustrative examples, other displays that are operably connected to the content visualization device 200, such as smart phones and eyeglass displays (EGDs), can be used.

[0101] The content visualization device 200 is implemented to provide augmented reality (AR) to the user. The content visualization device 200 displays the content to a depth within a range beyond the hood of the vehicle (for example, 5m to 70m from the vehicle). The content visualization device 200 reduces the user's visual fatigue by visualizing the content to a depth similar to the position where the user is currently looking at, and by visualizing the content to a more accurate depth (which will be referred to below). Figures 3 to 24 This is described) to help users focus on driving.

[0102] Figure 3 An example of a configuration of a content visualization system including a HUD is shown.

[0103] The content visualization system 300 provides content 361 to a user 390 , and is, for example, a device in which a content visualization apparatus 310 is installed.

[0104] Content visualization device 310 includes a sensor 311 , a processor 312 , and a HUD 313 .

[0105] In the example, the sensor 311 detects an object in front of the content visualization device 310. The sensor 311 measures the distance to the front object. However, the example is not limited to this. The sensor 311 measures the distance to the object located near the vehicle and generates a nearby distance map representing the distance to the object located near the vehicle. The sensor 311 generates an image by capturing the environment in front, behind, on the left hand side, and on the right hand side of the vehicle.

[0106] The processor 312 generates content 361 to be provided to the user 390 based on the detected object. The content 361 is data for providing information to the user. Information associated with driving (hereinafter referred to as driving information) is information required for the user's driving, such as route guidance information and driving related information. The processor 312 models the object, detects the position of the object, or recognizes the object by analyzing nearby information sensed by the sensor 311 (e.g., the distance to the surrounding objects and the image including the object).

[0107] The HUD 313 visualizes the content 361 in a visible area located in front of the user 390. The HUD 313 visualizes the content 361 on a window (e.g., a windshield of a vehicle) arranged in front of the user 390. The HUD 313 forms a virtual projection surface 350. The projection surface 350 is a plane on which a virtual image including the content 361 generated by the HUD 313 is displayed. The user 390 recognizes the virtual image as being displayed on the projection surface 350.

[0108] The HUD 313 visualizes content 361 having depth on the projection surface 350. The processor 312 determines the depth to which the content 361 will be visualized based on the projection surface 350. In the example, the HUD 313 visualizes the content 361 as having a depth relatively far from or close to the projection surface 350 or the user 390 based on the determined depth. The HUD 313 visualizes the content 361 having the corresponding depth in the virtual area 360 on the projection surface 350. In the example, the processor 312 renders the content 361 as a 3D graphic representation based on the optical system of the HUD 313. The 3D graphic representation embodies a stereoscopic graphic representation having depth, and will be referred to as a graphic object in the following example. The HUD 313 forms a projection surface 350 including a left image and a right image based on the depth of the content 361, and provides the left image to the left eye of the user 390 through the projection surface 350, and provides the right image to the right eye of the user 390. Therefore, the user 390 recognizes the depth of the stereoscopic rendered content 361.

[0109] For example, the HUD 313 includes an image generator 314, a fold mirror 315, and a concave mirror 316. However, the configuration of the HUD 313 is not limited thereto, and may include various elements designed to form a projection surface 350 on which a virtual image is focused by projecting toward a window disposed in front of the user 390.

[0110] Although an example in which the content visualization device 310 is mounted on a vehicle is described herein, the example is not limited thereto. The content visualization device 310 may be applied to a technology that combines information of a real world and information of a virtual world, such as AR glasses or mixed reality (MR).

[0111] In the example, the content visualization device 310 continuously expresses the depth by adjusting the depth of the content 361 without changing the position of the projection surface 350 formed by the HUD 313. Since there is no need to change the position of the projection surface 350, the content visualization device 310 does not need to physically control the elements included in the HUD 313. When the content visualization device 310 is installed on the vehicle, the content visualization device 310 dynamically visualizes the 3D content 361 in front of the driver.

[0112] The content visualization device 310 continuously expresses the depth through the HUD 313, and visualizes the content 361 having a depth of several hundred meters in front of the user 390. However, when there is an obstacle (for example, a blind spot due to a building or an uphill road) between the user 390 and the content 361 visualized as having depth, an overlap occurs between the 3D graphic representation corresponding to the content 361 visualized as having depth and the obstacle. This overlap causes visual fatigue of the user 390, or causes interference, so that the user 390 recognizes an unnatural image.

[0113] The content visualization device 310 adjusts the content 361 to be more naturally visualized. The operation of the content visualization device 310 will be further described below.

[0114] Figure 4 and Figure 5 An example of a content visualization method is shown.

[0115] Figure 4 is a diagram illustrating a content visualization method. Figure 4 The operations in the embodiment may be performed in the order and manner shown; however, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the described illustrative examples. Figure 4 Many of the operations shown may be performed in parallel or simultaneously. Figure 4 One or more blocks and combinations of these blocks can be implemented by computers based on special hardware that perform the specified functions or a combination of special hardware and computer instructions. Figure 4 In addition to the description of Figures 1 to 3 The description also applies to Figure 4 , and are incorporated herein by reference. Therefore, the above description may not be repeated here.

[0116] refer to Figure 4 In operation 410, the content visualization device generates an object arrangement space based on the estimated road area ahead. The content visualization device estimates the road area ahead based on data (e.g., an image) sensed by a sensor. The content visualization device determines the space corresponding to the road area ahead as the object arrangement space in the space corresponding to the field of view (FOV). The content visualization device generates the object arrangement space based on the user's FOV and the distance from the user's viewpoint to the obstacle. The content visualization device determines the space between the content visualization device and the obstacle as the object arrangement space, and excludes the space outside the obstacle. Therefore, the content visualization device determines only the space in the user's line of sight as the object arrangement space.

[0117] In operation 420, the content visualization device determines the display position of the route guidance content based on the indication point of the route guidance content and the object layout space. When the indication point is in the object layout space, the content visualization device determines the indication point as the display position. In another example, when the indication point is outside the object layout space, the content visualization device determines the position corresponding to the indication point in the object layout space as the display position.

[0118] For example, the point in the object arrangement space corresponding to the indicated point is the point where the visual axis of the user (e.g., the axis from the center of the eye box toward the indicated point) intersects with the limiting surface of the object arrangement space. In another example, the display position is a point in the object arrangement space among the points on the visual axis toward the indicated point. In the example, the display position is a point within a certain margin distance from the limiting surface of the object arrangement space among the points on the visual axis toward the indicated point. The margin distance represents the margin set to prevent the representation of the graphic object from being cut due to the limitation of the object arrangement space. In other words, in the example, in response to the point being outside the object arrangement space, the determination of the display position includes determining the position corresponding to the point in the object arrangement space as the display position. In the example, the position corresponding to the point is positioned substantially close to the limiting plane of the object arrangement space and on the axis from the center of the user's eye box toward the point.

[0119] In operation 430, the content visualization apparatus visualizes the graphic object corresponding to the route guidance content at the determined display position. Therefore, the content visualization apparatus visualizes the graphic object in the object arrangement space, wherein the object arrangement space is a space recognizable by the user.

[0120] For example, the content visualization device visualizes a graphic object corresponding to the route guidance content as being fixed to a display position. In the example, the display position of the graphic object is fixed to a plane coordinate. As the content visualization device approaches the display position, the content visualization device gradually reduces the depth of the graphic object. Therefore, the user recognizes that the content visualization device is close to the graphic object fixed to the display position. However, the example is not limited to this. The content visualization device dynamically adjusts the display position to visualize the graphic object while keeping the relative distance from the content visualization device to the graphic object unchanged. The content visualization device dynamically adjusts the display position to provide an animation effect. Fig.17 , Fig. 20 and Fig.23 The implementation of this visual effect is further described.

[0121] For reference, a position corresponds to an absolute coordinate. In addition, a relative coordinate based on a content visualization device is referred to as a relative position.

[0122] Figure 5 is a diagram illustrating the content visualization method in more detail. Figure 5 The operations in the embodiment may be performed in the order and manner shown; however, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the described illustrative examples. Figure 5 Many of the operations shown may be performed in parallel or simultaneously. Figure 5 One or more blocks and combinations of these blocks can be implemented by computers based on special hardware that perform the specified functions or a combination of special hardware and computer instructions. Figure 5 In addition to the description of Figures 1 to 4 The description also applies to Figure 5 , and are incorporated herein by reference. Therefore, the above description may not be repeated here.

[0123] refer to Figure 5 In operation 501, the content visualization device obtains route guidance content. The content visualization device generates route guidance information based on the route to the destination, and generates route guidance content based on the generated route guidance information. When the distance from the indicated point included in the route guidance information is less than the threshold distance, the content visualization device generates the route guidance content corresponding to the indicated point.

[0124] In operation 510, the content visualization device generates an object arrangement space based on the route guidance content and the estimated road area ahead. In the example, the content visualization device determines whether the road ahead is uphill or downhill according to the route guidance information. When the road is uphill, the content visualization device can limit the object arrangement space based on the intersection of the object arrangement space and the uphill road surface. In addition, the content visualization device limits the object arrangement space based on the road surface detected by the depth sensor or the road surface estimated according to the image data. The content visualization device determines a driving area based on the guidance path indicated by the route guidance information, wherein the content visualization device is predicted to travel in the driving area. When the route guidance information indicates a right turn at the intersection, the driving area includes an area corresponding to the right turn and excludes an area corresponding to the left turn. In the example, the content visualization device determines the space corresponding to the driving area in the road area ahead in the space corresponding to the FOV as the object arrangement space. Since the user mainly looks at the driving area, the content visualization device generates the area where the user's attention is focused as the object arrangement space. However, the example is not limited to this. In another example, the content visualization device generates the object arrangement space without considering the route guidance information.

[0125] In operation 521, the content visualization device determines the display position of the route guidance content. When the indication point of the route guidance content is in the object layout space, the content visualization device determines the corresponding indication point as the display position of the route guidance content. When the indication point is outside the object layout space, the content visualization device determines a position in the object layout space as the display position. In this example, when the distance from the user to the display position is greater than the maximum distance of the object layout space, the content visualization device updates the display position to be within the object layout space. The maximum distance of the object layout space represents the distance corresponding to the maximum depth of the object layout space. In the example, the content visualization device only adjusts (e.g., reduces) the depth of the display position and maintains the remaining coordinates (e.g., height), thereby determining the position in the object layout space as the display position.

[0126] In operation 522, the content visualization device determines a content movement trajectory. In an example, the content visualization device determines the content movement trajectory based on the user's FOV and the body of the vehicle. For example, the content movement trajectory is a trajectory from the road surface through the front edge of the vehicle body to the area corresponding to the user's eyes (e.g., eye box). The content movement trajectory defines the height of the graphic object according to the distance between the display position and the content visualization device. This will be referred to below. Fig.14 The content movement trajectory is further described.

[0127] In operation 530, the content visualization device generates and visualizes a graphic object corresponding to the route guidance content. The content visualization device visualizes the graphic object at a fixed display position, or visualizes the graphic object along a moving display position. In the example, the content visualization device visualizes the graphic object to a height determined based on the content movement trajectory. The content movement trajectory is a trajectory determined to be suitable for the user's FOV. Therefore, even when the display position of the route guidance content is close to the user, the content visualization device stably provides the user with the graphic object.

[0128] In operation 540, the content visualization device determines and visualizes a graphic object corresponding to driving related information. The content visualization device visualizes the graphic object corresponding to the additional information other than the route guidance content. The content visualization device visualizes the graphic object corresponding to the driving related information in a corner space (e.g., a lower end portion of the projection surface) adjacent to the user in the object arrangement space.

[0129] Figure 6 An example of obtaining route guidance content is shown. Figure 6 The operations in the embodiment may be performed in the order and manner shown; however, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the described illustrative examples. Figure 6 Many of the operations shown may be performed in parallel or simultaneously. Figure 6 One or more blocks and combinations of these blocks can be implemented by computers based on special hardware that perform the specified functions or a combination of special hardware and computer instructions. Figure 6 In addition to the description of Figures 1 to 5 The description also applies to Figure 6 , and are incorporated herein by reference. Therefore, the above description may not be repeated here.

[0130] refer to Figure 6 In operation 601, the content visualization apparatus receives a destination from a user. In an example, in response to user input, the content visualization apparatus obtains a destination desired by the user. For example, the destination is a geographic location.

[0131] In operation 602, the content visualization device generates a nearby distance map based on a sensor. The nearby distance map is a map indicating the distance to a target point (e.g., an obstacle) present near the content visualization device. The content visualization device generates the nearby distance map based on a color image, a depth image, and LIDAR data.

[0132] In operation 603, the content visualization apparatus calculates a guidance path to the destination. The content visualization apparatus determines the guidance path from the current position to the destination based on a lane in which the vehicle can travel.

[0133] In operation 604, the content visualization device obtains route guidance content corresponding to the calculated guidance path. The user is provided with directions for traveling along the guidance path calculated in operation 603 at various guidance points (e.g., intersections). The content visualization device obtains route guidance content for a plurality of indication points corresponding to the guidance path.

[0134] Figure 7 An example of determining a display position and visualizing a graphical object is shown. Figure 7 The operations in the embodiment may be performed in the order and manner shown; however, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the described illustrative examples. Figure 7 Many of the operations shown may be performed in parallel or simultaneously. Figure 7 One or more blocks and combinations of these blocks can be implemented by computers based on special hardware that perform the specified functions or a combination of special hardware and computer instructions. Figure 7 In addition to the description of Figures 1 to 6 The description also applies to Figure 7 , and are incorporated herein by reference. Therefore, the above description may not be repeated here.

[0135] refer to Figure 7In operation 721, the content visualization apparatus determines whether the map data is an HD map. As described above, the HD map is map data including precise information (eg, the number of lanes).

[0136] When the map data is an HD map, the content visualization device determines a display position at which the route guidance content is to be visualized. In operation 722, the content visualization device determines whether the indication point is outside the object layout space. When the indication point of the route guidance content is outside the object layout space, in operation 723, the content visualization device adjusts the display position of the route guidance content to be in the object layout space. As described above, the content visualization device determines a point located in the object layout space among the points on the visual axis toward the indication point as the display position. When the indication point of the route guidance content is in the object layout space, the content visualization device determines the indication point as the display position.

[0137] When the map data is not an HD map, in operation 724, the content visualization device determines the center of the projection surface as the display position. When the map data is a standard definition (SD) map, it is difficult to accurately adapt the object layout space in the physical world. Therefore, in the example, the content visualization device randomly determines the display position of the route guidance content. However, the example is not limited to this. In the example, the content visualization device determines the display position of the route guidance content as in the object layout space without considering the map data.

[0138] In operation 731, the content visualization device determines the transparency of the route indication content based on the distance. For example, the route indication content is a path indication line indicating the guidance path from the current position to the destination. For example, the route indication content is a set of points that the center point of the vehicle is predicted to pass through. In a straight section, the route indication content is an indication line passing through the center of the lane. In a turning section (e.g., a right turn section or a left turn section), the route indication content is a curved indication line entering the turning section through the center of the lane and leaving the turning section through the center of the lane.

[0139] The content visualization device assigns a relatively high transparency to a portion of the route indication content that is adjacent to a boundary surface of the object arrangement space with a greater depth. The content visualization device visualizes the end of the route indication content as fading out by gradually increasing the transparency of the route indication content in a direction toward the boundary surface of the object arrangement space with a greater depth. Therefore, the content visualization device naturally combines the route indication content with the real world.

[0140] In operation 732, the content visualization device visualizes the shadow object. The content visualization device displays the shadow object below the graphic object corresponding to the route guidance content, thereby improving the realism of the corresponding graphic object. Fig.24 Visualization of shadow objects is further described.

[0141] Figures 8 to 10 An example of a graphic object having a display position determined to be in the object arrangement space is shown.

[0142] Figure 8 890 is a side view showing an example of a content visualization device mounted on a vehicle 890 .

[0143] The content visualization device determines the display position of the route guidance content in the object layout space 810. The content visualization device generates the object layout space 810 based on the estimated road area near the vehicle 890 so that the bottom of the object layout space 810 is adapted to the road surface 809 of the road area. The content visualization device adapts the bottom of the object layout space 810 to the road surface 809 until the maximum depth of the object layout space 810.

[0144] The content visualization means visualizes a graphic object 820 corresponding to the route guidance content at the bottom of the object arrangement space 810. The object arrangement space 810 is a space in which the user's sight line 811 is ensured, and thus the user gazes at the graphic object 820.

[0145] Fig. 9 is a side view showing the display position of a graphical object relative to a vehicle 990 traveling on an uphill road.

[0146] For example, when the content visualization device enters an uphill road, the indication point of the route guidance content may be located at a position farther from the road surface 909 relative to the vehicle 990. The graphic object 921 visualized at a position farther than the road surface 909 is blocked by the road surface 909, which is located closer to the vehicle 990, and thus causes interference to the user. The graphic object 922 visualized at a position higher than the road surface 909 exceeds the user's FOV while maintaining a horizontal distance, so the user may not be able to observe the corresponding graphic object 922.

[0147] In the example, the content visualization device generates an object arrangement space 910 based on the road information and the line of sight information, so that the object arrangement space 910 is adapted to the road surface 909. When the height of the indication point is lower than the bottom of the object arrangement space 910, the content visualization device adapts the height of the display position to the bottom of the object arrangement space 910. The content visualization device visualizes a graphic object 923 corresponding to the route guidance content at the bottom of the object arrangement space 910. Therefore, even when the indication point is blocked by an obstacle (e.g., an uphill road), the content visualization device determines a point closer to the vehicle 990 than the indication point as the display position, thereby stably providing the route guidance content to the user.

[0148] Fig.10 Shows Fig. 9The object arrangement space is determined from a space 1010 corresponding to the user's line of sight above the vehicle's windshield 1090. The space 1010 corresponding to the line of sight is a quadrangular pyramid-shaped space, such as Fig.10 as shown in .

[0149] For example, the content visualization device determines an intersection surface 1015 where the space 1010 corresponding to the line of sight and the road surface 1009 intersect. For example, the intersection surface 1015 is also called a far clipping surface. The content visualization device determines the intersection surface 1015 as a boundary surface of the object arrangement space. The content visualization device determines that the display position of the route guidance content 1020 is on the boundary surface determined as described above, thereby visualizing the graphic object suitable for the road surface 1009. In the example, the content visualization device visualizes the path indication line 1030 as a bottom surface (e.g., the boundary surface) suitable for the object arrangement space. The content visualization device visualizes the driving related information 1040 in the space adjacent to the projection surface.

[0150] On an uphill road, the boundary surface of the object arrangement space 1010 is limited by the road surface, thereby reducing the distance over which objects can be arranged. Fig.10 In the example of FIG. 1 , the maximum distance of the object arrangement space 1010 can be visualized to be reduced to 40 m.

[0151] Fig.11 and Fig.12 An example of determining the display position of a graphic object corresponding to route guidance content when the indication point is blocked is shown.

[0152] Fig.11 is a bird's-eye view showing a situation where the user's view of the designated point is blocked.

[0153] At the first viewpoint 1101, the content visualization apparatus determines whether the user's gaze reaches the route guidance content 1121 visualized at the indication point. When there is an obstacle 1180 between the vehicle 1190 and the route guidance content 1121 corresponding to the indication point, the content visualization apparatus determines that the user's sight is blocked.

[0154] At the second viewpoint 1102, the content visualization device regenerates the object layout space 1110 based on the obstacle. The content visualization device determines the surface blocked by the obstacle as the boundary surface of the object layout space. In the example, the content visualization device determines the point adjacent to the boundary surface in the object layout space along the visual axis from the vehicle 1190 toward the indicated point as the display position. The content visualization device visualizes the graphic object corresponding to the route guidance content 1122 at the determined display position.

[0155] At the third viewpoint 1103, when the line of sight to the indicated point is ensured, the content visualization means visualizes, at the indicated point, a graphic object corresponding to the route guidance content 1123. The content visualization means pre-arranges the graphic object at a location to be reached by the driver (e.g., a turning point, a detour point, and an intersection entrance point) to notify the user of the corresponding location in advance.

[0156] Fig.12 is a bird's-eye view showing the visualization operation when the indication point is different from the display position.

[0157] When the indication point is different from the display position, the content visualization device obtains the temporary guide content corresponding to the display position. The content visualization device visualizes the graphic object corresponding to the temporary guide content at the display position until the object arrangement space includes the indication point. As the content visualization device moves, the content visualization device changes the display position of the graphic object corresponding to the temporary guide content, thereby maintaining the relative distance from the content visualization device to the graphic object.

[0158] For example, at the first viewpoint 1201, the line of sight from the vehicle 1290 to the indication point cannot be ensured. Since the indication point is different from the display position, the content visualization device generates an object arrangement space limited by obstacles. The content visualization device determines the display position to be in the object arrangement space. Fig.12 In the example of , vehicle 1290 needs to turn right at the intersection. However, since the display position determined at the first viewpoint 1201 is on a straight road, the content visualization device obtains temporary guidance content 1221 indicating straight forward. The content visualization device determines the display position of the temporary guidance content 1221 as a point on the path indication line. However, the example is not limited to this.

[0159] At the second viewpoint 1202 , the line of sight from the vehicle 1290 to the indication point is still not ensured. The content visualization apparatus visualizes the temporary guidance content 1221 previously generated at the first viewpoint 1201 while maintaining the distance from the vehicle 1290 to the temporary guidance content 1221 .

[0160] At the third viewpoint 1203, the line of sight from the vehicle 1290 to the indicated point is ensured. However, in response to the display position changing while the graphic object is being visualized, the content visualization device visualizes the graphic object while gradually moving the graphic object from the previous display position to the new display position. The content visualization device gradually moves the temporary guidance content 1221 from the previous display position to the next display position. In addition, the content visualization device gradually transforms the graphic object from the temporary guidance content 1221 to the intermediate content 1222, and then to the graphic object corresponding to the route guidance content 1223.

[0161] Herein, gradual deformation and gradual change (eg, movement) of a display position of a graphic object are continuously performed. The content visualization device naturally visualizes a graphic object that is gradually deformed or moved for each frame at a specified frame rate (eg, 30 frames per second (fps), 60fps, or 120fps).

[0162] At the fourth viewpoint 1204, the content visualization device visualizes a graphic object corresponding to the route guidance content 1223 at the indicated point. The content visualization device visualizes the route guidance content 1223 as being fixed to the indicated point until the vehicle 1290 reaches the indicated point. At the fourth viewpoint 1204, the content visualization device visualizes the deformed graphic object.

[0163] Fig.13 and Fig.14 An example of visualizing route guidance content based on a content movement trajectory is shown. Fig.13 The operations in the embodiment may be performed in the order and manner shown; however, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the described illustrative examples. Fig.13 Many of the operations shown may be performed in parallel or simultaneously. Fig.13 One or more blocks and combinations of these blocks can be implemented by a computer based on special hardware that performs the specified functions or a combination of special hardware and computer instructions. Fig.13 In addition to the description of Figures 1 to 12 The description also applies to Fig.13 , and are incorporated herein by reference. Therefore, the above description may not be repeated here.

[0164] like Figure 5 As described in operation 522 of , the content visualization device determines the content movement trajectory 1419 based on the user's FOV and the body of the vehicle 1490 .

[0165] In operation 1321, the content visualization apparatus determines whether the route guidance content is blocked by the FOV and the body of the vehicle 1490. When the distance between the vehicle 1490 and the indicated point is less than a threshold distance, the route guidance content is blocked by the body (e.g., hood) of the vehicle 1490. When the distance between the indicated point and the current position is less than a threshold distance, the content visualization apparatus determines that the route guidance content is blocked.

[0166] In operation 1322, when the route guidance content is blocked, the content visualization device adjusts the height of the route guidance content based on the body of the vehicle 1490 and the content movement trajectory 1419. The content visualization device generates an object arrangement space based on the body of the vehicle 1490 on which the content visualization device is installed and the user's FOV. The content visualization device also naturally changes the height of the graphic object in the generated object arrangement space using the content movement trajectory 1419. When the content visualization device approaches the indication point, the content visualization device visualizes the graphic object by adjusting the height of the graphic object based on the content movement trajectory 1419.

[0167] like Fig.14 As shown in , a vehicle 1490 approaches a graphic object 1421 visualized at an indication point. The content visualization device provides the user with a graphic object 1422 visualized at a height determined based on a content movement trajectory 1419 in response to the approach. The content movement trajectory 1419 is a trajectory in which, for an area exceeding a threshold distance from the content visualization device, the height of the route guidance content or the graphic object corresponding to the route guidance content is adapted to the road surface. The content movement trajectory 1419 is also a trajectory in which, for an area within the threshold distance, the trajectory passes from the road surface through the front edge of the body of the vehicle 1490 to an area corresponding to the user's eyes.

[0168] Fig.15 An example of determining the display position of the route guidance content when the content visualization device enters and exits an uphill road is shown.

[0169] The content visualization device may visualize the route guidance content at the display point, and may fix the display position until the content visualization device reaches the display point. Fig.15 As shown in , the content visualization apparatus visualizes the first content 1521 in the object arrangement space, and maintains the position of the first content 1521 until the content visualization apparatus reaches the first content 1521. Therefore, the user may recognize that the content visualization apparatus is approaching the first content 1521.

[0170] In an example, when the content visualization device reaches the current display point of the current route guidance content, the content visualization device determines the subsequent display position of the subsequent route guidance content. The content visualization device visualizes the graphic object corresponding to the subsequent route guidance content at the subsequent display position. Fig.15 As shown in , when the content visualization apparatus reaches a point corresponding to the first content 1521 , the content visualization apparatus visualizes the second content 1522 .

[0171] Since the line of sight is limited on the uphill road, the second content 1522 is visualized at a position close to the vehicle 1590. In response to the content visualization device reaching the second content 1522, the content visualization device visualizes the third content 1523. Fig.15 As shown in , when the content visualization device enters the uphill road, the line of sight is limited, and a relatively small object arrangement space is generated. After entering the uphill road, the content visualization device visualizes the fourth content 1524 at a distance similar to that on flat ground.

[0172] When the road surface and the space corresponding to the user's line of sight do not intersect each other, the content visualization device adapts the height of the display position to the bottom of the object arrangement space. Fig.15 In the example of , when the vehicle 1590 leaves the uphill road, the bottom surface of the object arrangement space is separated from the road surface. That is, the road surface is out of the user's sight. The content visualization device visualizes the fifth content 1525 as being suitable for the bottom surface of the object arrangement space, rather than being suitable for the road surface.

[0173] although Fig.15 An uphill road is shown, but the content visualization device visualizes the route guidance content on a downhill road in a similar manner. For example, entering an uphill road corresponds to leaving a downhill road, and leaving an uphill road corresponds to entering a downhill road.

[0174] Fig.16 An example of a content visualization method is shown. Fig.16 The operations in the embodiment may be performed in the order and manner shown; however, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the described illustrative examples. Fig.16 Many of the operations shown may be performed in parallel or simultaneously. Fig.16 One or more blocks and combinations of these blocks can be implemented by a computer based on special hardware that performs the specified functions or a combination of special hardware and computer instructions. Fig.16 In addition to the description of Figures 1 to 15 The description also applies to Fig.16 , and are incorporated herein by reference. Therefore, the above description may not be repeated here.

[0175] refer to Fig.16 In operation 1621, the content visualization apparatus determines whether the vehicle is traveling at a high speed. The content visualization apparatus determines whether the speed of the vehicle exceeds a threshold speed.

[0176] In operation 1622, when the vehicle is traveling at a high speed, the content visualization device visualizes the graphic object at a relatively far position (e.g., a second position). When the vehicle is traveling at a high speed, it is expected that the driver will look at a relatively far position. Therefore, the content visualization device determines the position corresponding to the driver's gaze as the display position. For example, the second position is a position 50m ahead. However, the example is not limited to this.

[0177] In operation 1623, when the vehicle is not traveling at high speed, the content visualization device visualizes the graphic object at a relatively close position (e.g., a first position). When the vehicle is traveling at a low speed, the driver is expected to look at a relatively close position. Therefore, the content visualization device determines the position corresponding to the driver's gaze as the display position. For example, the first position is a position 15m ahead. However, the example is not limited to this.

[0178] In the above example, the speed is divided into two stages. However, the example is not limited to this. In another example, the content visualization device classifies the speed of the vehicle into one of n speed intervals, and visualizes the graphic object at a display position corresponding to the classified speed interval. Here, n is an integer greater than or equal to "1".

[0179] In operation 1624, the content visualization device determines whether a lane change is required. The content visualization device determines whether to change lanes based on the route guidance information. For example, when a turn is required but it is difficult to turn (e.g., turn left or turn right) in the current lane, the content visualization device determines that the vehicle needs to move to a lane where the turn can be made.

[0180] In operation 1625, when a lane change is required, the content visualization device dynamically visualizes the graphic object. The content visualization device visualizes a continuous trajectory of the graphic object moving from the content visualization device to the indication point, thereby inducing an action of the driver.

[0181] However, the dynamic visualization of the graphic objects is not limited to lane change events. When an event that requires active user action (e.g., turning or changing lanes) is detected, the content visualization device dynamically visualizes the graphic objects corresponding to the route guidance content. The dynamic visualization of the graphic objects includes the following operations: continuously visualizing the graphic objects while repeatedly moving the display position from the current position of the content visualization device to the indicated point along the path indication line. Fig. 20 Describe dynamic visualization.

[0182] In operation 1626, the content visualization device determines whether the content visualization device is close to the indication point. When the content visualization device is close to the indication point, in operation 1627, the content visualization device visualizes the graphic object as fixed to the indication point. When the distance between the indication point and the position of the content visualization device is less than the distance required for the indication, the content visualization device visualizes the route guidance content at the indication point, that is, the content visualization device can pin the route guidance content at the indication point. When the indication point is outside the object arrangement space, the content visualization device redetermines the display point in the object arrangement space and visualizes the graphic object at the display point. When multiple information related to the route to the destination is provided at one time, driving is interrupted by the graphic objects corresponding to the information. Therefore, in order to prevent multiple indications from being included in the guidance path at the same time, when the content visualization device approaches the distance required for indication, the content visualization device provides the route guidance content corresponding to the indication.

[0183] In operation 1628, the content visualization device visualizes the path indication line using a dotted line. The content visualization device displays the route indication content using dots arranged at intervals so that the user can recognize the distance. Fig.18 Describes the visualization of the path indicator lines.

[0184] In operation 1629 , the content visualization apparatus determines whether the destination is reached. When the destination is reached, the content visualization apparatus terminates the operation associated with the route guidance. When the destination is not reached, the content visualization apparatus performs operation 510 .

[0185] In addition, when the vehicle leaves the guidance path, the content visualization device calculates a detour and visualizes a path indication line corresponding to the detour. For example, when the vehicle leaves the guidance path due to the driver's error, the content visualization device pre-visualizes the detour using the route guidance content and the route indication content, thereby suggesting a new route to the driver.

[0186] Fig.17 An example is shown in which the content visualization device changes the position of the route guidance content and deforms the graphic object when the line of sight is ensured.

[0187] exist Fig.16 In operations 1626 and 1627 of the method, when the content visualization apparatus approaches the indication point, the content visualization apparatus visualizes a graphic object corresponding to the route guidance content at the indication point.

[0188] At the first viewpoint 1701, the content visualization device determines whether the distance to the indication point is less than the distance required for indication. In the example, the content visualization device also determines whether the indication point is included in the object arrangement space. In the example, for example, the distance required for indication is the maximum distance of the object arrangement space. However, the example is not limited to this. In response to determining that the indication point is included in the object arrangement space, the content visualization device visualizes the first route guidance content 1721.

[0189] The content visualization device visualizes the second route guidance content 1722 at the second viewpoint 1702 by moving the first route guidance content 1721 of the first viewpoint 1701. The content visualization device visualizes the third route guidance content 1723 at the third viewpoint 1703 by moving the second route guidance content 1722. At the fourth viewpoint 1704, the content visualization device visualizes the fourth route guidance content 1724 at the indicated point. The content visualization device gradually deforms the shape of the graphic object from the first route guidance content 1721 to the fourth route guidance content 1724 for each frame. In addition, the content visualization device visualizes the first route guidance content 1721 to the fourth route guidance content 1724 moving along the path indication line for each frame, thereby visualizing the route guidance content as deformed and moved. Therefore, the content visualization device causes the user to travel along the route that the content visualization device needs to travel by moving the route guidance content, and guides the point (e.g., turning point) at which the action is to be performed.

[0190] Fig.18 An example of a graphic object corresponding to a path indication line is shown.

[0191] The content visualization device visualizes the path indication line 1820 as route guidance content. As described above, the path indication line 1820 is a line indicating the guide path along which the vehicle will travel, and corresponds to, for example, the center line of the lane. The route guidance content includes the path indication line 1820, and the path indication line 1820 includes a plurality of point objects 1821 arranged along the guide path. The plurality of point objects 1821 are spaced apart from each other at unit intervals. For example, the unit interval is 1m. However, the example is not limited thereto. For example, with respect to a close distance (e.g., a distance within 30m of the vehicle), the unit interval is set to 1m, and with respect to a long distance (e.g., a distance 60m away from the vehicle), the unit interval is set to 10m. The unit interval enhances the user's visual perception.

[0192] The content visualization device intuitively provides the user with information related to the distance to surrounding obstacles through the point objects 1821 arranged at even intervals. Fig.18 In the example of FIG. 1 , point object 1821 is a circle. However, the size and shape of point object 1821 may vary without departing from the spirit and scope of the illustrative examples described.

[0193] In the example, the content visualization device determines the transparency of the graphic object based on the distance from the maximum distance of the object arrangement space to the graphic object. In the example, the transparency is determined based on alpha blending. The content visualization device gradually increases the transparency of the distal end 1840 of the path indication line 1820 away from the content visualization device, that is, gradually increases the transparency in the direction away from the content visualization device. The content visualization device visualizes the portion of the path indication line 1820 corresponding to the maximum depth of the object arrangement space as completely transparent. Therefore, the content visualization device visualizes the path indication line 1820 as a natural fade-out. In the example, the content visualization device gradually increases the transparency of the proximal end 1830 of the path indication line 1820 close to the content visualization device in the direction toward the content visualization device.

[0194] Fig.19 An example of visualizing lane information is shown.

[0195] The content visualization device visualizes the graphic object corresponding to the lane information as driving related information. Fig.19 As shown in , the content visualization device visualizes driving-related content at the lower end of the FOV. As described above, the content visualization device visualizes the driving-related content at a display position corresponding to the projection surface, thereby providing driving-related information in a manner similar to a user watching a dashboard while driving. The driving-related content is set at a size that does not interrupt driving. The content visualization device updates the driving-related content in real time based on the state of the vehicle. For example, when a school area appears 50m ahead, the content visualization device arranges and matches a graphic object of a 30km speed limit sign at a front position similar to the driver's gaze. The content visualization device visualizes the content corresponding to the important information in the driving-related content at the center of the view for a period of time, and then moves the content to the driving information provision area at the lower end.

[0196] Fig.19 The example of FIG. 4 shows a situation where a vehicle enters a four-lane road.

[0197] A vehicle 1991 of a first viewpoint is traveling on a highway ramp. With respect to the vehicle 1991 of the first viewpoint, the content visualization device provides a view image 1901 of the first viewpoint. The view image 1901 shows a vision recognized by one of the user's eyes, and is, for example, an image in which the real environment in front of the vehicle overlaps with a graphic object visualized by the content visualization device. As shown in the view image 1901 of the first viewpoint, the content visualization device visualizes a graphic object 1931 indicating a single lane in which the vehicle is currently traveling. In the example, the graphic object 1931 also includes the current time. In addition, the content visualization device visualizes a graphic object 1942 indicating the speed of the vehicle and a graphic object 1941 corresponding to summary content summarizing road information.

[0198] The summary content is content corresponding to the summary of the road information, and indicates, for example, the line shape, type, number of lanes, and attributes of the road. Fig.19 , a graphic object 1941 corresponding to the summary content indicates that the lane in which the vehicle is traveling is the lane in which it is expected to merge into the highway.

[0199] The vehicle 1992 of the second viewpoint travels a section of the road to merge into the highway. As shown in the view image 1902 of the second viewpoint, the content visualization means visualizes a graphic object 1932 corresponding to the number of lanes of the road and the position of the lane in which the vehicle is currently traveling. For example, at the second viewpoint, the graphic object 1932 provides information indicating that the vehicle is traveling on the rightmost lane among a total of five lanes.

[0200] The vehicle 1993 of the third viewpoint travels a distance after merging into the highway. As shown in the view image 1903 of the third viewpoint, the content visualization device visualizes a graphic object 1933 associated with lane information updated when the vehicle changes lanes. For example, at the third viewpoint, the lane information provides information indicating that the lane in which the vehicle is currently traveling is the fourth lane among a total of five lanes.

[0201] The vehicle 1994 of the fourth viewpoint travels a distance after passing the merged section. As shown in the view image 1904 of the fourth viewpoint, the content visualization device visualizes a graphic object 1934 associated with lane information updated at the end of the merged lane. For example, at the fourth viewpoint, the lane information provides information indicating that the lane in which the vehicle is currently traveling is the rightmost lane among a total of four lanes.

[0202] supply Fig.19 The examples are for better understanding, and thus the lane information and the corresponding graphic objects are not limited thereto.

[0203] Fig. 20 An example of dynamic visualization of route guidance content is shown.

[0204] In response to detecting a dynamic event, the content visualization device dynamically visualizes a graphic object of route guidance content corresponding to the dynamic event.

[0205] For example, the content visualization device moves the graphic object being visualized by continuously changing the display point from the position of the vehicle to the indication point for each frame. The content visualization device moves the graphic object along the path indication line 2080. Therefore, in the user's sight, the graphic object moves forward quickly along the path indication line 2080, and the content visualization device causes a user action corresponding to the dynamic event.

[0206] exist Fig. 20 In the example of , the dynamic event is a lane change event. Fig. 20 In the example of , the route guidance information includes guidance indicating that the user needs to merge into the left lane, and a path indication line 2080 is formed from the current lane to the left lane where the lane change is to be performed. At the first viewpoint, the content visualization device generates a first route guidance content 2021, and visualizes the first route guidance content 2021 at a display position corresponding to the path indication line 2080. At the second viewpoint, the content visualization device moves the display position forward along the path indication line 2080, and visualizes the second route guidance content 2022 at the new display position. At the third viewpoint, the content visualization device determines a position farther in front of the second route guidance content 2022 as the display position, and visualizes the third route guidance content 2023 at the corresponding display position. When the display position of the route guidance content moves forward along the path indication line 2080 and reaches the boundary surface of the object arrangement space, the content visualization device cancels the existing display position and determines a new display position. For example, the content visualization device uses the position of the content visualization device as a starting point to generate a new graphic object corresponding to the route guidance content, and moves the graphic object to an indication point, for example, an end point corresponding to the boundary surface of the object arrangement space.

[0207] Fig.21 An example of a static visualization of route guidance content is shown.

[0208] When a static event is detected, the content visualization means statically visualizes a graphic object of route guidance content corresponding to the static event.

[0209] The content visualization device visualizes the graphic object by fixing the display position to the indication point. The content visualization device visualizes the graphic object at the fixed display position until the content visualization device passes through the display position. The user recognizes that the content visualization device seems to be close to the position where the graphic object is arranged.

[0210] exist Fig.21In the example of , the static event is a lane keeping event. Fig.21 In the example of , the route guidance information includes guidance indicating that the user needs to keep driving in the current lane, and a path indication line is formed along the center line of the current lane. At the first viewpoint 2101, the content visualization device visualizes the route guidance content 2121 for going straight at the indication point. At the second viewpoint 2102, even when the distance between the content visualization device and the indication point decreases, the content visualization device visualizes the route guidance content 2121 at the indication point.

[0211] Fig. 22 An example of warning of a dangerous object is shown.

[0212] The content visualization device visualizes a graphic object corresponding to information about a lane on which the content visualization device is traveling, and visualizes a graphic object corresponding to a dangerous object near the content visualization device on the road on which the content visualization device is located.

[0213] The content visualization device detects whether there is a dangerous object within the danger distance. When there is a dangerous object within the distance, the content visualization device determines the display position based on the physical coordinates of the dangerous object. For example, the content visualization device determines the point corresponding to the back side of the bounding box detected for the dangerous object as the display position. The content visualization device visualizes a first dangerous graphic object 2221 on the back side of the dangerous object. In addition, the content visualization device visualizes a dangerous area 2222 indicating a potential danger on the rear side of the dangerous object. The dangerous area 2222 is suitable for the bottom surface of the object arrangement space. In addition, the content visualization device visualizes a graphic object 2223 corresponding to relevant information of the lane where the dangerous object exists.

[0214] Fig.23 An example of providing speed limit information is shown.

[0215] When a driving-related event is detected, the content visualization device visualizes a graphic object corresponding to the detected event. In an example, when a road section is specified for a driving-related event, the content visualization device determines the starting point of the road section as a display position and fixes the display position. In an example, the content visualization device visualizes the graphic object corresponding to the driving-related event at a fixed display position. In an example, when the content visualization device passes over the starting point of the road section, the content visualization device adds the graphic object corresponding to the driving-related event to a corner of the object arrangement space.

[0216] In response to detecting an event of entering a speed limit area, the content visualization means visualizes a graphic object of driving-related content corresponding to the speed limit area. Fig.23In the example, the content visualization device determines the starting point of the speed limit area as the display position. The content visualization device visualizes the speed limit content 2321 at the starting point of the speed limit area. The content visualization device fixes the position of the speed limit content 2321. Therefore, as the vehicle travels, the content visualization device provides the speed limit content 2322 visualized in an enlarged manner. In the example, after the content visualization device enters the speed limit area, the content visualization device visualizes the speed limit content 2323 in the driving information providing area (e.g., the lower end area in the object arrangement space).

[0217] However, driving related events are not limited to speed limit events. Various events can be set according to design, for example, entering a work area.

[0218] Fig.24 An example of visualizing a shadow object corresponding to route guidance content is shown.

[0219] The content visualization device visualizes the shadow object in the area below the graphic object 2420 in the object arrangement space. The content visualization device maintains the brightness of the shadow area 2431 below the graphic object 2420. Regarding the area 2432 around the shadow area 2431, the content visualization device visualizes the object with a higher brightness than the shadow area 2431. The area 2432 is an area that surrounds at least the shadow area 2431. The area 2432 is a circular shape. However, the example is not limited to this.

[0220] It is difficult for the transparent HUD to reduce the brightness of the graphic object 2420, and therefore has limitations in expressing shadows. The content visualization device increases the brightness of the area 2432 instead of the shadow area 2431 to express the shadow, thereby realizing the shadow through an optical illusion that makes the brightness of the shadow area 2431 appear relatively low. The road surface 2490 is generally a relatively dark color, and therefore actually shows a shadow. The shadow of the 3D object expressed in a three-dimensional space provides a more realistic feeling to the user.

[0221] Content visualization apparatus 200 and 310 and the present invention with respect to Figures 1 to 24Other devices, units, modules, equipment and other components described are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application, where appropriate. In other examples, one or more hardware components for performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements, such as a logic gate array, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this application. Hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the example described in this application, but multiple processors or computers can be used in other examples, or processors or computers can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component, or two or more hardware components. Hardware components can have any one or more of different processing configurations, and the example of processing configuration includes a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, a multiple instruction single data (MISD) multiprocessing, and a multiple instruction multiple data (MIMD) multiprocessing.

[0222] Perform the operations described in this application Figures 4 to 24The methods shown in are performed by computing hardware, for example, by one or more processors or computers, wherein the computing hardware is implemented as described above to execute instructions or software to perform the operations performed by these methods described in the present application. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can perform a single operation or two or more operations.

[0223] The instructions or software for controlling a processor or computer to implement a hardware component and perform the method as described above are written as a computer program, a code segment, an instruction, or any combination thereof, to individually or collectively instruct or configure a processor or computer to operate as a machine or a special-purpose computer to perform the operations performed by the hardware component and the above-mentioned method. In an example, the instructions or software include at least one of the following: an applet, a dynamic link library (DLL), middleware, firmware, a device driver, an application for a method of storing visual content. In one example, the instructions or software include machine code directly executed by a processor or computer, such as machine code generated by a compiler. In another example, the instructions or software include high-level code executed by a processor or computer using an interpreter. An ordinary programmer in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the corresponding description discloses an algorithm for performing the operations performed by the hardware component and the above-mentioned method.

[0224] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card type storage (e.g., multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device, any other device configured to store instructions or software and any associated data, data files and data structures in a non-temporary manner, and provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that one or more processors or computers store, access and execute the instructions and software and any associated data, data files and data structures in a distributed manner.

[0225] Although the present disclosure includes specific examples, it will be apparent after understanding the disclosure of the present application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered to be descriptive only and not for limiting purposes. The description of the features or aspects in each example should be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved. Therefore, the scope of the present disclosure is not limited by the detailed description, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents should be interpreted as included in the present disclosure.

Claims

1. A method for visualizing content of a head-up display (HUD) of a vehicle based on augmented reality, comprising: generating a three-dimensional 3D object arrangement space based on a road area in front of the vehicle estimated by a sensor, wherein a graphic object is stereoscopically visualized in the 3D object arrangement space; determining a display position of the route guidance content within the 3D object arrangement space based on an indication point of the route guidance content at which a user should perform an operation corresponding to the route guidance content; as well as three-dimensionally visualizing a graphic object corresponding to the route guidance content at the determined display position, The content visualization method further includes: in response to the 3D object arrangement space intersecting with a road surface, limiting the 3D object arrangement space.

2. The content visualization method according to claim 1, wherein: The generating includes generating the 3D object arrangement space based on a field of view FOV of a user provided by the HUD and a distance from a viewpoint of the user to an obstacle.

3. The content visualization method according to claim 1, wherein: The three-dimensional visualization includes: generating a first graphical representation corresponding to a left eye of a user and a second graphical representation corresponding to a right eye of the user; and A disparity between the first graphical representation and the second graphical representation is provided.

4. The content visualization method according to claim 3, wherein: The providing includes tracking the positions of the left eye and the right eye of the user.

5. The content visualization method according to claim 1, wherein: The determining includes: in response to the indication point being outside the 3D object arrangement space, determining a position in the 3D object arrangement space as the display position.

6. The content visualization method according to claim 1, wherein: The determining includes: in response to the road surface being a part of an uphill road or a downhill road, adapting the height of the display position at which the graphic object is visualized in the 3D object arrangement space to a bottom of the 3D object arrangement space.

7. The content visualization method according to claim 1, wherein: The determining includes: in response to the road surface and the space corresponding to the user's line of sight not intersecting, adapting the height of the display position at which the graphic object is visualized in the 3D object arrangement space to the bottom of the 3D object arrangement space.

8. The content visualization method according to claim 1, wherein: The determining includes: in response to a distance from the user to the display position being greater than a maximum distance of the 3D object arrangement space, updating the display position to be located within the 3D object arrangement space.

9. The content visualization method according to claim 1, wherein: The generating includes generating the 3D object arrangement space based on a field of view FOV of a user provided by the HUD and a body of the vehicle.

10. The content visualization method according to claim 1, wherein: The three-dimensional visualization includes: determining a content movement trajectory defining a height of the graphic object according to a distance between the display position and the HUD relative to a field of view FOV of a user provided by the HUD and a body of the vehicle; and In response to the HUD approaching the indication point, the graphic object is visualized by adjusting a height of the graphic object according to a distance between the display position and the HUD based on the content movement trajectory.

11. The content visualization method according to claim 1, wherein: The bottom of the 3D object arrangement space is adapted to the surface of a road area estimated in the vicinity of the vehicle.

12. The content visualization method according to claim 11, wherein: The generating of the 3D object arrangement space includes adapting a bottom of the 3D object arrangement space to a road surface until a maximum depth of the 3D object arrangement space for visualizing the graphic object is determined based on the HUD.

13. The content visualization method according to claim 1, wherein: The three-dimensionally visualizing includes visualizing a shadow object in an area below the graphic object in the 3D object arrangement space.

14. The content visualization method according to claim 13, wherein: The visualization of the shadow object includes: maintaining the brightness of the shadow area relative to an area surrounding the shadow area below the graphical object; and The graphic object is visualized with a brightness higher than that of the shadow area.

15. The content visualization method according to claim 1, further comprising: In response to the HUD reaching the display position of the route guidance content, determining a subsequent display position of subsequent route guidance content; as well as Another graphical object corresponding to the subsequent route guidance content is visualized at the subsequent display position.

16. The content visualization method according to claim 1, wherein: The three-dimensionally visualizing includes determining a transparency of the graphic object based on a distance from a maximum distance of the 3D object arrangement space to the graphic object.

17. The content visualization method according to claim 1, wherein: The route guidance content includes a plurality of point objects arranged along the route, The plurality of point objects are spaced apart from each other at unit intervals.

18. The content visualization method according to claim 1, wherein: The determining includes: in response to the indication point being different from the display position, obtaining temporary guidance content corresponding to the display position, The three-dimensionally visualizing includes: visualizing a graphic object corresponding to the temporary guide content at the display position until the 3D object arrangement space includes the indication point.

19. The content visualization method according to claim 1, further comprising: Graphic objects corresponding to information about a lane in which the vehicle is traveling and graphic objects corresponding to dangerous objects on a road on which the vehicle is located are visualized.

20. The content visualization method according to claim 1, wherein: The three-dimensionally visualizing includes visualizing the graphic object while gradually moving the graphic object from a previous display position to a new display position in response to the display position being changed while the graphic object is being visualized.

21. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the content visualization method of claim 1.

22. A content visualization device for a vehicle, comprising: a sensor configured to estimate a road area in front of the vehicle; a processor configured to generate a three-dimensional 3D object arrangement space based on the road area, wherein the graphic objects are stereoscopically visualized in the 3D object arrangement space; and determine a display position of the route guidance content in the 3D object arrangement space based on an indication point of the route guidance content, wherein the user should perform an operation corresponding to the route guidance content at the indication point; as well as a display configured to three-dimensionally visualize a graphic object corresponding to the route guidance content at the determined display position, The processor is further configured to: in response to the 3D object arrangement space intersecting with a road surface, limit the 3D object arrangement space.

23. A content visualization device, comprising: a sensor disposed on a device and configured to sense information related to a state of the device and a distance to an obstacle near the device, and estimate a road area in front of the device; a processor configured to generate a three-dimensional 3D object arrangement space based on the road area, wherein the graphic objects are stereoscopically visualized in the 3D object arrangement space; and determine a display position of the route guidance content in the 3D object arrangement space based on an indication point of the route guidance content, wherein the user should perform an operation corresponding to the route guidance content at the indication point; as well as a head-up display HUD configured to visualize a 3D graphic object corresponding to the route guidance content at the determined display position, The processor is further configured to: in response to the 3D object arrangement space intersecting with a road surface, limit the 3D object arrangement space.

24. The content visualization apparatus according to claim 23, further comprising: The memory is configured to store the route guidance content, information related to the state of the device, a map database, and instructions, which when executed configure the processor to determine any one or any combination of the 3D object arrangement space and the display position.

25. The content visualization device according to claim 24, wherein: The processor is further configured to determine the indication point based on road information in the map database.

26. The content visualization device according to claim 23, wherein: The device includes a vehicle, and the information related to the state of the device includes any one or any combination of the following: location information of the vehicle, current geographic coordinates of the vehicle, road information corresponding to the location of the vehicle, a lane in which the vehicle is traveling, a width of the lane in which the vehicle is traveling, a width of the road on which the vehicle is traveling, the number of lanes of the road, a center line of the lane, a turning point on the road, a traffic signal on the road, a speed of the vehicle, fuel of the vehicle, a fuel level of the vehicle, and maintenance information of the vehicle.

27. The content visualization apparatus according to claim 26, wherein: The sensor is further configured to track the position of a left eye and a right eye of a user in response to the vehicle being in motion; and The processor is further configured to: generating a first graphical representation corresponding to the position of the user's left eye, generating a second graphical representation corresponding to the position of the user's right eye, and The 3D graphical object is presented on the HUD based on a disparity between the first graphical representation and the second graphical representation.

28. The content visualization device according to claim 23, wherein: The 3D object arrangement space includes a rectangular cone-shaped area extending from both eyes of the user toward a road area in front of the device.

29. A method for visualizing content of a head-up display (HUD) of a vehicle based on augmented reality, comprising: generating a three-dimensional 3D object arrangement space based on a field of view FOV of a user provided by the HUD and a distance from obstacles near the user, wherein a graphic object is stereoscopically visualized in the 3D object arrangement space; determining a display position of the route guidance content within the 3D object arrangement space based on a point at which an operation corresponding to the route guidance content is to be performed; as well as visualizing a 3D graphic object corresponding to the route guidance content at the determined display position, The content visualization method further includes: in response to the 3D object arrangement space intersecting with a road surface, limiting the 3D object arrangement space.

30. The content visualization method according to claim 29, wherein: The determining of the display position includes: in response to the point being located outside the 3D object arrangement space, determining a position corresponding to the point in the 3D object arrangement space as the display position.

31. The content visualization method according to claim 30, wherein: The position corresponding to the point is located substantially close to a limiting surface of the 3D object arrangement space and on an axis from a center of an eye box of the user toward the point.

32. The content visualization method according to claim 29, wherein: The visualization of the 3D graphic object includes: visualizing a shadow object under the 3D graphic object in the 3D object arrangement space, and increasing brightness of a portion of the 3D object arrangement space arranged around the 3D graphic object.

33. The content visualization method according to claim 29, further comprising: The transparency of the route guidance content adjacent to a boundary of the 3D object arrangement space is increased.

Images