Device for generating 3D map view based on mixed reality and control method thereof

Abstract

The present invention comprises: an interface unit for receiving sensing information collected from at least one sensor provided in a vehicle; a memory for storing map data for a plurality of map tiles; and a processor for controlling the interface unit to receive driving information of the vehicle including the position, speed, and moving direction of the vehicle, and modeling and rendering terrain, road objects, and building objects included in the map data of the plurality of map tiles determined on the basis of the driving information, wherein the processor distorts the terrain and the road objects so that road and ground shapes outside an area within a preconfigured first distance from the vehicle are warped with a preconfigured curvature.

Description

A device for generating a 3D map view based on mixed reality and a method for controlling the device

[0001] The present invention relates to a device for generating a 3D map view based on mixed reality.

[0002] Recently, following Augmented Reality (AR), which outputs graphic objects through a vehicle's windshield or HUD (Head Up Display), or uses images captured by a camera to additionally output graphic objects in the real world, technology development related to Mixed Reality (MR), which can provide various simulation information related to vehicles by applying Digital Twin (DT) technology, is actively underway.

[0003] As part of the technological development related to this mixed reality, active research is being conducted on methods that utilize mixed reality to provide drivers with route guidance-related information. Route guidance using mixed reality, for example, utilizes digital twin technology to display graphical objects corresponding to vehicles on a digitized 3D map. This map and graphical objects can be used to provide information on driving routes the driver has not yet driven, or provide a bird's-eye view, offering various information that the driver in the cockpit cannot view.

[0004] Meanwhile, for drivers to obtain accurate information through route guidance using mixed reality, the digital twin technology used to create the map—the digital twin map—must have a high degree of consistency with the real world. This requires the provision of a 3D map, not a 2D one. Therefore, service providers currently offering digital twin maps are offering 3D maps, which are essentially 2D maps mapped into 3D.

[0005] However, current service providers rely on building modeling using satellite or aerial photography to provide 3D mapping. However, the high angles at which satellite or aerial photography is used make it difficult to capture images of various aspects of a building, making it difficult to provide accurate textures. Therefore, conventional 3D maps are provided in the form of polygon data containing the building's floor coordinates (footprint) and height.

[0006] Meanwhile, 3D maps containing models provided in the form of polygon data like this have the problem of not including accurate building textures. These polygonal maps only represent the location and 3D volume of buildings, differing from the actual building shapes. This discrepancy creates a gap between the digital twin map and the real world, making it difficult to provide accurate information to drivers.

[0007] Additionally, in the case of a 3D map, a number of building objects may be displayed between guidance locations along the currently set route from the camera providing the 3D map view. Therefore, the 3D map may have a problem in that the guidance location may be at least partially obscured by the building objects, or the route guidance may be provided in a form that is difficult for the user to identify due to the distance between the guidance location and the 3D map view camera.

[0008] The present invention aims to solve the above-mentioned problems and other problems.

[0009] One object of the present invention is to provide a 3D map view generation device capable of displaying virtual objects having high-quality texture materials similar to real-world buildings in real time without delay according to changes in the driving condition of a vehicle, and a method for controlling the device.

[0010] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof capable of providing a more realistic mixed reality-based route guidance service through a digital twin map including virtual objects with more realistic textures corresponding to the shape of each side of a building.

[0011] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof in which the guidance location is not obscured by building objects located between the guidance location and a 3D map view camera providing a field of view of the 3D map view.

[0012] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof that provide a 3D map that allows a user to more intuitively and more easily identify the guidance location even when the distance between the guidance location and the 3D map view camera is greater than a certain distance.

[0013] According to one embodiment of the present invention, a 3D map view generation device includes an interface unit that receives sensing information collected from at least one sensor equipped in a vehicle, a memory that stores map data for a plurality of map tiles, and a processor that controls the interface unit to receive driving information of the vehicle including the location, speed, and moving direction of the vehicle, and models and renders terrain and road objects and building objects included in the map data of the plurality of map tiles determined based on the driving information, wherein the processor is characterized in that it distorts the terrain and road objects such that a road and ground shape outside an area within a first preset distance from the vehicle is bent at a preset curvature.

[0014] In one embodiment, the processor is characterized in that it determines the first distance based on at least one of a height of a 3D map view camera providing a field of view of the 3D map view, a viewing direction vector of the 3D map view camera, and a driving state of the vehicle.

[0015] In one embodiment, the processor distorts at least a portion of the terrain and road objects by modifying the positions of at least a portion of the vertices constituting the terrain and road objects detected from the map data based on a vertex coordinate transformation function defined by a preset distortion model and distortion parameters, wherein the distortion model is defined in the form of an exponential function, and the distortion parameter includes a slope value of the exponential function of the distortion model determined according to a scenario in which the 3D map view is provided.

[0016] In one embodiment, the processor varies the curvature of the terrain and road objects according to the distance between the guidance location for guiding the driving route of the vehicle and the vehicle or the driving speed of the vehicle, and the guidance location is characterized in that it is at least one of a destination, a waypoint for guiding the route, or a specific location set by a user.

[0017] In one embodiment, the processor distorts the terrain and road objects with a distortion rate greater than or equal to a preset minimum distortion rate, wherein the minimum distortion rate is a distortion rate that distorts the terrain object so that the guidance location exists within a preset distance from an intersection of an upper vector of a field of view of a 3D map view camera providing a field of view of the 3D map view and the distorted terrain object.

[0018] In one embodiment, the processor controls the interface unit to distort a terrain object corresponding to a map area within a preset distance centered on the current location of the vehicle, and to display virtual objects included in the distorted map area on a display unit of the vehicle centered on the location of the vehicle, wherein the distorted map area is characterized by being a distorted area that is bent into a hemisphere centered on the location of the vehicle so that the ground level becomes higher as it gets farther away from the vehicle object corresponding to the vehicle.

[0019] In one embodiment, when there are multiple guidance locations for guiding the driving route of the vehicle, the processor divides a map area according to each of the multiple guidance locations based on a distance from the vehicle, and distorts the terrain and road objects so as to be bent according to different curvatures for each of the divided areas, and the different curvatures include curvatures having different polarities (positive (+), negative (-)).

[0020] In one embodiment, the processor is characterized in that it models building objects detected from the map data, places the modeled building object models under a footprint area, which is a shape of a ground area occupied by each building object, included in a map area that is bent according to the distortion of the terrain object, and protrudes at least a portion of the placed building object models from the footprint area according to the driving information of the vehicle, and renders at least a portion of the building object models that protrude from the footprint area, thereby displaying a 3D map view including the building objects included in the bent map area.

[0021] In one embodiment, the processor is characterized in that it controls the building object models included in the area where the height of the curved map area exceeds a preset height so that they do not protrude from the footprint area, thereby limiting the rendering of the building objects included in the area where the height exceeds the preset height.

[0022] In one embodiment, the processor is characterized in that, among the curved map areas, the maximum height protruding from the footprint area corresponding to each building object is differently limited according to the distance exceeding the first distance, for building object models included in an area outside the area corresponding to the first distance.

[0023] In one embodiment, the processor controls the building object model not to protrude from the footprint area corresponding to the building object in an area where the protrusion of the building object model from the footprint area is maximally limited among the curved map areas, and the area where the protrusion of the building object model is maximally limited is a map area where the height of the curved map area reaches a preset height.

[0024] In one embodiment, the preset height is characterized by being the height of a point corresponding to a guidance position for guiding the driving path of the vehicle in the curved map area.

[0025] In one embodiment, the processor is characterized in that building objects included in an area above the preset height in the curved map area do not have a maximum height that they protrude from the footprint area.

[0026] In one embodiment, the processor is characterized in that it protrudes each building object positioned below the footprint area from the footprint area in a direction perpendicular to a horizontal plane parallel to the driving surface of the vehicle or a ground shape inclined with respect to the horizontal plane along the curved map area.

[0027] In one embodiment, the processor is characterized in that it enlarges the size of modeled building objects included in a map area outside the area within the first distance in proportion to the distance from the vehicle.

[0028] In one embodiment, the processor is characterized in that it determines the size magnification ratio of modeled building objects included in a map area outside the area within the first distance based on a straight-line distance between the center of a 3D map view camera providing a field of view of the 3D map view and each of the modeled building objects.

[0029] In one embodiment, the building objects whose size is enlarged are characterized by being building objects around a guidance location for guiding the driving path of the vehicle, building objects corresponding to a preset type of building, or building objects according to a preset POI (Point Of Interest).

[0030] In one embodiment, the processor is characterized in that, after the terrain and road objects distorted to bend at the preset curvature are modeled and rendered, other modeled objects are rendered on top of the distorted terrain and road objects, and the other modeled objects include dynamic objects including building objects and surrounding vehicles recognized based on information obtained from an Advanced Driver Assistance System (ADAS).

[0031] In one embodiment, the processor is characterized in that, when terrain objects and road objects included in the map data are modeled and rendered, building objects included in the map data are modeled, and a plurality of texture images including shapes of different building elements are tiled on each side of the modeled building objects to generate the map view image including the rendered building objects.

[0032] In addition, a control method of a 3D map view generation device according to an embodiment of the present invention is characterized by including the steps of: receiving driving information of a vehicle; determining at least one map tile included in a field of view based on the driving information, detecting map data of the determined map tile; detecting vertices constituting terrain and road objects from the map data; modifying coordinates of at least some of the vertices constituting the terrain and road objects based on a vertex coordinate transformation function defined by a preset distortion model and distortion parameters; modeling and rendering distorted terrain and road objects so that at least some of the vertices whose coordinates have been modified are bent at a preset curvature; modeling a building object under the terrain and road objects of which at least some are distorted; rendering a building object on at least some of the terrain and road objects of which at least some are distorted in response to a user request for displaying a map view image; and displaying a map view image in which building objects are rendered on the terrain and road objects of which at least some are distorted, on a display unit of the vehicle.

[0033] The effects of a 3D map view generation device and a control method thereof according to an embodiment of the present invention are described as follows.

[0034] First, the present invention can pre-load map data of a region corresponding to an area around a vehicle based on the vehicle's location according to vehicle driving information from map data of a wide area, and model virtual objects corresponding to buildings and roads included in the loaded region in advance. In addition, by rendering only at least a portion of the modeled virtual objects according to the vehicle's driving information, it has the advantage of providing a digital twin map containing high-quality virtual objects similar to real-world buildings and roads in real time according to the vehicle's driving information.

[0035] Second, for at least some of the modeled building objects, at least one mask map corresponding to different parts of a texture image that shapes each part of the building, and a normal map, which is an image that is superimposed on the mask map and has 3D vector values ​​that form different textures, are superimposed on the texture image so that each component of the building shaped by the texture image has different materials, and the texture image in which the mask map and the normal map are superimposed to form a surface material can be tiled on each face of at least some of the building objects. Then, by rendering and displaying on a display unit at least some of the building objects in which the texture image is tiled, it is possible to perform texturing of the building objects with a small amount of computation while including building objects having high-quality texture materials similar to the actual building, and there is an advantage in that a digital twin map can be provided that includes a building model in which a high-quality texture similar to an actual building is synthesized.

[0036] In addition, the present invention has the effect of enabling a user to more intuitively and more easily identify the guidance location even when the distance between the guidance location in the 3D map view and the 3D map view camera is greater than a certain distance by distorting the terrain object to be bent according to the distance from the vehicle based on the location of the vehicle on the 3D map.

[0037] In addition, the present invention has the effect of preventing the guidance location from being obscured by the building objects by limiting the height of building objects located within a certain distance from the guidance location among building objects located between the guidance location of the 3D map view and the 3D map view camera.

[0038] FIG. 1 is a drawing showing the exterior of a vehicle according to an embodiment of the present invention.

[0039] FIG. 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles.

[0040] Figures 3 and 4 are drawings showing the interior of a vehicle according to an embodiment of the present invention.

[0041] FIGS. 5 and 6 are drawings for reference in explaining an object according to an embodiment of the present invention.

[0042] FIG. 7 is a block diagram for reference in explaining a vehicle according to an embodiment of the present invention.

[0043] Figure 8a is a conceptual diagram for explaining the AR service platform of the present invention.

[0044] Figure 8b is a conceptual diagram for explaining an MR service platform for providing the MR service of the present invention.

[0045] Figure 8c is a conceptual diagram for explaining the MR AMS client of the present invention.

[0046] Figure 8d is a conceptual diagram for explaining the MR AMS server of the present invention.

[0047] Figure 9 is a conceptual diagram for explaining the DTaaS server of the present invention.

[0048] FIG. 10 is a block diagram illustrating the structure of a 3D map view generation device according to an embodiment of the present invention.

[0049] FIG. 11 is a block diagram illustrating in more detail the structure of a 3D map view generation device according to an embodiment of the present invention.

[0050] FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view generation device and an MR service device according to an embodiment of the present invention.

[0051] FIG. 13 is a flowchart illustrating an operation process of a processor of a 3D map view generation device according to an embodiment of the present invention, modeling virtual objects included in a map area and rendering and displaying the modeled virtual objects.

[0052] FIG. 14 is an exemplary diagram illustrating an example in which a processor of a 3D map view generation device according to an embodiment of the present invention updates a map area according to driving information of a vehicle.

[0053] FIG. 15 is a block diagram illustrating a configuration in which modeling and rendering of virtual objects are performed by different processes in a 3D map view generation device according to an embodiment of the present invention having multiple processors.

[0054] FIG. 16 is an exemplary diagram showing an example of simultaneously displaying a screen showing information collected around a vehicle according to an ADAS system and an MR view screen according to a digital twin map in a 3D map view generation device according to an embodiment of the present invention.

[0055] FIG. 17a is an exemplary diagram illustrating a frustum-shaped visible area formed across a plurality of map tiles or one map tile, depending on the position of the vehicle and the driving direction of the vehicle, in a 3D map view generation device according to an embodiment of the present invention.

[0056] Figure 17b is an exemplary diagram showing an example in which the visible area of ​​the frustum shape illustrated in Figure 17a includes a plurality of map tiles.

[0057] FIG. 18a is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention, in which texture images are tiled in a margin width and at least one preset texturing area.

[0058] FIG. 18b is an exemplary diagram showing examples of facade designs of a building virtual object expressed in various ways by varying partial areas and tiled texture images in a 3D map view generation device according to an embodiment of the present invention.

[0059] FIG. 19 is a flowchart illustrating a texturing process for forming different surface textures on each component of a building in a 3D map view generation device according to an embodiment of the present invention.

[0060] FIG. 20 is an exemplary diagram showing an example of normal maps blended to mask maps corresponding to different building elements and a multi-blended normal map generated by blending normal maps together in a 3D map view generation device according to an embodiment of the present invention.

[0061] FIG. 21 is a flowchart illustrating an operation process of mapping the colors of virtual building objects based on the colors of actual buildings within a visible area in a 3D map view generation device according to an embodiment of the present invention.

[0062] FIG. 22 is an exemplary diagram showing an example of determining colors mapped to texture images through images captured from a vehicle in a 3D map view generation device according to an embodiment of the present invention.

[0063] FIG. 23 is a conceptual diagram for explaining the concept of GPU shader processing that distorts a terrain object in a 3D map view generation device according to an embodiment of the present invention.

[0064] FIG. 24 is an exemplary diagram showing an example of a terrain object being distorted according to the distance from the vehicle in a 3D map view generation device according to an embodiment of the present invention.

[0065] FIG. 25 is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention in which a terrain object is distorted differently depending on the driving speed of the vehicle.

[0066] FIG. 26 is an exemplary diagram showing an example of providing path guidance according to a distorted terrain object in a 3D map view generation device according to an embodiment of the present invention.

[0067] FIG. 27 is an exemplary diagram showing an example of distorting a terrain object so that the terrain between multiple guidance locations differs depending on the distance from the vehicle in a 3D map view generation device according to an embodiment of the present invention.

[0068] FIG. 28 illustrates an example of a top-down view provided in a 3D map view generation device according to an embodiment of the present invention, in which a terrain object is distorted according to the distance from the vehicle.

[0069] FIG. 29 is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention, in which a building object is displayed on a distorted terrain object according to the height of the distorted terrain object.

[0070] Figure 30 illustrates an example where the guidance location is obscured by building objects when the terrain object is not distorted.

[0071] FIG. 31 is an exemplary diagram showing different examples of displaying a building object displayed on a distorted terrain object in a 3D map view generation device according to an embodiment of the present invention.

[0072] FIG. 32 is an exemplary diagram showing examples in which the height of building objects is limited according to the display method of building objects displayed on a distorted terrain object in a 3D map view generation device according to an embodiment of the present invention.

[0073] FIG. 33 is an exemplary diagram showing an example of displaying building objects displayed on a distorted terrain object differently according to distance in a 3D map view generation device according to an embodiment of the present invention.

[0074] FIG. 34 is an exemplary diagram showing an example of dynamic objects being displayed according to distorted terrain in a 3D map view generation device according to an embodiment of the present invention.

[0075] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0076] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0077] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0078] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0079] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0080] The vehicle described in this specification may include a concept that includes automobiles and motorcycles. In the following, the vehicle will be described primarily with automobiles.

[0081] The vehicle described in this specification may be a concept that includes all types of vehicles, including internal combustion engine vehicles equipped with an engine as a power source, hybrid vehicles equipped with an engine and an electric motor as power sources, and electric vehicles equipped with an electric motor as a power source.

[0082] In the following description, the left side of the vehicle means the left side of the vehicle's driving direction, and the right side of the vehicle means the right side of the vehicle's driving direction.

[0083] FIG. 1 is a drawing showing the exterior of a vehicle according to an embodiment of the present invention.

[0084] FIG. 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles.

[0085] Figures 3 and 4 are drawings showing the interior of a vehicle according to an embodiment of the present invention.

[0086] FIGS. 5 and 6 are drawings for reference in explaining an object according to an embodiment of the present invention.

[0087] FIG. 7 is a block diagram for reference in explaining a vehicle according to an embodiment of the present invention.

[0088] Referring to FIGS. 1 to 7, the vehicle (100) may include wheels that rotate by a power source and a steering input device (510) for controlling the direction of travel of the vehicle (100).

[0089] The vehicle (100) may be an autonomous vehicle.

[0090] The vehicle (100) can be switched to autonomous driving mode or manual mode based on user input.

[0091] For example, the vehicle (100) may be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on user input received through the user interface device (200).

[0092] The vehicle (100) can be switched to autonomous driving mode or manual mode based on driving situation information. The driving situation information can be generated based on object information provided by the object detection device (300).

[0093] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information generated by the object detection device (300).

[0094] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information received through the communication device (400).

[0095] The vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on information, data, and signals provided from an external device.

[0096] When the vehicle (100) is operated in autonomous driving mode, the autonomous vehicle (100) can be operated based on the driving system (700).

[0097] For example, an autonomous vehicle (100) may be driven based on information, data, or signals generated from a driving system (710), an exit system (740), or a parking system (750).

[0098] When the vehicle (100) is driven in manual mode, the autonomous vehicle (100) can receive user input for driving through the driving control device (500). Based on the user input received through the driving control device (500), the vehicle (100) can be driven.

[0099] The overall length refers to the length from the front to the rear of the vehicle (100), the overall width refers to the width of the vehicle (100), and the overall height refers to the length from the bottom of the wheel to the roof. In the following description, the overall length direction (L) may refer to the direction that serves as a reference for measuring the overall length of the vehicle (100), the overall width direction (W) may refer to the direction that serves as a reference for measuring the overall width of the vehicle (100), and the overall height direction (H) may refer to the direction that serves as a reference for measuring the overall height of the vehicle (100).

[0100] As illustrated in FIG. 7, the vehicle (100) may include a user interface device (200), an object detection device (300), a communication device (400), a driving operation device (500), a vehicle driving device (600), a driving system (700), a navigation system (770), a sensing unit (120), a vehicle interface unit (130), a memory (140), a control unit (170), and a power supply unit (190).

[0101] Depending on the embodiment, the vehicle (100) may include additional components other than the components described herein, or may not include some of the components described herein.

[0102] The user interface device (200) is a device for communication between a vehicle (100) and a user. The user interface device (200) can receive user input and provide information generated in the vehicle (100) to the user. The vehicle (100) can implement a UI (User Interface) or UX (User Experience) through the user interface device (200).

[0103] The user interface device (200) may include an input unit (210), an internal camera (220), a biometric detection unit (230), an output unit (250), and a processor (270).

[0104] Depending on the embodiment, the user interface device (200) may include additional components other than the described components, or may not include some of the described components.

[0105] The input unit (200) is for receiving information from a user, and data collected from the input unit (120) can be analyzed by a processor (270) and processed into a user's control command.

[0106] The input unit (200) may be placed inside the vehicle. For example, the input unit (200) may be placed in an area of ​​a steering wheel, an area of ​​an instrument panel, an area of ​​a seat, an area of ​​each pillar, an area of ​​a door, an area of ​​a center console, an area of ​​a head lining, an area of ​​a sun visor, an area of ​​a windshield, or an area of ​​a window.

[0107] The input unit (200) may include a voice input unit (211), a gesture input unit (212), a touch input unit (213), and a mechanical input unit (214).

[0108] The voice input unit (211) can convert a user's voice input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0109] The voice input unit (211) may include one or more microphones.

[0110] The gesture input unit (212) can convert a user's gesture input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0111] The gesture input unit (212) may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input.

[0112] According to an embodiment, the gesture input unit (212) can detect a user's three-dimensional gesture input. To this end, the gesture input unit (212) can include a light output unit that outputs a plurality of infrared lights or a plurality of image sensors.

[0113] The gesture input unit (212) can detect a user's 3D gesture input through a TOF (Time of Flight) method, a structured light method, or a disparity method.

[0114] The touch input unit (213) can convert a user's touch input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0115] The touch input unit (213) may include a touch sensor for detecting a user's touch input.

[0116] According to an embodiment, the touch input unit (213) may be formed integrally with the display unit (251), thereby implementing a touch screen. Such a touch screen may provide both an input interface and an output interface between the vehicle (100) and the user.

[0117] The mechanical input unit (214) may include at least one of a button, a dome switch, a jog wheel, and a jog switch. An electrical signal generated by the mechanical input unit (214) may be provided to a processor (270) or a control unit (170).

[0118] The mechanical input unit (214) can be placed on a steering wheel, center fascia, center console, cockpit module, door, etc.

[0119] The internal camera (220) can capture images of the vehicle interior. The processor (270) can detect the user's status based on the images of the vehicle interior. The processor (270) can obtain information about the user's gaze from the images of the vehicle interior. The processor (270) can detect the user's gestures from the images of the vehicle interior.

[0120] The biometric detection unit (230) can obtain the user's biometric information. The biometric detection unit (230) includes a sensor capable of obtaining the user's biometric information, and can use the sensor to obtain the user's fingerprint information, heartbeat information, etc. The biometric information can be used for user authentication.

[0121] The output unit (250) is for generating output related to vision, hearing, or touch.

[0122] The output unit (250) may include at least one of a display unit (251), an audio output unit (252), and a haptic output unit (253).

[0123] The display unit (251) can display graphic objects corresponding to various information.

[0124] The display unit (251) may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0125] The display unit (251) can be formed as a layer structure with the touch input unit (213) or as an integral part, thereby implementing a touch screen.

[0126] The display unit (251) may be implemented as a HUD (Head Up Display). When the display unit (251) is implemented as a HUD, the display unit (251) may be equipped with a projection module to output information through an image projected onto a windshield or window.

[0127] The display unit (251) may include a transparent display. The transparent display may be attached to a windshield or window.

[0128] A transparent display can display a predetermined screen while having a predetermined transparency. To have transparency, the transparent display may include at least one of a transparent TFEL (Thin Film Electroluminescent), a transparent OLED (Organic Light-Emitting Diode), a transparent LCD (Liquid Crystal Display), a transmissive transparent display, and a transparent LED (Light Emitting Diode) display. The transparency of the transparent display can be adjusted.

[0129] Meanwhile, the user interface device (200) may include a plurality of display units (251a to 251g).

[0130] The display unit (251) may be arranged in one area of ​​the steering wheel, one area of ​​the instrument panel (521a, 251b, 251e), one area of ​​the seat (251d), one area of ​​each pillar (251f), one area of ​​the door (251g), one area of ​​the center console, one area of ​​the head lining, one area of ​​the sun visor, or may be implemented in one area of ​​the windshield (251c), one area of ​​the window (251h).

[0131] The audio output unit (252) converts an electric signal provided from the processor (270) or the control unit (170) into an audio signal and outputs the converted signal. To this end, the audio output unit (252) may include one or more speakers.

[0132] The haptic output unit (253) generates a tactile output. For example, the haptic output unit (253) can operate by vibrating a steering wheel, a seat belt, or a seat (110FL, 110FR, 110RL, 110RR) so that the user can perceive the output.

[0133] The processor (270) can control the overall operation of each unit of the user interface device (200).

[0134] Depending on the embodiment, the user interface device (200) may include a plurality of processors (270) or may not include a processor (270).

[0135] If the user interface device (200) does not include a processor (270), the user interface device (200) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0136] Meanwhile, the user interface device (200) may be referred to as a vehicle display device.

[0137] The user interface device (200) can be operated under the control of the control unit (170).

[0138] The object detection device (300) is a device for detecting an object located outside a vehicle (100).

[0139] Objects may be various objects related to the operation of the vehicle (100).

[0140] Referring to FIGS. 5 and 6, objects may include lanes (OB10), other vehicles (OB11), pedestrians (OB12), two-wheeled vehicles (OB13), traffic signals (OB14, OB15), lights, roads, structures, speed bumps, terrain, animals, etc.

[0141] A lane (OB10) may be a driving lane, a lane adjacent to a driving lane, or a lane in which opposing vehicles drive. A lane (OB10) may be a concept that includes lines on the left and right sides that form a lane.

[0142] Another vehicle (OB11) may be a vehicle driving around the vehicle (100). The other vehicle may be a vehicle located within a predetermined distance from the vehicle (100). For example, the other vehicle (OB11) may be a vehicle preceding or following the vehicle (100).

[0143] A pedestrian (OB12) may be a person located around a vehicle (100). A pedestrian (OB12) may be a person located within a predetermined distance from a vehicle (100). For example, a pedestrian (OB12) may be a person located on a sidewalk or roadway.

[0144] A two-wheeled vehicle (OB12) may refer to a vehicle that is positioned around a vehicle (100) and moves using two wheels. The two-wheeled vehicle (OB12) may be a vehicle with two wheels that is positioned within a predetermined distance from the vehicle (100). For example, the two-wheeled vehicle (OB13) may be a motorcycle or bicycle positioned on a sidewalk or road.

[0145] Traffic signals may include traffic lights (OB15), traffic signs (OB14), and patterns or text painted on the road surface.

[0146] The light may be generated from a lamp installed in another vehicle. The light may be generated from a streetlight. The light may be sunlight.

[0147] A road may include slopes such as road surfaces, curves, uphill and downhill slopes, etc.

[0148] Structures may be objects located along roads and fixed to the ground. For example, structures may include streetlights, street trees, buildings, utility poles, traffic lights, and bridges.

[0149] Landforms may include mountains, hills, etc.

[0150] Meanwhile, objects can be classified into moving objects and fixed objects. For example, moving objects may include concepts such as other vehicles and pedestrians. For example, fixed objects may include concepts such as traffic signals, roads, and structures.

[0151] The object detection device (300) may include a camera (310), a radar (320), a lidar (330), an ultrasonic sensor (340), an infrared sensor (350), and a processor (370).

[0152] Depending on the embodiment, the object detection device (300) may include additional components other than the described components, or may not include some of the described components.

[0153] The camera (310) may be positioned at an appropriate location outside the vehicle to capture images of the vehicle's exterior. The camera (310) may be a mono camera, a stereo camera (310a), an AVM (Around View Monitoring) camera (310b), or a 360-degree camera.

[0154] For example, the camera (310) may be positioned inside the vehicle, close to the front windshield, to capture an image of the front of the vehicle. Alternatively, the camera (310) may be positioned around the front bumper or radiator grill.

[0155] For example, the camera (310) may be positioned inside the vehicle, close to the rear glass, to capture images of the rear of the vehicle. Alternatively, the camera (310) may be positioned around the rear bumper, trunk, or tailgate.

[0156] For example, the camera (310) may be positioned close to at least one of the side windows inside the vehicle to obtain an image of the side of the vehicle. Alternatively, the camera (310) may be positioned around a side mirror, fender, or door.

[0157] The camera (310) can provide the acquired image to the processor (370).

[0158] The radar (320) may include an electromagnetic wave transmitter and receiver. The radar (320) may be implemented in a pulse radar or continuous wave radar manner based on the principle of radio wave emission. Among continuous wave radar methods, the radar (320) may be implemented in an FMCW (Frequency Modulated Continuous Wave) manner or an FSK (Frequency Shift Keying) manner depending on the signal waveform.

[0159] The radar (320) can detect an object using electromagnetic waves, based on a TOF (Time of Flight) method or a phase-shift method, and can detect the location of the detected object, the distance to the detected object, and the relative speed.

[0160] The radar (320) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0161] The lidar (330) may include a laser transmitter and receiver. The lidar (330) may be implemented using a TOF (Time of Flight) method or a phase-shift method.

[0162] The lidar (330) can be implemented as a driven or non-driven type.

[0163] When implemented as a drive type, the lidar (330) is rotated by a motor and can detect objects around the vehicle (100).

[0164] When implemented in a non-driven manner, the lidar (330) can detect an object located within a predetermined range relative to the vehicle (100) through optical steering. The vehicle (100) can include a plurality of non-driven lidars (330).

[0165] Lidar (330) can detect an object based on a time-of-flight (TOF) method or a phase-shift method using laser light as a parameter, and can detect the position of the detected object, the distance to the detected object, and the relative speed.

[0166] The lidar (330) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.

[0167] The ultrasonic sensor (340) may include an ultrasonic transmitter and a receiver. The ultrasonic sensor (340) may detect an object based on ultrasonic waves, and may detect the location of the detected object, the distance from the detected object, and the relative speed.

[0168] The ultrasonic sensor (340) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0169] The infrared sensor (350) may include an infrared transmitter and a receiver. The infrared sensor (340) may detect an object based on infrared light, and may detect the location of the detected object, the distance to the detected object, and the relative speed.

[0170] The infrared sensor (350) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0171] The processor (370) can control the overall operation of each unit of the object detection device (300).

[0172] The processor (370) can detect and track an object based on the acquired image. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object through an image processing algorithm.

[0173] The processor (370) can detect and track an object based on the reflected electromagnetic waves that are returned when the transmitted electromagnetic waves are reflected by the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the electromagnetic waves.

[0174] The processor (370) can detect and track an object based on the reflected laser light that is reflected back by the transmitted laser beam from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the laser light.

[0175] The processor (370) can detect and track an object based on the reflected ultrasonic waves that are returned when the transmitted ultrasonic waves are reflected off the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the ultrasonic waves.

[0176] The processor (370) can detect and track an object based on the reflected infrared light that is reflected back by the transmitted infrared light from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the infrared light.

[0177] Depending on the embodiment, the object detection device (300) may include a plurality of processors (370) or may not include a processor (370). For example, each of the camera (310), radar (320), lidar (330), ultrasonic sensor (340), and infrared sensor (350) may individually include a processor.

[0178] If the object detection device (300) does not include a processor (370), the object detection device (300) can be operated under the control of the processor or control unit (170) of the device in the vehicle (100).

[0179] The object detection device (400) can be operated under the control of the control unit (170).

[0180] The communication device (400) is a device for communicating with an external device. Here, the external device may be another vehicle, a mobile terminal, or a server.

[0181] The communication device (400) may include at least one of a transmitting antenna, a receiving antenna, an RF (Radio Frequency) circuit capable of implementing various communication protocols, and an RF element to perform communication.

[0182] The communication device (400) may include a short-range communication unit (410), a location information unit (420), a V2X communication unit (430), an optical communication unit (440), a broadcast transmission / reception unit (450), and a processor (470).

[0183] Depending on the embodiment, the communication device (400) may include additional components other than the described components, or may not include some of the described components.

[0184] The short-range communication unit (410) is a unit for short-range communication. The short-range communication unit (410) can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0185] The short-range communication unit (410) can form a short-range wireless communication network (Wireless Area Network) to perform short-range communication between the vehicle (100) and at least one external device.

[0186] The location information unit (420) is a unit for obtaining location information of a vehicle (100). For example, the location information unit (420) may include a GPS (Global Positioning System) module or a DGPS (Differential Global Positioning System) module.

[0187] The V2X communication unit (430) is a unit for performing wireless communication with a server (V2I: Vehicle to Infrastructure), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). The V2X communication unit (430) may include an RF circuit capable of implementing protocols for communication with infrastructure (V2I), communication between vehicles (V2V), and communication with pedestrians (V2P).

[0188] The optical communication unit (440) is a unit for communicating with an external device via light. The optical communication unit (440) may include an optical transmission unit that converts an electrical signal into an optical signal and transmits it to the outside, and an optical reception unit that converts a received optical signal into an electrical signal.

[0189] According to an embodiment, the light transmitting unit may be formed to be integrated with a lamp included in the vehicle (100).

[0190] The broadcast transmitter / receiver (450) is a unit for receiving broadcast signals from an external broadcast management server via a broadcast channel, or transmitting broadcast signals to the broadcast management server. The broadcast channels may include satellite channels and terrestrial channels. The broadcast signals may include TV broadcast signals, radio broadcast signals, and data broadcast signals.

[0191] The processor (470) can control the overall operation of each unit of the communication device (400).

[0192] Depending on the embodiment, the communication device (400) may include a plurality of processors (470) or may not include a processor (470).

[0193] If the communication device (400) does not include a processor (470), the communication device (400) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0194] Meanwhile, the communication device (400) may implement a vehicle display device together with the user interface device (200). In this case, the vehicle display device may be referred to as a telematics device or an AVN (Audio Video Navigation) device.

[0195] The communication device (400) can be operated under the control of the control unit (170).

[0196] The driving control device (500) is a device that receives user input for driving.

[0197] In manual mode, the vehicle (100) can be driven based on signals provided by the driving control device (500).

[0198] The driving control device (500) may include a steering input device (510), an acceleration input device (530), and a brake input device (570).

[0199] The steering input device (510) can receive input for the direction of travel of the vehicle (100) from the user. The steering input device (510) is preferably formed in the form of a wheel so that steering input can be provided by rotation. Depending on the embodiment, the steering input device may be formed in the form of a touch screen, a touch pad, or a button.

[0200] The acceleration input device (530) can receive an input from a user for accelerating the vehicle (100). The brake input device (570) can receive an input from a user for decelerating the vehicle (100). The acceleration input device (530) and the brake input device (570) are preferably formed in the form of a pedal. Depending on the embodiment, the acceleration input device or the brake input device may also be formed in the form of a touch screen, a touch pad, or a button.

[0201] The driving operation device (500) can be operated under the control of the control unit (170).

[0202] The vehicle driving device (600) is a device that electrically controls the driving of various devices in the vehicle (100).

[0203] The vehicle driving device (600) may include a power train driving unit (610), a chassis driving unit (620), a door / window driving unit (630), a safety device driving unit (640), a lamp driving unit (650), and an air conditioning driving unit (660).

[0204] Depending on the embodiment, the vehicle drive device (600) may include additional components other than the described components, or may not include some of the described components.

[0205] Meanwhile, the vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0206] The power train drive unit (610) can control the operation of the power train device.

[0207] The power train drive unit (610) may include a power source drive unit (611) and a transmission drive unit (612).

[0208] The power source driving unit (611) can perform control over the power source of the vehicle (100).

[0209] For example, if a fossil fuel-based engine is the power source, the power source drive unit (610) can perform electronic control of the engine. This can control the engine output torque, etc. The power source drive unit (611) can adjust the engine output torque according to the control of the control unit (170).

[0210] For example, if an electric energy-based motor is the power source, the power source driving unit (610) can perform control over the motor. The power source driving unit (610) can adjust the rotation speed, torque, etc. of the motor according to the control of the control unit (170).

[0211] The transmission drive unit (612) can perform control over the transmission.

[0212] The transmission drive unit (612) can adjust the state of the transmission. The transmission drive unit (612) can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or parking (P).

[0213] Meanwhile, when the engine is the power source, the transmission drive unit (612) can adjust the gear engagement state in the forward (D) state.

[0214] The chassis drive unit (620) can control the operation of the chassis device.

[0215] The chassis drive unit (620) may include a steering drive unit (621), a brake drive unit (622), and a suspension drive unit (623).

[0216] The steering drive unit (621) can perform electronic control of the steering apparatus within the vehicle (100). The steering drive unit (621) can change the direction of travel of the vehicle.

[0217] The brake drive unit (622) can perform electronic control of the brake apparatus within the vehicle (100). For example, the speed of the vehicle (100) can be reduced by controlling the operation of the brakes placed on the wheels.

[0218] Meanwhile, the brake driving unit (622) can individually control each of the plurality of brakes. The brake driving unit (622) can control the braking force applied to the plurality of wheels differently.

[0219] The suspension drive unit (623) can perform electronic control of the suspension apparatus within the vehicle (100). For example, when there is a curve in the road surface, the suspension drive unit (623) can control the suspension apparatus to reduce vibration of the vehicle (100).

[0220] Meanwhile, the suspension drive unit (623) can individually control each of the plurality of suspensions.

[0221] The door / window actuator (630) can perform electronic control of a door apparatus or window apparatus in a vehicle (100).

[0222] The door / window driving unit (630) may include a door driving unit (631) and a window driving unit (632).

[0223] The door driving unit (631) can control the door device. The door driving unit (631) can control the opening and closing of a plurality of doors included in the vehicle (100). The door driving unit (631) can control the opening or closing of a trunk or tail gate. The door driving unit (631) can control the opening or closing of a sunroof.

[0224] The window driving unit (632) can perform electronic control of a window apparatus. It can control the opening or closing of a plurality of windows included in a vehicle (100).

[0225] The safety device driving unit (640) can perform electronic control of various safety devices in the vehicle (100).

[0226] The safety device drive unit (640) may include an airbag drive unit (641), a seat belt drive unit (642), and a pedestrian protection device drive unit (643).

[0227] The airbag driving unit (641) can perform electronic control of the airbag apparatus within the vehicle (100). For example, the airbag driving unit (641) can control the airbag to deploy when a danger is detected.

[0228] The seat belt drive unit (642) can perform electronic control of the seat belt apparatus within the vehicle (100). For example, the seat belt drive unit (642) can control the passenger to be secured to the seat (110FL, 110FR, 110RL, 110RR) using the seat belt when a danger is detected.

[0229] The pedestrian protection device drive unit (643) can perform electronic control of the hood lift and pedestrian airbag. For example, the pedestrian protection device drive unit (643) can control the hood lift up and the pedestrian airbag to deploy when a collision with a pedestrian is detected.

[0230] The lamp driving unit (650) can perform electronic control of various lamp apparatuses within the vehicle (100).

[0231] The air conditioning drive unit (660) can perform electronic control of the air conditioning device (air conditioner) within the vehicle (100). For example, the air conditioning drive unit (660) can control the air conditioning device to operate and supply cool air to the vehicle when the temperature inside the vehicle is high.

[0232] The vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0233] The vehicle driving device (600) can be operated under the control of the control unit (170).

[0234] The driving system (700) is a system that controls various operations of the vehicle (100). The driving system (700) can be operated in autonomous driving mode.

[0235] The driving system (700) may include a driving system (710), an exiting system (740), and a parking system (750).

[0236] Depending on the embodiment, the driving system (700) may include additional components other than the described components, or may not include some of the described components.

[0237] Meanwhile, the driving system (700) may include a processor. Each unit of the driving system (700) may individually include a processor.

[0238] Meanwhile, depending on the embodiment, if the operation system (700) is implemented in software, it may be a sub-concept of the control unit (170).

[0239] Meanwhile, according to an embodiment, the driving system (700) may be a concept including at least one of a user interface device (200), an object detection device (300), a communication device (400), a vehicle driving device (600), and a control unit (170).

[0240] The driving system (710) can drive the vehicle (100).

[0241] The driving system (710) receives navigation information from the navigation system (770) and provides a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0242] The driving system (710) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to enable driving of the vehicle (100).

[0243] The driving system (710) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0244] The exit system (740) can perform exit of a vehicle (100).

[0245] The exit system (740) receives navigation information from the navigation system (770) and provides a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0246] The exit system (740) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0247] The exit system (740) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0248] The parking system (750) can perform parking of a vehicle (100).

[0249] The parking system (750) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0250] The parking system (750) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0251] The parking system (750) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0252] A navigation system (770) can provide navigation information. The navigation information can include at least one of map information, set destination information, route information based on the set destination, information on various objects along the route, lane information, and current vehicle location information.

[0253] The navigation system (770) may include memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system (770).

[0254] According to an embodiment, the navigation system (770) may receive information from an external device through a communication device (400) and update previously stored information.

[0255] Depending on the embodiment, the navigation system (770) may be classified as a sub-component of the user interface device (200).

[0256] The sensing unit (120) can sense the status of the vehicle. The sensing unit (120) can include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight detection sensor, a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0257] The sensing unit (120) can obtain sensing signals for vehicle attitude information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle inclination information, vehicle forward / backward information, battery information, fuel information, tire information, vehicle lamp information, vehicle internal temperature information, vehicle internal humidity information, steering wheel rotation angle, vehicle external illumination, pressure applied to an accelerator pedal, pressure applied to a brake pedal, etc.

[0258] The sensing unit (120) may further include, in addition, an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0259] The vehicle interface unit (130) can serve as a conduit for various types of external devices connected to the vehicle (100). For example, the vehicle interface unit (130) may be equipped with a port capable of connecting to a mobile terminal, and may be connected to the mobile terminal through the port. In this case, the vehicle interface unit (130) can exchange data with the mobile terminal.

[0260] Meanwhile, the vehicle interface unit (130) may serve as a conduit for supplying electrical energy to a connected mobile terminal. When the mobile terminal is electrically connected to the vehicle interface unit (130), the vehicle interface unit (130) may provide the mobile terminal with electrical energy supplied from the power supply unit (190) under the control of the control unit (170).

[0261] The memory (140) is electrically connected to the control unit (170). The memory (140) can store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory (140) can be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware. The memory (140) can store various data for the overall operation of the vehicle (100), such as programs for processing or controlling the control unit (170).

[0262] Depending on the embodiment, the memory (140) may be formed integrally with the control unit (170) or implemented as a sub-component of the control unit (170).

[0263] The control unit (170) can control the overall operation of each unit within the vehicle (100). The control unit (170) can be referred to as an ECU (Electronic Control Unit).

[0264] The power supply unit (190) can supply power required for the operation of each component under the control of the control unit (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0265] One or more processors and control units (170) included in the vehicle (100) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0266] Figure 8a is a conceptual diagram for explaining the AR service platform of the present invention.

[0267] The AR service platform that provides the AR service of the present invention may be called an AR service system.

[0268] The above AR service platform may include a server (850) installed outside the vehicle, which collects and processes information required for AR service and transmits it to the vehicle, and an AR service device (800) installed in the vehicle, which provides AR service using information transmitted from the server.

[0269] The fact that the server (850) collects and processes information required for the AR service and transmits it to the vehicle may include the meaning that the server (850) collects and processes information required for the AR service and transmits it to the AR service device (800) equipped in the vehicle.

[0270] The above AR service device (800) can change the information provided as an AR service based on the situation of the vehicle.

[0271] That is, the AR service device (800) of the present invention can dynamically adjust (variably) the information and amount of information to be displayed in AR according to the situation of the vehicle, and select information to be emphasized.

[0272] In addition, the AR service platform of the present invention can control the AR service provided in the vehicle to vary depending on specific conditions such as vehicle conditions and advertisement exposure conditions.

[0273] In the case of conventional AR navigation, when displaying destinations or major POIs (Points of Interest) in AR navigation, it is difficult to reflect the latest information because it uses information stored in map data, and there are limitations in that POIs with real-time attributes such as fueling / parking cannot be provided.

[0274] On the other hand, the AR service platform of the present invention can display vehicle location information, map data, multiple sensor data, real-time POI information, advertisement / event information, etc., by integrating them in AR navigation.

[0275] For example, to display AR information, the AR service device (800) of the present invention may receive AR service information from a server based on the current location of the vehicle and navigation route / guidance information, and process the information into a form for displaying on an AR navigation screen.

[0276] For example, the AR service device (800) of the present invention can reconstruct real-time AR display information. The AR service device (800) can reconstruct service data received from a server to display on an AR navigation screen by determining the display format, size, location, exposure method, etc. of AR content by considering the driving situation (e.g., varying the POI exposure location and size according to driving speed, changing the service information exposure location according to traffic conditions, adjusting the AR Wall display location and exposure time, etc.).

[0277] In addition, the AR service device (800) of the present invention can analyze the frequency of exposure of AR display information through user feedback.

[0278] The server (850) collects user input information (input information such as touch, order, etc.) for AR service content, performs content exposure frequency analysis, and can adjust the service content exposure policy based on the information.

[0279] Through this configuration, the present invention can express various external service contents in AR navigation by integrating them, and can provide various services through POI information including real-time properties.

[0280] In addition, the present invention enables display of various forms of AR content, such as POI information, advertisements, events, and major landmark information.

[0281] In addition, a new user experience of AR navigation can be presented through an embodiment based on the UX scenario proposed in the present invention.

[0282] The present invention can provide a service platform structure and an AR information display method (UX) that dynamically adjusts the amount of information (POI data, advertisement) to be displayed in AR according to vehicle conditions and advertisement exposure conditions, a module that collects POI information and commerce service information for AR expression and processes them into a form that is easy to render in an AR engine, a module that processes specific POI information to be emphasized according to the inside / outside vehicle conditions, a module that collects vehicle situation information and applies a UX policy according to the situation, and an AR engine module that renders AR objects (Group POI, Mini POI, 3D Object, Event wall, etc.) according to the UX policy.

[0283] In addition, the present invention can provide a client module for transmitting and receiving interaction and data between the front and rear displays of a vehicle, a Service App module for exposing commerce service information linked to POI, a client module for collecting user actions for advertisements such as AR advertisement object exposure results and clicks, and a cloud module for collecting / analyzing user actions for advertisements such as AR advertisement object exposure results and clicks.

[0284] Referring to FIG. 8a, the AR service platform of the present invention may include a server (850) that is a configuration existing outside the vehicle (off-board) and an AR service device (800) that is a configuration equipped in the vehicle (on-board).

[0285] First, the server (850) may include a POI data aggregator (851), an Ads manager (852), an Ads monitoring unit (853), a Service & Ads Manager (854), a Commerce Manager (855), a DB Connector (856), and a Dashboard (857).

[0286] The POI Data Aggregator (851) can receive information required for an AR service from multiple external servers and convert / integrate it into a message format of the AR service platform.

[0287] The Ads manager (852) can manage advertising information / content and advertising campaigns (advertisement exposure conditions).

[0288] The Ads Monitoring Department (853) can collect / save ad exposure and click results.

[0289] The Service & Ads Manager (854) can insert advertising information that meets the exposure conditions into service information and provide it to the client.

[0290] The Commerce Manager (855) can collect commerce service linkage / payment information.

[0291] The database connector (856) can store / query advertising content, advertising exposure result information, and commerce payment information.

[0292] The Dashboard (857) can display real-time AR service status visualized with advertisement exposure results / payment history results.

[0293] In addition, the server (850) may further include an AR service cloud API (or data conversion unit) for converting information transmitted from the vehicle's AR service device (800) into a data format available to the server, and for converting information processed / generated by the server into a data format available to the AR service device (800).

[0294] Meanwhile, the AR service device (800) may include a client (810) including a cloud interface, a commerce app, a CID-RSE interaction manager, a policy manager, advertisement monitoring, a driving context, personalized recommendations, etc., and an AR engine (820) including a POI renderer, a display manager, a touch manager, etc.

[0295] The client (810) can receive POI information, advertisements, etc. from the server.

[0296] Additionally, the client (810) can transmit and receive order / payment information to and from the server (850) and transmit advertisement exposure results to the server (850).

[0297] The AR engine (820) can transmit data such as the number of times an AR object output as AR is touched and the number of times it is exposed to the client (810).

[0298] Additionally, the AR engine (820) can transmit and receive front / rear seat (CID, RSE) linkage data to and from the client (810), and output an AR object according to the AR display policy received from the client (810).

[0299] In addition, the AR engine (820) can determine the type, output location, POI type, output size, etc. of AR objects provided through the AR service based on data collected from a gyro sensor, camera, communication unit, navigation, ADAS (Advanced Driver Assistance Systems), GPS, etc. equipped in the vehicle.

[0300] The AR service device (800) installed in the vehicle can AR render service content to display data transmitted from a cloud server in AR on the front camera image.

[0301] Additionally, the AR service device (800) can mediate data transmission between the server and the AR engine, such as collecting advertisement posting result data from the AR engine and transmitting it to the server.

[0302] Additionally, the AR service device (800) can link data generated by AR between CID-RSE (i.e., front / rear seats).

[0303] In addition, the AR service device (800) can perform data management for AR display policies, and specifically, can provide AR display policy data according to driving conditions to the AR engine.

[0304] In addition, the AR service device (800) can provide situational awareness and personalized services, and specifically, can provide AR objects to the AR engine according to driving situations (speed, TBT (Turn-by-Turn), etc.) using in-vehicle data.

[0305] In this specification, an example of providing an AR service by overlapping and outputting AR information (or AR object, AR content, POI information, etc.) on an image captured (received, processed) by a camera installed in a vehicle will be described.

[0306] However, the AR service described in this specification is not limited thereto, and can be applied in a similar or identical manner to various methods of implementing augmented reality, such as outputting AR information directly on the windshield of a vehicle to overlap with the real world space for a driver or passenger, or outputting AR information through a HUD (Head-up Display).

[0307] The input data (input information) used to provide AR services and the output data (output information) provided through the AR service platform are as follows.

[0308] First, the types of input data may include map data (navigation information), service content information (POI, advertisement, etc.), dynamic information, vehicle sensor information, historical information, and driving-related information.

[0309] Map data (navigation information) may include information such as route information to the destination (navigation route), guidance information (turn-by-turn), road shape ahead (road / lane), multiple map attribute information (road type / attribute, road and lane width, curvature, slope, speed limit, etc.), and localization object information (road marking, traffic sign, etc.).

[0310] Service content information (POI, advertisements, etc.) may include POI information received from multiple service providers, advertisement information available at the current location, and real-time information for reservation / payment services such as gas stations, charging stations, and parking lots.

[0311] Dynamic information may include traffic information (road-level traffic, lane-level traffic), event information (accidents, hazard warnings, etc.), weather information, V2X (V2V, V2I) (Vehicle to Everything, Vehicle to Vehicle, Vehicle to Infra), etc.

[0312] Vehicle sensor information may include current location information (GPS / DR), camera input information (ADAS information, object recognition information), and V2X (real-time surrounding situation information that can be collected through V2V and V2I).

[0313] Historical information may include information about past driving routes, traffic history (e.g., traffic volume by time of day), and communication speeds by zone and time of day.

[0314] Driving-related information may include driving mode (manual, autonomous driving, semi-autonomous driving, ADAS function operation, etc.), whether the vehicle has entered a destination or waypoint, and whether the vehicle has entered a parking lot.

[0315] Output information that can be provided through the AR service platform may include AR service display data based on current location / route.

[0316] Current location / route-based AR service display data may include AR advertisement display points along the route (AR Wall, POI building highlights), selectable AR building information (selectable main building information such as landmarks), general POI information (POI summary information such as icons or speech bubbles), remote POI information (distance / direction of important POI information that does not exist on the route but is helpful for driving), display information displayed when multiple POIs exist in the same building, destination building and real-time parking lot status information, real-time status information of gas stations / charging stations, and location-based advertisement / event information.

[0317] The AR service platform of the present invention can filter AR service information according to real-time information and determine a display method.

[0318] Specifically, the AR service platform can determine the number of real-time POI exposures based on driving speed, POI overlap removal, size adjustment, exposure time, etc.

[0319] In addition, the AR service platform can determine the POI exposure method according to risk information recognition, and specifically, can dynamically change the POI display method according to situations such as accidents, construction, and recognition of multiple moving objects.

[0320] Additionally, the AR service platform can dynamically change the POI display location when AR display visibility is reduced due to traffic.

[0321] In addition, the AR service platform can reconfigure front / rear seat AR display data. For example, the AR service information can be minimized on the front seat display and the rear seat display can be reconfigured to display all displayable information by considering driving speed, hazard information, weather information, etc.

[0322] The operation / function / control method of such an AR service platform may be implemented by a server or AR service device included in the AR service platform, or by organic interaction between the server and the AR service device.

[0323] Referring to FIG. 8a, the configuration of the server (850) of the AR service platform is described in more detail as follows.

[0324] The service and advertisement manager (854) can perform a client linkage (request) function, a POI information and advertisement information integration (data processing & aggregation) function, and a client linkage (respond) function.

[0325] Specifically, the client connection (request) function may include requesting / receiving POI information (location, category) through Unified API, or requesting / receiving destination entrance location information (select one of destination coordinates / address / id) through Unified API.

[0326] Here, Unified API refers to an API defined in the AR service cloud that is independent of a specific data provider (to minimize client changes).

[0327] The POI information and advertisement information integration (data processing & aggregation) function may include integrating POI information and advertisement information within a radius of 000m from the location requested by the client (from data manager, ads manager) or integrating the entrance location of the destination requested by the client and POI advertisement information (from data manager, ads manager).

[0328] Specifically, the POI information and advertisement information integration function may include a function to align advertisement information including building wall and event wall information with POI information, or, if there are multiple POIs in the same building, to filter them by priority on the server (e.g., leaving only partner companies and excluding other POI information).

[0329] Here, the filtering criteria may include assigning priority scores to each POI and comparing them.

[0330] Client response functionality may include transmitting POI information and advertisement information via Unified API, or transmitting destination entrance location and advertisement information via Unified API.

[0331] The data manager (not shown) included in the server (850) may include a POI information collection / transmission function, a building shape (polygon) information collection / transmission function, and a destination entrance information collection / transmission function.

[0332] The POI information collection / transfer function can perform the function of requesting POI information from a 3rd party API or transmitting (to Service & Ads Aggregator) POI information received from a 3rd party API (by converting it into a Unified API response format).

[0333] The building shape (polygon) information collection / transfer function can request building exterior shape information from a 3rd party API / data set, or perform the function of transmitting (to Service & Ads Aggregator) POI information received from a 3rd party API (by converting it into a Unified API response format).

[0334] The destination entrance information collection / transfer function can request destination entrance information from a 3rd party API or transmit destination entrance information received from a 3rd party API (by converting it into a Unified API response format) (to Service & Ads Aggregator).

[0335] The Ads Manager (852) can provide a partner (advertising) company management interface, an advertising campaign management interface, and an advertising content management interface.

[0336] The Ads Monitoring Department (853) can perform the function of receiving advertising effectiveness measurement feedback and the function of transmitting advertising information.

[0337] The partner (advertiser) management interface can perform POI advertiser management (add / edit / delete advertiser information) and general advertiser management (add / delete advertiser information).

[0338] POI supported ad formats may include Brand poi pin, Building wall, 3D rendering, Event wall, etc., and the supported ad format (general supported ad format) for brand advertisements that are not related to actual POI / location (e.g. Coca-Cola advertisements) may be Event wall.

[0339] The ad campaign management interface allows you to add / edit / delete ad campaigns (ad location, type, time, etc.).

[0340] The advertising content management interface can add / edit / view / delete content by advertising format (POI brand icon image, building wall image, event wall image / video, 3D rendering image).

[0341] The advertising effectiveness measurement feedback reception function may include a function for receiving advertising exposure feedback sent by the client and transmitting it to the DB Manager (CPC / CMP / CPT&P).

[0342] The advertising information delivery function may include a function that searches for and delivers advertising campaign information that should be displayed within a radius of 000m from the location requested by the Service & Ads Aggregator (in the case of CPT&P, only deliver advertisements that meet the time conditions).

[0343] The Commerce Manager (855) can perform client linkage functions, external commerce service linkage functions, and payment information management functions.

[0344] The client linkage function may include receiving requests by linking the client with Unified API, converting the request content received with Unified API into external commerce API specifications, converting data received from the external API into the Unified API message format, and transmitting data to the client.

[0345] The Commerce Manager can convert the request content received through the Unified API into an external commerce API specification and then perform external service linkage functions using the converted content.

[0346] Converting data received from an external API into the message format of the Unified API may mean converting data received from an external service connection into the Unified API.

[0347] The external commerce service linkage function may include functions for requesting a list of nearby stores and meta information from the current location and receiving results, requesting detailed information about a specific store from the list and receiving results, requesting reservations / orders and receiving results, requesting service usage status and receiving results, and linking member information of commerce services and receiving results.

[0348] Here, the service usage status request & result reception can be used for sequence management and AR message pop-up according to the service usage status (reservation completed / parking lot entry / parking in progress / parking lot exit / reservation cancellation).

[0349] Service member information linking & result reception can be used to link commerce service member ↔ AR service member (OEM Connectivity service member) information.

[0350] The payment information management function may include a function to collect payment details (contents, amount) from external commerce services and charge fees to external commerce companies based on the payment details.

[0351] The database connector (856) can perform advertising effectiveness measurement data management functions, commerce data management functions, advertiser data management functions, advertising content data management functions, and advertising location data management functions.

[0352] The advertising effectiveness measurement data management function can save / delete CPC / CPM / CPT&P related log data and view data (by POI, brand, time, and ad type).

[0353] The commerce data management function can save / delete payment history (content, amount) from external commerce services and perform data search (by POI, brand, time, and ad type).

[0354] The advertiser data management function can save / edit / delete / view advertiser information and advertising campaign settings for each advertiser.

[0355] The advertising content data management function can save / edit / delete / view advertising content in conjunction with advertiser information.

[0356] The advertising location data management function can manage the Event wall section coordinates and Building wall coordinates (by brand) for displaying AR advertisements, and can be distinguished by coordinates directly registered by the user or specific coordinates obtained through company API linkage.

[0357] The Service Dashboard (857) can perform advertising effectiveness measurement data visualization functions and commerce service data visualization functions.

[0358] The advertising effectiveness measurement data visualization function can provide the following charts: CPC: Total number of ad clicks by company / brand (viewable by period), CPC: Aggregated chart of total number of ad clicks (viewable by period), CPM: Total number of ad impressions by company / brand (viewable by period), CPM: Aggregated chart of total number of ad impressions (viewable by period), CPT&P: Chart of number of ad clicks by company / brand (viewable by period), CPT&P: Chart of number of ad impressions by company / brand (viewable by period).

[0359] These charts can be presented in a variety of formats, including Bar graph, Line graph, Pie Chart, Word graph, and Geospatial graph.

[0360] Although CPT&P is billed per hour rather than per click or impression, it can be used as data for measuring the effectiveness of exposure.

[0361] The commerce service data visualization feature can provide a chart of the cumulative payment amount by company (viewable by period) and a chart of the total cumulative payment amount (viewable by period).

[0362] Figure 8b is a conceptual diagram illustrating an MR service platform for providing the MR service of the present invention.

[0363] The present invention can provide an MR service platform capable of providing a mixed reality automotive meta service (MR AMS) (hereinafter, used interchangeably with MR service).

[0364] The MR service platform may be named as an MR service system, an MR navigation system, an MR platform, an MR system, etc.

[0365] An MR service platform refers to a platform capable of providing services based on mixed reality, and may include multiple independent components.

[0366] For example, the MR service platform may include an MR service device (900) (or referred to as an MR navigation engine) equipped on the vehicle, an MR AMS (hereinafter referred to as an MR AMS server) (1100) equipped on a server (or cloud) outside the vehicle (offboard), and a DTaaS (Digital Twin as a Service) server (1200).

[0367] The above MR service device (900) may include an MR AMS client (910) and an MR renderer (920).

[0368] The MR service described herein can be understood as a mixed reality navigation service for vehicles. That is, the MR service platform of the present invention can provide a vehicle interface implemented in mixed reality to a user riding in a vehicle.

[0369] The MR service provided by the MR service platform of the present invention can provide an experience of the digital world through a display inside the vehicle while driving in the real world.

[0370] Specifically, MR services can interactively provide drivers with navigation, safe driving, POI (Point of Interest), and entertainment user experiences in a virtual 3D space that combines real-world information with the digital world.

[0371] Through this, the MR service platform of the present invention can provide various UX (User Experience) that are free from the spatial and temporal constraints compared to existing camera-based (or HUD (Head up Display)-based) AR (Augmented Reality).

[0372] Here, the digital world refers to a digital twin or a digital twin smart city, and real-world information may include infrastructure data such as V2X (Vehicle to Everything) and C-ITS (Cooperative-Intelligent transport Systems) and / or surrounding perception data sensed by sensors equipped in autonomous vehicles.

[0373] In addition, the fusion described above may include the concept of fusion of vehicle and infrastructure sensor data as well as the MR service cloud (1000) (or MR AMS cloud) and MR service device (900) for implementing the present MR service platform.

[0374] Additionally, interactivity should be understood as a concept that includes not only mixed reality navigation, but also high-quality 3D image rendering and user interaction.

[0375] Meanwhile, mixed reality (MR) described in this specification may mean an environment in which virtual reality is grafted onto the real world, allowing real physical objects and virtual objects to interact.

[0376] Mixed reality (MR) can include augmented reality (AR), which adds virtual information based on reality, and augmented virtuality (AV), which adds real information to a virtual environment.

[0377] In other words, it provides a smart environment where the real and virtual are seamlessly connected, allowing users to experience richer experiences. For example, users can interact with a virtual pet in the palm of their hand, create a virtual game environment within their real-world room, play games, virtually rearrange furniture in their home, or create an environment where remote people can gather and work together.

[0378] A mixed reality vehicle meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention can help users prepare road shapes and actions in advance by displaying a preview of a future driving path.

[0379] In addition, a mixed reality automobile meta service (MR AMS) using mixed reality (MR) according to an embodiment of the present invention can improve advertising effectiveness or induce increased service usage by focusing on a specific POI (Point of Interest).

[0380] In addition, the mixed reality automobile meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention is not dependent on a specific map company and can also merge data from various map companies.

[0381] The present invention can provide an MR navigation function as one of the mixed reality automotive meta services.

[0382] In the MR navigation function, since it is implemented in the virtual world rather than overlapping augmented reality objects on the real world, it can solve difficulties such as forward occlusion problems and achieving matching quality that occur in AR navigation.

[0383] Accordingly, the present invention can improve user UX by effectively expressing various contexts that were difficult to express in existing navigation through the MR navigation function.

[0384] To this end, the MR service platform of the present invention can provide a method for managing an MR context, and a method and system for obtaining metadata and 3D assets for providing an MR AMS.

[0385] The MR service platform of the present invention can render related service metadata and 3D assets to MR navigation by defining the MR context required in the digital world and modeling service metadata for obtaining the same.

[0386] Accordingly, the present invention provides a digital world experience through a display within a vehicle while driving in the real world, and can recommend and provide various additional HMI (Human Machine Interface) services to the user by utilizing the improved visibility and visibility of MR navigation.

[0387] Hereinafter, an MR service platform according to an embodiment of the present invention for providing the MR service described above will be described.

[0388] Referring to FIG. 8b, the MR service platform (or MR service system) of the present invention may include an MR service cloud (1000) provided outside the vehicle and an MR service device (900) provided in the vehicle.

[0389] The MR service cloud (1100) installed outside the vehicle (Offboard) may include at least one of an MR AMS (Mixed Reality Automotive Meta Service) server (1100) and a DTaaS (Digital Twin as a Service) server (1200).

[0390] The MR service device (900) equipped on the above vehicle may include an MR AMS client (910) and an MR renderer (920).

[0391] The above MR service device (900) can provide a navigation function (or navigation application) by interacting with the AR service device (800) and the navigation system (770).

[0392] Information required for the above navigation function can be received through user input (or user request) input through a camera (310), sensing unit (120), and user input unit (210) provided in the vehicle.

[0393] In addition, information required for the above navigation function can be received through an MR service cloud (1000) provided offboard the vehicle, and in order to receive the information required for each vehicle, information sensed or processed in the vehicle can be transmitted to the MR service cloud (1000).

[0394] The MR AMS (Mixed Reality Automotive Meta Service) server (1100) can be connected to various service providers (1300a, 1300b, 1300c) that provide online map services such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP, as illustrated in FIG. 8d. Then, based on the results of aggregating map data provided from the connected service providers (1300a, 1300b, 1300c), the server can aggregate shape information (e.g., building floor coordinates (footprint) information and height information) of each building included in the map and provide the aggregated information to the DTaaS server (1200). Here, the DTaaS server (1200) may refer to a server or device that provides a service using DTaaS, i.e., a digital twin map.

[0395] The above DTaaS may mean Digital Twin as a Service or Digital Transformation as a Service.

[0396] Meanwhile, the DTaaS server (1200) may be connected to a POI database in which POI service data for each building or each area included in the map data is stored. In addition, the DTaaS server (1200) may be connected to a 3D model database in which data of a 3D polygon model (or a 3D polygon map) for each building included in the map data is stored. Here, the 3D polygon model is a polygon model that can provide a building volume and may be a polygon model in which no texture is formed on the surface. The DTaaS server (1200) may receive service data related to POI from the connected POI database, and may receive data of 3D polygon models of each building included in the map data of an area from the connected 3D model database.

[0397] The processor (not shown) of the MR service device (900) can receive various information related to the driving of the vehicle from the object detection device (300), the sensing unit (120), and the navigation system (770). For example, the processor can receive information on an object detected in front, behind, or to the side of the vehicle from the camera (310) of the object detection device (300).

[0398] Additionally, the processor may receive information related to the vehicle's speed, driving direction, current vehicle location (GPS), etc. from a sensing unit (120) including sensors connected to each component of the vehicle, including the driving system (710). Additionally, the processor may receive information related to the vehicle's driving path from a navigation system (770).

[0399] Meanwhile, the MR service device (900) and the DTaaS server (1200) may be connected via the interface (Interface APIs) of the MR AMS. Here, the connection between the MR service device (900) and the interface (Interface APIs) of the MR AMS may be established via a wireless network connection. In this case, the MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the MR service device (900).

[0400] In this way, when connected to the MR AMS server (1100), the MR service device (900) can provide at least some of the information received from the connected components (e.g., the vehicle's camera (310), the sensing unit (120), or the user input input through the user input unit (210)) to the MR AMS server (1100) via a network connection. Then, the MR AMS server (1100) can provide 3D map data for providing mixed reality to the MR service device (900) in response to the provided information.

[0401] For example, the MR service device (900) can transmit information about objects detected around the vehicle, as well as information about the vehicle's speed, direction, and current location. In addition, it can provide information about the driving path to the MR AMS server (1100). Then, the MR AMS server (1100) can provide 3D map data of an area according to the current location of the vehicle to the MR service device (900) based on the information provided by the MR service device (900).

[0402] In this case, the MR AMS server (1100) may determine POI information based on the current vehicle location, vehicle speed, and vehicle driving path, and may provide 3D map data that further includes the determined POI information to the 3D building map. In addition, the MR AMS server (1100) may provide 3D map data that further includes information on the situation around the vehicle to the MR service device (900) based on the information on objects around the vehicle provided.

[0403] Meanwhile, the MR service device (900) can render a mixed reality image based on the 3D map data provided from the MR AMS server (1100). For example, the MR service device (900) can control the MR renderer (920) to display a 3D map screen including models of buildings around the vehicle based on the provided 3D map data. In addition, the MR renderer (920) can display a graphic object corresponding to the vehicle on the 3D map screen, and display graphic objects corresponding to the provided POI data and situation information around the vehicle on the 3D map screen.

[0404] Accordingly, an image of a virtual environment (in other words, a mixed reality (MR) image, or an MR navigation screen, or an MR navigation interface) including a three-dimensional building model similar to the shape of the current vehicle and buildings around the vehicle and graphic objects corresponding to the vehicle can be output to a display (251) equipped in the vehicle, for example, a Center Information Display (CID), a Head Up Display (HUD), a Rear Sheet Information (RSI), or a Rear Sheet Entertainment (RSE).

[0405] In this case, information related to the vehicle's driving and the surroundings of the vehicle can be provided to the driver through the virtual environment. Through the 3D map data, i.e., the digital twin map (hereinafter referred to as the "DT map"), the MR service device (900) according to an embodiment of the present invention can provide the driver with a mixed reality service.

[0406] Meanwhile, the MR AMS server (1100) can determine 3D map data and POI information or situation information around each vehicle that can be provided together with the 3D map data based on information collected from not only the MR service device (900) equipped in one vehicle but also the MR service devices (900) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect from multiple vehicles in the form of a cloud server and generate 3D map data for mixed reality based on the collected information. In addition, it can be implemented to provide mixed reality service to at least one MR service device (900) equipped in different vehicles based on the generated 3D map data.

[0407] Therefore, for convenience of explanation, in the following, a cloud or server that includes the MR AMS server (1100) and the DTaaS server (1200), and provides information such as metadata (e.g., service metadata, 3D assets), 3D polygon maps, and digital twin maps for providing mixed reality services, i.e., a digital twin map (DT map), is referred to as an MR service cloud (1000).

[0408] As illustrated in FIG. 8b, the MR service device (900) (or MR navigation engine) may include an MR AMS client (910) and an MR renderer (920).

[0409] In addition, in order to implement the MR navigation function, which is one of the MR services, the MR service device (900) can transmit and receive data with the AR service device (800) (or AR engine) and the navigation system (770) equipped in the vehicle.

[0410] The MR AMS client (910) may include a context manager (911), a scene manager (913), and a UX scenario database (914).

[0411] Additionally, the MR renderer (920) may include a DTaaS client (921), an MR visualization unit (922), and a 3D HMI framework (923).

[0412] The MR AMS client (910) can collect vehicle location information, user input, user feedback information, payment information, etc. and transmit them to the MR AMS server (1100) located outside the vehicle.

[0413] The MR AMS server (1100) can transmit at least one of metadata, service metadata, and 3D assets required to provide MR service to the MR AMS client (910) based on information received from the MR AMS client.

[0414] The MR AMS client (910) can transmit data received from the MR AMS server (910) to the MR renderer (920).

[0415] The MR renderer (920) can create a digital twin map using a 3D polygon map received from a DTaaS server (1200) and an image received from an MR AMS client (910) or a camera (310) installed in a vehicle.

[0416] Additionally, the MR renderer (920) can render data received from the MR AMS client (920) into an MR object that can be overlapped on a digital twin map, and can create a mixed reality (MR) image by overlapping the rendered MR object on the digital twin map.

[0417] Thereafter, the MR renderer (920) can output the generated mixed reality image to a display (251) equipped in the vehicle.

[0418] All components described in this specification can be implemented as separate hardware modules, and can be understood as components implemented in software block units as needed.

[0419] Below, each component that constitutes the MR service platform will be described in more detail with reference to the attached drawings.

[0420] Figure 8c is a conceptual diagram for explaining the MR AMS client of the present invention.

[0421] The MR AMS client (910) is equipped in a vehicle and can provide a mixed reality automotive meta service (MR AMS).

[0422] The MR AMS client (910) may include a context manager (911) that requests a context corresponding to a user request (or user input) from an MR AMS server (1100) installed outside the vehicle, a scene manager (913) that manages MR scene information provided to a display (251) installed in the vehicle, and a UX scenario database (914) that provides a UX rule to at least one of the context manager (911) and the scene manager (913).

[0423] Additionally, the MR AMS client (910) may further include an interface API (912) that calls a function for communicating with the MR AMS server (1100) provided outside the vehicle.

[0424] The above interface API (912) may be configured with one or more functions configured to perform communication with the MR AMS server (1100), and may use these functions to convert a data format or message format to transmit data to the MR AMS server (1100), or to convert the format of data received from the MR AMS server (1100).

[0425] The interface API (921) can transmit a context request output from the context manager (911) to the MR AMS server (1100) and receive a 3D asset corresponding to the requested context from the MR AMS server (912).

[0426] Here, the context may refer to situational information and information corresponding to the situation the vehicle is in. Furthermore, the context may also include the meaning of content.

[0427] The above 3D asset may refer to 3D object data corresponding to the requested context. Furthermore, the 3D asset may refer to a 3D graphic object that overlaps or can be updated on a digital twin image (or digital twin map).

[0428] The MR AMS client (910) may be included in the MR service device (900).

[0429] The MR service device (900) may include a user interaction handler (901) that, when a user input is received through an input unit (210) provided in the vehicle, generates an action corresponding to the user input and transmits the action to the context manager.

[0430] The above user interaction handler (901) may be included in the MR service device (900) or may be included in the MR AMS client (910).

[0431] For example, when a user input of “Find nearby Starbucks” is received through the vehicle input unit (210), the user interaction handler (901) can generate an action corresponding to the user input (e.g., “Search POI”) and transmit it to the context manager (911) provided in the MR AMS client (910).

[0432] For example, the action may be determined by a motion matching a word included in the user input, and the action may be named a command or control command.

[0433] The above context manager (911) can generate a command to request a context corresponding to an action received from a user interaction handler (901) and transmit the command to the MR AMS server (1100) via an interface API (912).

[0434] The above command may be generated based on an action (e.g., “Search POI”) received from the user interaction handler (901), and may be formed to include, for example, the current vehicle location, the type of POI to be found, and radius information (e.g., GET “Starbucks” (type of POI) WITHIN “500m” (radius) FROM “37.7795, -122.4201” (current vehicle location (latitude, longitude)).

[0435] The context manager (911) can receive current scene information (current scene) currently being output from the vehicle from the scene manager (913) and receive UX rules from the UX scenario database (914).

[0436] Additionally, the context manager (911) can receive navigation information including the current route and current location from the navigation handler (902) that handles information of the navigation system (770).

[0437] The navigation handler (902) may be provided in the MR service device (900) or may be provided in the MR AMS client (910).

[0438] The context manager (911) can generate a command to request the context based on at least one of the current scene information, the UX rule, and the navigation information.

[0439] The current scene information may include screen information currently being output on the vehicle's display (251). For example, the current scene information may include information about a mixed reality image in which an MR object and an MR interface overlap on a digital twin map.

[0440] Additionally, at least one of the context manager (911) and the scene manager (913) of the present invention can receive sensor data processed through a sensor data adapter (903) that processes information sensed through the vehicle's sensing unit (120).

[0441] The above sensor data adapter (903) may be provided in the MR service device (900) or may be provided in the MR AMS client (910). The sensor data adapter (903) may transmit processed sensor data to an AR engine handler (904) that handles data transmitted to the AR engine (or AR service device) (800).

[0442] The interface API (912) can receive metadata of a context corresponding to the command and / or a 3D asset corresponding to the context from the MR AMS server (1100).

[0443] Thereafter, the interface API (912) can transmit the received metadata and / or 3D assets to the scene manager (913).

[0444] The scene manager (913) can generate UI data using the UX rules received from the UX scenario database (914) and the metadata and 3D assets received from the interface API (912).

[0445] Thereafter, the scene manager (913) can transmit the generated UI data to the MR renderer (920) that renders the generated UI data to be output as mixed reality (MR) or a mixed reality image on the display (251) provided in the vehicle.

[0446] Additionally, the scene manager (913) can further transmit the generated UI data to an AR engine handler (904) configured to handle an AR service device (800) equipped in the vehicle.

[0447] The UX rule stored in the UX scenario database (914) may refer to information about the rules, forms, formats or templates for creating a screen, UX or user interface provided by the MR service device, and these UX rules may be predefined for each type of data.

[0448] Additionally, UX rules can be updated or modified by users or administrators.

[0449] Figure 8d is a conceptual diagram for explaining the MR AMS server of the present invention.

[0450] Referring to FIG. 8d, an MR AMS server (1100) provided offboard in a vehicle and providing a Mixed Reality Automotive Meta Service (MR AMS) may include an interface API (1101) for calling a function for communicating with an MR AMS client provided in the vehicle, a service aggregation manager (1110) for requesting and receiving a context corresponding to a request received from the MR AMS client from a service provider, and a data integration manager (1120) for loading a 3D asset corresponding to the received context from a database (3D Assets for MR Navigation Database) (1130).

[0451] The above interface API (1101) may be named a server interface API (1101) to distinguish it from the interface API (912) of the MR AMS client (910) equipped in the vehicle.

[0452] Additionally, the interface API (912) of the MR AMS client (910) may be named a vehicle interface API or an MR AMS client interface API.

[0453] The interface API (1101) provided in the MR AMS server (1100) can transmit a user request (or context request) received from the MR AMS client to the service aggregation manager (1110).

[0454] The above interface API may include a first interface API (1101) that calls a function for performing communication with the MR AMS client (910) and a second interface API (1102a, 1102b, 1102c) that calls a function for performing communication with the service aggregation manager (1110) and the service provider (1300a, 1300b, 1300c).

[0455] The above second interface API (1102a, 1102b, 1102c) can receive service data and / or map data through the interface API provided to the service provider (1300a, 1300b, 1300c).

[0456] The interface APIs provided in the second interface API (1102a, 1102b, 1102c) and the service provider (1300a, 1300b, 1300c) may include functions configured to perform mutual data transmission and reception and to convert data formats or message formats, and these functions may be used to convert data formats or message formats to enable mutual data transmission and reception.

[0457] The service aggregation manager (1110) can request the requested context from different service providers based on the type of context requested from the MR AMS client (910) equipped in the vehicle.

[0458] Specifically, the service aggregation manager (1110) may, if the type of the requested context is a first type of context, request the first type of context from a first service provider (1300a) that provides the first type of context, and if the type of the requested context is a second type of context different from the first type, request the second type of context from a second service provider (1300b) that provides the second type of context.

[0459] For example, if the type of requested context is about a POI (e.g., “Starbucks”), the service aggregation manager (1110) can request and receive context (or POI data) about the POI from the first service provider (1300a) that provides information about the POI.

[0460] Additionally, if the type of the requested context is a view of a street, the service aggregation manager (1110) can request and receive context (or imagery data) for the view of a street from a second service provider (1300b) that provides information about the view of the street.

[0461] Additionally, if the type of the requested context is a service, the service aggregation manager (1110) may request and receive context related to the service (or data about the service (e.g., service ratings or prices)) from a third-party service provider (1300c) that provides information related to the service.

[0462] Additionally, the interface API (1101) can request a confirmed service API (Expand service API calls) to the service aggregation manager (1110) based on a service (or context request) requested from the MR AMS client (910).

[0463] The service aggregation manager (1110) can request and receive information corresponding to a confirmed service from a service provider (1300a, 1300b, 1300c) based on the extended service API request, and use the information to create a service API and output it to the data integration manager (1120).

[0464] The above data integration manager (1120) can perform data enhancement based on the service API received from the service aggregation manager (1110), generate a metadata package for the requested context, and transmit it to the vehicle's MR AMS client (910) through the interface API (1101).

[0465] The above metadata package may include the 3D asset and service metadata described above. Here, the service metadata may refer to metadata for providing a service corresponding to the requested context.

[0466] Meanwhile, the interface API (1101) can transmit a 3D asset loaded from the data integration manager (1120) to the MR AMS client (910).

[0467] Meanwhile, the MR AMS server (1100) of the present invention may further include the context manager (911) described above.

[0468] That is, the context manager (911) may be included in the MR AMS client (910) and installed in the vehicle unit, may be included in the MR AMS server (1100) and installed in the server (cloud) unit, or may be installed in both units.

[0469] The above context manager (911), when equipped in the MR AMS server (1100), can be configured to manage a context corresponding to a request received from the MR AMS client (910).

[0470] The above context manager (911) may include a context handler (911a) that handles and parses a context request, a context interpreter (911b) that manages a session for interpreting a context request and creates a context set using a data model, and a context graph database (Context Graph DB or MR Context DB) (911c) that stores the data model.

[0471] Here, the context handler (911a) can receive a user request input to the MR AMS client through the interface API (1101), parse the received user request, and transmit it to the context interpreter (911b).

[0472] The above context interpreter (911b) can create a query for a context request corresponding to the user request after creating a session, and request and receive a context data model corresponding to the query from the context graph database (911c).

[0473] The context interpreter (911b) requests a context corresponding to the context data model from the service aggregation manager (1110), and the service aggregation manager (1110) can request and receive context data corresponding to the context data model from the service provider (1300a, 1300b, 1300c).

[0474] The above service aggregation manager (1110) can request and receive a 3D asset (and / or service metadata) corresponding to the requested context from the data integration manager (1120), and transmit the context data received from the service provider and the 3D asset (and / or service metadata) received from the data integration manager to the context interpreter (911b).

[0475] The context interpreter (911b) can transmit the received context data and the 3D asset to the MR AMS client (910) equipped in the vehicle through the context handler (911a) and the interface API (1101).

[0476] Meanwhile, the context manager (911) may further include a context recorder (911d) that extracts a recommended context based on the above-mentioned generated context set, and a context controller (911e) (or context tracker) that manages contexts that must be periodically acquired.

[0477] The above context recorder (911d) may request the context interpreter (911b) to generate a query to recommend a service that can replace the specific service when the completed context data contains information that a specific service is unavailable.

[0478] Figure 9 is a conceptual diagram for explaining the DTaaS server of the present invention.

[0479] Referring to FIG. 9, the DTaaS (Digital Twin as a Service, or Digital Transformation as a Service) server (1200) of the present invention is installed outside a vehicle and can provide a mixed reality automotive meta service (MR AMS). Specifically, the DTaaS server (1200) can provide a digital twin map or data necessary for creating a digital twin map (e.g., all types of information regarding a 3D polygon map or an object overlapping on a digital twin).

[0480] The DTaaS server (1200) may include a DTaaS API (1210) that calls a function for communicating with an MR service device (900) equipped in a vehicle, a database (Digital Twins Maps DB) (1220) that stores a digital twin map and a renderable three-dimensional polygon map provided to the MR service device, and a processor (1280) that transmits a three-dimensional polygon map corresponding to the location information of the vehicle received from the MR service device to the MR service device through the DTaaS API.

[0481] In addition, the DTaaS server (1200) may further include a communication unit (TeleCommunicatino Unit, TCU) (1290) that performs communication with the MR AMS server (1100) that is installed outside the vehicle and provides MR AMS service.

[0482] In addition, the DTaaS server (1200) may further include a digital twin map generation unit (Digital Twin Representation and Update Unit) (1230) that creates a digital twin map by matching an actual captured image to a 3D polygon map stored in the database (1220).

[0483] In addition, the DTaaS server (1200) may further include a dynamic model database (Dynamics Modeling DB) (1240) that stores dynamic information about moving objects received from at least one of the MR service device (900) and the MR AMS server (1100) and a scenario database (Scenarios DB) (1250) that stores information related to scenarios that can be implemented in a digital twin.

[0484] In addition, the DTaaS server (1200) may further include a simulation unit (1260) that performs a simulation corresponding to a user request on the digital twin and a visualization unit (1270) that visualizes information to be implemented on the digital twin.

[0485] All of the components described above can be implemented as independent hardware (e.g., chips or modules), and can also be implemented as software-blocked components as needed.

[0486] The above DTaaS server (1200) can transmit and receive data through the DTaaS API (1210) not only with the vehicle (100), but also with a server (FMS Server) (1280) that provides a fleet management service (or vehicle fleet management service) and a server (1290) that provides a city planning service.

[0487] For example, the DTaaS server (1200) can collect log information collected from each server from at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290).

[0488] Afterwards, the DTaaS server (1200) can store the collected log information in a log database.

[0489] The DTaaS server (1200) can provide a digital twin map for visualization in at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290) based on the collected log information.

[0490] Additionally, the DTaaS server (1200) may transmit at least one of event notification information, simulation information, and visualization information to at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290) based on the received log information.

[0491] FIG. 10 is a block diagram illustrating the structure of a 3D map view generation device (1300) connected to a cloud server (1350) according to an embodiment of the present invention.

[0492] Referring to FIG. 10, the MR AMS (Mixed Reality Automotive Meta Service) server (1100) described in FIG. 8d can be connected to various service providers (1351) that provide online map services, such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP. Furthermore, based on the results of aggregating map data provided from the connected service providers (1351), shape information of each building included in the map, for example, floor coordinate (footprint) information and height information of a building, can be aggregated and the aggregated information can be provided to DTaaS (1352). Here, DTaaS (1352) may refer to a DTaaS server (1200), i.e., a server or device that provides a service using a digital twin map.

[0493] Meanwhile, DTaaS (1352) may be connected to a POI database in which POI service data for each building or each area included in the map data is stored. In addition, it may be connected to a 3D model database in which data of a 2.5D polygon model for each building included in the map data is stored. Here, the 2.5D polygon model is a polygon model that can provide a building volume and may be a polygon model without a texture formed on the surface. The DTaaS (1352) may receive service data related to POI from the connected POI database, and may receive data of 2.5D polygon models of each building included in the map data of an area from the connected 3D model database.

[0494] The processor (1330) of the 3D map view generation device (1300) can receive various information related to the driving of the vehicle from the camera (310), the sensing unit (120), and the navigation system (770). For example, the processor (1330) can receive information on objects detected in front, behind, or to the side of the vehicle from the camera (310). In addition, the processor (1330) can receive information related to the speed, driving direction, and current location (GPS) of the vehicle from the sensing unit (120) including sensors connected to each component of the vehicle including the driving system (710). In addition, the processor (1330) can receive information related to the driving path of the vehicle from the navigation system (770).

[0495] Meanwhile, the 3D map view creation device (1300) and DTaaS (1352) may be connected through the interface (Interface APIs) of the MR AMS server (1100). In this case, the 3D map view creation device (1300) may be configured to correspond to the MR AMS client (910).

[0496] Here, the interface (Interface APIs) of the 3D map view creation device (1300) and the MR AMS server (1100) can be connected via a wireless network connection. In this case, the MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the 3D map view creation device (1300).

[0497] In this way, when connected to the MR AMS server (1100), the 3D map view generation device (1300) can provide at least some of the information received from the connected components to the MR AMS server (1100) via a network connection. Then, the MR AMS server (1100) can provide 3D map data for providing mixed reality to the 3D map view generation device (1300) in response to the provided information.

[0498] The 3D map data provided here is map data including a polygon model provided in the 3D model DB, and may include 2.5D map data providing a simple polygon model or 3D map data providing a three-dimensional virtual object.

[0499] For example, the 3D map view generation device (1300) can provide information on objects detected around the vehicle, information on the speed and direction of the vehicle, information on the current location of the vehicle, and information on the driving path of the vehicle to the MR AMS server (1100). Then, the MR AMS server (1100) can provide 2.5D map data of an area according to the current location of the vehicle to the 3D map view generation device (1300) based on the information provided from the 3D map view generation device (1300).

[0500] In this case, the MR AMS server (1100) may determine POI information based on the current vehicle location, direction, speed, and driving path, and may provide 3D map data that further includes the determined POI information to the 3D building map. In addition, the MR AMS server (1100) may provide 3D map data that further includes information on the situation around the vehicle to the 3D map view generation device (1300) based on the information on objects around the vehicle provided.

[0501] Meanwhile, the 3D map view generation device (1300) can render a mixed reality image based on the 3D map data provided from the MR AMS server (1100). For example, the 3D map view generation device (1300) can control an MR renderer to display a 3D map screen including models of buildings around the vehicle based on the provided 3D map data. In addition, the device can display a graphic object corresponding to the vehicle on the 3D map screen, and display graphic objects corresponding to the provided POI data and situation information around the vehicle on the 3D map screen.

[0502] Accordingly, an image of a virtual environment including a three-dimensional building model similar to the shape of the current vehicle and the buildings around the vehicle and graphic objects corresponding to the vehicle can be output to a display unit (251) such as a CID (Center Information Display), HUD (Head Up Display), RSI (Rear Sheet Information), or RSE (Rear Sheet Entertainment).

[0503] In this case, information related to the vehicle's driving and the surroundings of the vehicle can be provided to the driver through the virtual environment. Through the 3D map data, i.e., the digital twin map (hereinafter referred to as the "DT map"), the 3D map view generation device (1300) according to an embodiment of the present invention can provide the driver with a mixed reality service.

[0504] Meanwhile, the MR AMS server (1100) can determine 3D map data and POI information or situation information around each vehicle that can be provided together with the 3D map data based on information collected from not only a 3D map view generation device (1300) equipped in one vehicle but also 3D map view generation devices (1300) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect information from multiple vehicles in the form of a cloud server and generate 3D map data for mixed reality based on the collected information. In addition, it can be implemented to transmit MR information for providing a mixed reality service to at least one 3D map view generation device (1300) equipped in different vehicles based on the generated 3D map data.

[0505] For convenience of explanation, in the following, the MR AMS server (1100) that provides 3D map data, i.e., a digital twin map (DT map), for providing a mixed reality service by being connected to the DTaaS (1352) and the DTaaS (1352) are collectively referred to as a cloud server (1350).

[0506] Meanwhile, the 3D map view generation device (1300) may be a device that controls a display equipped in a vehicle through an interface unit. Alternatively, the 3D map view generation device (1300) may of course be a display device equipped in the vehicle. In this case, the 3D map view generation device (1300), that is, the 3D map view generation device (1300), may be equipped with a display unit, and may receive the updated DT map on which the real-world texturing has been performed from a cloud server, and directly display an MR view image including the received DT map on the display unit equipped in the 3D map view generation device (1300).

[0507] In the following description, for convenience of explanation, it is referred to as a 3D map view generation device (1300).

[0508] Meanwhile, a vehicle (100) related to the present invention may include a 3D map view generation device (1300).

[0509] The 3D map view generation device (1300) is capable of controlling at least one of the components described in FIG. 7. From this perspective, the 3D map view generation device (1300) may be a control unit (170).

[0510] Without being limited thereto, the 3D map view generation device (1300) may be a separate configuration independent from the control unit (170). When the 3D map view generation device (1300) is implemented as a component independent from the control unit (170), the 3D map view generation device (1300) may be provided in a part of the vehicle (100).

[0511] Hereinafter, for convenience of explanation, the 3D map view generation device (1300) will be described as a separate component independent from the control unit (170). The functions (operations) and control methods described for the 3D map view generation device (1300) in this specification can be performed by the control unit (170) of the vehicle. That is, all contents described in relation to the 3D map view generation device (1300) can be analogically applied to the control unit (170) in the same / similar manner.

[0512] In addition, the 3D map view generation device (1300) described in this specification may include some of the components described in FIG. 7 and various components provided in the vehicle. In this specification, for convenience of explanation, the components described in FIG. 7 and various components provided in the vehicle will be described by assigning separate names and drawing numbers.

[0513] Figure 11 is a conceptual diagram for explaining the above 3D map view generation device (1300).

[0514] A 3D map view generation device (1300) according to one embodiment of the present invention may include a communication unit (1310), an interface unit (1320), a memory (1340), and a processor (1330).

[0515] The above communication unit (1310) may be configured to perform wireless communication with at least one of the electrical components equipped in the vehicle (for example, the electrical components equipped in the vehicle as shown in FIG. 7).

[0516] In addition, the communication unit (1310) may be configured to perform communication with devices other than the vehicle, such as a mobile terminal, a server, another vehicle, infrastructure installed on the road, etc.

[0517] The above communication unit (1310) may be the communication device (400) described above, and may include at least one of the components included in the communication device (400).

[0518] The interface unit (1320) can communicate with at least one of the components provided in the vehicle.

[0519] Specifically, the interface unit (1320) can be configured to perform wired communication with at least one of the components provided in the vehicle as shown in FIG. 7.

[0520] The above interface unit (1320) receives sensing information from one or more sensors provided in the vehicle (100).

[0521] The above interface unit (1320) may be referred to as a sensor data collector.

[0522] The above interface unit (1320) collects (receives) information sensed through sensors provided in the vehicle (e.g., sensors for detecting vehicle operation (V.Sensors) (e.g., heading, throttle, break, wheel, etc.) and sensors for sensing vehicle surrounding information (S.Sensors) (e.g., Camera, Radar, LiDAR, Sonar, etc.)).

[0523] The above interface unit (1320) can transmit information sensed through a sensor equipped in the vehicle to the communication unit (1310) (or processor (1330)) so that it is reflected in a high-precision map.

[0524] The interface unit (1320) may, for example, serve as a passageway with electrical components installed in the vehicle through the vehicle interface unit (130).

[0525] The interface unit (1320) can exchange data with the vehicle interface unit (130).

[0526] The interface unit (1320) can be connected to a vehicle and serve as a passage for supplying electric energy.

[0527] For example, the route guidance device can be powered on by receiving electric energy from the vehicle's power supply unit (190) through the interface unit (1320).

[0528] Meanwhile, the present invention may include a memory (1340) that stores data supporting various functions of the 3D map view creation device (1300). For example, the memory (1340) may store a plurality of application programs (or applications) that can be executed by the processor (1330), data for the operation of the 3D map view creation device (1300), and commands.

[0529] As an example of such data, the memory (1340) may store building reference information, which is information on actual buildings corresponding to each of the building virtual objects included in the map data. The building reference information may store information on the height of the actual building, planar shape information (e.g., footprint information) which is the shape of the land occupied by the actual building, and information on the actual location of each building, i.e., the latitude and longitude of each building. In addition, the building reference information may include information on the shape of each side of each building or various three-dimensional shapes of the building. In addition, the building reference information may include information related to the use, type, and age of each building corresponding to each building virtual object.

[0530] The above memory (1340) can store building reference information corresponding to at least one of the building virtual objects included in the map data. Hereinafter, an area of ​​the memory (1340) where the building reference information is stored will be referred to as a building reference information storage unit (1343).

[0531] In addition, the memory (1340) may store building image setting values ​​including setting values ​​for angles and projection directions for generating building virtual objects, modeling setting values ​​for 2.5D building virtual objects formed of polygons, or modeling setting values ​​for 3D building virtual objects having a three-dimensional shape, as information for modeling the building virtual objects. In this way, based on the building reference information and building image setting values ​​stored in the memory (1340), the processor (1330) may calculate various building setting values ​​such as height, area, size, area of ​​each side, margin value, and offset value of the building virtual object to be generated, and may generate each building virtual object based on the calculated building setting values. Here, the building setting value may include a value related to at least one of the height, location, planar shape, three-dimensional shape, and shape of each side of the building included in the building reference information.

[0532] In addition, the memory (1340) can further store data for generating surface materials, i.e., surface textures, of each building virtual object. For example, the memory (1340) can store images of a portion of a building including shapes of various building elements. The images are images for forming the surface texture of the building (hereinafter, texture images) and can be formed as 2D images. In addition, the memory (1340) can store data of images (hereinafter, normal maps) corresponding to normal maps having different vector values ​​according to different materials so as to provide texture to the texture images. In addition, images (hereinafter, albedo maps) corresponding to albedo maps having material-specific color values ​​for each different material can be stored.

[0533] Meanwhile, the memory (1340) may store 2D images including shapes corresponding to building components. The 2D image may be an image including images corresponding to at least one building element for describing the texture of the building. In addition, each 2D image may include different building elements depending on the type, and the shapes of the included building elements may also be different. The 2D image may be a texture image that can form a surface texture (material) of the building virtual object in a form that is tiled on the surface of the building virtual object.

[0534] Meanwhile, the texture image may be synthesized with the outer surface of the building virtual object in a tiled form as described above. Here, the tiling may refer to a process of filling the surface area of ​​the building virtual object with tiles normalized to a certain size in a horizontal and vertical direction without overlapping each other. Accordingly, the texture image may refer to a tile, which is a unit image synthesized on the outer surface of the building virtual object through the tiling. Therefore, in the following description, a texture image texturing the outer surface of the building virtual object in a tiling manner may also be referred to as a tile. Here, the texturing may refer to an operation process of adding a texture representing a surface texture (material) to the surface area of ​​the building virtual object and synthesizing a texture on the surface area of ​​the building virtual object using a tile.

[0535] Additionally, the surface area of ​​the virtual object may be composed of a plurality of different sub-areas. In this case, each of the plurality of sub-areas may be matched with a different tile, and each of the plurality of areas may be tiled using different matching tiles.

[0536] Meanwhile, the above tiles may be grouped into different groups according to the characteristics of the building to which the tiles can be applied, such as the type, purpose, or size of the building. For example, the tiles may be grouped into residential buildings, commercial buildings, or officetel buildings. In this case, the tiles grouped into different groups, i.e., the tiles grouped into the residential building group, the tiles grouped into the commercial building group, and the tiles grouped into the officetel building group, may be tiles normalized to different sizes according to each group.

[0537] In addition, some of the tiles may be associated with at least one other tile. For example, the associated tiles may be tiles that are similar in shape and color. That is, the associated tiles may be tiles that differ only in at least a portion of their shape or in color. Alternatively, the associated tiles may be tiles that are similar to a certain degree in at least one of their shape and color. These associated tiles may reflect changes over time. For example, tiles that have the same shape but differ only in color may be used to reflect changes over time in a portion of a specific building.

[0538] Meanwhile, memory (1340) may contain association information regarding tiles that are interconnected in this manner. Furthermore, through association information, a specific tile can be associated with another tile. Therefore, by utilizing this association information, even tiles with different shapes or colors can be associated with each other.

[0539] The tiles stored in the above memory (1340) may form a database including each tile and tile information corresponding to each tile. In this case, the tile information stored in the database may include address information of each tile, information on a group to which each tile is included (group information), and, if there is another tile associated with the corresponding tile, association information including information on the other associated tile. Hereinafter, a database including tiles and tile information corresponding to each tile will be referred to as a tile database (DB) (hereinafter referred to as a tile DB) (1341).

[0540] Meanwhile, the memory (1340) may store map data including virtual objects. Here, the map data may be formed by connecting map data from multiple different regions in tile form. In this case, each map tile may represent a logical rectangular section corresponding to a certain area, and may be used to subdivide a wide map area. In other words, the map data may be formed by data from map tiles each corresponding to a different region.

[0541] Meanwhile, the above map data may be map data including data of a so-called 2.5D polygon model. In addition, the map data may include high-definition (HD) map data having high accuracy. The map data may be linked to a navigation system (770) and a driving system (700) provided in a vehicle (100), and the memory (1340) may provide the map data to various linked systems and components under the control of the processor (1330). Such map data may be stored in a map data storage unit (1342).

[0542] Meanwhile, a 3D map view generation device (1300) according to an embodiment of the present invention may include a processor (1330) that generates a 3D map as a digital twin using at least one of an image captured by a camera (310) provided in a vehicle (100), 2D map data (e.g., HD map data), and 3D map data (e.g., map data including data of a 2.5D polygon model). The processor (1330) controls each connected component and may control the overall operation of the 3D map view generation device (1300).

[0543] Additionally, the processor (1330) can overlap (or superimpose, output) graphic objects related to route guidance on a digital twinned 3D map.

[0544] Here, the graphic object related to route guidance refers to an object output as mixed reality (MR), and may include various types of objects (e.g., POI objects, carpet-shaped objects, 3D objects, etc.) required to perform route guidance. Here, the graphic object related to the route guidance may also be referred to as an MR object.

[0545] When the destination of the vehicle (100) is set, the processor (1330) can determine a route for the vehicle (100) to drive through the navigation system (770). And, when the driving route of the vehicle (100) is determined, the processor (1330) can determine at least one virtual object (e.g., an object output as the mixed reality (MR)) to perform texturing on a digital twin 3D map for route guidance.

[0546] Meanwhile, in order to determine virtual objects to be displayed on the digital twinned 3D map, the processor (1330) may first determine a map area in which to generate the virtual objects, i.e., to model. To this end, the processor (1330) may first detect, from map data stored in the memory (1340), at least one map tile around a map tile corresponding to an area in which the vehicle (100) is located, according to the current location of the vehicle (100) and the driving direction of the vehicle (100). For example, the processor (1330) may determine an area corresponding to 5 horizontal and 5 vertical maps centered on the map tile in which the vehicle (100) is located, i.e., a total of 25 map tile data, as a map area, according to a zoom level of a specified map, and may detect map tile data corresponding to the determined map area from the map data.

[0547] And the processor (1330) can detect buildings included in the detected map area. And it can detect building reference information corresponding to the detected buildings from the building reference information storage unit (1343). And the setting values ​​of each building image related to the detected building reference information can be determined. Here, the building image setting values ​​are information for modeling each building virtual object, and may include a setting value of a projection direction according to an angle or distance for generating the building virtual objects according to the driving direction of the vehicle (100) and / or the angle at which the MR view is displayed, a setting value of modeling according to a 2.5D virtual object or a setting value of modeling according to a 3D virtual object, for example, a vector value such as a coordinate value of a vertex or a normal line.

[0548] Once the building image setting values ​​are determined, the processor (1330) can calculate building setting values ​​corresponding to each building included in the map area based on the determined building image setting values ​​and building reference information of each building. Here, the building setting values ​​can include the height of the building virtual object to be created, the side area of ​​each side, the plane area, etc. Then, once the calculation of the building setting values ​​is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included in the map area based on the building setting values.

[0549] And when the modeling of virtual objects included in the above map area is completed, the processor (1330) can determine virtual objects to be displayed on the display unit (251) through rendering among the modeled virtual objects. To this end, the processor (1330) can determine a visible area of ​​a frustum shape formed according to the driving direction of the vehicle (100) based on the current position of the vehicle (100), determine building virtual objects included in the visible area of ​​the frustum shape as objects to be rendered, and perform texturing on the determined building virtual objects.

[0550] Meanwhile, the processor (1330) may pre-model building virtual objects corresponding to each building included in the map area, and render and display only at least some of the modeled building virtual objects on the display unit according to the visible area of ​​the frustum shape determined according to the position and driving direction of the vehicle (100). In this case, the texturing may be performed during the rendering process. Therefore, by pre-modeling objects to be rendered and selectively rendering only some of the modeled objects as needed, texturing of virtual objects that are not displayed on the display unit can be prevented. Therefore, the amount of computation required for the texturing can be reduced, and the time for virtual objects to be displayed on the display unit can be significantly reduced.

[0551] Moreover, the processor (1330) may be implemented as a plurality of processors. For example, the processor (1330) may be implemented as a processor that models the building virtual objects and a processor that renders some of the modeled virtual objects as different processors. For example, the processor that models the virtual objects may be a central processing unit (Central Processing Unit) that interprets various program commands of the 3D map view generation device (1300) according to an embodiment of the present invention and performs data calculations and processing. On the other hand, the processor that renders some of the pre-modeled virtual objects may be a GPU (Graphics Processing Unit) that processes graphic calculations in the 3D map view generation device (1300) according to an embodiment of the present invention and outputs result values. In this case, since the modeling of the virtual object and the rendering of the modeled virtual object can be processed in parallel on different processors, the computational load on the processor can be significantly reduced compared to the case where all processes are performed on a single processor, and by displaying the virtual object by performing rendering from objects that have been modeled in advance, the time required until the virtual object is displayed on the display unit can also be significantly shortened. Furthermore, the 3D map view generation device (1300) according to the embodiment of the present invention described above generates modeling of building virtual objects corresponding to each building included in the map area in advance, and renders only some virtual objects according to some areas (e.g., the visible area of ​​the frustum shape) among the generated building virtual objects, thereby further shortening the time for displaying the building virtual objects, and thus real-time provision of an MR view according to the driving of the vehicle (100) can be made possible.

[0552] In this way, since the present invention has a configuration that pre-generates modeling of building virtual objects corresponding to each building included in the map area, when the map area needs to be updated according to the location and driving direction of the vehicle (100), the processor (1330) can additionally detect at least one map tile data from the map data through the update. In addition, the virtual objects of the buildings included in the area of ​​the additionally detected map tile can be further modeled. On the other hand, the processor (1330) can delete virtual objects of the buildings included in the area of ​​the map tile that has deviated from the map area according to the update of the map area for efficient memory utilization.

[0553] Meanwhile, when the destination of the vehicle (100) or the route to the destination is determined, the processor (1330) may additionally detect at least one map tile data from the map data based on the determined destination or the route to the destination. In this case, the additionally detected at least one map tile data may be map tile data that are connected to each other. In addition, when the additionally detected at least one map tile data is additionally detected, the processor (1330) may model virtual objects corresponding to each of the buildings included in the additionally detected map tile data.

[0554] In this case, virtual objects modeled from the additionally detected map tile data may be rendered and displayed on the display unit (251) regardless of the driving information of the vehicle (100) when a preset condition is met. For example, the preset condition may be met when a user requests a search for a route to a destination or a check of the surroundings of the destination. In this case, regardless of the current driving status of the vehicle (100), building virtual objects modeled from the additionally detected map tile data may be rendered and displayed along the shortest path to the set destination or the searched driving path to the destination. In this case, texturing of the building virtual objects may be performed during the rendering process.

[0555] Meanwhile, texturing of the above-mentioned modeled virtual objects can be performed in a manner in which texture images are tiled, and can be performed by sequentially combining multiple sub-regions constituting the virtual object.

[0556] For example, the virtual object corresponding to the target building may have a surface area divided into multiple layers according to the exterior appearance of the target building, and a partial area may be formed for each of the divided surface areas. For example, the processor (1330) may divide the target building into a bottom layer, a top layer, and at least one middle layer, and different tiles may be matched to each partial area.

[0557] In this case, the top floor is distinguished when it has a shape or form that is distinct from the middle floor, such as a roof, an antenna, or a spire. If the target building does not have a structure placed on the roof of the building, such as a roof, an antenna, or a spire, the processor (1330) may distinguish the target building only by the bottom floor and at least one middle floor.

[0558] Meanwhile, in the case of an intermediate layer, the processor (1330) can be divided into at least one layer according to a preset floor height or window arrangement, etc. For example, if the intermediate layers have different colors or shapes (e.g., different window arrangements), the layers with the different colors or shapes can be distinguished as different intermediate layers. That is, in the case of floors designated as commercial spaces and floors designated as residential spaces, such as in a mixed-use building, the floors designated as commercial spaces and the floors designated as residential spaces can each be distinguished as different intermediate layers.

[0559] In this case, different layers, for example, the ground floor, at least one intermediate layer, and the top layer, can be matched with different tiles. And based on the different matching tiles, tiling can be performed for each layer, i.e., each sub-region (e.g., the ground floor, at least one intermediate layer, and the top layer). Then, each sub-region for which tiling has been performed can be sequentially combined in a preset order based on combination information specified in the building reference information corresponding to the target building.

[0560] Meanwhile, the tiling may be sequentially performed based on a tiling order specified in the building reference information. For example, each of the sub-regions may be sequentially combined starting from a sub-region corresponding to a floor close to the ground. In this case, tiling may be performed first for a sub-region corresponding to an area in contact with the ground, i.e., the bottom floor, and then tiling may be performed for a sub-region corresponding to the next order (e.g., the first middle floor) on top of the sub-region already tiled. Finally, tiling may be performed for a sub-region corresponding to the uppermost floor (e.g., the top floor). In this way, by sequentially performing tiling for each sub-region in a preset order for the virtual object, texturing may be performed for the entire surface area of ​​the virtual object.

[0561] In this way, texturing of a virtual object according to an embodiment of the present invention can be performed based on the sequential combination of each tiled sub-region. Therefore, texturing of a virtual object according to an embodiment of the present invention can also be referred to as procedural texturing.

[0562] Meanwhile, the tile DB (1341) may be provided from a preset server that is communicatively connected to the 3D map view generation device (1300). For example, the preset server may be a cloud server. In this case, the cloud server may be the same server as the cloud server that provides map data (e.g., map data including 2.5D polygon objects), but of course, they may be different servers. In this case, that is, if they are different servers, the cloud server that provides the map data will be described as a first cloud server, and the cloud server that provides the tile DB (1341) and building reference information will be described as a second cloud server.

[0563] When the above tile DB (1341) is provided, the processor (1330) can determine tiles to be tiled on the virtual object surface area of ​​each building included in an area defined based on driving information of the vehicle (100) based on information collected around the vehicle (100).

[0564] For example, the processor (1330) can segment an image of a building around a vehicle (100) acquired through a camera (310) into a plurality of partial regions and detect a tile matching each of the segmented images (i.e., partial region images of the target building). To this end, the processor (1330) can calculate a similarity for each tile included in the tile DB (1341) for each of the partial region images, and determine a tile matching each of the partial region images based on the calculated similarity.

[0565] For example, the processor (1330) can detect feature information for a certain partial region image. And, among the tiles included in the tile DB (1341), at least one tile having a feature similar to a feature detected from the certain partial region image can be detected. And, for each of the at least one detected tiles, a similarity to the certain partial region image can be calculated, and a certain tile having the highest calculated similarity can be matched as a tile corresponding to the certain partial region image.

[0566] For this tile matching, the processor (1330) may utilize a pre-trained artificial intelligence (AI) model. In this case, the AI ​​model may recognize a partial region image from features detected in the partial region image, and detect a tile having features most similar to the feature points of the recognized partial region image.

[0567] Here, the feature points of the partial region image can be detected in various ways. For example, the shape, form, or color pattern of the image can be detected as the feature point. In this case, the more distinct the shape, form, or color pattern of the partial region image, the more clearly the AI ​​model can detect the pattern from the partial region image, resulting in a higher recognition rate.

[0568] On the other hand, if it is difficult to detect a pattern of shape, form or color from the partial region image due to low image quality, brightness difference, shaking, etc., the artificial intelligence model may not be able to detect the feature points of the partial region image, in which case a low recognition rate may be produced. Then, based on the recognition rate, the artificial intelligence model may not detect a tile corresponding to the partial region image. In other words, the artificial intelligence model may determine whether to detect a tile corresponding to any one of the partial region images based on the produced recognition rate.

[0569] Here, the artificial intelligence model may be included in the 3D map view generation device (1300) according to an embodiment of the present invention as a separate component from the processor (1330) (not shown). Alternatively, the artificial intelligence model may be part of the processor (1330), or the processor (1330) may perform the function of the artificial intelligence model. In this case, the artificial intelligence model may be understood as the same component as the processor (1330).

[0570] Meanwhile, when the tiles corresponding to each partial region image are determined, the processor (1330) can generate combination information related to the combination of each partial region image based on the acquired building image. For example, the combination information may be information about the location of each partial region image or the order in which tiling is performed for the acquired building image.

[0571] And the processor (1330) can model virtual objects of buildings located in an area defined according to the location of the vehicle (100) from map data. And, among the area, building reference information of buildings located in at least a part of the area (hereinafter referred to as a map area) defined according to the driving information of the vehicle (100) can be detected. And, the processor (1330) can perform texturing on virtual objects corresponding to the detected building reference information among the modeled virtual objects.

[0572] In more detail, the processor (1330) can sequentially tile surface areas of virtual objects corresponding to the detected building reference information using tiles pre-designated for each subarea.

[0573] For example, if the target building is formed of three layers, including a ground floor, a first intermediate floor, and a second intermediate floor, the processor (1330) may sequentially tile tiles matching each sub-area in a preset order onto the surface area of ​​a virtual object corresponding to the target building. Here, the preset order may be an order from the ground.

[0574] Accordingly, tiles matching the bottom area, i.e., the partial area (bottom layer) included in the building reference information, can be tiled on the surface of a virtual object in an area in contact with the ground in the virtual object in the map data corresponding to the target building. Then, when tiling for the bottom layer is completed, the processor (1330) can tile tiles matching the next partial area (first intermediate layer) in a preset order on the partial area (bottom layer) for which tiling has been completed. Then, when tiling for the first intermediate layer is completed, the processor (1330) can tile tiles matching the next partial area (second intermediate layer) in a preset order on the partial area (first intermediate layer) for which tiling has been completed. In this way, by sequentially tiling the tiles matching each partial area on the surface area of ​​the target building, texturing can be performed on the virtual object in the map data corresponding to the target building.

[0575] Here, the height between each floor of the target building may correspond to an integer multiple of the tile. For example, if the height between each floor of the target building is equal to the height of the tile, the processor (1330) may perform tiling for each sub-region by arranging tiles matching each sub-region in a non-overlapping manner.

[0576] To this end, the processor (1330) may determine an area in which the tiling is to be performed based on the left and right margin values ​​preset from the corners of each side area of ​​each virtual object (building virtual object) and the horizontal and vertical lengths of each side area of ​​the building according to the building reference information. In this case, each building surface may be divided into a plurality of sub-areas based on a floor divider that divides the top and bottom, the top layer, and at least one middle layer. In addition, the processor (1330) may configure the left and right margin values ​​and the top and bottom margin values ​​or the width of the floor divider as margins for each of the plurality of sub-areas, and determine the remaining area excluding the margins as an area in which the tiling is to be performed. In addition, in this case, the inter-floor height or inter-floor width of the bottom or top layer may be formed differently from the inter-floor height or inter-floor width of the middle layer. Additionally, among the tiles stored in the tile DB (1341), the tiles corresponding to the middle layer may be tiles that differ in at least one of width or height from the tiles corresponding to the bottom layer and the top layer.

[0577] And the processor (1330) can determine the number of tiles to be tiled horizontally or the number of tiles to be tiled vertically (tiling offset) in each sub-area where the tiling is to be performed according to the width or height of the determined area. And the processor (1330) can distinguish tiles that are tiled continuously in the surface area of ​​the building virtual object, i.e., each building element included in the texture image.

[0578] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different areas of the texture image by each color value, and accordingly, different areas of the texture image, i.e., different building elements, can be distinguished by color channel.

[0579] And the processor (1330) can generate mask maps for masking building elements for each distinguished building element in each surface area of ​​the tiled building virtual object. The mask map is a map formed so that building elements other than a certain building element are hidden in the surface area of ​​the tiled building virtual object, and may mean an image used to specify only one building element among each building element included in the texture image. In other words, the mask map corresponding to a specific building element may be an image in which other areas are hidden except for an area corresponding only to the specific building element, and may be an image for applying a specific effect only to an area corresponding to the specific building element.

[0580] Meanwhile, when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps with shading effects so that each mask map can display a different material texture.

[0581] Normally, when there is a curve in the surface of an object, the direction of the surface changes due to the curve, and light is reflected differently depending on the changed surface direction, and the curve of the surface of the object can be felt by the reflected light that is reflected differently. The normal map is used to express the curved surface using the reflected light of the curve rather than the actual curve, and may mean an image having a 3D vector value that represents the 3D direction of the surface of the object required for calculating the degree of shading corresponding to the reflected light reflected by the surface curve of the specific material when light incident on a specific material is incident on the surface of the specific material. When such a normal map is used, even if it is a 2D image that does not have a three-dimensional shape, the reflected light corresponding to the curve according to the specific material can be expressed for the light incident on the building, so the texture of the specific material can be formed.

[0582] Accordingly, the processor (1330) can blend normal maps corresponding to different materials with respect to the mask maps corresponding to each building element. Then, a 3D vector value according to the normal map can be applied to the building element corresponding to each mask map, so that a material texture according to the normal map can be expressed.

[0583] Here, in the case of an area covered by the mask map, the 3D vector value may not be applied even when the normal map is blended. Therefore, the normal map to which the mask map is blended may be textured with a specific material only in an area not covered by the blended mask map. In this case, the area not covered by the mask map may be an area corresponding to a specific building element specified by the mask map. Therefore, the normal map to which the mask map is blended may be an image to which a 3D vector value is applied so that a surface texture of a specific material is expressed only in a specific building element that is not masked by the mask map.

[0584] Meanwhile, the mask map may be performed for each distinguished building element in each surface area of ​​the building virtual object in which texture images are tiled. Accordingly, multiple mask maps may be generated for each building element in the surface area of ​​the building virtual object. Each mask map may be blended with a normal map of a different material. The processor (1330) may perform blending between normal maps to which the mask maps are blended.

[0585] Accordingly, the processor (1330) can generate a single normal map (hereinafter referred to as a multi-blended normal map) in which 3D vector values ​​corresponding to surface textures of different materials are applied to areas corresponding to different building elements, respectively, for the surface area of ​​the building virtual object through blending between normal maps to which the mask map is blended. Then, by synthesizing the generated multi-blended normal map to the surface area of ​​the building virtual object, the areas corresponding to each building element of the building virtual object can be textured to have different surface textures.

[0586] Here, the processor (1330) can generate a normal map in which 3D vectors corresponding to different surface textures are applied to areas corresponding to each building element included in the texture image for each texture image tiled on the surface area of ​​the building virtual object. Then, the generated normal map, i.e., the multi-blended normal map, can be synthesized for each texture image arranged on the surface area of ​​the building virtual object. In this case, since the texture image is synthesized in a tiling manner on the surface area of ​​the building virtual object, the multi-blended normal map synthesized for each texture image can also be synthesized in a tiling manner on the surface area of ​​the building virtual object.

[0587] Meanwhile, a normal map having 3D vector values ​​corresponding to the surface texture of the specific material applied may be blended with an albedo map for representing the color of the specific material. In this case, the albedo map may be an image having a color value unique to the specific material. Accordingly, a normal map having 3D vector values ​​corresponding to the texture of the specific surface material may form a pair with an albedo map having a color value unique to the specific surface material, and may be blended together.

[0588] Therefore, when an albedo map is blended with a normal map, the normal map may have a 3D vector value corresponding to the surface texture of a specific material as well as a color value unique to the specific material. Therefore, when a mask map corresponding to a specific building element and a normal map are blended, and the albedo map is further blended to the blended normal map, the specific building element corresponding to the mask map may have a color value unique to the specific material as well as a surface texture of the specific material. In this case, a multi-blended normal map corresponding to the entire surface area of ​​a building virtual object or a texture image tiled on the building virtual object may be an image in which 3D vector values ​​having different material textures are applied to different areas corresponding to a plurality of building elements, and each area is expressed with a color of a different material.

[0589] Meanwhile, when texturing of a surface area of ​​a building virtual object is completed through the multi-blending normal map, the processor (1330) can control the MR service device (900) to MR render the building virtual object for which texturing has been completed and display it on the display unit (251). In this case, the processor (1330) can control the MR service device (900) and the display unit (251) of the vehicle (100) through the interface unit (1320).

[0590] Meanwhile, the processor (1330) may determine at least one of the colors of the normal map corresponding to each building element based on an image captured from the camera of the vehicle (100). For example, the processor (1330) may obtain an image corresponding to a visible area of ​​a frustum shape according to the current position of the vehicle (100) and the driving direction of the vehicle (100) from the camera of the vehicle (100). Then, the objects included in the received image may be classified by type and a representative color may be determined for each type of the classified object. In this case, the representative color may be the RGB average value of the color of each object classified as a specific type, or the color having the highest ratio among the colors of each object classified as the specific type.

[0591] For example, the processor (1330) can detect colors corresponding to building objects from the acquired image, and determine a specific color having the highest RGB average value or ratio of the detected colors as a representative color of the building objects included in the image.

[0592] And the processor (1330) can determine at least one color among the colors of the normal map corresponding to each building element as the determined representative color. Accordingly, at least one color among the colors of the normal map corresponding to each building element of the building virtual object can be determined from an image acquired from the camera of the vehicle (100).

[0593] Meanwhile, the above description describes modeling building virtual objects corresponding to buildings included in the map area and texturing and rendering some of the modeled building virtual objects. However, it is also possible to model and render road virtual objects corresponding to road shapes included in the map area.

[0594] To model the above road virtual object, the processor (1330) may generate a plurality of nodes along a road shape included in the map area from map data. Then, the plurality of generated nodes may be defined as segments, and polygon objects having a specific width may be generated for each segment, and the generated polygon objects may be combined to model the road virtual object. In this case, the polygon objects may be round objects having a specific width.

[0595] Meanwhile, the processor (1330) may utilize a morphology operation to more smoothly express the edge of the road object. The morphology operation is one of the techniques for analyzing and transforming the shape and structure of an object within an image, and may include erosion, dilation, opening, and closing operations. Here, the processor (1330) may more smoothly express the road virtual object composed of the polygon objects through a closing operation used when filling a groove or cutting off a protruding part among the morphology operation techniques.

[0596] Meanwhile, FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view creation device (1300) and an MR service device (900) according to an embodiment of the present invention.

[0597] First, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can collect driving information of the vehicle (100). Here, the driving information of the vehicle (100) can include information on the speed, location, and driving direction of the vehicle (100) detected through the sensing unit (120). In addition, a map area for modeling virtual objects can be determined based on the collected driving information of the vehicle (100).

[0598] Meanwhile, when a map area is determined, the processor (1330) can detect building reference information of buildings included in the determined map area from the object reference information storage unit (1343). Then, when the building reference information is detected, building image setting values, which are information for modeling building virtual objects corresponding to each building, are determined, and building virtual objects corresponding to each building can be modeled based on the determined building image setting values ​​and the building reference information of each building.

[0599] And the processor (1330) can determine any texture image as a tile from the tile DB (1341) for each building or for each partial region (e.g., floor region, at least one middle region, top region) of each building. And the texture image determined as a tile can be tiled for each building or for each partial region of each building. And mask maps corresponding to different components of the building included in the tiled texture image are blended with normal maps corresponding to different materials, and each normal map blended with the mask map is blended with an albedo map having a color corresponding to each material, so that the different components of the building can be textured to have different materials and different surface textures. And by blending the textured normal maps, a multi-blended normal map corresponding to a specific surface region or partial region of the building virtual object or a specific texture image can be generated. And, by synthesizing the generated multi-blended normal map to a specific surface area or partial area of ​​the building virtual object, or by tiling a specific surface area or partial area of ​​the building virtual object along a tiled texture image, a surface texture for a surface area of ​​the building virtual object can be formed. That is, a textured virtual object can be generated.

[0600] Meanwhile, the MR service device (900) can model the location of the self-vehicle (100), pedestrians and other vehicles around the vehicle (100), POI (Position Of Interest) information, route information, and road information and terrain information around the vehicle (100) based on information provided from the vehicle system, i.e., GPS, ADAS, and navigation system of the vehicle (100). In addition, the MR service device (900) can configure a UX scene including the textured virtual objects using the modeled information and information about the modeled virtual objects provided from the 3D map view generation device (1300). In addition, the MR renderer can be controlled to generate a map image including a camera view, 3D effects, and GUI (graphics rendering), and the image can be displayed through the display unit (251).

[0601] Hereinafter, a method of determining a map area, modeling virtual objects included in the determined map area, and rendering and displaying the modeled virtual objects by a 3D map view generation device (1300) according to an embodiment of the present invention will be described in more detail with reference to the attached drawings.

[0602] FIG. 13 is a flowchart illustrating an operation process of a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention for determining a map area, modeling virtual objects included in the determined map area, blending the modeled virtual objects into a normal map having 3D vector values ​​of different materials to texture them, and rendering and displaying the textured virtual objects. FIG. 14 is an exemplary diagram illustrating an example of a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention for updating a map area according to vehicle driving information. FIG. 15 is a conceptual diagram illustrating map tile background modeling described in FIG. 14 in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0603] First, referring to FIG. 13, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can obtain driving information about the vehicle's location, speed, driving direction, destination, and path to the destination through the navigation system (770) and the vehicle's sensing unit (120) (S1300).

[0604] When driving information of a vehicle (100) is acquired, the processor (1330) can determine a map area based on a map tile corresponding to the location of the vehicle (100). The map area can be determined differently depending on the preset zoom level of the map.

[0605] For example, the map area may be an area formed by 25 map tiles centered on the map tile according to the location of the current vehicle (100) when the zoom level of the currently set map is 17. In this case, when the zoom level of the map is lowered, an area formed by a greater number of map tiles centered on the map tile according to the location of the current vehicle (100) may be determined as the map area, and when the zoom level of the map is higher, a map area formed by a smaller number of map tiles (e.g., 9) centered on the map tile according to the location of the current vehicle (100) may be determined. When the map area is determined, the processor (1330) may detect data of map tiles included in the determined map area from map data stored in the memory (1340).

[0606] Meanwhile, the above map area may be updated according to driving information of the vehicle (100). For example, when the vehicle (100) leaves the map tile where it is currently located due to driving of the vehicle (100), the processor (1330) may update the map area so that the map area includes map tiles within a range according to the zoom level of the currently set map based on the map tile into which the vehicle (100) newly enters.

[0607] FIG. 14 is an exemplary diagram showing an example in which a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention updates a map area according to driving information of a vehicle (100).

[0608] First, referring to (a) of FIG. 14, (a) of FIG. 14 is a drawing showing an example of a map area (1402) formed with 25 map tiles centered on a map tile according to the location of the current vehicle (100) according to the currently set zoom level '17' of the map. In this case, the map area (1402) may be formed centered on the first map tile (1400) in which the current vehicle (100) is located.

[0609] In this state, if the vehicle (100) moves from the first map tile (1400) to the second map tile (1410) according to the movement of the vehicle (100), the processor (1330) can update the map area (1402) including a number of map tiles (25) determined according to the preset zoom level of the map centered on the first map tile (1400), to a map area (1412) including a number of map tiles (25) determined according to the preset zoom level of the map centered on the second map tile (1410) according to the position of the moved vehicle (100), as shown in (b) of FIG. 14. Accordingly, the processor (1330) can detect data of map tiles corresponding to the updated map area (1412) from map data stored in the memory (1340). In this case, the processor (1330) can detect at least one map tile (1420) that is connected to each other from the map data of the memory (1340), as shown in (b) of FIG. 14.

[0610] Meanwhile, when a map area is determined from data of detected map tiles, the processor (1330) can model virtual objects corresponding to buildings and roads included in the determined map area (S1304).

[0611] In the above step S1304, the processor (1330) can detect buildings included in the detected map area. And, building reference information corresponding to the detected buildings can be detected from the building reference information storage unit (1343). And, the setting values ​​of each building image related to the detected building reference information can be determined. Here, the building image setting values ​​are information for modeling each building virtual object, and may include a setting value of a projection direction according to an angle or distance for generating the building virtual objects according to the driving direction of the vehicle (100) and / or an angle at which the MR view is displayed, a setting value of modeling according to a 2.5D virtual object or a setting value of modeling according to a 3D virtual object, for example, a vector value such as a coordinate value of a vertex or a normal line.

[0612] Once the building image setting values ​​are determined, the processor (1330) can calculate building setting values ​​corresponding to each building included in the map area based on the determined building image setting values ​​and building reference information of each building. Here, the building setting values ​​can include the height of the building virtual object to be created, the side area of ​​each side, the plane area, etc. Then, once the calculation of the building setting values ​​is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included in the map area based on the building setting values.

[0613] Meanwhile, as mentioned in step S1302, the processor (1330) may update the map area according to the driving information of the vehicle (100). In this case, if new map tile data is additionally detected as shown in (b) of FIG. 14 due to the update of the map area, the processor (1330) may additionally model virtual objects corresponding to buildings and roads included in the added map tiles. On the other hand, if the map area is updated as shown in (b) of FIG. 14, map tiles (1430) may be excluded from the map area. In this case, the processor (1330) may delete virtual objects generated from map tiles (1430) excluded from the map area, thereby increasing the efficiency of the memory (1340).

[0614] Here, the processor (1330) may maintain at least some of the virtual objects generated from the excluded map tiles, depending on the building type, user-specified status, POI status, etc. In this case, the maintained virtual objects may be stored in the memory (1340) and may be reused when the corresponding map tile is detected again in the future. In other words, in the case of a building virtual object corresponding to a building frequently visited by the user or a building specified by the user, the virtual object may be stored and reused regardless of whether the map tile containing the building leaves the map area.

[0615] Meanwhile, the step S1304 may include a process of modeling a road virtual object corresponding to the road shape of each map tile included in the map area.

[0616] In this case, the processor (1330) can generate a plurality of nodes along the road shape included in the map area in step S1304. Then, the generated plurality of nodes can be defined as each segment, and the road virtual object can be modeled by generating polygon objects having a specific width for each segment and combining the generated polygon objects. In this case, the polygon objects can be round objects having a specific width.

[0617] Meanwhile, the processor (1330) may utilize morphological operations to more smoothly express the edges of the road virtual object. In particular, the processor (1330) may temporarily increase the radius of the round polygon objects and then restore it to its original state in order to perform a closing operation among the morphological operations. In this case, the degree of smoothness of the edge portion of the road virtual object may vary depending on the radius of the polygon objects and the number of times the closing operation is performed.

[0618] Additionally, the processor (1330) can raise the generated road virtual object above the ground or form it lower than the ground level. In this case, if the height of the road virtual object is lower than the ground level, the ground around the road can exhibit a curb effect, forming a curb depending on the height difference between the ground level and the road virtual object. In this case, by varying the height difference between the road virtual object and the ground level depending on the width or length of the road or the characteristics of the road, the road virtual object can be expressed differently depending on the characteristics of the actual road.

[0619] In addition, the processor (1330) may also form an object corresponding to a river or stream similar to the method of generating the road virtual object. In this case, the height of the virtual object corresponding to the river or stream may be formed so as to be lower than the height of the ground by a preset depth. In this case, similar to the road virtual object, by making the height difference between the virtual object corresponding to the river or stream and the ground different depending on the width or length of the river or stream, the virtual object corresponding to the river or stream may be expressed differently depending on the characteristics of the actual river or stream.

[0620] Meanwhile, in step S1304, when virtual objects such as buildings and roads of each map tile included in the map area are modeled, the processor (1330) can determine an area according to driving information obtained from the vehicle (100) (S1306).

[0621] The area determined in the above step S1306 is an area determined based on the location of the vehicle (100) or the driving direction of the vehicle (100), and may be a map area determined based on the current driving information of the vehicle (100), that is, an area corresponding to at least a portion of at least one map tile detected from the map data storage unit (1342) of the memory (1340). For example, the area may be an area corresponding to the surroundings of the current location of the vehicle (100) within a first map tile according to the current location of the vehicle (100). Alternatively, the area may be an area corresponding to the front of the vehicle (100) in a first map tile including the current location of the vehicle (100) according to the driving direction of the vehicle (100). Alternatively, the area may be an area including a portion of the first map tile and an area corresponding to a portion of at least one other map tile.

[0622] Meanwhile, the area determined in the above step S1306 may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), i.e., a user field of view area according to the driving direction of the vehicle (100). In this case, the user field of view area has a form that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to perspective projection that can form a sense of perspective, and thus may have the shape of a frustum with the location of the vehicle (100) as a vanishing point.

[0623] Here, an area determined according to the driving information of the vehicle (100) in the step S1306 may be an area where virtual objects displayed as a digital twin map can be displayed on the display unit (251). That is, it may be an area displayed as a digital twin map to the user of the vehicle (100) through the MR view, and may be a map area shown to the user of the vehicle (100), i.e., a visible area. Hereinafter, among the map areas determined according to the driving information of the current vehicle (100), an area determined in the step S1306 that is displayed on the display unit according to the driving information of the vehicle (100) will be referred to as a visible area.

[0624] When the visible area is determined in step S1306, the processor (1330) may determine some of the virtual objects to be rendered among the virtual objects modeled in step S1304 according to the determined visible area (S1308). The objects determined in step S1308 may be only some of the virtual objects included in the visible area among the virtual objects modeled in step S1304, that is, the map tiles detected as the current map area. In this case, the modeled objects determined as the rendering target in step S1308, that is, the virtual objects, may include a plurality of building virtual objects and a plurality of road virtual objects or parts of road virtual objects.

[0625] Meanwhile, when the virtual objects to be rendered are determined in step S1308, the processor (1330) can tile a preset texture image, i.e., a tile, for each of the building virtual objects among the determined virtual objects (S1310). Here, the tiling can be performed for each side of each building virtual object determined as the rendering target, and can be performed for the side areas of each building virtual object excluding a preset offset area (or margin area) from the corner of each side.

[0626] Here, at least one of the above-described building virtual objects may have its side region divided into multiple sub-regions. For example, the side region may be divided into a bottom region, a middle region, and a top region, and at least one of the bottom region, the middle region, and the top region may be further divided into multiple regions. In this case, the processor (1330) may tile different texture images for each of the multiple divided regions.

[0627] In addition, the processor (1330) may tile an arbitrary texture image for each building virtual object or for each partial region of each building virtual object. In this case, texture images corresponding to each building virtual object or each partial region of each building virtual object may be determined in advance. In this case, the building reference information, which is information on an actual building corresponding to each building virtual object and stored in the memory (1340), may include information on a texture image corresponding to each building virtual object or each partial region of each building virtual object, and the processor (1330) may determine a texture image corresponding to each building virtual object or each partial region of each building virtual object when determining a building image setting value based on the driving information of the vehicle (100).

[0628] And, based on the height, area, side area of ​​the building included in the reference information of each building virtual object, and the offset area (or margin area), at least one of the horizontal length and the vertical length of the area in which the texture image is to be tiled in each building virtual object can be calculated, and the number of texture images to be tiled in the horizontal or vertical direction can be calculated according to the calculated length. That is, the processor (1330) can calculate information for modeling building virtual objects based on the reference information of the building including the building image setting value and the height of the actual building, and at the same time calculate a building setting value including information on the area of ​​each side area in which the texture images are to be tiled in each side area for each building virtual object and the number of texture images to be tiled in each side area.

[0629] And the processor (1330) can tile texture images in the side areas of each rendering target building virtual object based on information about texture images corresponding to each building virtual object or a partial area of ​​each building virtual object, and information about the number of tilings of the side areas of each virtual object and the texture images included in the calculated building setting value.

[0630] Meanwhile, when tiling of texture images for side areas of each building virtual object is completed in step S1310, the processor (1330) can generate a normal map including 3D vector values ​​that form different textures for each building element included in the texture image, which is a tiled 2D image (S1312).

[0631] In this case, the normal map may be a normal map (hereinafter referred to as a multi-blended normal map) obtained by blending multiple normal maps generated for each building element included in the texture image. In order to generate the multi-blended normal map, the processor (1330) may generate mask maps for masking building elements for each distinguished building element in each surface area of ​​the building virtual object in which the texture images are tiled.

[0632] In this case, the mask map is a map formed so that building elements other than one building element are hidden in the surface area of ​​the building virtual object on which the texture images are tiled, and may mean an image used to specify only one building element among each building element included in the texture image.

[0633] And when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps to which shading effects are applied so that the textures of different materials can be represented in each mask map. In this case, as described above, since the mask map is an image that covers the remaining area except for the corresponding specific building element, when the normal map is blended, a 3D vector value forming a texture corresponding to the specific material corresponding to the normal map can be blended only in an area not covered by the mask map and an area corresponding to the specific building element corresponding to the mask map.

[0634] Meanwhile, the processor (1330) may generate a mask map for each distinguished building element in each surface area of ​​the building virtual object in which texture images are tiled. Accordingly, multiple mask maps may be generated for each building element, and each mask map may be blended with a normal map of a different material. In addition, the processor (1330) may generate a multi-blended normal map in which 3D vector values ​​corresponding to surface textures of different materials are applied to areas corresponding to different building elements, respectively, with respect to the surface area of ​​the building virtual object, by blending normal maps in which the mask maps are blended with each other. In addition, by synthesizing the generated multi-blended normal map to the surface area of ​​the building virtual object, the processor (1330) may texture the surface areas of each building virtual object so that the areas corresponding to each building element of the building virtual object each have a different surface texture.

[0635] Here, when generating a mask map for each building element, the processor (1330) may further blend at least some of the plurality of mask maps generated for each building element with an albedo map, which is a map image having a color corresponding to a specific material. In this case, as at least some of the plurality of mask maps generated for each building element further have a color according to the albedo map, color mapping can be performed with a color corresponding to a specific material.

[0636] In this case, the processor (1330) may determine a color corresponding to at least some of the plurality of mask maps generated for each building element based on the results of analyzing an image captured from the camera of the vehicle (100). For example, the processor (1330) may detect objects corresponding to buildings among image objects included in an image captured from the camera of the vehicle (100). Then, colors may be detected from the detected building image objects, and based on the detected colors, a dominant color or an average color of actual buildings around the vehicle (100) may be determined as a representative color. And, by blending an albedo map corresponding to a color determined as a representative color with at least some of a plurality of mask maps generated for each building element, blending mask maps to which the albedo map is blended to generate the multi-blended normal map, and synthesizing the generated multi-blended normal map to each building virtual object, at least some of the building elements of the building virtual objects currently determined as rendering targets in step S1308 can be textured to have colors similar to those of actual buildings around the vehicle (100).

[0637] Meanwhile, when texturing is completed in step S1312, the processor (1330) may render the building virtual objects for which texturing has been completed to generate images corresponding to the building virtual objects. In addition, the processor (1330) may render other virtual objects, including at least one road virtual object included in the visible area, to generate images corresponding to each virtual object (S1314).

[0638] And the processor (1330) can control the vehicle interface unit (1320) to display virtual objects including each building virtual object and road virtual object generated through the above rendering on the display unit (251) (S1316).

[0639] As described above, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention has a configuration that detects map tiles within a certain area (map area) set centered on the vehicle (100), models virtual objects within the detected map tiles in advance, and renders and images only some of the objects in the driving direction of the vehicle (100) among the pre-modeled objects. In addition, the processor (1330) synthesizes a normal map so that a 3D vector value corresponding to a specific material for each building element is applied only to some of the objects to be rendered, so that a texture according to a specific material for each building element can be formed by the applied 3D vector value during the rendering. In this way, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can shorten the time required for modeling by rendering pre-modeled objects when displaying virtual objects as an MR view using a digital twin map. In addition, the number of virtual objects to be rendered is also limited to the visible area to minimize the number, and the surface area of ​​each building virtual object can be textured to have the three-dimensional effect of various materials using a texture image, which is a 2D image, and a normal map having 3D vector values ​​corresponding to different materials. In this way, by performing texturing using only 2D images, the amount of computation required for texturing can be greatly reduced, and as a result, the time required for rendering can be greatly shortened.

[0640] In this way, since the amount of computation and time required to display virtual objects in an MR view using the digital twin map can be significantly reduced, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can perform rendering of virtual objects to be displayed through the MR view during the time when the app for displaying the MR view using the digital twin map is running, i.e., during the run time. In other words, real-time run-time 3D map modeling can be possible.

[0641] Meanwhile, as described above, the present invention has a configuration for pre-modeling virtual objects included in a plurality of map tile data according to the location of the vehicle (100) and rendering some of the modeled virtual objects, including texturing. In this case, the map tile data can be updated according to the location of the vehicle (100), and the detection of such map tile data can be performed through background processing. In this case, since the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention pre-models virtual objects included in the map tile data when the map tile data is detected according to the determined map area, the modeling of the virtual objects can also be performed through the background processing. The modeling of virtual objects performed through background processing in this way will be referred to as background modeling.

[0642] Meanwhile, the processor (1330) according to an embodiment of the present invention has a configuration that models virtual objects in a map area in advance through background modeling, and renders a portion of virtual objects that have already been modeled based on a visible area determined according to driving information of the vehicle (100). Here, the modeling of the virtual objects is performed through background processing, and the rendering of the modeled virtual objects is performed through foreground processing according to the execution of the user's MR view app, so the 3D map view generation device (1300) according to an embodiment of the present invention has a configuration in which the modeling and rendering of virtual objects are performed separately. Therefore, when the 3D map view generation device (1300) according to an embodiment of the present invention has a plurality of processors, the modeling of the virtual objects and the rendering of the modeled virtual objects may be performed by different processors, respectively.

[0643] FIG. 15 is a block diagram illustrating a configuration in which modeling and rendering of virtual objects are performed by different processes in a 3D map view generation device (1300) according to an embodiment of the present invention having multiple processors.

[0644] Referring to FIG. 15, a 3D map view generation device (1300) according to an embodiment of the present invention, i.e., an embedded system, may include a first processor that loads a plurality of map tiles through an API based on the current location (GPS Position) of a vehicle (100) in conjunction with a navigation map, and a second processor that renders an image of a virtual object displayed on a display unit (251). Here, the first processor may be a central processing unit (CPU) of the 3D map view generation device (1300) according to the embodiment of the present invention, and the second processor may be a GPU (Graphics Processing Unit) that processes graphic operations on the display device and outputs a result value.

[0645] In this case, the first processor may load a plurality of map tiles according to the location of the vehicle (100) and perform modeling on virtual objects included in the loaded plurality of map tiles. In this case, the first processor may perform modeling of the virtual objects in a background modeling manner. In addition, the modeled virtual objects may be stored in a buffer. In this case, the modeled objects may be simple polygonal (2.5D) virtual objects without surface texturing.

[0646] In this case, at least one of the plurality of map tiles may be added when the map area is updated according to the location of the vehicle (100). Then, the first processor may additionally model virtual objects included in at least one additional map tile according to the updated map area. Then, the buffer may be updated so that the modeled virtual objects are added. Accordingly, virtual objects corresponding to each object, including buildings and roads included in the entire updated map area, may be stored in the buffer.

[0647] Meanwhile, when an app for displaying an MR view is executed by a user, the second processor can render the modeled virtual objects stored in the buffer. During the rendering process, the second processor can perform texturing by tiling a texture image to a side area of ​​each modeled building virtual object, and synthesizing a normal map to which a 3D vector corresponding to a different material is applied to each area of ​​each building element included in the tiled texture image to the side area of ​​each building virtual object. In this case, the 3D vector is for providing a shading effect that forms a texture of a specific material through a shading effect, and the second processor, as a custom shader, can perform rendering to generate an image of each building virtual object, including texturing that includes shading according to the 3D vector applied to each building element.

[0648] Therefore, the 3D map view generation device (1300) according to an embodiment of the present invention can process modeling of virtual objects and rendering of modeled virtual objects in parallel on different processors. Therefore, the computational load applied to the processor can be significantly reduced compared to a case where all processes are performed on a single processor, and by displaying virtual objects by performing rendering from pre-modeled objects, the time required until the virtual objects are displayed on the display unit can also be significantly shortened. Accordingly, when an app related to the MR view is executed, rendering of 3D virtual objects included in the digital twin map can be performed during the runtime time, which is the time when the app is executed, and real-time provision of a digital twin map including virtual objects with high-quality textures can be possible.

[0649] However, for convenience of explanation, the following description assumes that modeling and rendering are performed by a single processor (1330). However, it should be understood that the present invention is not limited thereto.

[0650] Meanwhile, in step S1316 of FIG. 13, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can simultaneously display a screen showing information collected around the vehicle according to the ADAS system and an MR view screen according to the digital twin map. FIG. 16 is an exemplary diagram illustrating such an example.

[0651] Referring to FIG. 16, the processor (1330) of the 3D map view generation device (1300) can display information on surrounding vehicles and obstacles based on information collected through the vehicle interface unit (1320) and an ADAS (Advanced Driver Assistance Systems) view image based on driving information of the vehicle (100) in one area (1600) of the display unit (251). ADAS refers to an active safety device that detects a dangerous situation with a sensor or a camera, warns the driver of the risk of an accident, and helps the driver make a judgment and respond. The ADAS view image can include information on objects corresponding to the surrounding situation of the vehicle, for example, a vehicle object and other vehicles and obstacles detected around the vehicle, and can include route information such as the current driving speed or driving time of the vehicle (100) or the expected time required to reach the destination.

[0652] Meanwhile, the processor (1330) can display an MR view image in another area (1610) on the display unit (251). The MR view image is a view image including a building virtual object or a road virtual object created as a 3D object, and can display a digital twin map formed by the virtual objects.

[0653] Meanwhile, according to the above description, it has been mentioned that the processor (1330) of the 3D map view generation device (1300) according to the embodiment of the present invention can determine an area to display virtual objects by rendering them according to the driving information of the vehicle (100), i.e., a visible area, in step S1306 of FIG. 13. Here, the visible area may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), i.e., a user viewing area according to the driving direction of the vehicle (100). In this case, the user viewing area has a form that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to perspective projection that can form a sense of perspective, and thus may have the shape of a frustum with the location of the vehicle (100) as a vanishing point. Therefore, the above-mentioned visible area can be formed in a frustum shape with the position of the vehicle (100) as the vanishing point.

[0654] FIG. 17a is an exemplary diagram illustrating a frustum-shaped visible area formed across a plurality of map tiles or a single map tile, depending on the position of the vehicle and the driving direction of the vehicle, in a 3D map view generation device (1300) according to an embodiment of the present invention. FIG. 17b illustrates an example of a visible area formed to include a plurality of map tiles.

[0655] First, referring to (a) of FIG. 17A, the processor (1330) can form a frustum-shaped visible area (1700) formed along the driving direction of the vehicle (100) with the current location (1720) of the vehicle (100) as a reference point. In this case, the processor (1330) can perform rendering only on virtual objects located within the formed visible area (1700). That is, even within one map tile, virtual objects located in areas other than the visible area (1700) may not be rendered.

[0656] Here, rendering may refer to a process or calculation process of generating an image using a computer program. Therefore, even if it is a modeled virtual object, that is, even if a virtual object is created, if the rendering is not performed, an image corresponding to the virtual object may not be created. In addition, if an image is not created, it may not be displayed on the display unit (251). That is, the processor (1330) may selectively determine objects to be displayed on the display unit (251) among the modeled virtual objects based on a visible area formed according to the driving direction of the vehicle (100).

[0657] Accordingly, as shown in (a) of FIG. 17A, images corresponding to each virtual object can be generated through a rendering process for virtual objects within an area included in the visible area (1700). However, images of virtual objects within an area included in the visible area (1700) may not be generated. Therefore, the processor (1330) can reduce the amount of computation required for rendering and shorten the computation time by rendering only some of the virtual objects modeled based on the visible area.

[0658] Meanwhile, the visible area (1700) may be formed across a plurality of map tiles, as shown in (b) of FIG. 17, depending on the zoom level of the set map. In addition, the plurality of map tiles may be a part of a map area set around the current location (1720) of the vehicle (100). In this case, since the processor (1330) models virtual objects based on the map area, images of virtual objects in an area included in the visible area (1700) among the plurality of map tiles may be generated through a rendering process according to the visible area (1700) formed across the plurality of map tiles, and may be displayed on the display unit (251).

[0659] Meanwhile, Fig. 17b is an example diagram showing an example in which a visible area formed in a frustum shape includes multiple map tiles.

[0660] As shown in Fig. 17b, when a plurality of map tiles are included in the visible area (1700), the processor (1330) can perform modeling on virtual objects included in each of the plurality of map tiles included in the visible area (1700). In addition, the virtual objects for which modeling has been performed can be rendered in a preset order or by selecting only some of the virtual objects.

[0661] For example, the processor (1330) may perform modeling and rendering in order of proximity to the vehicle based on the current location (1720) of the vehicle. Accordingly, objects included in the first map tiles (1751) that are close to the current location (1720) of the vehicle may be modeled and rendered before objects included in the second map tiles (1752) that are further away from the current location (1720) of the vehicle.

[0662] Alternatively, the processor (1330) may model and render only a portion of the objects included in each of the map tiles included within the visible area (1700) based on a distance from the current location (1720) of the vehicle.

[0663] For example, the processor (1330) may perform modeling and rendering for all objects included in each map tile of the first map tiles (1751), but may perform modeling and rendering for only some of the objects included in each map tile of the second map tiles (1752). In this case, the processor (1330) may model and render only objects corresponding to roads and terrain among the objects included in the second map tiles (1752), and may not perform modeling and rendering for general building objects.

[0664] In this way, the modeling and rendering priorities for objects included in each map tile can be determined based on their type. These object type-specific priorities can be included in the data for the objects for the map tile (hereinafter referred to as "map tile data"), along with geometry data containing the shape information of each object, as attribute data corresponding to each object.

[0665] In this way, the order in which objects included in a map tile are modeled and rendered according to their type is determined by attribute information, and the processor (1330) can sequentially model (procedural modeling) and render (procedural rendering) objects included in a map tile according to the order, i.e., priority.

[0666] Meanwhile, by using the above priority, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention may model and render a specific object with priority over other objects. For example, the processor (1330) may give a higher priority to objects such as landmarks than to general building objects. In this case, the building object corresponding to the landmark may be modeled and rendered with priority over or earlier than other general building objects. For example, if the second map tiles (1752) include a building object corresponding to a landmark, the processor (1330) may perform modeling and rendering for the landmark building object together with the road objects and terrain objects included in the second map tiles (1752).

[0667] Meanwhile, FIG. 18a is an exemplary diagram showing an example of texture images being tiled on modeled building virtual objects in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0668] When the building virtual objects determined as rendering targets (hereinafter referred to as rendering target virtual objects) in step S1308 of the above-described FIG. 13 are determined, the processor (1330) may tile a preset texture image in each side area for each of the building virtual objects among the determined virtual objects. In this case, the processor (1330) may detect a side area of ​​the building virtual object excluding a preset offset area (or margin area) inward from each side edge of the building virtual object, and may tile the texture image in the detected side area.

[0669] First, referring to (a) of FIG. 18a, the processor (1330) can determine the left and right margins (1811, 1812) of the side area of ​​the building virtual object determined as the rendering target according to the preset left and right margin values. According to the left and right margin values, the coordinate range of the area (1810) where the texture image is to be tiled in the horizontal direction can be determined.

[0670] For example, if the left and right margin values ​​are 0.1, the lower left point of the side area of ​​the building virtual object (2520) can be defined as (0, 0) and the upper right point can be defined as (1, 1). Then, as shown in Fig. 18a, the width of the tiling area (1810) excluding the left and right margins can be calculated as 0.8. In addition, if the building width of the actual building corresponding to the building virtual object (2520) is 14 m, the actual horizontal width of the tiling area (1810) can be calculated as 11.2 m.

[0671] Meanwhile, the texture image (1800) may be a normalized tile image and may have a preset width. In this case, if the horizontal width of the texture image (1800) is 4 m, dividing the actual horizontal width of the tiling area (1810), 11.2 m, by the horizontal width of the texture image (1800), 4 m, may yield a value of 2.8. In this case, since the number of texture images (1800) is an integer, rounding may yield a value of 3, and the yielded value 3 may be the number of texture images (1800) to be tiled in the tiling area (1810) of the side area of ​​the building virtual object (2520) set according to the preset left and right margin values. And the processor (1330) inputs (0.1, 0.9), which are the left and right X-axis coordinates of the tiling area (1810), as input values ​​of the remap function and inputs (0, 3) as a new range according to the remap function, so that up to three tiles (texture images (1800)) can be successively attached to the tiling area (1810) in the horizontal direction in the tiling area (1810).

[0672] In addition, the processor (1330) can set multiple partial regions based on the bottom layer, which is the first floor, according to preset upper and lower margin values. For example, the processor (1330) can set a floor divider corresponding to the space between the bottom layer and the upper layer of the partial region. In this case, the space between the set floor dividers based on the bottom layer can be determined as the partial region where texture images are to be tiled.

[0673] Here, the side region of the building virtual object may include multiple subregions. For example, the side region of the building virtual object may be divided into a bottom region (1852), a middle region (1850), and a top region (1851), and at least one of the bottom region (1852), the middle region (1850), and the top region (1851) may be further divided into multiple regions. In this case, the processor (1330) may tile different texture images for each of the multiple divided regions.

[0674] In this case, the processor (1330) can obtain a mask map corresponding to each sub-region based on the UV coordinate system corresponding to each sub-region on each building side. For example, the processor (1330) can define the lower left point of the side region of the building virtual object as (0, 0) and the upper right point as (1, 1). Then, by defining the Y-axis coordinate in the UV coordinate system according to the margin value of the sub-region in which the layer division is formed, the vertical width corresponding to the sub-region can be defined, and the mask map corresponding to the sub-region can be obtained according to the defined vertical width. In this case, the mask map corresponding to the sub-region can be a mask map in which a height lower than the Y-axis coordinate according to the margin value of the sub-region has a value of 0, and a height higher than the Y-axis coordinate has a value of 1. In this case, when blending with a normal map is performed, a part having a value of 0 maintains a value of 0, but a part having a value of 1 can have a 3D vector value of the normal map.

[0675] As an example of such a partial region, if the margin value of the partial region where the layer division is formed is 0.8, the processor (1330) can determine the coordinates of the upper right point of the partial region as (1, 0.8). Then, the processor (1330) can obtain a mask map in which an area corresponding to a height less than 80% of the total height of the building virtual object from the floor surface of the building virtual object has a value of 0, and an area corresponding to a height greater than that has a value of 1. That is, an area corresponding to the upper edge of the building virtual object from a height of 80% or more of the total height of the building virtual object can be formed as a partial region (e.g., a tower region).

[0676] Meanwhile, the processor (1330) can tile different texture images for each sub-region (1850, 1851, 1852) of the building virtual object (2560) in the same manner as the method described in (a) of FIG. 18A. In this case, as shown in (b) of FIG. 18A, the top region (1851) can be tiled with the first texture image (1801), the bottom region (1852) can be tiled with the second texture image (1802), and the middle region (1850) can be tiled with the third texture image (1800). Accordingly, the building virtual object (2650) can be tiled with different texture images for each different sub-region.

[0677] In this way, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can diversify the exterior shape (facade design) of the building virtual object by dividing each building virtual object into a plurality of different regions and tiling different texture images in each of the divided regions.

[0678] FIG. 18b illustrates examples of facade designs of a building virtual object expressed in various ways by using different partial regions and tiled texture images in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0679] First, Figure 18b (a) illustrates an example of a building virtual object formed by three sub-regions: a ground floor, a middle floor, and a top floor. Figures 18b (b) and (c) illustrate examples of a building virtual object formed by only the ground and middle floors. In this way, by varying the number of sub-regions, building virtual objects can be expressed with different exterior shapes.

[0680] In addition, the processor (1330) can vary the texture images tiled in each sub-region for each building virtual object. In this case, since not only the number of sub-regions but also the tiled texture images vary, the number of possible combinations can increase. Therefore, a more diverse exterior representation of the building virtual object can be achieved.

[0681] Meanwhile, the processor (1330) can blend normal maps having 3D vector values ​​corresponding to different materials for each building element in step S1312 of FIG. 13 so that the images can be displayed with different materials for each building element included in the tiled texture image of the building virtual object during rendering.

[0682] FIG. 19 is a flowchart illustrating in more detail the texturing process of step S1312 of FIG. 13, which forms different surface textures for each component of a building in a 3D map view generation device (1300) according to an embodiment of the present invention. FIG. 20 is an exemplary diagram illustrating an example of normal maps blended to mask maps corresponding to different building elements and a multi-blended normal map generated by blending normal maps together, according to the process described in FIG. 19.

[0683] First, referring to FIG. 19, the processor (1330) can identify and distinguish building elements included in each texture image for each texture image tiled in the tiling area of ​​the building virtual object (S1900).

[0684] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different areas of the texture image by each color value, and accordingly, different areas of the texture image, i.e., different building elements, can be distinguished by color channels. For example, if the texture image includes at least one window element, the window element may have different color values ​​from the rest of the building background. In addition, if the window element includes a glass element corresponding to a glass area, a window frame element corresponding to a window frame area, and a blind element corresponding to a blind area, the glass element, the window frame element, and the blind element may each have different color values. Therefore, the processor (1330) can distinguish and identify areas corresponding to a plurality of different building elements included in the texture image based on the color values. In this case, the texture image may be referred to as a labeling map, which is a map image that labels a plurality of different building elements.

[0685] Meanwhile, when each building element included in the texture image is distinguished and identified, the processor (1330) can generate mask maps corresponding to each building element (S1902).

[0686] In this case, the mask map corresponding to a specific building element may be an image having a value of 1 in an area corresponding to a specific building element identified by a specific color value in the texture image, and a value of 0 in the remaining areas.

[0687] For example, if the texture image includes at least one window element, the mask map corresponding to the window element may be a mask map that has a value of 1 only in the area corresponding to the window element and a value of 0 in the remaining area. On the other hand, the mask map corresponding to the building wall may be a mask map that has a value of 1 only in the building wall portion, i.e., in the remaining area of ​​the texture image excluding the window element, and a value of 0 in the remaining area.

[0688] In addition, if the window element includes the glass element, the window frame element, and the blind element, the processor (1330) can generate a glass mask map in which only the area corresponding to the glass element in each window element has a value of 1 and the rest has a value of 0, a window frame mask map in which only the area corresponding to the window frame element in each window element has a value of 1 and the rest has a value of 0, and a blind mask map in which only the area corresponding to the blind element in each window element has a value of 1 and the rest has a value of 0.

[0689] When mask maps corresponding to each building element are generated for each building element included in the tiled texture image in this way, the processor (1330) can blend each generated mask map with a normal map having a 3D vector value corresponding to a different material (S1904). In this case, the normal map means an image having a 3D vector value indicating a 3D direction of the surface of an object required for calculating the degree of shading corresponding to the reflected light reflected by the surface curvature of the specific material when light incident on the specific material is incident on the specific material, in a graphic processing device, and may be a 2D image capable of expressing the texture of the specific material through the degree of shading, i.e., the shading effect.

[0690] The blending of these normal maps and mask maps may be a process similar to a multiplication process. Therefore, when the normal map and the mask map are blended, the 3D vector value applied to the normal map may be applied only to the area that is not masked with a value of 0 in the mask map, i.e., only to the area corresponding to a specific building element. Therefore, only the area corresponding to a specific building element corresponding to the mask map may have a texture according to a specific material expressed. In this case, the processor (1330) may blend the mask map corresponding to each building element with a normal map having a 3D vector value of a different material, thereby allowing each building element to have a texture of a different material.

[0691] In the above step S1904, when each mask map is blended with each normal map, the processor (1330) may further blend an albedo map having a different color for each of the mask maps and the blended normal maps. In this case, the albedo map may be an image having a color value unique to a specific material. In addition, a pair may be formed with a normal map having a 3D vector value corresponding to the texture of a specific surface material.

[0692] Then, the processor (1330) can further blend the albedo map forming a pair with the normal map for each normal map blended with the mask map (S1906). In this case, blending with the albedo map can be applied similarly to the multiplication process as with blending with the normal map. Therefore, a color value can be applied only to a part having a value of 1 in the mask map, and accordingly, a normal map capable of expressing the texture of a specific material and the unique color of the material can be generated for a specific building element.

[0693] Meanwhile, in step S1906, when the mask maps and blended normal maps corresponding to each component of the building are blended with the albedo maps, the processor (1330) can blend the blended normal maps of the building elements with each other (S1908). In this case, the blending of the normal maps may be a union concept, which may mean a process of combining the areas to which 3D vector values ​​(and color values ​​according to the albedo map) are applied between the normal maps into a single map image, a multi-blended normal map. Therefore, when the blending process of step S1906 is completed, a normal map (multi-blended normal map) to which 3D vector values ​​(and albedo color values) of different normal maps are applied for each building element included in a single texture image may be generated.

[0694] And the processor (1330) can perform texturing on the side area of ​​the building virtual object by synthesizing the multi-blending normal map into the tiling area among the side areas of the building virtual object (S1910).

[0695] In this case, the multi-blending normal map may be generated for the entire tiled area in which texture images are tiled in the side area or partial area of ​​the building virtual object. In this case, the processor (1330) may synthesize the multi-blending normal map generated in a size corresponding to the entire side area or partial area of ​​the building virtual object onto the side area or partial area of ​​the building virtual object.

[0696] Alternatively, the multi-blending normal map may be generated for any one texture image tiled in a side area or partial area of ​​the building virtual object. In this case, the processor (1330) may synthesize the multi-blending normal map for the side area or partial area of ​​the building virtual object by tiling the multi-blending normal map for each of the texture images tiled in the side area or partial area of ​​the building virtual object.

[0697] Meanwhile, FIG. 20 illustrates an example of generating a multi-blending normal map including areas to which different 3D vector values ​​corresponding to each of a plurality of building elements included in a texture image are applied according to the operation process described in FIG. 19.

[0698] Referring to FIG. 20, the processor (1330) can distinguish and identify each building element from a texture image (1800) including a window element. In this case, if the window element includes a glass element and a window frame element (2002), the processor (1330) can identify the wall element (2001), the glass element, and the window frame element (2002) from the texture image (1800).

[0699] And the processor (1330) can generate a mask map corresponding to each identified building element. In this case, as shown in FIG. 20, a mask map (2011) corresponding to the wall element can be generated in which only the area corresponding to the wall element (2001) has a value of 1 and the remaining areas, for example, the areas corresponding to the glass element and the window frame element (2002), have a value of 0. In addition, a mask map (2012) corresponding to the window frame element can be generated in which only the area corresponding to the window frame element (2002) has a value of 1 and the remaining areas, for example, the areas corresponding to the glass element and the wall element (2001), have a value of 0.

[0700] And the processor (1330) can blend the first normal map (2021) and the second normal map (2022), which are map images to which 3D vector values ​​corresponding to different materials are applied, for each generated mask map (2011, 2012). In this case, the first normal map (2021) may be an image to which a 3D vector value capable of expressing the first material with a shading effect is applied, and the second normal map (2022) may be an image to which a 3D vector value capable of expressing the second material with a shading effect is applied.

[0701] Then, the 3D vector value of the blended normal map can be synthesized only in the area having the value 1, excluding the area having the value 0 in each mask map. Therefore, when the mask map (2011) corresponding to the wall element and the first normal map (2021) are blended, a normal map (2031) synthesized with a 3D vector value that can express the first material only in the area corresponding to the wall element in the texture image (1800) can be generated, as shown in FIG. 20. In addition, when the mask map (2012) corresponding to the window frame element and the second normal map (2022) are blended, a normal map (2032) synthesized with a 3D vector value that can express the second material only in the area corresponding to the window frame element in the texture image (1800) can be generated, as shown in FIG. 20.

[0702] Then, the processor (1330) can blend a normal map (2031) in which 3D vector values ​​that can express a first material only in an area corresponding to the wall element are synthesized, and a normal map (2032) in which 3D vector values ​​that can express a second material only in an area corresponding to the window frame element are synthesized. In this case, blending is performed between normal maps, and each normal map can be synthesized into a single multi-blended normal map. Therefore, as shown in FIG. 20, a multi-blended normal map (2050) can be generated in which 3D vector values ​​that can express a first material in an area corresponding to the wall element (2001) are synthesized, and 3D vector values ​​that can express a second material in an area corresponding to the window frame element (2002) are synthesized. That is, a normal map can be generated in which 3D vector values ​​that can express different materials in each area corresponding to each building element included in the texture image are synthesized.

[0703] Meanwhile, in the description of Fig. 19 described above, the albedo map having a material-specific color is blended into the normal map to which the mask map is blended, thereby mapping the material-specific color to each building element. However, in contrast to this, the colors of virtual objects within the visible area can also be determined as colors similar to the colors of buildings within the actual visible area based on images captured from the camera of the vehicle (100).

[0704] FIG. 21 is a flowchart illustrating an operation process of mapping the colors of virtual building objects based on the colors of actual buildings within a visible area in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0705] Referring to FIG. 21, when step S1904 of blending the mask maps generated for each building element in FIG. 19 into a normal map having 3D vector values ​​corresponding to different materials is performed, the processor (1330) can acquire an image of the surroundings of the vehicle (100) through the camera of the vehicle (100) (S2100). In this case, the camera may be a camera that captures an image of the front area of ​​the vehicle (100), and the acquired image may be an image of the front of the vehicle (100) acquired according to the driving direction of the vehicle (100), i.e., an image of the visible area.

[0706] And the processor (1330) can identify each object included in the acquired image and partition the acquired image into multiple regions according to the type of each identified object. That is, the processor (1330) can partition the regions where each object included in the acquired image is displayed on the image into different regions for each object (S2102).

[0707] For example, if the acquired image is an image including trees, buildings, the sky, and a road, the processor (1330) may, in step S2102, segment the image into an area where each tree is displayed, an area where each building is displayed, an area where the sky is displayed, and an area where the road is displayed. Then, a dominant color may be detected in each segmented area, and each area may be displayed with the detected dominant color. In this case, an image may be acquired in which the area where each object included in the image is displayed is segmented with the dominant color in each area.

[0708] To this end, the processor (1330) may utilize an artificial intelligence module (not shown) trained through preset learning data. In this case, the training may be training to identify each region on the image as a different object based on shape and color. Accordingly, the artificial intelligence module, upon completion of the training, may identify a region of the input image as belonging to one of the preset object types based on the pattern formed by the shape and color displayed in the region.

[0709] And the processor (1330) can classify the regions on the image divided by each object by type of the object. And a representative color can be determined from the regions on the image divided by type of each object (S2104).

[0710] In this case, one or more areas on the segmented image can be classified as having the same object type. For example, an area where each building is displayed on the image can be classified into areas corresponding to each building, and a dominant color can be detected for each segmented area. In this state, the processor (1330) can classify each area on the image classified by each object according to the type of each object. Accordingly, at least one area identified as a 'building' can be classified as an area corresponding to the object type 'building'. In addition, the processor (1330) can determine a representative color corresponding to the object type 'building' based on the color of each of the plurality of areas classified as areas corresponding to the object type 'building'.

[0711] For example, the processor (1330) may determine the color with the largest proportion among the colors of each of the areas corresponding to the object type 'building' as the representative color corresponding to the object type 'building'. Alternatively, the processor (1330) may calculate the average color of the colors of each of the areas corresponding to the object type 'building' and determine the calculated average color as the representative color corresponding to the object type 'building'.

[0712] When a representative color corresponding to the object type 'building' is determined, the processor (1330) can map at least some of the building elements of the building virtual object within the visible area to the determined representative color (S2106). For example, the processor (1330) can blend at least some of the normal maps blended to the mask maps corresponding to each of the different building elements with an albedo map having a representative color corresponding to the object type 'building'.

[0713] FIG. 22 is an exemplary diagram showing an example of determining colors mapped to texture images through images captured from a vehicle in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0714] First, (a) of FIG. 22 illustrates an example of an image captured from a camera of a vehicle (100). In this case, the image may be an image captured from the front of the vehicle (100) along the driving direction of the vehicle (100).

[0715] Then, the processor (1330) can identify objects included in the acquired image. And, as shown in (b) of Fig. 22, the areas identified as each object can generate a color map image in which each area is displayed with a dominant color.

[0716] As shown in (b) of FIG. 22, when a color map image is generated, the processor (1330) can distinguish the area of ​​each object identified in the generated color map image by the type of each object. That is, as shown in (b) of FIG. 22, in the color map image, a first area (2201) corresponding to a first building object, a second area (2202) corresponding to a second building object, a third area (2203) corresponding to a third building object, a fourth area (2204) corresponding to a fourth building object, and a fifth area (2205) corresponding to a fifth building object can be distinguished as areas corresponding to the object type 'building'.

[0717] Then, the processor (1330) can calculate the color proportion of each area corresponding to the object type "building." In this case, areas with the same color can be considered as a single area. Accordingly, the first area (2201) and the fifth area (2205) with the same color can be considered as a single area.

[0718] In this case, as shown in (c) of Fig. 22, the color proportions determined from each area identified as a building object in the image can be calculated. In this case, similar to the building object, the color proportions of road objects or sky objects can also be calculated.

[0719] In this case, as shown in (c) of FIG. 22, if the first color, which is the dominant color of the first region (2201) and the fifth region (2205), has the largest proportion, the processor (1330) can determine the representative color of the building virtual object with the first color. Then, the processor (1330) can generate an albedo map having the first color, and blend at least some of the normal maps blended with the albedo map having the first color and the mask map corresponding to each different building element. Accordingly, a multi-blended normal map can be generated by blending at least one normal map to which the albedo map of the first color is blended. Therefore, texturing of the building virtual object can be performed so that at least some of the building elements of the building virtual object are displayed with the first color.

[0720] Meanwhile, in the 3D map view generation device (1300) according to an embodiment of the present invention, when a plurality of map tiles are included in a visible area (1700) as shown in FIG. 17b, the processor (1330) can perform modeling on virtual objects included in each of the plurality of map tiles included in the visible area (1700). In addition, the processor (1330) can generate an original mesh, which is a set of vertices having a polygonal shape of objects, through the modeling. In addition, by modifying the coordinates of the vertices constituting the original mesh, the shape of at least one object included in the plurality of map tiles can be provided as a distorted 3D map view.

[0721] In this case, the distorted object may be a terrain object included in the plurality of map tiles. Then, the 3D map view generation device (1300) according to an embodiment of the present invention can provide a 3D map view in which a plurality of objects, including building objects, are displayed on a distorted ground according to the distorted terrain object.

[0722] FIG. 23 is a conceptual diagram for explaining the concept of GPU shader processing that distorts an object in a 3D map view generation device according to an embodiment of the present invention.

[0723] Referring to FIG. 23, map data corresponding to each of a plurality of map tiles included in a visible area (1700) may include information on vertices for forming a polygonal shape of objects included in each map tile. Then, the processor (1330) may generate meshes (original mesh - mesh before distortion) which are a set of vertices from the map data. In this case, the original mesh may be classified into a terrain mesh expressing a road, curb, and ground within a map tile area, and a building mesh expressing a plurality of buildings and 3D landmarks within a map tile area.

[0724] Then, the processor (1330) can receive the vertex coordinates of each original mesh. Then, based on a preset distortion model (2302) and a vertex coordinate transformation function defined by distortion parameters, the processor can perform vertex distortion processing (2303) to modify the coordinates of vertices included in the original mesh, i.e., the original vertices.

[0725] Here, the distortion model (2302) can define a function that can express the distorted shape of a vertex. For example, when a terrain object in the form of a horizontal plane is distorted into a hemispherical shape, the distortion model (2302) can be defined in the form of an exponential function with the current position of the vehicle as the origin. In this case, the distortion parameter is a coefficient and a constant such as the slope of the function, and can be a slope value of the exponential function of the distortion model (2302), and can be a value controlled according to the UX scenario situation. In addition, the vertex coordinate transformation function can be an exponential function defined by the slope value of the exponential function. Here, the UX scenario is a scenario in which a 3D map view is provided, and can include a method in which the 3D map view is provided (e.g., a front view, a top down view, etc.).

[0726] Meanwhile, when the distance between vertices constituting the original mesh exceeds a certain distance during the vertex distortion processing (2303), smooth distortion processing may not be possible. For example, in the case of a road object in the shape of a straight road with a distance of 1 km, the original mesh may be expressed with two vertices, one at the beginning and one at the end. In this case, since there is no vertex that can specify the middle area of ​​the road object, distortion of the road object may not be possible.

[0727] Accordingly, the processor (1330) may add a tessellation process (2301) to the original mesh detected from the map data, if necessary. In this case, the tessellation process may mean a process of adding vertices between vertices having a distance greater than a certain distance so that the distance between the vertices is less than the certain distance.

[0728] The process of FIG. 23 described above may be performed in the processor (1330). Alternatively, it may be a process performed in a GPU (Graphic Processing Unit) that performs graphic processing in the 3D map view generation device (1300) according to an embodiment of the present invention under the control of the processor (1330). For example, the process of FIG. 23 may represent shader processing, which is a set of software commands used to calculate the rendering effect of graphic hardware in the GPU. Therefore, in the following description, the processor (1330) may be a GPU.

[0729] FIG. 24 illustrates a cross-section of a 3D map in which a plurality of building objects (2411) are rendered on a distorted terrain object (2500) when the terrain object (2500) is distorted according to the distance from the vehicle (2410) in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0730] First, referring to (a) of FIG. 24, the terrain object (2500) may not be distorted from the position of the ego vehicle (2310) to the first area (2401) that is a preset first distance. In this case, the first distance may be a distance at which navigation information and visibility of the terrain shape can be secured. Accordingly, the terrain object may not be distorted from the position of the ego vehicle (2410) to the first area (2401) corresponding to the first distance, and the positions of the vertices of the original mesh may be maintained as is.

[0731] And in the case of the second area (2402) where the distance from the self-driving car (2410) exceeds the first distance, the processor (1330) can change the vertex positions of the original mesh (hereinafter referred to as the terrain object mesh) that constitutes the terrain object (2500) based on a predefined coordinate transformation function.

[0732] In this case, the coordinate transformation function can modify the vertex positions of the terrain object mesh with the starting point of the second region (2402) as the initial value so that the displacement increases as the distance from the initial value increases. Accordingly, the processor (1330) can modify the position of the vertex with a smaller displacement value as the distance from the starting point of the second region (2402), that is, the ego (2410), among the vertices of the terrain object mesh, is closer to the point corresponding to the first distance, and can modify the position of the vertex with a larger displacement value as the distance from the ego (2410) is farther from the point corresponding to the first distance.

[0733] Accordingly, as shown in (a) of FIG. 24, distortion may begin at a point where the distance from the ego vehicle (2410) exceeds a first distance, and the terrain object (2500) may be distorted to have a greater curvature as the distance from the ego vehicle (2410) increases. In other words, distortion of the terrain object (2500) may begin from an area (second area (2402)) that exceeds the first distance from the ego vehicle (2410).

[0734] Meanwhile, the first distance may refer to a distance at which visibility of navigation information and terrain shape can be secured as described above. In addition, the visibility of the terrain shape may be determined according to the height of the 3D map view camera (2400) or the angle of the 3D map view camera (2400). Here, the 3D map view camera (2400) may provide a field of view (2550) of a 3D map view including a navigation guidance location (2451). In addition, the first distance may be determined according to the driving state of the vehicle (2410), for example, the driving speed, etc. Therefore, the first distance may be determined based on the height of the 3D map view camera (2400), the orientation angle of the gaze point vector of the 3D map view camera (2400), or driving information including the driving state of the vehicle (2410).

[0735] Meanwhile, the processor (1330) can determine in real time a distortion model (exponential function) and a distortion parameter (slope) that define a coordinate transformation function that distorts the terrain object (2500) according to the driving situation of the vehicle (2410).

[0736] For example, the processor (1330) may determine the first distance based on the driving path of the vehicle (2410). Furthermore, the processor may determine the distortion parameter. For example, the processor (1330) may set the first distance to 300 m. Furthermore, the processor may determine the degree of distortion of the terrain object (2500) by setting the distortion parameter.

[0737] In this case, when a right turn situation occurs 700 m ahead, the ground shape corresponding to the right turn section (700 m ahead of the vehicle (2410)) exceeding 300 m (first distance) from the vehicle (2410) may be distorted with a slope according to the distortion parameter, as shown in (a) of Fig. 24. Accordingly, the road and terrain shapes of the area from a point exceeding 300 m from the vehicle (2410) to the right turn section 700 m ahead may be distorted, and a 3D map view screen including the distorted road and terrain shapes may be displayed on the display unit.

[0738] Alternatively, the processor (1330) may determine the distortion rate of the terrain object (2500) based on the distance between the vehicle (2410) and the navigation guidance location (2451). Here, the navigation guidance location (2451) may be a destination or a waypoint for route guidance. Alternatively, it may be a specific location that is preset or arbitrarily set by the user.

[0739] In this case, the processor (1330) can increase the distortion rate of the terrain object as the distance between the navigation guidance location (2451) and the vehicle (2410) becomes closer. Accordingly, as the distance to the vehicle (2410) becomes closer, the terrain object (2500) becomes more distorted with a higher curvature, so the straight-line distance between the 3D map view camera (2400) and the navigation guidance location (2451) can become closer. Accordingly, the user can identify and observe the navigation guidance location (2451) at a closer distance.

[0740] On the other hand, the processor (1330) can reduce the distortion rate of the terrain object as the distance between the navigation guidance location (2451) and the vehicle (2410) increases. Accordingly, as the distance from the vehicle (2410) increases, the terrain object (2500) is distorted with a lower curvature, and thus the straight-line distance between the 3D map view camera (2400) and the navigation guidance location (2451) can increase.

[0741] Meanwhile, the processor (1330) can determine a minimum distortion rate based on the distance between the vehicle (2410) and the navigation guidance location (2451), and distort the terrain object (2500) (the area outside the first distance from the vehicle (2410) - the second area (2402)) with a distortion rate greater than the minimum distortion rate.

[0742] In this case, the processor (1330) can calculate a distortion rate for distorting the terrain object (2500) such that the navigation guidance location (2451) exists within a preset distance from the intersection point (2460) of the upper vector (2405) of the field of view of the 3D map view camera (2400) and the distortion plane of the terrain object (2500) corresponding to the second area (2402), as shown in (b) of FIG. 24. Then, the calculated distortion rate can be determined as the minimum distortion rate. Then, the terrain object (2500) can be distorted such that the ground shape of an area (the second area (2402)) in which the distance from the vehicle (2410) exceeds the first distance is distorted with a distortion rate equal to or greater than the determined minimum distortion rate.

[0743] As described above, it has been mentioned that the curvature of the shape of the road and the ground (hereinafter referred to as the ground shape) distorted at a first distance or after the first distance from the self-vehicle (2410) based on the driving situation of the self-vehicle (2410), for example, the driving speed, may vary depending on the driving situation of the self-vehicle (2410). Fig. 25 is an exemplary diagram illustrating an example in which a terrain object is distorted differently depending on the speed of the vehicle during the driving situation of the self-vehicle (2410).

[0744] First, referring to FIG. 25, (a) of FIG. 25 illustrates an example of a ground shape according to a distorted terrain object (2500) in an area after the first distance from the self-vehicle (2410) when the driving speed of the self-vehicle (2410) is a first speed or higher. And (b) of FIG. 25 illustrates an example of a ground shape according to a distorted terrain object (2500) in an area after the first distance from the self-vehicle (2410) when the driving speed of the self-vehicle (2410) is a second speed or lower. In this case, the first speed may be a speed faster than the second speed.

[0745] In this case, when the speed of the self-vehicle (2410) is slow (below the second speed), the ground shape after the first distance from the self-vehicle (2410) may be bent with a low curvature, as shown in (b) of FIG. 25. On the other hand, when the speed of the self-vehicle (2410) is fast (aor above the first speed), the ground shape after the first distance from the self-vehicle (2410) may be bent with a high curvature, as shown in (a) of FIG. 25. That is, the higher the driving speed of the self-vehicle (2410), the higher the distortion rate of the terrain object (2500) may be.

[0746] In this case, when the distortion rate of the terrain object (2500) increases, the distance between objects (e.g., building objects) located near the ego vehicle (2410) and the 3D map view camera (2400) may be shortened due to the bending of the ground shape caused by the distortion. Therefore, the visibility of the area near the ego vehicle (2410) may be improved. On the other hand, when the distortion rate of the terrain object (2500) decreases, a wider field of view is secured due to the ground shape bent with a low curvature, so that a wider map area may be displayed on the 3D map view screen. Therefore, the user can more easily observe an area far from the ego vehicle (2410).

[0747] FIG. 26 is an exemplary diagram showing an example of providing path guidance according to the distorted terrain object in a 3D map view generation device according to an embodiment of the present invention.

[0748] First, referring to (a) of Fig. 26, (a) of Fig. 26 illustrates an example of distorting a terrain object detected from map data of map tiles included in a field of view into a preset three-dimensional shape.

[0749] In this case, the processor (1330) may distort the map area visible within the field of view by bending the terrain object hemispherically so that the ground level becomes higher the farther away from the vehicle object (3000) with the location of the vehicle object (3000) corresponding to the location of the vehicle (100) as the center, as shown in (a) of the above-described FIG. 26. In this case, the vehicle object (3000) may be located at the center of the hemispherically distorted map area, and may be at the lowest height among the hemispherically distorted map areas.

[0750] Accordingly, as the distance from the vehicle object (3000) increases, the height of the ground displayed on the display unit (251) may increase, and in this case, the obscuration between building virtual objects due to the step may be reduced. In addition, as the height of the ground increases as the distance from the vehicle object (3000) increases, there may be no horizontal line, and thus, the map area may be displayed without restrictions due to vanishing points.

[0751] Referring to (b) of Fig. 26, (b) of Fig. 26 shows an example in which an MR view screen according to a digital twin map is displayed in a state in which a terrain object is not distorted according to the location of the vehicle (100). In this case, since the terrain object has a map horizontal plane shape, the building objects displayed on the terrain object may be smaller according to the perspective projection principle as the distance from the vehicle object (3000) increases. In addition, since the building objects become smaller as they get closer to the horizon, virtual objects may no longer be displayed from the horizon. In addition, the guidance location displayed on the display unit (251) may also be covered by other building objects, and objects in front of the vehicle that can be identified by the user on the 3D map view may be limited to within a certain distance from the vehicle object (3000).

[0752] Meanwhile, referring to (c) of FIG. 26, where the terrain object is distorted hemispherically around the position of the vehicle (100), the terrain object may be distorted so that the ground level becomes higher the farther away from the vehicle object (3000) is from the vehicle object. In this case, the terrain object may be distorted hemispherically as shown in (a) of FIG. 26. In addition, when the terrain object is distorted hemispherically, a horizontal line may not be formed due to the shape of the ground area according to the distorted terrain object.

[0753] Accordingly, since the limitation of the horizontal line is eliminated, the field of view in front of the vehicle that can be displayed within the field of view in front of the vehicle can be extended. In addition, since the distortion creates a step difference the farther away from the vehicle object (3000), even when the zoom level of the map is the same, 3D objects located at a further distance can be displayed on the display unit (251) screen without being obscured.

[0754] Meanwhile, the above description has been given as an example of a case where there is a single navigation guidance location, but it is of course possible that there may be multiple navigation guidance locations. In this case, the 3D map view generation device (1300) according to an embodiment of the present invention may distort the ground shape between each of the multiple navigation guidance locations with different curvatures.

[0755] FIG. 27 illustrates an example of distorting a terrain object (2500) so that the terrain between multiple guidance locations differs depending on the distance from the vehicle in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0756] Referring to FIG. 27, when there are multiple navigation guidance locations (first guidance location (2451), second guidance location (2452)) after a first distance, the processor (1330) can divide the regions into a region (first guidance region (2701)) corresponding to the distance (first guidance distance) from after the first distance to the first guidance location (2451) and a region (second guidance region (2702)) corresponding to the distance (second guidance distance) from after the first guidance location (2451) to the second guidance location (2452). In addition, for each of the divided regions, at least a portion of the terrain object (2500) can be distorted so that the ground shape is distorted with different curvatures.

[0757] In this case, the processor (1330) may not distort the ground shape of the area within the first distance (the first area (2401)). In addition, the processor may distort a part of the terrain object (2500) so that the ground shape of the area corresponding to the distance after the first distance among the distances from the self-vehicle (2410) to the first guidance position (2451), that is, the first guidance area (2701), is distorted with the first curvature. In addition, the ground shape of the area after the first guidance distance among the distances from the self-vehicle (2410) to the second guidance position (2452), that is, the second guidance area (2702) corresponding to the second guidance distance, is distorted with the second curvature. In addition, the processor (1330) may distort a part of the terrain object (2500) so that the ground shape of the area corresponding to the second guidance distance, that is, the second guidance area (2702), is distorted with the second curvature.

[0758] For example, the first curvature may have a smaller value than the second curvature. Alternatively, the first curvature may have a larger value than the second curvature. Alternatively, the first curvature and the second curvature may be curvatures that distort the ground shape in different directions.

[0759] For example, as shown in the above-described FIG. 27, the first curvature may be a curvature that distorts the terrain object (2500) such that the ground shape is distorted in the direction of the sky (e.g., a curvature having a positive value). Conversely, the second curvature may be a curvature that distorts the terrain object (2500) such that the ground shape is distorted in the direction of the ground downward (e.g., a curvature having a negative value).

[0760] In this case, as shown in Fig. 27, due to the curvature having the negative value, the ground shape in the area (first guidance area (2701)) between the first guidance position (2451) and the second guidance position (2452) may be bent in the opposite direction to the direction in which the ground shape is bent in the second guidance area (2702). Therefore, as shown in Fig. 27, the distance between the first guidance position (2701) and the second guidance position (2702) may be shortened along the viewing direction of the 3D map view camera (2400). Therefore, the second guidance position (2452), which is spaced apart from the vehicle (2410) by the first distance, the first guidance distance, and the second guidance distance, may be included within the field of view (2405) of the 3D map view camera (2400) together with the first guidance position (2451). Accordingly, multiple guidance locations (first guidance location (2451), second guidance location (2452)) can be displayed on one screen.

[0761] In this way, the processor (1330) can determine a curvature for distorting the ground shape between each of the plurality of guidance locations, so that the plurality of guidance locations having different distances from the self-driving car (2410) can all be displayed on a single screen. That is, the processor (1330) can determine at least one of the size of the area where distortion occurs and the direction in which the distortion occurs by setting different curvatures for the ground areas between the plurality of guidance locations having different distances from the self-driving car (2410).

[0762] In addition, although the above description has been given as an example that the first curvature and the second curvature have different polarities (positive or negative), it is obvious that the first curvature and the second curvature may have the same polarity. For example, if both the first curvature and the second curvature have positive values, in FIG. 27, the ground shape between the first guide position (2701) and the second guide position (2702) may be distorted in a downward direction toward the ground.

[0763] Meanwhile, the above terrain distortion can be applied to various types of 3D map views.

[0764] For example, in the above description, when a front view image corresponding to the forward direction of the vehicle is provided from a 3D map view camera positioned at a certain height from the vehicle, an example is described in which the shape of the ground in an area (second area) exceeding an area (first area) up to a first distance from the vehicle is distorted. However, unlike this front view, it is obvious that the distortion of the terrain object may also be applied even when a top-down view in the ground direction is provided from above the vehicle (2410).

[0765] FIG. 28 illustrates an example of a ground shape being distorted when a top-down view is provided in this manner in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0766] First, the upper drawing of Fig. 28 is an example of a cross-section when a top-down view is provided centered on the position of the vehicle (2810). For example, when a top-down view is selected at the user's request, the processor (1330) can align the line of sight of the 3D map view camera (2800) perpendicular to the ground and move it to a preset height from the vehicle (2810). Accordingly, the 3D map view camera (2800) can provide a 3D map view in the form of looking down on the ground from the air centered on the position of the vehicle (2810), as shown in the lower drawing of Fig. 28.

[0767] In this case, the area shown by the 3D map view, i.e., the field of view, can be determined according to the field of view (2830) of the 3D map view camera (2800) and the height of the 3D map view camera (2800). In this case, if the field of view (2830) of the 3D map view camera (2800) is fixed, the area shown by the 3D map view can be determined according to the height of the 3D map view camera (2800). In addition, the processor (1330) determines at least one map tile as the field of view range based on the determined field of view of the 3D map view camera (2800), and detects meshes for configuring each object, i.e., original meshes, from information of objects included in the map data of each of the determined at least one map tile.

[0768] Meanwhile, as described above, the 3D map view generation device (1300) according to an embodiment of the present invention can prevent distortion of the ground shape of the first area (2801) within the first distance from the vehicle (2810). However, the ground shape of the second area (2802) corresponding to the second distance, that is, the distance exceeding the first distance from the vehicle (2810) and reaching the field of view limit of the 3D map view camera (2800), can distort the terrain object (2500) to be bent according to a preset curvature.

[0769] Accordingly, objects (e.g., building objects) included in a map area corresponding to a first area (2801) whose diameter corresponds to a first distance centered on a vehicle (2810) may be displayed in the form of a conventional top-down map view, as shown in the lower drawing of Fig. 28. On the other hand, objects included in a map area between the field of view limit of the 3D map view and the first area (2801) may be displayed in a distorted manner according to a ground shape that is bent according to a preset curvature.

[0770] Meanwhile, as described above, the field of view of the 3D map view can be determined based on the height from the ground at which the 3D map view camera (2800) is positioned. Accordingly, a range within which the ground shape is not distorted, i.e., a first distance, can be determined. In other words, in the case of a top-down view, the first distance may vary depending on the height at which the 3D map view camera (2800) is positioned, and accordingly, the location and range of the area where the ground shape is distorted may vary.

[0771] Meanwhile, as described above, the 3D map view generation device (1300) according to an embodiment of the present invention can provide a 3D map view with a distorted ground shape. In addition, since the distorted ground shape is bent at a certain curvature depending on the distance from the ego (2410), the ground shape can become more bent as the distance from the ego (2410) increases.

[0772] In this case, if the ground shape is bent from the ground toward the sky due to the distortion of the terrain object (2500), building objects located farther away from the magnetic field (2410) may be displayed at a higher position than the height of the magnetic field (2410). In this case, the processor (1330) may display building objects rendered on the distorted ground shape differently depending on the distance from the magnetic field (2410).

[0773] FIG. 29 is an exemplary diagram showing an example in which building objects are displayed on a distorted terrain object according to the height of the distorted terrain object in a 3D map view generation device according to an embodiment of the present invention.

[0774] Referring to FIG. 29, the processor (1330) can distort the terrain object (2500) so that the ground shape is bent according to a certain curvature in a second area (2402) that deviates from the first area (2401) that is within a first distance from the ego vehicle (2410). Accordingly, as shown in FIG. 29, building objects displayed on the terrain object (2500) that becomes distorted as the distance from the ego vehicle (2410) increases can be displayed at a position higher than the height of the ego vehicle (2410).

[0775] In this case, the processor (1330) can render building objects only up to an area corresponding to a preset height (2900). That is, the processor (1330) can normally render modeled building objects for which the height of the distorted ground shape is displayed at a height (first height) that is lower than or equal to the preset height (2900).

[0776] Here, the preset height (2900) may be determined according to the position at which the 3D map view camera (2400) is positioned. For example, the height at which the 3D map view camera (2400) is positioned may be determined as the preset height (2900), or the preset height (2900) may be determined according to the upper limit of the field of view provided through the 3D map view camera (2400).

[0777] On the other hand, the processor (1330) may limit the rendering of building objects among the building objects included in the distorted ground shape that have a height (second height) exceeding the first height. In this case, only the shape of the area occupied by the building objects, i.e., the footprint shape, may be displayed for the building objects that have a height (second height) exceeding the first height. Accordingly, in the distorted ground shape that has a height exceeding the first height, only the footprint shapes of the building objects and the road objects that are distorted together with the ground shape may be displayed. Accordingly, a map area that is sufficiently far from the vehicle (2410) may be displayed with route guidance information, the shape of a road section, and the footprint shapes of the building objects that are displayed similarly to a top-down view.

[0778] Meanwhile, when the processor (1330) completes parsing and modeling of building objects from map data, the modeled building objects can be placed under the ground object, i.e., on the lower surface of the area (footprint area) corresponding to each building object on the ground object. Then, when rendering begins, the processor (1330) can smoothly move the modeled building objects from under the ground object toward the sky. Then, the building objects can be sequentially rendered from the portion exposed above the ground object. Accordingly, the sense of time axis segmentation in which the building objects are rendered can be eliminated.

[0779] In this case, when building objects are rendered, the processor (1330) may restrict the rendering of building objects in an area where the height of the ground shape bent by the distortion is greater than a certain height from the self-driving car (2410), i.e., greater than the second height. In this case, the building objects parsed and modeled from the map data may not be elevated to the ground object while being placed on the lower surface of the terrain object bent by the distortion. In other words, the rendering of the modeled building objects may not be performed.

[0780] Meanwhile, similarly, the processor (1330) may limit the height of building objects displayed on the curved ground shape based on the height of the curved ground shape according to the distance from the vehicle (2410).

[0781] As described above, the processor (1330) can distort the terrain object (2500) so that the ground shape of the second area (2402) that is a first distance away from the ego (2410) is bent according to a certain curvature. In this case, the processor (1330) can limit the rendering of the building objects included in the second area (2402) so that the height of the building objects decreases according to the distance from the ego (2410). That is, the processor (1330) can limit the height protruding above the terrain object (2500) to a certain ratio or a maximum height according to the distance from the ego (2410) in a state where models of the building objects included in the second area are placed on the lower surface of the terrain object that is bent due to distortion. Therefore, the greater the distance from the ego (2410), the lower the height at which the building object protrudes from the terrain object (2500) can be. And some of the building object models placed on the terrain object (2500) may not be rendered.

[0782] The rendering of the above building objects (2412) can be restricted by restricting them from protruding.

[0783] That is, the processor (1330) can limit the rendering of the building objects by restricting some of the building objects included in the curved ground shape from protruding from the ground, and accordingly, the height at which the building objects are displayed can be limited according to the distance from the vehicle (2410).

[0784] Here, the processor (1330) may limit the rendering of the building objects so that the height of the building objects decreases as the height of the curved ground shape approaches the first height. Accordingly, the building objects included in the ground shape that is curved due to distortion are displayed so that their height gradually decreases as they move away from the vehicle (2410), and when the preset first height is reached, a 3D map view may be provided in which the display of the building objects smoothly transitions to only the footprint area without the building objects.

[0785] Meanwhile, in the above-described FIG. 29, the rendering of building objects included in the distorted ground shape is limited according to the height from the vehicle (2410), but it is of course also possible to limit the rendering of building objects included in the distorted ground shape based on a route guidance location such as a destination or waypoint.

[0786] First, referring to FIG. 30, which illustrates an example in which a path guidance location (2451) is obscured by building objects when the terrain object is not distorted, multiple building objects (2411, 2412) may exist between the path guidance location (2451) and the vehicle (2410) when the terrain object (2500) is not distorted.

[0787] And the 3D map view camera (2400) that provides the field of view of the 3D map view can look at the route guidance location (2451) for route guidance. Then, a vector corresponding to the viewing direction of the 3D map view camera (2400) can be aligned toward the route guidance location (2451). In this case, a plurality of building objects may exist between the 3D map view camera (2400) and the route guidance location (2451), and in this case, even if the 3D map view camera (2400) is placed at a certain height, in this case, the route guidance location (2451) may be obscured by building objects colliding with the viewing direction vector (3000) as shown in FIG. 30.

[0788] In order to prevent the path guidance location (2451) from being obscured by building objects, the 3D map view generation device (1300) according to an embodiment of the present invention can distort the terrain object (2500) so that the ground shape of a second area spaced apart from the vehicle (2410) by a first distance or more is bent.

[0789] Then, the building objects (2412) included in the second area where the ground shape is distorted can be displayed to be protruded over the distorted ground shape. In this case, the building objects (2412) can be protruded in a direction perpendicular to the horizontal plane that is horizontal to the driving plane of the vehicle (2410), which is the ground shape on which the vehicle (2410) is driving, according to the ground direction according to the undistorted terrain object, as disclosed in (a) of FIG. 31. That is, although the ground shape is distorted to be bent, the building objects (2412) can be protruded in a direction perpendicular to the horizontal plane. In this case, if the building object model parsed and modeled from the map data as described above is placed under the terrain object (2500), and the rendering of the building object is performed while the building object model rises from the terrain object (2500), the building objects (2412) can be placed under the distorted terrain object (2500) in a direction perpendicular to the horizontal plane in the area where each building object (2412) is placed, i.e., the footprint area, as shown in (a) of FIG. 31. In addition, the processor (1330) can sequentially perform rendering from the portion exposed above the terrain object (2500) while moving the modeled object in a direction perpendicular to the horizontal plane. Therefore, as shown in (a) of FIG. 31, the building objects (2412) can be rendered in a curved ground shape and at an angle.

[0790] On the other hand, in contrast to this, the processor (1330) may protrude in the ground direction according to the distorted terrain object, i.e., in the direction perpendicular to the curved ground shape, as disclosed in (b) of FIG. 31. In this case, if the building object model parsed and modeled from the map data as described above is placed under the terrain object (2500), and the rendering of the building object is performed while the building object model protrudes from the terrain object (2500), the building objects (2412) may be placed under the distorted terrain object (2500) in the footprint area included in the curved ground shape in the direction perpendicular to the curved ground shape, as shown in (b) of FIG. 31. In addition, the processor (1330) may sequentially perform rendering from the portion exposed above the terrain object (2500) while moving the modeled object in the direction perpendicular to the footprint area of ​​each building object. Accordingly, as shown in (b) of Fig. 31, each building object (2412) can be rendered by rising at different angles depending on the height of the curved ground shape.

[0791] Meanwhile, when building objects (2412) are rendered in this manner, the processor (1330) can limit the height at which the building objects (2412) included in the second area (2402) are displayed according to the distance to the path guidance location (2451).

[0792] In this case, the processor (1330) may limit the rendering of the building objects (2412) included in the second area (2402) so that the displayed height of the building objects (2412) decreases as the distance from the ego vehicle (2410) increases. That is, the processor (1330) may limit the height of the building objects (2412) protruding above the terrain object (2500) to a certain ratio or a maximum height according to the distance from the ego vehicle (2410) in a state where models of the building objects (2412) included in the second area (2402) are placed on the lower surface of the terrain object that is bent due to distortion.

[0793] Accordingly, the farther the distance from the ego vehicle (2410), the lower the height at which the building object (2412) of the second area (2402) protrudes from the terrain object (2500). Accordingly, a part of the building object model placed on the bottom of the terrain object (2500) may not be rendered. That is, the processor (1330) may limit the rendering of the building objects (2412) by limiting a part of the building objects (2412) included in the curved ground shape from protruding from the ground. Accordingly, the height at which the building objects (2412) are displayed may be limited depending on the distance from the ego vehicle (2410).

[0794] Here, the processor (1330) may limit the rendering of the building objects (2412) so that the height of the building objects (2412) decreases as the height of the curved ground shape approaches the path guidance location (2451). Accordingly, the building objects (2412) included in the ground shape that is bent due to distortion are displayed so that their heights gradually decrease as they move away from the vehicle (2410), and the displayed height may be the lowest when reaching the path guidance location (2451).

[0795] Here, the building object close to the above-mentioned path guidance location (2451) may have its height limited by the height of the building object. Then, the building object close to the above-mentioned path guidance location (2451) may not protrude above the terrain object (2500). In this case, the building object close to the above-mentioned path guidance location (2451) may only display the shape of the footprint area.

[0796] Meanwhile, for building objects (2413) located after the above-described path guidance location (2451), the processor (1330) may not limit rendering. Accordingly, for building objects (2413) located after the above-described path guidance location (2451), the processor (1330) may move each building object so that it rises toward the sky by the height of each building object, and for the portion exposed above the terrain object (2500), rendering may be performed through tiling of the texture image according to the processes described in the above-described FIGS. 18a to 21.

[0797] In this case, as described in (a) and (b) of the above-described FIG. 31, building objects may protrude above the curved ground shape in a direction perpendicular to the horizontal plane, or in a direction perpendicular to the ground curved by distortion.

[0798] FIG. 32 (a) illustrates an example in which building objects protrude above a distorted terrain object (2500) in a direction perpendicular to the horizontal plane. In this case, the processor (1330) may place models of each building object, for which modeling has been completed, below the terrain object (2500) in a direction perpendicular to the horizontal plane, in the footprint area of ​​each building object included in the ground shape bent by the distortion. Then, rendering may be performed while raising each building object model in a direction perpendicular to the horizontal plane. Accordingly, each building object protruding above the distorted terrain object (2500) may protrude in a tilted manner in the direction in which the ground shape is bent due to the distortion. Accordingly, as shown in FIG. 32 (a), the building objects may be rendered as if they are lying on the bent ground shape.

[0799] In contrast, (a) of FIG. 32 illustrates an example in which building objects are projected in a direction perpendicular to the direction of the ground that has been bent due to distortion. In this case, the processor (1330) may place models of each building object, for which modeling has been completed, under the terrain object (2500) in a direction perpendicular to the direction of the bent ground in the footprint area of ​​each building object included in the ground shape that has been bent due to distortion. Then, rendering may be performed while raising each building object model in a direction perpendicular to the direction of the bent ground. Accordingly, each building object projected on the distorted terrain object (2500) may be rendered by projecting each building object on the terrain object (2500) at different angles depending on the distance from the ego (2410), as shown in (a) of FIG. 32.

[0800] Meanwhile, since 3D map views typically display objects in perspective projection, their sizes may decrease as their distance from the ego-car increases. Furthermore, the smaller the displayed size, the more difficult it is to identify objects, which poses the problem of making identification more difficult the further away an object is from the ego-car (2410).

[0801] However, as described above, when a terrain object (2500) is distorted, in the case of an object that is far from the ego vehicle (2410), even if the actual distance is far, the distance to the 3D map view camera (2400) may become closer due to the distortion. Accordingly, the 3D map view generation device (1300) according to an embodiment of the present invention may reflect the distance difference caused by such distortion and display the size of objects such as terrain, buildings, or roads that are far from the ego vehicle (2410) as larger.

[0802] FIG. 33 is an exemplary diagram illustrating an example of displaying building objects displayed on a distorted terrain object in a larger size according to distance in a 3D map view generation device (1300) according to an embodiment of the present invention. While FIG. 33 assumes a building object for convenience, it is of course not limited thereto. That is, the invention can be applied to various objects, including not only building objects but also static objects such as road objects and automobiles.

[0803] Referring to FIG. 33, the processor (1330) may distort the terrain object (2500) so that the ground shape is curved in the second area (2402) where the distance from the ego vehicle (2410) is outside the first area (2401) of the first distance. In this case, the building objects (2412) placed in the second area (2402) may be displayed with different sizes depending on the distance from the ego vehicle (2410).

[0804] For example, the processor (1330) can greatly enlarge the size of building objects (2412) arranged in the second area (2402) according to the distance from the ego-car (2410), as shown in FIG. 33. Accordingly, the building objects (2412) can increase in size simultaneously with the vertical position, i.e., height, being changed due to distortion. Accordingly, not only can the size reduced due to perspective projection depending on the distance from the ego-car (2410) be compensated for, but the size can also be increased to a size that is easy for a user to identify. Accordingly, the user can more easily observe distant objects.

[0805] Meanwhile, according to the above-described explanation, it has been explained that the increase in the size of the objects is due to the closer distance between the 3D map view camera (2400) and each object that has changed due to distortion.

[0806] Accordingly, the processor (1330) can determine the size increase rate of the objects according to the straight-line distance between the ground curved due to distortion and the 3D map view camera (2400) and each building object. In this case, regardless of the distance from the ego (2410), the closer the distance between the object protruding on the ground shape curved due to distortion and the 3D map view camera (2400), the larger the size of the object may be. On the other hand, the longer the distance between the object protruding on the ground shape curved due to distortion and the 3D map view camera (2400), the smaller the size of the object may be according to the perspective projection principle.

[0807] Alternatively, the processor (1330) may increase the size of only those building objects that satisfy preset conditions among the building objects placed on the curved ground shape for the distortion. Here, the objects that satisfy the preset conditions may be objects corresponding to route guidance, object types, or preset POIs (Points Of Interest). For example, the processor (1330) may increase the displayed size only of preset objects, for example, preset route guidance locations such as destinations or waypoints, or objects around the route guidance locations (2451). Alternatively, the processor (1330) may increase the size only of objects corresponding to preset POIs (Points Of Interest). Alternatively, the processor (1330) may increase the displayed size only of preset types of building objects, for example, landmark objects.

[0808] Meanwhile, in the above description, for convenience, it was explained assuming that the objects displayed (rendered) on the ground shape that is bent due to distortion are building objects. However, it is obvious that the present invention is not limited to this. That is, in addition to building objects, various dynamic objects such as vehicles may be displayed on the ground shape that is bent due to the distortion.

[0809] FIG. 34 is an exemplary diagram showing an example of a 3D map view generation device (1300) according to an embodiment of the present invention in which dynamic objects are displayed according to distorted terrain.

[0810] The processor (1330) of the above 3D map view generation device (1300) can detect information on fixed objects, such as terrain objects, road objects, and building objects, from map data of at least one map tile included in a viewing area. Then, the processor (1330) can model the detected objects and procedurally model and render them according to a preset order. For example, the processor (1330) can first model and render terrain objects and road objects. Then, the processor can perform modeling and rendering of building objects after the terrain and road objects have been rendered.

[0811] And the processor (1330) can collect information on dynamic objects that can move, such as vehicles, within the field of view. To this end, the processor (1330) can receive information on dynamic objects, such as the location of surrounding vehicles and traffic light status, through a multimodal recognition system that recognizes dynamic objects through their relationships by considering various types of data, such as text, images, audio, and video, acquired from an ADAS (Advanced Driver Assistance System). Then, a dynamic object can be created according to the information on the received dynamic object, and the created dynamic object can be modeled and rendered to be displayed on a 3D map. Accordingly, as shown in FIG. 34, a 3D map view including vehicle objects (3301, 3302, 3303, 3304), a traffic light object (3320), POI objects (3311, 3312), etc. can be displayed on the display unit.

[0812] In this ...

Claims

1. In the 3D map view generation device, An interface unit that receives sensing information collected from at least one sensor equipped in a vehicle; Memory for storing map data for multiple map tiles; and A processor that controls the interface unit to receive driving information of the vehicle including the location, speed, and direction of movement of the vehicle, and models and renders terrain and road objects and building objects included in map data of a plurality of map tiles determined based on the driving information, The above processor, A 3D map view generation device characterized in that it distorts the terrain and road objects so that the road and ground shapes outside an area within a preset first distance from the vehicle are bent at a preset curvature.

2. In the first paragraph, the processor, A 3D map view generation device characterized in that the first distance is determined based on at least one of a height of a 3D map view camera providing a field of view of the 3D map view, a viewing direction vector of the 3D map view camera, and a driving state of the vehicle.

3. In the first paragraph, the processor, The positions of at least some of the vertices constituting the terrain and road objects detected from the map data are modified based on a vertex coordinate transformation function defined by a preset distortion model and distortion parameters, thereby distorting at least some of the terrain and road objects. The above distortion model is, It is defined in the form of an exponential function, The above distortion parameters are, A 3D map view generation device characterized in that it includes an exponential function slope value of the distortion model determined according to a scenario in which the 3D map view is provided.

4. In the third paragraph, the processor, The curvature of the terrain and road objects is distorted differently depending on the distance between the guidance location for guiding the driving path of the vehicle and the vehicle or the driving speed of the vehicle. The above guidance location is, A 3D map view generation device characterized by at least one of a waypoint for destination or route guidance or a specific location set by a user.

5. In the fourth paragraph, the processor, Distorts the terrain and road objects with a distortion rate greater than the preset minimum distortion rate, The minimum distortion rate is, A 3D map view generation device characterized by a distortion rate that distorts the terrain object so that the guidance location exists within a preset distance from the intersection of the distorted terrain object and the upper vector of the field of view of the 3D map view camera that provides the field of view of the 3D map view.

6. In the first paragraph, the processor, Control the interface unit so that a terrain object corresponding to a map area within a preset distance centered on the current location of the vehicle is distorted, and virtual objects included in the distorted map area are displayed on the display unit of the vehicle centered on the location of the vehicle. The above distorted map area is, A 3D map view generation device characterized in that the area is distorted to be hemispherically curved around the position of the vehicle so that the ground level becomes higher as one moves away from the vehicle object corresponding to the vehicle.

7. In paragraph 1, The above processor, When there are multiple guidance locations for guiding the driving route of the vehicle, the map area is divided into each of the multiple guidance locations based on the distance from the vehicle, and the terrain and road objects are distorted to be bent at different curvatures for each divided area. The above different curvatures are, A 3D map view generation device characterized by including curvatures having different polarities (positive (+), negative (-)).

8. In the first paragraph, the processor, Modeling building objects detected from the above map data and placing the modeled building object models under a footprint area, which is the shape of the ground area occupied by each building object, included in a map area that is bent according to the distortion of the terrain object. A 3D map view generation device characterized in that it displays a 3D map view including building objects included in the curved map area by rendering at least a portion of the deployed building object models protruding from the footprint area according to the driving information of the vehicle and at least a portion of the building object models protruding from the footprint area.

9. In paragraph 8, the processor, A 3D map view generation device characterized in that the building object models included in the area where the height of the curved map area exceeds a preset height are controlled so as not to protrude from the footprint area, thereby limiting the rendering of the building objects included in the area where the height exceeds the preset height.

10. In the 8th paragraph, the processor, A 3D map view generation device characterized in that, among the above-mentioned curved map areas, the maximum height protruding from the footprint area corresponding to each building object is differently limited according to the distance exceeding the first distance, for building object models included in an area that exceeds the area corresponding to the first distance.

11. In the 10th paragraph, the processor, Among the above curved map areas, in the area where the protrusion of the building object model from the footprint area is maximally limited, the building object model is controlled so as not to protrude from the footprint area corresponding to the building object, The area where the protrusion of the above building object model is maximally limited is: A 3D map view generation device characterized in that the height of the above-mentioned curved map area is a map area that has reached a preset height.

12. In the 9th or 10th paragraph, the preset height is A 3D map view generation device characterized in that the height of a point corresponding to a guidance position for guiding the driving path of the vehicle in the above-mentioned curved map area is.

13. In the 12th paragraph, the processor, A 3D map view generation device characterized in that building objects included in an area higher than the preset height in the above-described curved map area do not have a maximum height limit protruding from the above-described footprint area.

14. In paragraph 8, the processor, A 3D map view generation device characterized in that each building object placed under the above footprint area is protruded from the footprint area in a direction perpendicular to a horizontal plane parallel to a driving surface of a vehicle or a ground shape inclined with respect to the horizontal plane along the curved map area.

15. In the first paragraph, the processor, A 3D map view generation device characterized in that the size of modeled building objects included in a map area outside the area within the first distance is enlarged in proportion to the distance from the vehicle.

16. In paragraph 15, the processor, A 3D map view generation device characterized in that the size magnification ratio of modeled building objects included in a map area outside the area within the first distance is determined based on the straight-line distance between the center of a 3D map view camera providing a field of view of the 3D map view and each of the modeled building objects.

17. In paragraph 15, the building objects whose size is enlarged are: A 3D map view generation device characterized in that the building object is a building object corresponding to a preset type of building or a building object according to a preset POI (Point Of Interest) around a guidance location for guiding the driving route of the above vehicle.

18. In the first paragraph, the processor, After the above-described terrain and road objects are modeled and rendered so as to be distorted to the above-described curvature, other modeled objects are rendered on top of the distorted terrain and road objects. Other objects modeled above are: A 3D map view generation device characterized by including dynamic objects including building objects and surrounding vehicles recognized based on information obtained from an ADAS (Advanced Driver Assistance System).

19. In the first paragraph, the processor, A 3D map view generation device characterized in that when terrain objects and road objects included in the above map data are modeled and rendered, building objects included in the above map data are modeled, and a plurality of texture images including shapes of different building elements are tiled on each side of the modeled building objects to generate the map view image including the rendered building objects.

20. In a method for controlling a 3D map view generation device, A step of receiving driving information of a vehicle; A step of determining at least one map tile included in a field of view based on the driving information and detecting map data of the determined map tile; A step of detecting vertices constituting terrain and road objects from map data; A step of modifying the coordinates of at least some of the vertices constituting the terrain and road objects based on a vertex coordinate transformation function defined by a preset distortion model and distortion parameters; A step of modeling and rendering distorted terrain and road objects such that at least some of the vertices have their coordinates modified so that at least some of them are bent with a predetermined curvature; A step of modeling a building object under at least a portion of the above distorted terrain and road objects; In response to a user request to display a map view image, the step of rendering a building object on at least a portion of the at least partially distorted terrain and road objects; and A control method of a 3D map view generation device, characterized in that it comprises a step of displaying a map view image in which building objects are rendered on at least some of the distorted terrain and road objects on a display unit of the vehicle.

Images