Apparatus for generating 3D map view on basis of mixed reality, and control method for apparatus

Abstract

The present invention relates to an apparatus for generating a 3D map view, comprising: an interface unit for receiving sensing information collected from a sensor provided in a vehicle; a memory for storing map data; and a processor for controlling the interface unit such that travel information of the vehicle is received, determining a map area from the map data on the basis of the travel information, modeling building virtual objects corresponding to respective buildings included in the map area, determining, on the basis of the travel information, some of the modeled building virtual objects as building virtual objects to be rendered, when there is a user request, tiling, into a plurality of texture images including the shapes of different building elements, respective side areas of the building virtual objects to be rendered, mapping, by building colors extracted from an image captured by a camera of the vehicle, building elements formed in the respective side areas of the building virtual objects to be rendered, and controlling the interface unit such that the building virtual objects to be rendered to which the colors are mapped are rendered.

Description

A device for generating a 3D map view based on mixed reality and a method for controlling the device

[0001] The present invention relates to a device for generating a 3D map view based on mixed reality.

[0002] Recently, following Augmented Reality (AR), which outputs graphic objects through a vehicle's windshield or HUD (Head Up Display), or uses images captured by a camera to additionally output graphic objects in the real world, technology development related to Mixed Reality (MR), which can provide various simulation information related to vehicles by applying Digital Twin (DT) technology, is actively underway.

[0003] As part of the technological development related to this mixed reality, active research is being conducted on methods that utilize mixed reality to provide drivers with route guidance-related information. Route guidance using mixed reality, for example, utilizes digital twin technology to display graphical objects corresponding to vehicles on a digitized 3D map. This map and graphical objects can be used to provide information on driving routes the driver has not yet driven, or provide a bird's-eye view, offering various information that the driver in the cockpit cannot view.

[0004] Meanwhile, for drivers to obtain accurate information through route guidance using mixed reality, the digital twin technology used to create the map—the digital twin map—must have a high degree of consistency with the real world. This requires the provision of a 3D map, not a 2D one. Therefore, service providers currently offering digital twin maps are offering 3D maps, which are essentially 2D maps mapped into 3D.

[0005] However, current service providers rely on building modeling using satellite or aerial photography to provide 3D mapping. However, the high angles at which satellite or aerial photography is used make it difficult to capture images of various aspects of a building, making it difficult to provide accurate textures. Therefore, conventional 3D maps are provided in the form of polygon data containing the building's floor coordinates (footprint) and height.

[0006] Meanwhile, 3D maps containing models provided in the form of polygon data like this have the problem of not including accurate building textures. These polygonal maps only represent the location and 3D volume of buildings, differing from the actual building shapes. This discrepancy creates a gap between the digital twin map and the real world, making it difficult to provide accurate information to drivers.

[0007] In addition, in order to provide a 3D map that includes 3D virtual objects with high-quality textures similar to actual buildings, a texture model with a three-dimensional shape similar to a high-quality real-world building or road must be mapped to each side of the modeling provided in the form of polygon data. However, since the texture model with such a three-dimensional shape is large in size and has a large number of vertices, there is a problem in that the amount of computation required for mapping the texture model is very large. Accordingly, there is a problem that a delay may occur when providing a high-quality digital twin map.

[0008] Furthermore, to provide a digital twin map that more closely resembles the real world, not only the building virtual objects but also the road virtual objects on which the vehicle objects drive must have a similar shape to the actual road. However, in the case of road virtual objects, conventional display devices simply map them with a specific color to indicate that they are roads, which causes a gap between the real world and the digital twin map. In this case, forming the road virtual object in three dimensions can provide a high-quality map view that more closely resembles the real world. However, the texture model for the road virtual object increases the computational load and computation time.

[0009] The present invention aims to solve the above-mentioned problems and other problems.

[0010] One object of the present invention is to provide a 3D map view generation device capable of displaying virtual objects having high-quality texture materials similar to real-world buildings in real time without delay according to changes in the driving condition of a vehicle, and a method for controlling the device.

[0011] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof capable of providing a more realistic mixed reality-based route guidance service through a digital twin map including road virtual objects with more realistic textures mapped corresponding to actual roads.

[0012] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof that can provide a mixed reality-based route guidance service through a digital twin map that is more similar to the real world at a lower cost.

[0013] According to an embodiment of the present invention, a 3D map view generation device includes an interface unit for receiving sensing information collected from at least one sensor provided in a vehicle, a memory for storing map data, and a processor for controlling the interface unit to receive driving information of the vehicle including the location, speed, and moving direction of the vehicle, determine a map area from the map data based on the driving information, model building virtual objects corresponding to each of buildings included in the map area, determine some of the modeled building virtual objects as rendering targets based on the driving information, and, when a user requests, tile each side area of ​​the building virtual objects to be rendered with a plurality of texture images including shapes of different building elements, map at least one of the building elements formed in each side area of ​​the building virtual objects to be rendered with a building color extracted from an image captured by a camera of the vehicle, and control the interface unit to render the building virtual objects to be rendered in which colors are mapped to each building element.

[0014] In one embodiment, the processor is characterized in that it identifies building objects from the image, detects colors of each of the identified building objects, calculates a color density, which is an area ratio on the image corresponding to each color, for each detected color, determines at least one representative color according to the calculated color density, and maps the determined at least one representative color to the at least one building element.

[0015] In one embodiment, the processor is characterized in that it detects colors having a color density of a certain level or higher as the representative colors, forms each of the colors having a color density of less than the certain level into one cluster with the most similar representative color, and adds up the color densities of at least one color included in one cluster to determine the color density of the representative color corresponding to the one cluster.

[0016] In one embodiment, the processor is characterized in that, when the representative color of each cluster is determined, a color based on the average of RGB values ​​of colors included in each cluster is determined as the representative color of each cluster.

[0017] In one embodiment, the processor is characterized in that, when there are multiple determined representative colors, it determines the priority of each representative color according to color density, and maps each representative color to each building element in order of highest priority.

[0018] In one embodiment, the processor is characterized in that it determines a priority for mapping colors according to the area of ​​each building element formed in each side area of ​​the rendering target building virtual objects, and preferentially maps building elements with high priorities to a representative color with high priority.

[0019] In one embodiment, the processor is characterized in that it determines to divide the image into a plurality of regions, identify building objects in each region, detect colors of each of the identified building objects, calculate color densities for each region, and determine at least one representative color according to the calculated color densities.

[0020] In one embodiment, the processor is characterized in that it groups the virtual objects to be rendered into a plurality of groups according to each region on the image divided into the plurality of regions, and maps each building element to at least one representative color determined in the region on the image corresponding to each group for each group.

[0021] In one embodiment, the processor is characterized in that it divides the image into a plurality of regions based on a road image object among image objects included in the image or a vanishing point detected from the image.

[0022] In one embodiment, the processor is characterized in that it detects a point where extension lines of a road object or a curb object included in the image converge or a central point of the image as the vanishing point.

[0023] In one embodiment, the processor is characterized in that it divides the areas on the image where each object is displayed into a plurality of areas according to the direction in which each object is located based on the driving direction of the vehicle, or according to the distance from the vehicle estimated based on the vanishing point.

[0024] In one embodiment, the processor is characterized in that it detects areas in which the ratio of the area occupied by building objects identified for each area is greater than a certain level, calculates the color densities for each area only for the detected areas, and determines at least one representative color according to the calculated color density.

[0025] In one embodiment, the processor is characterized in that it divides the image into a plurality of regions based on at least one of saturation or brightness of colors of each building object detected from the image.

[0026] In one embodiment, the processor is characterized in that it forms a texture of a different material for each building element by blending a normal map to which vector values ​​for forming a texture of a different material are applied for each building element identified from the tiled texture images, and maps building colors extracted from the image to each building element on which the texture is formed.

[0027] In one embodiment, the processor determines a frustum-shaped visible area that becomes wider as it gets farther away from the vehicle based on the location of the vehicle included in the driving information, according to the driving direction of the vehicle, and determines building virtual objects included in the visible area among the modeled building virtual objects as rendering targets, and the captured image is characterized in that it is an image in front of the vehicle corresponding to the visible area.

[0028] In one embodiment, the user's request is execution of a program or application related to the map view image, and the processor is characterized in that, during a runtime in which the program or application is executed, the processor renders building virtual objects determined as rendering targets among the pre-modeled building virtual objects.

[0029] In one embodiment, the processor comprises a plurality of processors, some of the plurality of processors model a plurality of building virtual objects based on the map area determined according to the received driving information through background processing, and other of the plurality of processors render some of the modeled building virtual objects in response to a request from the user.

[0030] In addition, a control method of a 3D map view generation device according to an embodiment of the present invention comprises the steps of: receiving driving information of the vehicle; determining a map area from map data based on the driving information, and modeling building virtual objects corresponding to each of buildings included in the map area; determining some of the modeled building virtual objects based on the driving information as rendering targets; receiving a user request for displaying a map view image including 3D virtual objects; in response to the user request, tiling each side area of ​​the rendering target building virtual objects with a plurality of texture images including shapes of different building elements; acquiring an image of the surroundings of the vehicle; detecting areas corresponding to buildings from the acquired image, detecting colors of the detected building areas, and determining at least one representative color based on the colors of the detected building areas; mapping at least one building element formed on each side of the rendering target building virtual objects with the determined at least one representative color; performing rendering on the rendering target building virtual objects including at least one building element to which the representative color is mapped, and generating a map view image including images of the rendered building virtual objects. It is characterized by including a step of displaying on a display unit of the above vehicle.

[0031] In one embodiment, the step of determining at least one representative color is characterized by including the steps of dividing the acquired image into a plurality of regions, detecting building regions for each region, and calculating densities of the detected colors according to the area ratio of the region occupied by each region for each detected color, determining a preset number of representative colors in order of the highest calculated density, and determining priorities of the determined representative colors in order of the highest calculated density.

[0032] In one embodiment, the step of mapping the representative color to the building element is characterized by including the steps of grouping the rendering target building virtual objects into a plurality of groups according to each region of the image divided into the plurality of groups, matching each region of the image corresponding to each group, and matching representative colors determined from each matched region, determining the priority of each building element according to the area of ​​each building element formed in each building virtual object for each group, and mapping the representative color to each building element according to the determined priority of the building elements and the priority of each representative color.

[0033] The effects of a 3D map view generation device and a control method thereof according to an embodiment of the present invention are described as follows.

[0034] First, the present invention can pre-load map data of a region corresponding to an area around a vehicle based on the vehicle's location according to vehicle driving information from map data of a wide area, and model virtual objects corresponding to buildings and roads included in the loaded region in advance. In addition, by rendering only at least a portion of the modeled virtual objects according to the vehicle's driving information, it has the advantage of providing a digital twin map containing high-quality virtual objects similar to real-world buildings and roads in real time according to the vehicle's driving information.

[0035] Second, for at least some of the modeled building objects, at least one mask map corresponding to different parts of a texture image that shapes each part of the building, and a normal map, which is an image that is superimposed on the mask map and has 3D vector values ​​that form different textures, are superimposed on the texture image so that each component of the building shaped by the texture image has different materials, and the texture image in which the mask map and the normal map are superimposed to form a surface material can be tiled on each face of at least some of the building objects. Then, by rendering and displaying on a display unit at least some of the building objects in which the texture image is tiled, it is possible to perform texturing of the building objects with a small amount of computation while including building objects having high-quality texture materials similar to the actual building, and there is an advantage in that a digital twin map can be provided that includes a building model in which a high-quality texture similar to an actual building is synthesized.

[0036] Third, the present invention can color virtual building objects around the vehicle with colors similar to those of actual buildings around the vehicle by mapping tiled texture images of each building object around the vehicle to at least one color extracted from an actual image captured from the vehicle. This has the advantage of providing a digital twin map containing high-quality building models that are depicted more closely to the real world.

[0037] FIG. 1 is a drawing showing the exterior of a vehicle according to an embodiment of the present invention.

[0038] FIG. 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles.

[0039] Figures 3 and 4 are drawings showing the interior of a vehicle according to an embodiment of the present invention.

[0040] FIGS. 5 and 6 are drawings for reference in explaining an object according to an embodiment of the present invention.

[0041] FIG. 7 is a block diagram for reference in explaining a vehicle according to an embodiment of the present invention.

[0042] Figure 8a is a conceptual diagram for explaining the AR service platform of the present invention.

[0043] Figure 8b is a conceptual diagram for explaining an MR service platform for providing the MR service of the present invention.

[0044] Figure 8c is a conceptual diagram for explaining the MR AMS client of the present invention.

[0045] Figure 8d is a conceptual diagram for explaining the MR AMS server of the present invention.

[0046] Figure 9 is a conceptual diagram for explaining the DTaaS server of the present invention.

[0047] FIG. 10 is a block diagram illustrating the structure of a 3D map view generation device according to an embodiment of the present invention.

[0048] FIG. 11 is a block diagram illustrating in more detail the structure of a 3D map view generation device according to an embodiment of the present invention.

[0049] FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view generation device and an MR service device according to an embodiment of the present invention.

[0050] FIG. 13 is a flowchart illustrating an operation process of a processor of a 3D map view generation device according to an embodiment of the present invention, modeling virtual objects included in a map area and rendering and displaying the modeled virtual objects.

[0051] FIG. 14 is an exemplary diagram illustrating an example in which a processor of a 3D map view generation device according to an embodiment of the present invention updates a map area according to driving information of a vehicle.

[0052] FIG. 15 is a block diagram illustrating a configuration in which modeling and rendering of virtual objects are performed by different processes in a 3D map view generation device according to an embodiment of the present invention having multiple processors.

[0053] FIG. 16a is an exemplary diagram showing an example of simultaneously displaying a screen showing information collected around a vehicle according to an ADAS system and an MR view screen according to a digital twin map in a 3D map view generation device according to an embodiment of the present invention.

[0054] FIG. 16b is an exemplary diagram showing a frustum-shaped visible area formed across a plurality of map tiles or one map tile, depending on the position of the vehicle and the driving direction of the vehicle, in a 3D map view generation device according to an embodiment of the present invention.

[0055] FIG. 17 and FIG. 18 are flowcharts illustrating an operation process for creating a road object included in a vehicle's viewing area based on road-related information extracted from map data in a 3D map view creation device according to an embodiment of the present invention, and exemplary diagrams illustrating examples of created road objects.

[0056] FIG. 19 is a flowchart illustrating an operation process in which a road virtual object is modeled in a 3D map view generation device according to an embodiment of the present invention.

[0057] Figure 20 is an example diagram illustrating a process in which a road virtual object is modeled according to the operation process of Figure 19 above.

[0058] FIG. 21 is an exemplary diagram showing examples in which the results of morphological operations on a road virtual object differ depending on the radius of polygon objects in a 3D map view generation device according to an embodiment of the present invention.

[0059] FIG. 22 is a flowchart illustrating an operation process in which material mapping of a road virtual object is performed in a 3D map view generation device according to an embodiment of the present invention.

[0060] FIG. 23 is a flowchart illustrating an operation process of dividing a captured image into a plurality of regions and extracting colors of building elements from each region in a 3D map view generation device according to an embodiment of the present invention.

[0061] FIG. 24 is a drawing showing an example of colors extracted from each of a plurality of areas divided according to the operation process of FIG. 23 in the 3D map view generation device of the present invention.

[0062] Figure 25 is an example diagram showing an example in which multiple areas are distinguished based on the distance from the captured image based on the vanishing point from the captured image.

[0063] Figure 26 is an example diagram showing an example of dividing a captured image into multiple regions based on saturation.

[0064] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0065] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0066] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0067] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0068] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0069] The vehicle described in this specification may include a concept that includes automobiles and motorcycles. In the following, the vehicle will be described primarily with automobiles.

[0070] The vehicle described in this specification may be a concept that includes all types of vehicles, including internal combustion engine vehicles equipped with an engine as a power source, hybrid vehicles equipped with an engine and an electric motor as power sources, and electric vehicles equipped with an electric motor as a power source.

[0071] In the following description, the left side of the vehicle means the left side of the vehicle's driving direction, and the right side of the vehicle means the right side of the vehicle's driving direction.

[0072] FIG. 1 is a drawing showing the exterior of a vehicle according to an embodiment of the present invention.

[0073] FIG. 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles.

[0074] Figures 3 and 4 are drawings showing the interior of a vehicle according to an embodiment of the present invention.

[0075] FIGS. 5 and 6 are drawings for reference in explaining an object according to an embodiment of the present invention.

[0076] FIG. 7 is a block diagram for reference in explaining a vehicle according to an embodiment of the present invention.

[0077] Referring to FIGS. 1 to 7, the vehicle (100) may include wheels that rotate by a power source and a steering input device (510) for controlling the direction of travel of the vehicle (100).

[0078] The vehicle (100) may be an autonomous vehicle.

[0079] The vehicle (100) can be switched to autonomous driving mode or manual mode based on user input.

[0080] For example, the vehicle (100) may be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on user input received through the user interface device (200).

[0081] The vehicle (100) can be switched to autonomous driving mode or manual mode based on driving situation information. The driving situation information can be generated based on object information provided by the object detection device (300).

[0082] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information generated by the object detection device (300).

[0083] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information received through the communication device (400).

[0084] The vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on information, data, and signals provided from an external device.

[0085] When the vehicle (100) is operated in autonomous driving mode, the autonomous vehicle (100) can be operated based on the driving system (700).

[0086] For example, an autonomous vehicle (100) may be driven based on information, data, or signals generated from a driving system (710), an exit system (740), or a parking system (750).

[0087] When the vehicle (100) is driven in manual mode, the autonomous vehicle (100) can receive user input for driving through the driving control device (500). Based on the user input received through the driving control device (500), the vehicle (100) can be driven.

[0088] The overall length refers to the length from the front to the rear of the vehicle (100), the overall width refers to the width of the vehicle (100), and the overall height refers to the length from the bottom of the wheel to the roof. In the following description, the overall length direction (L) may refer to the direction that serves as a reference for measuring the overall length of the vehicle (100), the overall width direction (W) may refer to the direction that serves as a reference for measuring the overall width of the vehicle (100), and the overall height direction (H) may refer to the direction that serves as a reference for measuring the overall height of the vehicle (100).

[0089] As illustrated in FIG. 7, the vehicle (100) may include a user interface device (200), an object detection device (300), a communication device (400), a driving operation device (500), a vehicle driving device (600), a driving system (700), a navigation system (770), a sensing unit (120), a vehicle interface unit (130), a memory (140), a control unit (170), and a power supply unit (190).

[0090] Depending on the embodiment, the vehicle (100) may include other components in addition to the components described herein, or may not include some of the components described herein.

[0091] The user interface device (200) is a device for communication between a vehicle (100) and a user. The user interface device (200) can receive user input and provide information generated in the vehicle (100) to the user. The vehicle (100) can implement a UI (User Interface) or UX (User Experience) through the user interface device (200).

[0092] The user interface device (200) may include an input unit (210), an internal camera (220), a biometric detection unit (230), an output unit (250), and a processor (270).

[0093] Depending on the embodiment, the user interface device (200) may include additional components other than the described components, or may not include some of the described components.

[0094] The input unit (200) is for receiving information from a user, and data collected from the input unit (120) can be analyzed by a processor (270) and processed into a user's control command.

[0095] The input unit (200) may be placed inside the vehicle. For example, the input unit (200) may be placed in an area of ​​a steering wheel, an area of ​​an instrument panel, an area of ​​a seat, an area of ​​each pillar, an area of ​​a door, an area of ​​a center console, an area of ​​a head lining, an area of ​​a sun visor, an area of ​​a windshield, or an area of ​​a window.

[0096] The input unit (200) may include a voice input unit (211), a gesture input unit (212), a touch input unit (213), and a mechanical input unit (214).

[0097] The voice input unit (211) can convert a user's voice input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0098] The voice input unit (211) may include one or more microphones.

[0099] The gesture input unit (212) can convert a user's gesture input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0100] The gesture input unit (212) may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input.

[0101] According to an embodiment, the gesture input unit (212) can detect a user's three-dimensional gesture input. To this end, the gesture input unit (212) can include a light output unit that outputs a plurality of infrared lights or a plurality of image sensors.

[0102] The gesture input unit (212) can detect a user's 3D gesture input through a TOF (Time of Flight) method, a structured light method, or a disparity method.

[0103] The touch input unit (213) can convert a user's touch input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0104] The touch input unit (213) may include a touch sensor for detecting a user's touch input.

[0105] According to an embodiment, the touch input unit (213) may be formed integrally with the display unit (251), thereby implementing a touch screen. Such a touch screen may provide both an input interface and an output interface between the vehicle (100) and the user.

[0106] The mechanical input unit (214) may include at least one of a button, a dome switch, a jog wheel, and a jog switch. An electrical signal generated by the mechanical input unit (214) may be provided to a processor (270) or a control unit (170).

[0107] The mechanical input unit (214) can be placed on a steering wheel, center fascia, center console, cockpit module, door, etc.

[0108] The internal camera (220) can capture images of the vehicle interior. The processor (270) can detect the user's status based on the images of the vehicle interior. The processor (270) can obtain information about the user's gaze from the images of the vehicle interior. The processor (270) can detect the user's gestures from the images of the vehicle interior.

[0109] The biometric detection unit (230) can obtain the user's biometric information. The biometric detection unit (230) includes a sensor capable of obtaining the user's biometric information, and can use the sensor to obtain the user's fingerprint information, heartbeat information, etc. The biometric information can be used for user authentication.

[0110] The output unit (250) is for generating output related to vision, hearing, or touch.

[0111] The output unit (250) may include at least one of a display unit (251), an audio output unit (252), and a haptic output unit (253).

[0112] The display unit (251) can display graphic objects corresponding to various information.

[0113] The display unit (251) may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0114] The display unit (251) can implement a touch screen by forming a mutual layer structure with the touch input unit (213) or forming it as an integral part.

[0115] The display unit (251) may be implemented as a HUD (Head Up Display). When the display unit (251) is implemented as a HUD, the display unit (251) may be equipped with a projection module to output information through an image projected onto a windshield or window.

[0116] The display unit (251) may include a transparent display. The transparent display may be attached to a windshield or window.

[0117] A transparent display can display a predetermined screen while having a predetermined transparency. To have transparency, the transparent display may include at least one of a transparent TFEL (Thin Film Electroluminescent), a transparent OLED (Organic Light-Emitting Diode), a transparent LCD (Liquid Crystal Display), a transmissive transparent display, and a transparent LED (Light Emitting Diode) display. The transparency of the transparent display can be adjusted.

[0118] Meanwhile, the user interface device (200) may include a plurality of display units (251a to 251g).

[0119] The display unit (251) may be arranged in one area of ​​the steering wheel, one area of ​​the instrument panel (521a, 251b, 251e), one area of ​​the seat (251d), one area of ​​each pillar (251f), one area of ​​the door (251g), one area of ​​the center console, one area of ​​the head lining, one area of ​​the sun visor, or may be implemented in one area of ​​the windshield (251c), one area of ​​the window (251h).

[0120] The audio output unit (252) converts an electric signal provided from the processor (270) or the control unit (170) into an audio signal and outputs the converted signal. To this end, the audio output unit (252) may include one or more speakers.

[0121] The haptic output unit (253) generates a tactile output. For example, the haptic output unit (253) can operate by vibrating a steering wheel, a seat belt, or a seat (110FL, 110FR, 110RL, 110RR) so that the user can perceive the output.

[0122] The processor (270) can control the overall operation of each unit of the user interface device (200).

[0123] Depending on the embodiment, the user interface device (200) may include a plurality of processors (270) or may not include a processor (270).

[0124] If the user interface device (200) does not include a processor (270), the user interface device (200) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0125] Meanwhile, the user interface device (200) may be referred to as a vehicle display device.

[0126] The user interface device (200) can be operated under the control of the control unit (170).

[0127] The object detection device (300) is a device for detecting an object located outside a vehicle (100).

[0128] Objects may be various objects related to the operation of the vehicle (100).

[0129] Referring to FIGS. 5 and 6, objects may include lanes (OB10), other vehicles (OB11), pedestrians (OB12), two-wheeled vehicles (OB13), traffic signals (OB14, OB15), lights, roads, structures, speed bumps, terrain, animals, etc.

[0130] A lane (OB10) may be a driving lane, a lane adjacent to a driving lane, or a lane in which opposing vehicles drive. A lane (OB10) may be a concept that includes lines on the left and right sides that form a lane.

[0131] Another vehicle (OB11) may be a vehicle driving around the vehicle (100). The other vehicle may be a vehicle located within a predetermined distance from the vehicle (100). For example, the other vehicle (OB11) may be a vehicle preceding or following the vehicle (100).

[0132] A pedestrian (OB12) may be a person located around a vehicle (100). A pedestrian (OB12) may be a person located within a predetermined distance from a vehicle (100). For example, a pedestrian (OB12) may be a person located on a sidewalk or roadway.

[0133] A two-wheeled vehicle (OB12) may refer to a vehicle that is positioned around a vehicle (100) and moves using two wheels. The two-wheeled vehicle (OB12) may be a vehicle with two wheels that is positioned within a predetermined distance from the vehicle (100). For example, the two-wheeled vehicle (OB13) may be a motorcycle or bicycle positioned on a sidewalk or road.

[0134] Traffic signals may include traffic lights (OB15), traffic signs (OB14), and patterns or text painted on the road surface.

[0135] The light may be generated from a lamp installed in another vehicle. The light may be generated from a streetlight. The light may be sunlight.

[0136] A road may include slopes such as road surfaces, curves, uphill and downhill slopes, etc.

[0137] Structures may be objects located along roads and fixed to the ground. For example, structures may include streetlights, street trees, buildings, utility poles, traffic lights, and bridges.

[0138] Landforms may include mountains, hills, etc.

[0139] Meanwhile, objects can be classified into moving objects and fixed objects. For example, moving objects may include concepts such as other vehicles and pedestrians. For example, fixed objects may include concepts such as traffic signals, roads, and structures.

[0140] The object detection device (300) may include a camera (310), a radar (320), a lidar (330), an ultrasonic sensor (340), an infrared sensor (350), and a processor (370).

[0141] Depending on the embodiment, the object detection device (300) may include additional components other than the described components, or may not include some of the described components.

[0142] The camera (310) may be positioned at an appropriate location outside the vehicle to capture images of the vehicle's exterior. The camera (310) may be a mono camera, a stereo camera (310a), an AVM (Around View Monitoring) camera (310b), or a 360-degree camera.

[0143] For example, the camera (310) may be positioned inside the vehicle, close to the front windshield, to capture an image of the front of the vehicle. Alternatively, the camera (310) may be positioned around the front bumper or radiator grill.

[0144] For example, the camera (310) may be positioned inside the vehicle, close to the rear glass, to capture images of the rear of the vehicle. Alternatively, the camera (310) may be positioned around the rear bumper, trunk, or tailgate.

[0145] For example, the camera (310) may be positioned close to at least one of the side windows inside the vehicle to obtain an image of the side of the vehicle. Alternatively, the camera (310) may be positioned around a side mirror, fender, or door.

[0146] The camera (310) can provide the acquired image to the processor (370).

[0147] The radar (320) may include an electromagnetic wave transmitter and receiver. The radar (320) may be implemented in a pulse radar or continuous wave radar manner based on the principle of radio wave emission. Among continuous wave radar methods, the radar (320) may be implemented in a frequency modulated continuous wave (FMCW) manner or a frequency shift keying (FSK) manner depending on the signal waveform.

[0148] The radar (320) can detect an object using electromagnetic waves, based on a TOF (Time of Flight) method or a phase-shift method, and can detect the location of the detected object, the distance to the detected object, and the relative speed.

[0149] The radar (320) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0150] The lidar (330) may include a laser transmitter and receiver. The lidar (330) may be implemented using a TOF (Time of Flight) method or a phase-shift method.

[0151] The lidar (330) can be implemented as a driven or non-driven type.

[0152] When implemented as a drive type, the lidar (330) is rotated by a motor and can detect objects around the vehicle (100).

[0153] When implemented in a non-driven manner, the lidar (330) can detect an object located within a predetermined range relative to the vehicle (100) through optical steering. The vehicle (100) can include a plurality of non-driven lidars (330).

[0154] Lidar (330) can detect an object based on a time-of-flight (TOF) method or a phase-shift method using laser light as a parameter, and can detect the position of the detected object, the distance to the detected object, and the relative speed.

[0155] The lidar (330) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.

[0156] The ultrasonic sensor (340) may include an ultrasonic transmitter and a receiver. The ultrasonic sensor (340) may detect an object based on ultrasonic waves, and may detect the location of the detected object, the distance from the detected object, and the relative speed.

[0157] The ultrasonic sensor (340) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0158] The infrared sensor (350) may include an infrared transmitter and a receiver. The infrared sensor (340) may detect an object based on infrared light, and may detect the location of the detected object, the distance to the detected object, and the relative speed.

[0159] The infrared sensor (350) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0160] The processor (370) can control the overall operation of each unit of the object detection device (300).

[0161] The processor (370) can detect and track an object based on the acquired image. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object through an image processing algorithm.

[0162] The processor (370) can detect and track an object based on the reflected electromagnetic waves that are returned when the transmitted electromagnetic waves are reflected by the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the electromagnetic waves.

[0163] The processor (370) can detect and track an object based on the reflected laser light that is reflected back by the transmitted laser beam from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the laser light.

[0164] The processor (370) can detect and track an object based on the reflected ultrasonic waves that are returned when the transmitted ultrasonic waves are reflected off the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the ultrasonic waves.

[0165] The processor (370) can detect and track an object based on the reflected infrared light that is reflected back by the transmitted infrared light from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the infrared light.

[0166] Depending on the embodiment, the object detection device (300) may include a plurality of processors (370) or may not include a processor (370). For example, each of the camera (310), radar (320), lidar (330), ultrasonic sensor (340), and infrared sensor (350) may individually include a processor.

[0167] If the object detection device (300) does not include a processor (370), the object detection device (300) can be operated under the control of the processor or control unit (170) of the device in the vehicle (100).

[0168] The object detection device (400) can be operated under the control of the control unit (170).

[0169] The communication device (400) is a device for communicating with an external device. Here, the external device may be another vehicle, a mobile terminal, or a server.

[0170] The communication device (400) may include at least one of a transmitting antenna, a receiving antenna, an RF (Radio Frequency) circuit capable of implementing various communication protocols, and an RF element to perform communication.

[0171] The communication device (400) may include a short-range communication unit (410), a location information unit (420), a V2X communication unit (430), an optical communication unit (440), a broadcast transmission / reception unit (450), and a processor (470).

[0172] Depending on the embodiment, the communication device (400) may include additional components other than the described components, or may not include some of the described components.

[0173] The short-range communication unit (410) is a unit for short-range communication. The short-range communication unit (410) can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0174] The short-range communication unit (410) can form a short-range wireless communication network (Wireless Area Network) to perform short-range communication between the vehicle (100) and at least one external device.

[0175] The location information unit (420) is a unit for obtaining location information of a vehicle (100). For example, the location information unit (420) may include a GPS (Global Positioning System) module or a DGPS (Differential Global Positioning System) module.

[0176] The V2X communication unit (430) is a unit for performing wireless communication with a server (V2I: Vehicle to Infrastructure), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). The V2X communication unit (430) may include an RF circuit capable of implementing protocols for communication with infrastructure (V2I), communication between vehicles (V2V), and communication with pedestrians (V2P).

[0177] The optical communication unit (440) is a unit for communicating with an external device via light. The optical communication unit (440) may include an optical transmission unit that converts an electrical signal into an optical signal and transmits it to the outside, and an optical reception unit that converts a received optical signal into an electrical signal.

[0178] According to an embodiment, the light transmitting unit may be formed to be integrated with a lamp included in the vehicle (100).

[0179] The broadcast transmitter / receiver (450) is a unit for receiving broadcast signals from an external broadcast management server via a broadcast channel, or transmitting broadcast signals to the broadcast management server. The broadcast channels may include satellite channels and terrestrial channels. The broadcast signals may include TV broadcast signals, radio broadcast signals, and data broadcast signals.

[0180] The processor (470) can control the overall operation of each unit of the communication device (400).

[0181] Depending on the embodiment, the communication device (400) may include a plurality of processors (470) or may not include a processor (470).

[0182] If the communication device (400) does not include a processor (470), the communication device (400) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0183] Meanwhile, the communication device (400) may implement a vehicle display device together with the user interface device (200). In this case, the vehicle display device may be referred to as a telematics device or an AVN (Audio Video Navigation) device.

[0184] The communication device (400) can be operated under the control of the control unit (170).

[0185] The driving control device (500) is a device that receives user input for driving.

[0186] In manual mode, the vehicle (100) can be driven based on signals provided by the driving control device (500).

[0187] The driving control device (500) may include a steering input device (510), an acceleration input device (530), and a brake input device (570).

[0188] The steering input device (510) can receive input for the direction of travel of the vehicle (100) from the user. The steering input device (510) is preferably formed in the form of a wheel so that steering input can be provided by rotation. Depending on the embodiment, the steering input device may be formed in the form of a touch screen, a touch pad, or a button.

[0189] The acceleration input device (530) can receive an input from a user for accelerating the vehicle (100). The brake input device (570) can receive an input from a user for decelerating the vehicle (100). The acceleration input device (530) and the brake input device (570) are preferably formed in the form of a pedal. Depending on the embodiment, the acceleration input device or the brake input device may also be formed in the form of a touch screen, a touch pad, or a button.

[0190] The driving operation device (500) can be operated under the control of the control unit (170).

[0191] The vehicle driving device (600) is a device that electrically controls the driving of various devices in the vehicle (100).

[0192] The vehicle driving device (600) may include a power train driving unit (610), a chassis driving unit (620), a door / window driving unit (630), a safety device driving unit (640), a lamp driving unit (650), and an air conditioning driving unit (660).

[0193] Depending on the embodiment, the vehicle drive device (600) may include additional components other than the described components, or may not include some of the described components.

[0194] Meanwhile, the vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0195] The power train drive unit (610) can control the operation of the power train device.

[0196] The power train drive unit (610) may include a power source drive unit (611) and a transmission drive unit (612).

[0197] The power source driving unit (611) can perform control over the power source of the vehicle (100).

[0198] For example, if a fossil fuel-based engine is the power source, the power source drive unit (610) can perform electronic control of the engine. This can control the engine output torque, etc. The power source drive unit (611) can adjust the engine output torque according to the control of the control unit (170).

[0199] For example, if an electric energy-based motor is the power source, the power source driving unit (610) can perform control over the motor. The power source driving unit (610) can adjust the rotation speed, torque, etc. of the motor according to the control of the control unit (170).

[0200] The transmission drive unit (612) can perform control over the transmission.

[0201] The transmission drive unit (612) can adjust the state of the transmission. The transmission drive unit (612) can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or parking (P).

[0202] Meanwhile, when the engine is the power source, the transmission drive unit (612) can adjust the gear engagement state in the forward (D) state.

[0203] The chassis drive unit (620) can control the operation of the chassis device.

[0204] The chassis drive unit (620) may include a steering drive unit (621), a brake drive unit (622), and a suspension drive unit (623).

[0205] The steering drive unit (621) can perform electronic control of the steering apparatus within the vehicle (100). The steering drive unit (621) can change the direction of travel of the vehicle.

[0206] The brake drive unit (622) can perform electronic control of the brake apparatus within the vehicle (100). For example, the speed of the vehicle (100) can be reduced by controlling the operation of the brakes placed on the wheels.

[0207] Meanwhile, the brake driving unit (622) can individually control each of the plurality of brakes. The brake driving unit (622) can control the braking force applied to the plurality of wheels differently.

[0208] The suspension drive unit (623) can perform electronic control of the suspension apparatus within the vehicle (100). For example, when there is a curve in the road surface, the suspension drive unit (623) can control the suspension apparatus to reduce vibration of the vehicle (100).

[0209] Meanwhile, the suspension drive unit (623) can individually control each of the plurality of suspensions.

[0210] The door / window actuator (630) can perform electronic control of a door apparatus or window apparatus in a vehicle (100).

[0211] The door / window driving unit (630) may include a door driving unit (631) and a window driving unit (632).

[0212] The door driving unit (631) can control the door device. The door driving unit (631) can control the opening and closing of a plurality of doors included in the vehicle (100). The door driving unit (631) can control the opening or closing of a trunk or tail gate. The door driving unit (631) can control the opening or closing of a sunroof.

[0213] The window driving unit (632) can perform electronic control of a window apparatus. It can control the opening or closing of a plurality of windows included in a vehicle (100).

[0214] The safety device driving unit (640) can perform electronic control of various safety devices in the vehicle (100).

[0215] The safety device drive unit (640) may include an airbag drive unit (641), a seat belt drive unit (642), and a pedestrian protection device drive unit (643).

[0216] The airbag driving unit (641) can perform electronic control of the airbag apparatus within the vehicle (100). For example, the airbag driving unit (641) can control the airbag to deploy when a danger is detected.

[0217] The seat belt drive unit (642) can perform electronic control of the seat belt apparatus within the vehicle (100). For example, the seat belt drive unit (642) can control the passenger to be secured to the seat (110FL, 110FR, 110RL, 110RR) using the seat belt when a danger is detected.

[0218] The pedestrian protection device drive unit (643) can perform electronic control of the hood lift and pedestrian airbag. For example, the pedestrian protection device drive unit (643) can control the hood lift up and the pedestrian airbag to deploy when a collision with a pedestrian is detected.

[0219] The lamp driving unit (650) can perform electronic control of various lamp apparatuses within the vehicle (100).

[0220] The air conditioning drive unit (660) can perform electronic control of the air conditioning device (air conditioner) within the vehicle (100). For example, the air conditioning drive unit (660) can control the air conditioning device to operate and supply cool air to the vehicle when the temperature inside the vehicle is high.

[0221] The vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0222] The vehicle driving device (600) can be operated under the control of the control unit (170).

[0223] The driving system (700) is a system that controls various operations of the vehicle (100). The driving system (700) can be operated in autonomous driving mode.

[0224] The driving system (700) may include a driving system (710), an exiting system (740), and a parking system (750).

[0225] Depending on the embodiment, the driving system (700) may include additional components other than the described components, or may not include some of the described components.

[0226] Meanwhile, the driving system (700) may include a processor. Each unit of the driving system (700) may individually include a processor.

[0227] Meanwhile, depending on the embodiment, if the operation system (700) is implemented in software, it may be a sub-concept of the control unit (170).

[0228] Meanwhile, according to an embodiment, the driving system (700) may be a concept including at least one of a user interface device (200), an object detection device (300), a communication device (400), a vehicle driving device (600), and a control unit (170).

[0229] The driving system (710) can drive the vehicle (100).

[0230] The driving system (710) receives navigation information from the navigation system (770) and provides a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0231] The driving system (710) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to enable driving of the vehicle (100).

[0232] The driving system (710) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0233] The exit system (740) can perform exit of a vehicle (100).

[0234] The exit system (740) receives navigation information from the navigation system (770) and provides a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0235] The exit system (740) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0236] The exit system (740) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0237] The parking system (750) can perform parking of a vehicle (100).

[0238] The parking system (750) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0239] The parking system (750) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0240] The parking system (750) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0241] A navigation system (770) can provide navigation information. The navigation information can include at least one of map information, set destination information, route information based on the set destination, information on various objects along the route, lane information, and current vehicle location information.

[0242] The navigation system (770) may include a memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system (770).

[0243] According to an embodiment, the navigation system (770) may receive information from an external device through a communication device (400) and update previously stored information.

[0244] Depending on the embodiment, the navigation system (770) may be classified as a sub-component of the user interface device (200).

[0245] The sensing unit (120) can sense the status of the vehicle. The sensing unit (120) can include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight detection sensor, a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0246] The sensing unit (120) can obtain sensing signals for vehicle attitude information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle inclination information, vehicle forward / backward information, battery information, fuel information, tire information, vehicle lamp information, vehicle internal temperature information, vehicle internal humidity information, steering wheel rotation angle, vehicle external illumination, pressure applied to an accelerator pedal, pressure applied to a brake pedal, etc.

[0247] The sensing unit (120) may further include, in addition, an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0248] The vehicle interface unit (130) can serve as a conduit for various types of external devices connected to the vehicle (100). For example, the vehicle interface unit (130) may be equipped with a port capable of connecting to a mobile terminal, and may be connected to the mobile terminal through the port. In this case, the vehicle interface unit (130) can exchange data with the mobile terminal.

[0249] Meanwhile, the vehicle interface unit (130) may serve as a conduit for supplying electrical energy to a connected mobile terminal. When the mobile terminal is electrically connected to the vehicle interface unit (130), the vehicle interface unit (130) may provide the mobile terminal with electrical energy supplied from the power supply unit (190) under the control of the control unit (170).

[0250] The memory (140) is electrically connected to the control unit (170). The memory (140) can store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory (140) can be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware. The memory (140) can store various data for the overall operation of the vehicle (100), such as programs for processing or controlling the control unit (170).

[0251] Depending on the embodiment, the memory (140) may be formed integrally with the control unit (170) or implemented as a sub-component of the control unit (170).

[0252] The control unit (170) can control the overall operation of each unit within the vehicle (100). The control unit (170) can be referred to as an ECU (Electronic Control Unit).

[0253] The power supply unit (190) can supply power required for the operation of each component under the control of the control unit (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0254] One or more processors and control units (170) included in the vehicle (100) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0255] Figure 8a is a conceptual diagram for explaining the AR service platform of the present invention.

[0256] The AR service platform that provides the AR service of the present invention may be called an AR service system.

[0257] The above AR service platform may include a server (850) installed outside the vehicle, which collects and processes information required for AR service and transmits it to the vehicle, and an AR service device (800) installed in the vehicle, which provides AR service using information transmitted from the server.

[0258] The fact that the server (850) collects and processes information required for the AR service and transmits it to the vehicle may include the meaning that the server (850) collects and processes information required for the AR service and transmits it to the AR service device (800) equipped in the vehicle.

[0259] The above AR service device (800) can change the information provided as an AR service based on the situation of the vehicle.

[0260] That is, the AR service device (800) of the present invention can dynamically adjust (variably) the information and amount of information to be displayed in AR according to the situation of the vehicle, and select information to be emphasized.

[0261] In addition, the AR service platform of the present invention can control the AR service provided in the vehicle to vary depending on specific conditions such as vehicle conditions and advertisement exposure conditions.

[0262] In the case of conventional AR navigation, when displaying destinations or major POIs (Points of Interest) in AR navigation, it is difficult to reflect the latest information because it uses information stored in map data, and there are limitations in that POIs with real-time attributes such as fueling / parking cannot be provided.

[0263] On the other hand, the AR service platform of the present invention can display vehicle location information, map data, multiple sensor data, real-time POI information, advertisement / event information, etc., by integrating them in AR navigation.

[0264] For example, to display AR information, the AR service device (800) of the present invention may receive AR service information from a server based on the current location of the vehicle and navigation route / guidance information, and process the information into a form for displaying on an AR navigation screen.

[0265] For example, the AR service device (800) of the present invention can reconstruct real-time AR display information. The AR service device (800) can reconstruct service data received from a server to display on an AR navigation screen by determining the display format, size, location, exposure method, etc. of AR content by considering the driving situation (e.g., varying the POI exposure location and size according to driving speed, changing the service information exposure location according to traffic conditions, adjusting the AR Wall display location and exposure time, etc.).

[0266] In addition, the AR service device (800) of the present invention can analyze the frequency of exposure of AR display information through user feedback.

[0267] The server (850) collects user input information (input information such as touch, order, etc.) for AR service content, performs content exposure frequency analysis, and can adjust the service content exposure policy based on the information.

[0268] Through this configuration, the present invention can express various external service contents in AR navigation by integrating them, and can provide various services through POI information including real-time properties.

[0269] In addition, the present invention enables display of various forms of AR content, such as POI information, advertisements, events, and major landmark information.

[0270] In addition, a new user experience of AR navigation can be presented through an embodiment based on the UX scenario proposed in the present invention.

[0271] The present invention can provide a service platform structure and an AR information display method (UX) that dynamically adjusts the amount of information (POI data, advertisement) to be displayed in AR according to vehicle conditions and advertisement exposure conditions, a module that collects POI information and commerce service information for AR expression and processes them into a form that is easy to render in an AR engine, a module that processes specific POI information to be emphasized according to the inside / outside vehicle conditions, a module that collects vehicle situation information and applies a UX policy according to the situation, and an AR engine module that renders AR objects (Group POI, Mini POI, 3D Object, Event wall, etc.) according to the UX policy.

[0272] In addition, the present invention can provide a client module for transmitting and receiving interaction and data between the front and rear displays of a vehicle, a Service App module for exposing commerce service information linked to POI, a client module for collecting user actions for advertisements such as AR advertisement object exposure results and clicks, and a cloud module for collecting / analyzing user actions for advertisements such as AR advertisement object exposure results and clicks.

[0273] Referring to FIG. 8a, the AR service platform of the present invention may include a server (850) that is a configuration existing outside the vehicle (off-board) and an AR service device (800) that is a configuration equipped in the vehicle (on-board).

[0274] First, the server (850) may include a POI data aggregator (851), an Ads manager (852), an Ads monitoring unit (853), a Service & Ads Manager (854), a Commerce Manager (855), a DB Connector (856), and a Dashboard (857).

[0275] The POI Data Aggregator (851) can receive information required for an AR service from multiple external servers and convert / integrate it into a message format of the AR service platform.

[0276] The Ads manager (852) can manage advertising information / content and advertising campaigns (advertisement exposure conditions).

[0277] The Ads Monitoring Department (853) can collect / save ad exposure and click results.

[0278] The Service & Ads Manager (854) can insert advertising information that meets the exposure conditions into service information and provide it to the client.

[0279] The Commerce Manager (855) can collect commerce service linkage / payment information.

[0280] The database connector (856) can store / query advertising content, advertising exposure result information, and commerce payment information.

[0281] The Dashboard (857) can display real-time AR service status visualized with advertisement exposure results / payment history results.

[0282] In addition, the server (850) may further include an AR service cloud API (or data conversion unit) for converting information transmitted from the vehicle's AR service device (800) into a data format available to the server, and for converting information processed / generated by the server into a data format available to the AR service device (800).

[0283] Meanwhile, the AR service device (800) may include a client (810) including a cloud interface, a commerce app, a CID-RSE interaction manager, a policy manager, advertisement monitoring, a driving context, personalized recommendations, etc., and an AR engine (820) including a POI renderer, a display manager, a touch manager, etc.

[0284] The client (810) can receive POI information, advertisements, etc. from the server.

[0285] Additionally, the client (810) can transmit and receive order / payment information to and from the server (850) and transmit advertisement exposure results to the server (850).

[0286] The AR engine (820) can transmit data such as the number of times an AR object output as AR is touched and the number of times it is exposed to the client (810).

[0287] Additionally, the AR engine (820) can transmit and receive front / rear seat (CID, RSE) linkage data to and from the client (810), and output an AR object according to the AR display policy received from the client (810).

[0288] In addition, the AR engine (820) can determine the type, output location, POI type, output size, etc. of AR objects provided through the AR service based on data collected from a gyro sensor, camera, communication unit, navigation, ADAS (Advanced Driver Assistance Systems), GPS, etc. equipped in the vehicle.

[0289] The AR service device (800) installed in the vehicle can AR render service content to display data transmitted from a cloud server in AR on the front camera image.

[0290] Additionally, the AR service device (800) can mediate data transmission between the server and the AR engine, such as collecting advertisement posting result data from the AR engine and transmitting it to the server.

[0291] Additionally, the AR service device (800) can link data generated by AR between CID-RSE (i.e., front / rear seats).

[0292] In addition, the AR service device (800) can perform data management for AR display policies, and specifically, can provide AR display policy data according to driving conditions to the AR engine.

[0293] In addition, the AR service device (800) can provide situational awareness and personalized services, and specifically, can provide AR objects to the AR engine according to driving situations (speed, TBT (Turn-by-Turn), etc.) using in-vehicle data.

[0294] In this specification, an example of providing an AR service by overlapping and outputting AR information (or AR object, AR content, POI information, etc.) on an image captured (received, processed) by a camera installed in a vehicle will be described.

[0295] However, the AR service described in this specification is not limited thereto, and can be applied in a similar or identical manner to various methods of implementing augmented reality, such as outputting AR information directly on the windshield of a vehicle to overlap with the real world space for a driver or passenger, or outputting AR information through a HUD (Head-up Display).

[0296] The input data (input information) used to provide AR services and the output data (output information) provided through the AR service platform are as follows.

[0297] First, the types of input data may include map data (navigation information), service content information (POI, advertisement, etc.), dynamic information, vehicle sensor information, historical information, and driving-related information.

[0298] Map data (navigation information) may include information such as route information to the destination (navigation route), guidance information (turn-by-turn), road shape ahead (road / lane), multiple map attribute information (road type / attribute, road and lane width, curvature, slope, speed limit, etc.), and localization object information (road marking, traffic sign, etc.).

[0299] Service content information (POI, advertisements, etc.) may include POI information received from multiple service providers, advertisement information available at the current location, and real-time information for reservation / payment services such as gas stations, charging stations, and parking lots.

[0300] Dynamic information may include traffic information (road-level traffic, lane-level traffic), event information (accidents, hazard warnings, etc.), weather information, V2X (V2V, V2I) (Vehicle to Everything, Vehicle to Vehicle, Vehicle to Infra), etc.

[0301] Vehicle sensor information may include current location information (GPS / DR), camera input information (ADAS information, object recognition information), and V2X (real-time surrounding situation information that can be collected through V2V and V2I).

[0302] Historical information may include information about past driving routes, traffic history (e.g., traffic volume by time of day), and communication speeds by zone and time of day.

[0303] Driving-related information may include driving mode (manual, autonomous driving, semi-autonomous driving, ADAS function operation, etc.), whether the vehicle has entered a destination or waypoint, and whether the vehicle has entered a parking lot.

[0304] Output information that can be provided through the AR service platform may include AR service display data based on current location / route.

[0305] Current location / route-based AR service display data may include AR advertisement display points along the route (AR Wall, POI building highlights), selectable AR building information (selectable main building information such as landmarks), general POI information (POI summary information such as icons or speech bubbles), remote POI information (distance / direction of important POI information that does not exist on the route but is helpful for driving), display information displayed when multiple POIs exist in the same building, destination building and real-time parking lot status information, real-time status information of gas stations / charging stations, and location-based advertisement / event information.

[0306] The AR service platform of the present invention can filter AR service information according to real-time information and determine a display method.

[0307] Specifically, the AR service platform can determine the number of real-time POI exposures based on driving speed, POI overlap removal, size adjustment, exposure time, etc.

[0308] In addition, the AR service platform can determine the POI exposure method according to risk information recognition, and specifically, can dynamically change the POI display method according to situations such as accidents, construction, and recognition of multiple moving objects.

[0309] Additionally, the AR service platform can dynamically change the POI display location when AR display visibility is reduced due to traffic.

[0310] In addition, the AR service platform can reconfigure front / rear seat AR display data. For example, the AR service information can be minimized on the front seat display and the rear seat display can be reconfigured to display all displayable information by considering driving speed, hazard information, weather information, etc.

[0311] The operation / function / control method of such an AR service platform may be implemented by a server or AR service device included in the AR service platform, or by organic interaction between the server and the AR service device.

[0312] Referring to FIG. 8a, the configuration of the server (850) of the AR service platform is described in more detail as follows.

[0313] The service and advertisement manager (854) can perform a client linkage (request) function, a POI information and advertisement information integration (data processing & aggregation) function, and a client linkage (respond) function.

[0314] Specifically, the client connection (request) function may include requesting / receiving POI information (location, category) through Unified API, or requesting / receiving destination entrance location information (select one of destination coordinates / address / id) through Unified API.

[0315] Here, Unified API refers to an API defined in the AR service cloud that is independent of a specific data provider (to minimize client changes).

[0316] The POI information and advertisement information integration (data processing & aggregation) function may include integrating POI information and advertisement information within a radius of 000m from the location requested by the client (from data manager, ads manager) or integrating the entrance location of the destination requested by the client and POI advertisement information (from data manager, ads manager).

[0317] Specifically, the POI information and advertisement information integration function may include a function to align advertisement information including building wall and event wall information with POI information, or, if there are multiple POIs in the same building, to filter them by priority on the server (e.g., leaving only partner companies and excluding other POI information).

[0318] Here, the filtering criteria may include assigning priority scores to each POI and comparing them.

[0319] Client response functionality may include transmitting POI information and advertisement information via Unified API, or transmitting destination entrance location and advertisement information via Unified API.

[0320] The data manager (not shown) included in the server (850) may include a POI information collection / transmission function, a building shape (polygon) information collection / transmission function, and a destination entrance information collection / transmission function.

[0321] The POI information collection / transfer function can perform the function of requesting POI information from a 3rd party API or transmitting (to Service & Ads Aggregator) POI information received from a 3rd party API (by converting it into a Unified API response format).

[0322] The building shape (polygon) information collection / transfer function can request building exterior shape information from a 3rd party API / data set, or perform the function of transmitting (to Service & Ads Aggregator) POI information received from a 3rd party API (by converting it into a Unified API response format).

[0323] The destination entrance information collection / transfer function can request destination entrance information from a 3rd party API or transmit destination entrance information received from a 3rd party API (by converting it into a Unified API response format) (to Service & Ads Aggregator).

[0324] The Ads Manager (852) can provide a partner (advertising) company management interface, an advertising campaign management interface, and an advertising content management interface.

[0325] The Ads Monitoring Department (853) can perform the function of receiving advertising effectiveness measurement feedback and the function of transmitting advertising information.

[0326] The partner (advertiser) management interface can perform POI advertiser management (add / edit / delete advertiser information) and general advertiser management (add / delete advertiser information).

[0327] POI supported ad formats may include Brand poi pin, Building wall, 3D rendering, Event wall, etc., and the supported ad format (general supported ad format) for brand advertisements that are not related to actual POI / location (e.g. Coca-Cola advertisements) may be Event wall.

[0328] The ad campaign management interface allows you to add / edit / delete ad campaigns (ad location, type, time, etc.).

[0329] The advertising content management interface can add / edit / view / delete content by advertising format (POI brand icon image, building wall image, event wall image / video, 3D rendering image).

[0330] The advertising effectiveness measurement feedback reception function may include a function for receiving advertising exposure feedback sent by the client and transmitting it to the DB Manager (CPC / CMP / CPT&P).

[0331] The advertising information delivery function may include a function that searches for and delivers advertising campaign information that should be displayed within a radius of 000m from the location requested by the Service & Ads Aggregator (in the case of CPT&P, only deliver advertisements that meet the time conditions).

[0332] The Commerce Manager (855) can perform client linkage functions, external commerce service linkage functions, and payment information management functions.

[0333] The client linkage function may include receiving requests by linking the client with Unified API, converting the request contents received with Unified API into external commerce API specifications, converting data received from the external API into the Unified API message format, and transmitting data to the client.

[0334] The Commerce Manager can convert the request content received through the Unified API into an external commerce API specification and then perform external service linkage functions using the converted content.

[0335] Converting data received from an external API into the message format of the Unified API may mean converting data received from an external service connection into the Unified API.

[0336] The external commerce service linkage function may include functions for requesting a list of nearby stores and meta information from the current location and receiving results, requesting detailed information about a specific store from the list and receiving results, requesting reservations / orders and receiving results, requesting service usage status and receiving results, and linking member information of commerce services and receiving results.

[0337] Here, the service usage status request & result reception can be used for sequence management and AR message pop-up according to the service usage status (reservation completed / parking lot entry / parking in progress / parking lot exit / reservation cancellation).

[0338] Service member information linking & result reception can be used to link commerce service member ↔ AR service member (OEM Connectivity service member) information.

[0339] The payment information management function may include a function to collect payment details (contents, amount) from external commerce services and charge fees to external commerce companies based on the payment details.

[0340] The database connector (856) can perform advertising effectiveness measurement data management functions, commerce data management functions, advertiser data management functions, advertising content data management functions, and advertising location data management functions.

[0341] The advertising effectiveness measurement data management function can save / delete CPC / CPM / CPT&P related log data and view data (by POI, brand, time, and ad type).

[0342] The commerce data management function can save / delete payment history (content, amount) from external commerce services and perform data search (by POI, brand, time, and ad type).

[0343] The advertiser data management function can save / edit / delete / view advertiser information and advertising campaign settings for each advertiser.

[0344] The advertising content data management function can save / edit / delete / view advertising content in conjunction with advertiser information.

[0345] The advertising location data management function can manage the Event wall section coordinates and Building wall coordinates (by brand) for displaying AR advertisements, and can be distinguished by coordinates directly registered by the user or specific coordinates obtained through company API linkage.

[0346] The Service Dashboard (857) can perform advertising effectiveness measurement data visualization functions and commerce service data visualization functions.

[0347] The advertising effectiveness measurement data visualization function can provide the following charts: CPC: Total number of ad clicks by company / brand (viewable by period), CPC: Aggregated chart of total number of ad clicks (viewable by period), CPM: Total number of ad impressions by company / brand (viewable by period), CPM: Aggregated chart of total number of ad impressions (viewable by period), CPT&P: Chart of number of ad clicks by company / brand (viewable by period), CPT&P: Chart of number of ad impressions by company / brand (viewable by period).

[0348] These charts can be presented in a variety of formats, including Bar graph, Line graph, Pie Chart, Word graph, and Geospatial graph.

[0349] Although CPT&P is billed per hour rather than per click or impression, it can be used as data for measuring the effectiveness of exposure.

[0350] The commerce service data visualization feature can provide a chart of the cumulative payment amount by company (viewable by period) and a chart of the total cumulative payment amount (viewable by period).

[0351] Figure 8b is a conceptual diagram illustrating an MR service platform for providing the MR service of the present invention.

[0352] The present invention can provide an MR service platform capable of providing a mixed reality automotive meta service (MR AMS) (hereinafter, used interchangeably with MR service).

[0353] The MR service platform may be named as an MR service system, an MR navigation system, an MR platform, an MR system, etc.

[0354] An MR service platform refers to a platform capable of providing services based on mixed reality, and may include multiple independent components.

[0355] For example, the MR service platform may include an MR service device (900) (or referred to as an MR navigation engine) equipped on the vehicle, an MR AMS (hereinafter referred to as an MR AMS server) (1100) equipped on a server (or cloud) outside the vehicle (offboard), and a DTaaS (Digital Twin as a Service) server (1200).

[0356] The above MR service device (900) may include an MR AMS client (910) and an MR renderer (920).

[0357] The MR service described herein can be understood as a mixed reality navigation service for vehicles. That is, the MR service platform of the present invention can provide a vehicle interface implemented in mixed reality to a user riding in a vehicle.

[0358] The MR service provided by the MR service platform of the present invention can provide an experience of the digital world through a display inside the vehicle while driving in the real world.

[0359] Specifically, MR services can interactively provide drivers with navigation, safe driving, POI (Point of Interest), and entertainment user experiences in a virtual 3D space that combines real-world information with the digital world.

[0360] Through this, the MR service platform of the present invention can provide various UX (User Experience) that are free from the spatial and temporal constraints compared to existing camera-based (or HUD (Head up Display)-based) AR (Augmented Reality).

[0361] Here, the digital world refers to a digital twin or a digital twin smart city, and real-world information may include infrastructure data such as V2X (Vehicle to Everything) and C-ITS (Cooperative-Intelligent transport Systems) and / or surrounding perception data sensed by sensors equipped in autonomous vehicles.

[0362] In addition, the fusion described above may include the concept of fusion of vehicle and infrastructure sensor data as well as the MR service cloud (1000) (or MR AMS cloud) and MR service device (900) for implementing the present MR service platform.

[0363] Additionally, interactivity should be understood as a concept that includes not only mixed reality navigation, but also high-quality 3D image rendering and user interaction.

[0364] Meanwhile, mixed reality (MR) described in this specification may mean an environment in which virtual reality is grafted onto the real world, allowing real physical objects and virtual objects to interact.

[0365] Mixed reality (MR) can include augmented reality (AR), which adds virtual information based on reality, and augmented virtuality (AV), which adds real information to a virtual environment.

[0366] In other words, it provides a smart environment where the real and virtual are seamlessly connected, allowing users to experience richer experiences. For example, users can interact with a virtual pet in the palm of their hand, create a virtual game environment within their real-world room, play games, virtually rearrange furniture in their home, or create an environment where remote people can gather and work together.

[0367] A mixed reality vehicle meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention can help users prepare road shapes and actions in advance by displaying a preview of a future driving path.

[0368] In addition, a mixed reality automobile meta service (MR AMS) using mixed reality (MR) according to an embodiment of the present invention can improve advertising effectiveness or induce increased service usage by focusing on a specific POI (Point of Interest).

[0369] In addition, the mixed reality automobile meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention is not dependent on a specific map company and can also merge data from various map companies.

[0370] The present invention can provide an MR navigation function as one of the mixed reality automotive meta services.

[0371] In the MR navigation function, since it is implemented in the virtual world rather than overlapping the augmented reality object on the real world, it can solve the difficulties in achieving matching quality and the problem of forward occlusion that occurs in AR navigation.

[0372] Accordingly, the present invention can improve user UX by effectively expressing various contexts that were difficult to express in existing navigation through the MR navigation function.

[0373] To this end, the MR service platform of the present invention can provide a method for managing an MR context, and a method and system for obtaining metadata and 3D assets for providing an MR AMS.

[0374] The MR service platform of the present invention can render related service metadata and 3D assets to MR navigation by defining the MR context required in the digital world and modeling service metadata for obtaining the same.

[0375] Accordingly, the present invention provides a digital world experience through a display within a vehicle while driving in the real world, and can recommend and provide various additional HMI (Human Machine Interface) services to the user by utilizing the improved visibility and visibility of MR navigation.

[0376] Hereinafter, an MR service platform according to an embodiment of the present invention for providing the MR service described above will be described.

[0377] Referring to FIG. 8b, the MR service platform (or MR service system) of the present invention may include an MR service cloud (1000) provided outside the vehicle and an MR service device (900) provided in the vehicle.

[0378] The MR service cloud (1100) installed outside the vehicle (Offboard) may include at least one of an MR AMS (Mixed Reality Automotive Meta Service) server (1100) and a DTaaS (Digital Twin as a Service) server (1200).

[0379] The MR service device (900) equipped on the above vehicle may include an MR AMS client (910) and an MR renderer (920).

[0380] The above MR service device (900) can provide a navigation function (or navigation application) by interacting with the AR service device (800) and the navigation system (770).

[0381] Information required for the above navigation function can be received through user input (or user request) input through a camera (310), sensing unit (120), and user input unit (210) provided in the vehicle.

[0382] In addition, information required for the above navigation function can be received through an MR service cloud (1000) installed outside the vehicle (Offboard), and in order to receive the information required for each vehicle, information sensed or processed in the vehicle can be transmitted to the MR service cloud (1000).

[0383] The MR AMS (Mixed Reality Automotive Meta Service) server (1100) can be connected to various service providers (1300a, 1300b, 1300c) that provide online map services such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP, as illustrated in FIG. 8d. Then, based on the results of aggregating map data provided from the connected service providers (1300a, 1300b, 1300c), the server can aggregate shape information (e.g., building floor coordinates (footprint) information and height information) of each building included in the map and provide the aggregated information to the DTaaS server (1200). Here, the DTaaS server (1200) may refer to a server or device that provides a service using DTaaS, i.e., a digital twin map.

[0384] The above DTaaS may mean Digital Twin as a Service or Digital Transformation as a Service.

[0385] Meanwhile, the DTaaS server (1200) may be connected to a POI database in which POI service data for each building or each area included in the map data is stored. In addition, the DTaaS server (1200) may be connected to a 3D model database in which data of a 3D polygon model (or a 3D polygon map) for each building included in the map data is stored. Here, the 3D polygon model is a polygon model that can provide a building volume and may be a polygon model in which no texture is formed on the surface. The DTaaS server (1200) may receive service data related to POI from the connected POI database, and may receive data of 3D polygon models of each building included in the map data of an area from the connected 3D model database.

[0386] The processor (not shown) of the MR service device (900) can receive various information related to the driving of the vehicle from the object detection device (300), the sensing unit (120), and the navigation system (770). For example, the processor can receive information on an object detected in front, behind, or to the side of the vehicle from the camera (310) of the object detection device (300).

[0387] Additionally, the processor may receive information related to the vehicle's speed, driving direction, current vehicle location (GPS), etc. from a sensing unit (120) including sensors connected to each component of the vehicle, including the driving system (710). Additionally, the processor may receive information related to the vehicle's driving path from a navigation system (770).

[0388] Meanwhile, the MR service device (900) and the DTaaS server (1200) may be connected via the interface (Interface APIs) of the MR AMS. Here, the connection between the MR service device (900) and the interface (Interface APIs) of the MR AMS may be established via a wireless network connection. In this case, the MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the MR service device (900).

[0389] In this way, when connected to the MR AMS server (1100), the MR service device (900) can provide at least some of the information received from the connected components (e.g., the vehicle's camera (310), the sensing unit (120), or the user input input through the user input unit (210)) to the MR AMS server (1100) via a network connection. Then, the MR AMS server (1100) can provide 3D map data for providing mixed reality to the MR service device (900) in response to the provided information.

[0390] For example, the MR service device (900) can transmit information about objects detected around the vehicle, as well as information about the vehicle's speed, direction, and current location. In addition, it can provide information about the driving path to the MR AMS server (1100). Then, the MR AMS server (1100) can provide 3D map data of an area according to the current location of the vehicle to the MR service device (900) based on the information provided by the MR service device (900).

[0391] In this case, the MR AMS server (1100) may determine POI information based on the current vehicle location, vehicle speed, and vehicle driving path, and may provide 3D map data that further includes the determined POI information to the 3D building map. In addition, the MR AMS server (1100) may provide 3D map data that further includes information on the situation around the vehicle to the MR service device (900) based on the information on objects around the vehicle provided.

[0392] Meanwhile, the MR service device (900) can render a mixed reality image based on the 3D map data provided from the MR AMS server (1100). For example, the MR service device (900) can control the MR renderer (920) to display a 3D map screen including models of buildings around the vehicle based on the provided 3D map data. In addition, the MR renderer (920) can display a graphic object corresponding to the vehicle on the 3D map screen, and display graphic objects corresponding to the provided POI data and situation information around the vehicle on the 3D map screen.

[0393] Accordingly, an image of a virtual environment (in other words, a mixed reality (MR) image, or an MR navigation screen, or an MR navigation interface) including a three-dimensional building model similar to the shape of the current vehicle and buildings around the vehicle and graphic objects corresponding to the vehicle can be output to a display (251) equipped in the vehicle, for example, a Center Information Display (CID), a Head Up Display (HUD), a Rear Sheet Information (RSI), or a Rear Sheet Entertainment (RSE).

[0394] In this case, information related to the vehicle's driving and the surroundings of the vehicle can be provided to the driver through the virtual environment. Through the 3D map data, i.e., the digital twin map (hereinafter referred to as the "DT map"), the MR service device (900) according to an embodiment of the present invention can provide the driver with a mixed reality service.

[0395] Meanwhile, the MR AMS server (1100) can determine 3D map data and POI information or situation information around each vehicle that can be provided together with the 3D map data based on information collected from not only the MR service device (900) equipped in one vehicle but also the MR service devices (900) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect from multiple vehicles in the form of a cloud server and generate 3D map data for mixed reality based on the collected information. In addition, it can be implemented to provide mixed reality service to at least one MR service device (900) equipped in different vehicles based on the generated 3D map data.

[0396] Therefore, for convenience of explanation, in the following, a cloud or server that includes the MR AMS server (1100) and the DTaaS server (1200), and provides information such as metadata (e.g., service metadata, 3D assets), 3D polygon maps, and digital twin maps for providing mixed reality services, i.e., a digital twin map (DT map), is referred to as an MR service cloud (1000).

[0397] As illustrated in FIG. 8b, the MR service device (900) (or MR navigation engine) may include an MR AMS client (910) and an MR renderer (920).

[0398] In addition, in order to implement the MR navigation function, which is one of the MR services, the MR service device (900) can transmit and receive data with the AR service device (800) (or AR engine) and the navigation system (770) equipped in the vehicle.

[0399] The MR AMS client (910) may include a context manager (911), a scene manager (913), and a UX scenario database (914).

[0400] Additionally, the MR renderer (920) may include a DTaaS client (921), an MR visualization unit (922), and a 3D HMI framework (923).

[0401] The MR AMS client (910) can collect vehicle location information, user input, user feedback information, payment information, etc. and transmit them to the MR AMS server (1100) located outside the vehicle.

[0402] The MR AMS server (1100) can transmit at least one of metadata, service metadata, and 3D assets required to provide MR service to the MR AMS client (910) based on information received from the MR AMS client.

[0403] The MR AMS client (910) can transmit data received from the MR AMS server (910) to the MR renderer (920).

[0404] The MR renderer (920) can create a digital twin map using a 3D polygon map received from a DTaaS server (1200) and an image received from an MR AMS client (910) or a camera (310) installed in a vehicle.

[0405] Additionally, the MR renderer (920) can render data received from the MR AMS client (920) into an MR object that can be overlapped on a digital twin map, and can create a mixed reality (MR) image by overlapping the rendered MR object on the digital twin map.

[0406] Thereafter, the MR renderer (920) can output the generated mixed reality image to a display (251) equipped in the vehicle.

[0407] All components described in this specification can be implemented as separate hardware modules, and can be understood as components implemented in software block units as needed.

[0408] Below, each component that constitutes the MR service platform will be described in more detail with reference to the attached drawings.

[0409] Figure 8c is a conceptual diagram for explaining the MR AMS client of the present invention.

[0410] The MR AMS client (910) is equipped in a vehicle and can provide a mixed reality automotive meta service (MR AMS).

[0411] The MR AMS client (910) may include a context manager (911) that requests a context corresponding to a user request (or user input) from an MR AMS server (1100) installed outside the vehicle, a scene manager (913) that manages MR scene information provided to a display (251) installed in the vehicle, and a UX scenario database (914) that provides a UX rule to at least one of the context manager (911) and the scene manager (913).

[0412] Additionally, the MR AMS client (910) may further include an interface API (912) that calls a function for communicating with the MR AMS server (1100) provided outside the vehicle.

[0413] The above interface API (912) may be configured with one or more functions configured to perform communication with the MR AMS server (1100), and may use these functions to convert a data format or message format to transmit data to the MR AMS server (1100), or to convert the format of data received from the MR AMS server (1100).

[0414] The interface API (921) can transmit a context request output from the context manager (911) to the MR AMS server (1100) and receive a 3D asset corresponding to the requested context from the MR AMS server (912).

[0415] Here, the context may refer to situational information and information corresponding to the situation the vehicle is in. Furthermore, the context may also include the meaning of content.

[0416] The above 3D asset may refer to 3D object data corresponding to the requested context. Furthermore, the 3D asset may refer to a 3D graphic object that overlaps or can be updated on a digital twin image (or digital twin map).

[0417] The MR AMS client (910) may be included in the MR service device (900).

[0418] The MR service device (900) may include a user interaction handler (901) that, when a user input is received through an input unit (210) provided in the vehicle, generates an action corresponding to the user input and transmits the action to the context manager.

[0419] The above user interaction handler (901) may be included in the MR service device (900) or may be included in the MR AMS client (910).

[0420] For example, when a user input of “Find nearby Starbucks” is received through the vehicle input unit (210), the user interaction handler (901) can generate an action corresponding to the user input (e.g., “Search POI”) and transmit it to the context manager (911) provided in the MR AMS client (910).

[0421] For example, the action may be determined by a motion matching a word included in the user input, and the action may be named a command or control command.

[0422] The above context manager (911) can generate a command to request a context corresponding to an action received from a user interaction handler (901) and transmit the command to the MR AMS server (1100) via an interface API (912).

[0423] The above command may be generated based on an action (e.g., “Search POI”) received from the user interaction handler (901), and may be formed to include, for example, the current vehicle location, the type of POI to be found, and radius information (e.g., GET “Starbucks” (type of POI) WITHIN “500m” (radius) FROM “37.7795, -122.4201” (current vehicle location (latitude, longitude)).

[0424] The context manager (911) can receive current scene information (current scene) currently being output from the vehicle from the scene manager (913) and receive UX rules from the UX scenario database (914).

[0425] Additionally, the context manager (911) can receive navigation information including the current route and current location from the navigation handler (902) that handles information of the navigation system (770).

[0426] The navigation handler (902) may be provided in the MR service device (900) or may be provided in the MR AMS client (910).

[0427] The context manager (911) can generate a command to request the context based on at least one of the current scene information, the UX rule, and the navigation information.

[0428] The current scene information may include screen information currently being output on the vehicle's display (251). For example, the current scene information may include information about a mixed reality image in which an MR object and an MR interface overlap on a digital twin map.

[0429] Additionally, at least one of the context manager (911) and the scene manager (913) of the present invention can receive sensor data processed through a sensor data adapter (903) that processes information sensed through the vehicle's sensing unit (120).

[0430] The above sensor data adapter (903) may be provided in the MR service device (900) or may be provided in the MR AMS client (910). The sensor data adapter (903) may transmit processed sensor data to an AR engine handler (904) that handles data transmitted to the AR engine (or AR service device) (800).

[0431] The interface API (912) can receive metadata of a context corresponding to the command and / or a 3D asset corresponding to the context from the MR AMS server (1100).

[0432] Thereafter, the interface API (912) can transmit the received metadata and / or 3D assets to the scene manager (913).

[0433] The scene manager (913) can generate UI data using the UX rules received from the UX scenario database (914) and the metadata and 3D assets received from the interface API (912).

[0434] Thereafter, the scene manager (913) can transmit the generated UI data to the MR renderer (920) that renders the generated UI data to be output as mixed reality (MR) or a mixed reality image on the display (251) provided in the vehicle.

[0435] Additionally, the scene manager (913) can further transmit the generated UI data to an AR engine handler (904) configured to handle an AR service device (800) equipped in the vehicle.

[0436] The UX rule stored in the UX scenario database (914) may refer to information about the rules, forms, formats or templates for creating a screen, UX or user interface provided by the MR service device, and these UX rules may be predefined for each type of data.

[0437] Additionally, UX rules can be updated or modified by users or administrators.

[0438] Figure 8d is a conceptual diagram for explaining the MR AMS server of the present invention.

[0439] Referring to FIG. 8d, an MR AMS server (1100) provided offboard in a vehicle and providing a Mixed Reality Automotive Meta Service (MR AMS) may include an interface API (1101) for calling a function for communicating with an MR AMS client provided in the vehicle, a service aggregation manager (1110) for requesting and receiving a context corresponding to a request received from the MR AMS client from a service provider, and a data integration manager (1120) for loading a 3D asset corresponding to the received context from a database (3D Assets for MR Navigation Database) (1130).

[0440] The above interface API (1101) may be named a server interface API (1101) to distinguish it from the interface API (912) of the MR AMS client (910) equipped in the vehicle.

[0441] Additionally, the interface API (912) of the MR AMS client (910) may be named a vehicle interface API or an MR AMS client interface API.

[0442] The interface API (1101) provided in the MR AMS server (1100) can transmit a user request (or context request) received from the MR AMS client to the service aggregation manager (1110).

[0443] The above interface API may include a first interface API (1101) that calls a function for performing communication with the MR AMS client (910) and a second interface API (1102a, 1102b, 1102c) that calls a function for performing communication with the service aggregation manager (1110) and the service provider (1300a, 1300b, 1300c).

[0444] The above second interface API (1102a, 1102b, 1102c) can receive service data and / or map data through the interface API provided to the service provider (1300a, 1300b, 1300c).

[0445] The interface APIs provided in the second interface API (1102a, 1102b, 1102c) and the service provider (1300a, 1300b, 1300c) may include functions configured to perform mutual data transmission and reception and to convert data formats or message formats, and these functions may be used to convert data formats or message formats to enable mutual data transmission and reception.

[0446] The service aggregation manager (1110) can request the requested context from different service providers based on the type of context requested from the MR AMS client (910) equipped in the vehicle.

[0447] Specifically, the service aggregation manager (1110) may, if the type of the requested context is a first type of context, request the first type of context from a first service provider (1300a) that provides the first type of context, and if the type of the requested context is a second type of context different from the first type, request the second type of context from a second service provider (1300b) that provides the second type of context.

[0448] For example, if the type of requested context is about a POI (e.g., “Starbucks”), the service aggregation manager (1110) can request and receive context (or POI data) about the POI from the first service provider (1300a) that provides information about the POI.

[0449] Additionally, if the type of the requested context is a view of a street, the service aggregation manager (1110) can request and receive context (or imagery data) for the view of a street from a second service provider (1300b) that provides information about the view of the street.

[0450] Additionally, if the type of the requested context is a service, the service aggregation manager (1110) may request and receive context related to the service (or data about the service (e.g., service ratings or prices)) from a third-party service provider (1300c) that provides information related to the service.

[0451] Additionally, the interface API (1101) can request a confirmed service API (Expand service API calls) to the service aggregation manager (1110) based on a service (or context request) requested from the MR AMS client (910).

[0452] The service aggregation manager (1110) can request and receive information corresponding to a confirmed service from a service provider (1300a, 1300b, 1300c) based on the extended service API request, and use the information to create a service API and output it to the data integration manager (1120).

[0453] The above data integration manager (1120) can perform data enhancement based on the service API received from the service aggregation manager (1110), generate a metadata package for the requested context, and transmit it to the vehicle's MR AMS client (910) through the interface API (1101).

[0454] The above metadata package may include the 3D asset and service metadata described above. Here, the service metadata may refer to metadata for providing a service corresponding to the requested context.

[0455] Meanwhile, the interface API (1101) can transmit a 3D asset loaded from the data integration manager (1120) to the MR AMS client (910).

[0456] Meanwhile, the MR AMS server (1100) of the present invention may further include the context manager (911) described above.

[0457] That is, the context manager (911) may be included in the MR AMS client (910) and installed in the vehicle unit, may be included in the MR AMS server (1100) and installed in the server (cloud) unit, or may be installed in both units.

[0458] The above context manager (911), when equipped in the MR AMS server (1100), can be configured to manage a context corresponding to a request received from the MR AMS client (910).

[0459] The above context manager (911) may include a context handler (911a) that handles and parses a context request, a context interpreter (911b) that manages a session for interpreting a context request and creates a context set using a data model, and a context graph database (Context Graph DB or MR Context DB) (911c) that stores the data model.

[0460] Here, the context handler (911a) can receive a user request input to the MR AMS client through the interface API (1101), parse the received user request, and transmit it to the context interpreter (911b).

[0461] The above context interpreter (911b) can create a query for a context request corresponding to the user request after creating a session, and request and receive a context data model corresponding to the query from the context graph database (911c).

[0462] The context interpreter (911b) requests a context corresponding to the context data model from the service aggregation manager (1110), and the service aggregation manager (1110) can request and receive context data corresponding to the context data model from the service provider (1300a, 1300b, 1300c).

[0463] The above service aggregation manager (1110) can request and receive a 3D asset (and / or service metadata) corresponding to the requested context from the data integration manager (1120), and transmit the context data received from the service provider and the 3D asset (and / or service metadata) received from the data integration manager to the context interpreter (911b).

[0464] The context interpreter (911b) can transmit the received context data and the 3D asset to the MR AMS client (910) equipped in the vehicle through the context handler (911a) and the interface API (1101).

[0465] Meanwhile, the context manager (911) may further include a context recorder (911d) that extracts a recommended context based on the above-mentioned generated context set, and a context controller (911e) (or context tracker) that manages contexts that must be periodically acquired.

[0466] The above context recorder (911d) may request the context interpreter (911b) to generate a query to recommend a service that can replace the specific service when the completed context data contains information that a specific service is unavailable.

[0467] Figure 9 is a conceptual diagram for explaining the DTaaS server of the present invention.

[0468] Referring to FIG. 9, the DTaaS (Digital Twin as a Service, or Digital Transformation as a Service) server (1200) of the present invention is installed outside a vehicle and can provide a mixed reality automotive meta service (MR AMS). Specifically, the DTaaS server (1200) can provide a digital twin map or data necessary for creating a digital twin map (e.g., all types of information regarding a 3D polygon map or an object overlapping on a digital twin).

[0469] The DTaaS server (1200) may include a DTaaS API (1210) that calls a function for communicating with an MR service device (900) equipped in a vehicle, a database (Digital Twins Maps DB) (1220) that stores a digital twin map and a renderable three-dimensional polygon map provided to the MR service device, and a processor (1280) that transmits a three-dimensional polygon map corresponding to the location information of the vehicle received from the MR service device to the MR service device through the DTaaS API.

[0470] In addition, the DTaaS server (1200) may further include a communication unit (TeleCommunicatino Unit, TCU) (1290) that performs communication with the MR AMS server (1100) that is installed outside the vehicle and provides MR AMS service.

[0471] In addition, the DTaaS server (1200) may further include a digital twin map generation unit (Digital Twin Representation and Update Unit) (1230) that creates a digital twin map by matching an actual captured image to a 3D polygon map stored in the database (1220).

[0472] In addition, the DTaaS server (1200) may further include a dynamic model database (Dynamics Modeling DB) (1240) that stores dynamic information about moving objects received from at least one of the MR service device (900) and the MR AMS server (1100) and a scenario database (Scenarios DB) (1250) that stores information related to scenarios that can be implemented in a digital twin.

[0473] In addition, the DTaaS server (1200) may further include a simulation unit (1260) that performs a simulation corresponding to a user request on the digital twin and a visualization unit (1270) that visualizes information to be implemented on the digital twin.

[0474] All of the components described above can be implemented as independent hardware (e.g., chips or modules), and can also be implemented as software-blocked components as needed.

[0475] The above DTaaS server (1200) can transmit and receive data through the DTaaS API (1210) not only with the vehicle (100), but also with a server (FMS Server) (1280) that provides a fleet management service (or vehicle fleet management service) and a server (1290) that provides a city planning service.

[0476] For example, the DTaaS server (1200) can collect log information collected from each server from at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290).

[0477] Afterwards, the DTaaS server (1200) can store the collected log information in a log database.

[0478] The DTaaS server (1200) can provide a digital twin map for visualization in at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290) based on the collected log information.

[0479] Additionally, the DTaaS server (1200) may transmit at least one of event notification information, simulation information, and visualization information to at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290) based on the received log information.

[0480] FIG. 10 is a block diagram illustrating the structure of a 3D map view generation device (1300) connected to a cloud server (1350) according to an embodiment of the present invention.

[0481] Referring to FIG. 10, the MR AMS (Mixed Reality Automotive Meta Service) server (1100) described in FIG. 8d can be connected to various service providers (1351) that provide online map services, such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP. Furthermore, based on the results of aggregating map data provided from the connected service providers (1351), shape information of each building included in the map, for example, floor coordinate (footprint) information and height information of a building, can be aggregated and the aggregated information can be provided to DTaaS (1352). Here, DTaaS (1352) may refer to a DTaaS server (1200), i.e., a server or device that provides a service using a digital twin map.

[0482] Meanwhile, DTaaS (1352) may be connected to a POI database in which POI service data for each building or each area included in the map data is stored. In addition, it may be connected to a 3D model database in which data of a 2.5D polygon model for each building included in the map data is stored. Here, the 2.5D polygon model is a polygon model that can provide a building volume and may be a polygon model without a texture formed on the surface. The DTaaS (1352) may receive service data related to POI from the connected POI database, and may receive data of 2.5D polygon models of each building included in the map data of an area from the connected 3D model database.

[0483] The processor (1330) of the 3D map view generation device (1300) can receive various information related to the driving of the vehicle from the camera (310), the sensing unit (120), and the navigation system (770). For example, the processor (1330) can receive information on objects detected in front, behind, or to the side of the vehicle from the camera (310). In addition, the processor (1330) can receive information related to the speed, driving direction, and current location (GPS) of the vehicle from the sensing unit (120) including sensors connected to each component of the vehicle including the driving system (710). In addition, the processor (1330) can receive information related to the driving path of the vehicle from the navigation system (770).

[0484] Meanwhile, the 3D map view creation device (1300) and DTaaS (1352) may be connected through the interface (Interface APIs) of the MR AMS server (1100). In this case, the 3D map view creation device (1300) may be configured to correspond to the MR AMS client (910).

[0485] Here, the interface (Interface APIs) of the 3D map view creation device (1300) and the MR AMS server (1100) can be connected via a wireless network connection. In this case, the MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the 3D map view creation device (1300).

[0486] In this way, when connected to the MR AMS server (1100), the 3D map view generation device (1300) can provide at least some of the information received from the connected components to the MR AMS server (1100) via a network connection. Then, the MR AMS server (1100) can provide 3D map data for providing mixed reality to the 3D map view generation device (1300) in response to the provided information.

[0487] The 3D map data provided here is map data including a polygon model provided in the 3D model DB, and may include 2.5D map data providing a simple polygon model or 3D map data providing a three-dimensional virtual object.

[0488] For example, the 3D map view generation device (1300) can provide information on objects detected around the vehicle, information on the speed and direction of the vehicle, information on the current location of the vehicle, and information on the driving path of the vehicle to the MR AMS server (1100). Then, the MR AMS server (1100) can provide 2.5D map data of an area according to the current location of the vehicle to the 3D map view generation device (1300) based on the information provided from the 3D map view generation device (1300).

[0489] In this case, the MR AMS server (1100) may determine POI information based on the current vehicle location, direction, speed, and driving path, and may provide 3D map data that further includes the determined POI information to the 3D building map. In addition, the MR AMS server (1100) may provide 3D map data that further includes information on the situation around the vehicle to the 3D map view generation device (1300) based on the information on objects around the vehicle provided.

[0490] Meanwhile, the 3D map view generation device (1300) can render a mixed reality image based on the 3D map data provided from the MR AMS server (1100). For example, the 3D map view generation device (1300) can control an MR renderer to display a 3D map screen including models of buildings around the vehicle based on the provided 3D map data. In addition, the device can display a graphic object corresponding to the vehicle on the 3D map screen, and display graphic objects corresponding to the provided POI data and situation information around the vehicle on the 3D map screen.

[0491] Accordingly, an image of a virtual environment including a three-dimensional building model similar to the shape of the current vehicle and the buildings around the vehicle and graphic objects corresponding to the vehicle can be output to a display unit (251) such as a CID (Center Information Display), HUD (Head Up Display), RSI (Rear Sheet Information), or RSE (Rear Sheet Entertainment).

[0492] In this case, information related to the vehicle's driving and the surroundings of the vehicle can be provided to the driver through the virtual environment. Through the 3D map data, i.e., the digital twin map (hereinafter referred to as the "DT map"), the 3D map view generation device (1300) according to an embodiment of the present invention can provide the driver with a mixed reality service.

[0493] Meanwhile, the MR AMS server (1100) can determine 3D map data and POI information or situation information around each vehicle that can be provided together with the 3D map data based on information collected from not only a 3D map view generation device (1300) equipped in one vehicle but also 3D map view generation devices (1300) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect information from multiple vehicles in the form of a cloud server and generate 3D map data for mixed reality based on the collected information. In addition, it can be implemented to transmit MR information for providing a mixed reality service to at least one 3D map view generation device (1300) equipped in different vehicles based on the generated 3D map data.

[0494] For convenience of explanation, in the following, the MR AMS server (1100) that provides 3D map data, i.e., a digital twin map (DT map), for providing a mixed reality service by being connected to the DTaaS (1352) and the DTaaS (1352) are collectively referred to as a cloud server (1350).

[0495] Meanwhile, the 3D map view generation device (1300) may be a device that controls a display equipped in a vehicle through an interface unit. Alternatively, the 3D map view generation device (1300) may of course be a display device equipped in the vehicle. In this case, the 3D map view generation device (1300), that is, the 3D map view generation device (1300), may be equipped with a display unit, and may receive the updated DT map on which the real-world texturing has been performed from a cloud server, and directly display an MR view image including the received DT map on the display unit equipped in the 3D map view generation device (1300).

[0496] In the following description, for convenience of explanation, it is referred to as a 3D map view generation device (1300).

[0497] Meanwhile, a vehicle (100) related to the present invention may include a 3D map view generation device (1300).

[0498] The 3D map view generation device (1300) is capable of controlling at least one of the components described in FIG. 7. From this perspective, the 3D map view generation device (1300) may be a control unit (170).

[0499] Without being limited thereto, the 3D map view generation device (1300) may be a separate configuration independent from the control unit (170). When the 3D map view generation device (1300) is implemented as a component independent from the control unit (170), the 3D map view generation device (1300) may be provided in a part of the vehicle (100).

[0500] Hereinafter, for convenience of explanation, the 3D map view generation device (1300) will be described as a separate component independent from the control unit (170). The functions (operations) and control methods described for the 3D map view generation device (1300) in this specification can be performed by the control unit (170) of the vehicle. That is, all contents described in relation to the 3D map view generation device (1300) can be analogically applied to the control unit (170) in the same / similar manner.

[0501] In addition, the 3D map view generation device (1300) described in this specification may include some of the components described in FIG. 7 and various components provided in the vehicle. In this specification, for convenience of explanation, the components described in FIG. 7 and various components provided in the vehicle will be described by assigning separate names and drawing numbers.

[0502] Figure 11 is a conceptual diagram for explaining the above 3D map view generation device (1300).

[0503] A 3D map view generation device (1300) according to one embodiment of the present invention may include a communication unit (1310), an interface unit (1320), a memory (1340), and a processor (1330).

[0504] The above communication unit (1310) may be configured to perform wireless communication with at least one of the electrical components equipped in the vehicle (for example, the electrical components equipped in the vehicle as shown in FIG. 7).

[0505] In addition, the communication unit (1310) may be configured to perform communication with devices other than the vehicle, such as a mobile terminal, a server, another vehicle, infrastructure installed on the road, etc.

[0506] The above communication unit (1310) may be the communication device (400) described above, and may include at least one of the components included in the communication device (400).

[0507] The interface unit (1320) can communicate with at least one of the components provided in the vehicle.

[0508] Specifically, the interface unit (1320) can be configured to perform wired communication with at least one of the components provided in the vehicle as shown in FIG. 7.

[0509] The above interface unit (1320) receives sensing information from one or more sensors provided in the vehicle (100).

[0510] The above interface unit (1320) may be referred to as a sensor data collector.

[0511] The above interface unit (1320) collects (receives) information sensed through sensors provided in the vehicle (e.g., sensors for detecting vehicle operation (V.Sensors) (e.g., heading, throttle, break, wheel, etc.) and sensors for sensing vehicle surrounding information (S.Sensors) (e.g., Camera, Radar, LiDAR, Sonar, etc.)).

[0512] The above interface unit (1320) can transmit information sensed through a sensor equipped in the vehicle to the communication unit (1310) (or processor (1330)) so that it is reflected in a high-precision map.

[0513] The interface unit (1320) may, for example, serve as a passageway with electrical components installed in the vehicle through the vehicle interface unit (130).

[0514] The interface unit (1320) can exchange data with the vehicle interface unit (130).

[0515] The interface unit (1320) can be connected to a vehicle and serve as a passage for supplying electric energy.

[0516] For example, the route guidance device can be powered on by receiving electric energy from the vehicle's power supply unit (190) through the interface unit (1320).

[0517] Meanwhile, the present invention may include a memory (1340) that stores data supporting various functions of the 3D map view creation device (1300). For example, the memory (1340) may store a plurality of application programs (or applications) that can be executed by the processor (1330), data for the operation of the 3D map view creation device (1300), and commands.

[0518] As an example of such data, the memory (1340) may store building reference information, which is information on actual buildings corresponding to each of the building virtual objects included in the map data. The building reference information may store information on the height of the actual building, planar shape information (e.g., footprint information) which is the shape of the land occupied by the actual building, and information on the actual location of each building, i.e., the latitude and longitude of each building. In addition, the building reference information may include information on the shape of each side of each building or various three-dimensional shapes of the building. In addition, the building reference information may include information related to the use, type, and age of each building corresponding to each building virtual object.

[0519] The above memory (1340) can store building reference information corresponding to at least one of the building virtual objects included in the map data. Hereinafter, an area of ​​the memory (1340) where the building reference information is stored will be referred to as a building reference information storage unit (1343).

[0520] In addition, the memory (1340) may store building image setting values ​​including setting values ​​for angles and projection directions for generating building virtual objects, modeling setting values ​​for 2.5D building virtual objects formed of polygons, or modeling setting values ​​for 3D building virtual objects having a three-dimensional shape, as information for modeling the building virtual objects. In this way, based on the building reference information and building image setting values ​​stored in the memory (1340), the processor (1330) may calculate various building setting values ​​such as height, area, size, area of ​​each side, margin value, and offset value of the building virtual object to be generated, and may generate each building virtual object based on the calculated building setting values. Here, the building setting value may include a value related to at least one of the height, location, planar shape, three-dimensional shape, and shape of each side of the building included in the building reference information.

[0521] In addition, the memory (1340) can further store data for generating surface materials, i.e., surface textures, of each building virtual object. For example, the memory (1340) can store images of a portion of a building including shapes of various building elements. The images are images for forming the surface texture of the building (hereinafter, texture images) and can be formed as 2D images. In addition, the memory (1340) can store data of images (hereinafter, normal maps) corresponding to normal maps having different vector values ​​according to different materials so as to provide texture to the texture images. In addition, images (hereinafter, albedo maps) corresponding to albedo maps having material-specific color values ​​for each different material can be stored.

[0522] Meanwhile, the memory (1340) may store 2D images including shapes corresponding to building components. The 2D image may be an image including images corresponding to at least one building element for describing the texture of the building. In addition, each 2D image may include different building elements depending on the type, and the shapes of the included building elements may also be different. The 2D image may be a texture image that can form a surface texture (material) of the building virtual object in a form that is tiled on the surface of the building virtual object.

[0523] Meanwhile, the texture image may be synthesized with the outer surface of the building virtual object in a tiled form as described above. Here, the tiling may refer to a process of filling the surface area of ​​the building virtual object with tiles normalized to a certain size in a horizontal and vertical direction without overlapping each other. Accordingly, the texture image may refer to a tile, which is a unit image synthesized on the outer surface of the building virtual object through the tiling. Therefore, in the following description, a texture image texturing the outer surface of the building virtual object in a tiling manner may also be referred to as a tile. Here, the texturing may refer to an operation process of adding a texture representing a surface texture (material) to the surface area of ​​the building virtual object and synthesizing a texture on the surface area of ​​the building virtual object using a tile.

[0524] Additionally, the surface area of ​​the virtual object may be composed of a plurality of different sub-areas. In this case, each of the plurality of sub-areas may be matched with a different tile, and each of the plurality of areas may be tiled using different matching tiles.

[0525] Meanwhile, the above tiles may be grouped into different groups according to the characteristics of the building to which the tiles can be applied, such as the type, purpose, or size of the building. For example, the tiles may be grouped into residential buildings, commercial buildings, or officetel buildings. In this case, the tiles grouped into different groups, i.e., the tiles grouped into the residential building group, the tiles grouped into the commercial building group, and the tiles grouped into the officetel building group, may be tiles normalized to different sizes according to each group.

[0526] In addition, some of the tiles may be associated with at least one other tile. For example, the associated tiles may be tiles that are similar in shape and color. That is, the associated tiles may be tiles that differ only in at least a portion of their shape or in color. Alternatively, the associated tiles may be tiles that are similar to a certain degree in at least one of their shape and color. These associated tiles may reflect changes over time. For example, tiles that have the same shape but differ only in color may be used to reflect changes over time in a portion of a specific building.

[0527] Meanwhile, memory (1340) may contain association information regarding tiles that are interconnected in this manner. Furthermore, through association information, a specific tile can be associated with another tile. Therefore, by utilizing this association information, even tiles with different shapes or colors can be associated with each other.

[0528] The tiles stored in the above memory (1340) may form a database including each tile and tile information corresponding to each tile. In this case, the tile information stored in the database may include address information of each tile, information on a group to which each tile is included (group information), and, if there is another tile associated with the corresponding tile, association information including information on the other associated tile. Hereinafter, a database including tiles and tile information corresponding to each tile will be referred to as a tile database (DB) (hereinafter referred to as a tile DB) (1341).

[0529] Meanwhile, the memory (1340) may store map data including virtual objects. Here, the map data may be formed by connecting map data from multiple different regions in tile form. In this case, each map tile may represent a logical rectangular section corresponding to a certain area, and may be used to subdivide a wide map area. In other words, the map data may be formed by data from map tiles each corresponding to a different region.

[0530] Meanwhile, the above map data may be map data including data of a so-called 2.5D polygon model. In addition, the map data may include high-definition (HD) map data having high accuracy. The map data may be linked to a navigation system (770) and a driving system (700) provided in a vehicle (100), and the memory (1340) may provide the map data to various linked systems and components under the control of the processor (1330). Such map data may be stored in a map data storage unit (1342).

[0531] Meanwhile, a 3D map view generation device (1300) according to an embodiment of the present invention may include a processor (1330) that generates a 3D map as a digital twin using at least one of an image captured by a camera (310) provided in a vehicle (100), 2D map data (e.g., HD map data), and 3D map data (e.g., map data including data of a 2.5D polygon model). The processor (1330) controls each connected component and may control the overall operation of the 3D map view generation device (1300).

[0532] Additionally, the processor (1330) can overlap (or superimpose, output) graphic objects related to route guidance on a digital twinned 3D map.

[0533] Here, the graphic object related to route guidance refers to an object output as mixed reality (MR), and may include various types of objects (e.g., POI objects, carpet-shaped objects, 3D objects, etc.) required to perform route guidance. Here, the graphic object related to the route guidance may also be referred to as an MR object.

[0534] When the destination of the vehicle (100) is set, the processor (1330) can determine a route for the vehicle (100) to drive through the navigation system (770). And, when the driving route of the vehicle (100) is determined, the processor (1330) can determine at least one virtual object (e.g., an object output as the mixed reality (MR)) to perform texturing on a digital twin 3D map for route guidance.

[0535] Meanwhile, in order to determine virtual objects to be displayed on the digital twinned 3D map, the processor (1330) may first determine a map area in which to generate the virtual objects, i.e., to model. To this end, the processor (1330) may first detect, from map data stored in the memory (1340), at least one map tile around a map tile corresponding to an area in which the vehicle (100) is located, according to the current location of the vehicle (100) and the driving direction of the vehicle (100). For example, the processor (1330) may determine an area corresponding to 5 horizontal and 5 vertical maps centered on the map tile in which the vehicle (100) is located, i.e., a total of 25 map tile data, as a map area, according to a zoom level of a specified map, and may detect map tile data corresponding to the determined map area from the map data.

[0536] And the processor (1330) can detect buildings included in the detected map area. And it can detect building reference information corresponding to the detected buildings from the building reference information storage unit (1343). And the setting values ​​of each building image related to the detected building reference information can be determined. Here, the building image setting values ​​are information for modeling each building virtual object, and may include a setting value of a projection direction according to an angle or distance for generating the building virtual objects according to the driving direction of the vehicle (100) and / or the angle at which the MR view is displayed, a setting value of modeling according to a 2.5D virtual object or a setting value of modeling according to a 3D virtual object, for example, a vector value such as a coordinate value of a vertex or a normal line.

[0537] Once the building image setting values ​​are determined, the processor (1330) can calculate building setting values ​​corresponding to each building included in the map area based on the determined building image setting values ​​and building reference information of each building. Here, the building setting values ​​can include the height of the building virtual object to be created, the side area of ​​each side, the plane area, etc. Then, once the calculation of the building setting values ​​is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included in the map area based on the building setting values.

[0538] And when the modeling of virtual objects included in the above map area is completed, the processor (1330) can determine virtual objects to be displayed on the display unit (251) through rendering among the modeled virtual objects. To this end, the processor (1330) can determine a visible area of ​​a frustum shape formed according to the driving direction of the vehicle (100) based on the current position of the vehicle (100), determine building virtual objects included in the visible area of ​​the frustum shape as objects to be rendered, and perform texturing on the determined building virtual objects.

[0539] Meanwhile, the processor (1330) may pre-model building virtual objects corresponding to each building included in the map area, and render and display only at least some of the modeled building virtual objects on the display unit according to the visible area of ​​the frustum shape determined according to the position and driving direction of the vehicle (100). In this case, the texturing may be performed during the rendering process. Therefore, by pre-modeling objects to be rendered and selectively rendering only some of the modeled objects as needed, texturing of virtual objects that are not displayed on the display unit can be prevented. Therefore, the amount of computation required for the texturing can be reduced, and the time for virtual objects to be displayed on the display unit can be significantly reduced.

[0540] Moreover, the processor (1330) may be implemented as a plurality of processors. For example, the processor (1330) may be implemented as a processor that models the building virtual objects and a processor that renders some of the modeled virtual objects as different processors. For example, the processor that models the virtual objects may be a central processing unit (Central Processing Unit) that interprets various program commands of the 3D map view generation device (1300) according to an embodiment of the present invention and performs data calculations and processing. On the other hand, the processor that renders some of the pre-modeled virtual objects may be a GPU (Graphics Processing Unit) that processes graphic calculations in the 3D map view generation device (1300) according to an embodiment of the present invention and outputs result values. In this case, since the modeling of the virtual object and the rendering of the modeled virtual object can be processed in parallel on different processors, the computational load on the processor can be significantly reduced compared to the case where all processes are performed on a single processor, and by displaying the virtual object by performing rendering from objects that have been modeled in advance, the time required until the virtual object is displayed on the display unit can also be significantly shortened. Furthermore, the 3D map view generation device (1300) according to the embodiment of the present invention described above generates modeling of building virtual objects corresponding to each building included in the map area in advance, and renders only some virtual objects according to some areas (e.g., the visible area of ​​the frustum shape) among the generated building virtual objects, thereby further shortening the time for displaying the building virtual objects, and thus real-time provision of an MR view according to the driving of the vehicle (100) can be made possible.

[0541] In this way, since the present invention has a configuration that pre-generates modeling of building virtual objects corresponding to each building included in the map area, when the map area needs to be updated according to the location and driving direction of the vehicle (100), the processor (1330) can additionally detect at least one map tile data from the map data through the update. In addition, the virtual objects of the buildings included in the area of ​​the additionally detected map tile can be further modeled. On the other hand, the processor (1330) can delete virtual objects of the buildings included in the area of ​​the map tile that has deviated from the map area according to the update of the map area for efficient memory utilization.

[0542] Meanwhile, when the destination of the vehicle (100) or the route to the destination is determined, the processor (1330) may additionally detect at least one map tile data from the map data based on the determined destination or the route to the destination. In this case, the additionally detected at least one map tile data may be map tile data that are connected to each other. In addition, when the additionally detected at least one map tile data is additionally detected, the processor (1330) may model virtual objects corresponding to each of the buildings included in the additionally detected map tile data.

[0543] In this case, virtual objects modeled from the additionally detected map tile data may be rendered and displayed on the display unit (251) regardless of the driving information of the vehicle (100) when a preset condition is met. For example, the preset condition may be met when a user requests a search for a route to a destination or a check of the surroundings of the destination. In this case, regardless of the current driving status of the vehicle (100), building virtual objects modeled from the additionally detected map tile data may be rendered and displayed along the shortest path to the set destination or the searched driving path to the destination. In this case, texturing of the building virtual objects may be performed during the rendering process.

[0544] Meanwhile, texturing of the above-mentioned modeled virtual objects can be performed in a manner in which texture images are tiled, and can be performed by sequentially combining multiple sub-regions constituting the virtual object.

[0545] For example, the virtual object corresponding to the target building may have a surface area divided into multiple layers according to the exterior appearance of the target building, and a partial area may be formed for each of the divided surface areas. For example, the processor (1330) may divide the target building into a bottom layer, a top layer, and at least one middle layer, and different tiles may be matched to each partial area.

[0546] In this case, the top floor is distinguished when it has a shape or form that is distinct from the middle floor, such as a roof, an antenna, or a spire. If the target building does not have a structure placed on the roof of the building, such as a roof, an antenna, or a spire, the processor (1330) may distinguish the target building only by the bottom floor and at least one middle floor.

[0547] Meanwhile, in the case of an intermediate layer, the processor (1330) can be divided into at least one layer according to a preset floor height or window arrangement, etc. For example, if the intermediate layers have different colors or shapes (e.g., different window arrangements), the layers with the different colors or shapes can be distinguished as different intermediate layers. That is, in the case of floors designated as commercial spaces and floors designated as residential spaces, such as in a mixed-use building, the floors designated as commercial spaces and the floors designated as residential spaces can each be distinguished as different intermediate layers.

[0548] In this case, different layers, for example, the ground floor, at least one intermediate layer, and the top layer, can be matched with different tiles. And based on the different matching tiles, tiling can be performed for each layer, i.e., each sub-region (e.g., the ground floor, at least one intermediate layer, and the top layer). Then, each sub-region for which tiling has been performed can be sequentially combined in a preset order based on combination information specified in the building reference information corresponding to the target building.

[0549] Meanwhile, the tiling may be sequentially performed based on a tiling order specified in the building reference information. For example, each of the sub-regions may be sequentially combined starting from a sub-region corresponding to a floor close to the ground. In this case, tiling may be performed first for a sub-region corresponding to an area in contact with the ground, i.e., the bottom floor, and then tiling may be performed for a sub-region corresponding to the next order (e.g., the first middle floor) on top of the sub-region already tiled. Finally, tiling may be performed for a sub-region corresponding to the uppermost floor (e.g., the top floor). In this way, by sequentially performing tiling for each sub-region in a preset order for the virtual object, texturing may be performed for the entire surface area of ​​the virtual object.

[0550] In this way, texturing of a virtual object according to an embodiment of the present invention can be performed based on the sequential combination of each tiled sub-region. Therefore, texturing of a virtual object according to an embodiment of the present invention can also be referred to as procedural texturing.

[0551] Meanwhile, the tile DB (1341) may be provided from a preset server that is communicatively connected to the 3D map view generation device (1300). For example, the preset server may be a cloud server. In this case, the cloud server may be the same server as the cloud server that provides map data (e.g., map data including 2.5D polygon objects), but of course, they may be different servers. In this case, that is, if they are different servers, the cloud server that provides the map data will be described as a first cloud server, and the cloud server that provides the tile DB (1341) and building reference information will be described as a second cloud server.

[0552] When the above tile DB (1341) is provided, the processor (1330) can determine tiles to be tiled on the virtual object surface area of ​​each building included in an area defined based on driving information of the vehicle (100) based on information collected around the vehicle (100).

[0553] For example, the processor (1330) can segment an image of a building around a vehicle (100) acquired through a camera (310) into a plurality of partial regions and detect a tile matching each of the segmented images (i.e., partial region images of the target building). To this end, the processor (1330) can calculate a similarity for each tile included in the tile DB (1341) for each of the partial region images, and determine a tile matching each of the partial region images based on the calculated similarity.

[0554] For example, the processor (1330) can detect feature information for a certain partial region image. And, among the tiles included in the tile DB (1341), at least one tile having a feature similar to a feature detected from the certain partial region image can be detected. And, for each of the at least one detected tiles, a similarity to the certain partial region image can be calculated, and a certain tile having the highest calculated similarity can be matched as a tile corresponding to the certain partial region image.

[0555] For this tile matching, the processor (1330) may utilize a pre-trained artificial intelligence (AI) model. In this case, the AI ​​model may recognize a partial region image from features detected in the partial region image, and detect a tile having features most similar to the feature points of the recognized partial region image.

[0556] Here, the feature points of the partial region image can be detected in various ways. For example, the shape, form, or color pattern of the image can be detected as the feature point. In this case, the more distinct the shape, form, or color pattern of the partial region image, the more clearly the AI ​​model can detect the pattern from the partial region image, resulting in a higher recognition rate.

[0557] On the other hand, if it is difficult to detect a pattern of shape, form or color from the partial region image due to low image quality, brightness difference, shaking, etc., the artificial intelligence model may not be able to detect the feature points of the partial region image, in which case a low recognition rate may be produced. Then, based on the recognition rate, the artificial intelligence model may not detect a tile corresponding to the partial region image. In other words, the artificial intelligence model may determine whether to detect a tile corresponding to any one of the partial region images based on the produced recognition rate.

[0558] Here, the artificial intelligence model may be included in the 3D map view generation device (1300) according to an embodiment of the present invention as a separate component from the processor (1330) (not shown). Alternatively, the artificial intelligence model may be part of the processor (1330), or the processor (1330) may perform the function of the artificial intelligence model. In this case, the artificial intelligence model may be understood as the same component as the processor (1330).

[0559] Meanwhile, when the tiles corresponding to each partial region image are determined, the processor (1330) can generate combination information related to the combination of each partial region image based on the acquired building image. For example, the combination information may be information about the location of each partial region image or the order in which tiling is performed for the acquired building image.

[0560] And the processor (1330) can model virtual objects of buildings located in an area defined according to the location of the vehicle (100) from map data. And, among the area, building reference information of buildings located in at least a part of the area (hereinafter referred to as a map area) defined according to the driving information of the vehicle (100) can be detected. And, the processor (1330) can perform texturing on virtual objects corresponding to the detected building reference information among the modeled virtual objects.

[0561] In more detail, the processor (1330) can sequentially tile surface areas of virtual objects corresponding to the detected building reference information using tiles pre-designated for each sub-area.

[0562] For example, if the target building is formed of three layers, including a ground floor, a first intermediate floor, and a second intermediate floor, the processor (1330) may sequentially tile tiles matching each sub-area in a preset order onto the surface area of ​​a virtual object corresponding to the target building. Here, the preset order may be an order from the ground.

[0563] Accordingly, tiles matching the bottom area, i.e., the partial area (bottom layer) included in the building reference information, can be tiled on the surface of a virtual object in an area in contact with the ground in the virtual object in the map data corresponding to the target building. Then, when tiling for the bottom layer is completed, the processor (1330) can tile tiles matching the next partial area (first intermediate layer) in a preset order on the partial area (bottom layer) for which tiling has been completed. Then, when tiling for the first intermediate layer is completed, the processor (1330) can tile tiles matching the next partial area (second intermediate layer) in a preset order on the partial area (first intermediate layer) for which tiling has been completed. In this way, by sequentially tiling the tiles matching each partial area on the surface area of ​​the target building, texturing can be performed on the virtual object in the map data corresponding to the target building.

[0564] Here, the height between each floor of the target building may correspond to an integer multiple of the tile. For example, if the height between each floor of the target building is equal to the height of the tile, the processor (1330) may perform tiling for each sub-region by arranging tiles matching each sub-region in a non-overlapping manner.

[0565] To this end, the processor (1330) may determine an area in which the tiling is to be performed based on the left and right margin values ​​preset from the corners of each side area of ​​each virtual object (building virtual object) and the horizontal and vertical lengths of each side area of ​​the building according to the building reference information. In this case, each building surface may be divided into a plurality of sub-areas based on a floor divider that divides the top and bottom, the top layer, and at least one middle layer. In addition, the processor (1330) may configure the left and right margin values ​​and the top and bottom margin values ​​or the width of the floor divider as margins for each of the plurality of sub-areas, and determine the remaining area excluding the margins as an area in which the tiling is to be performed. In addition, in this case, the inter-floor height or inter-floor width of the bottom or top layer may be formed differently from the inter-floor height or inter-floor width of the middle layer. Additionally, among the tiles stored in the tile DB (1341), the tiles corresponding to the middle layer may be tiles that differ in at least one of width or height from the tiles corresponding to the bottom layer and the top layer.

[0566] And the processor (1330) can determine the number of tiles to be tiled horizontally or the number of tiles to be tiled vertically (tiling offset) in each sub-area where the tiling is to be performed according to the width or height of the determined area. And the processor (1330) can distinguish tiles that are tiled continuously in the surface area of ​​the building virtual object, i.e., each building element included in the texture image.

[0567] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different areas of the texture image by each color value, and accordingly, different areas of the texture image, i.e., different building elements, can be distinguished by color channel.

[0568] And the processor (1330) can generate mask maps for masking building elements for each distinguished building element in each surface area of ​​the tiled building virtual object. The mask map is a map formed so that building elements other than a certain building element are hidden in the surface area of ​​the tiled building virtual object, and may mean an image used to specify only one building element among each building element included in the texture image. In other words, the mask map corresponding to a specific building element may be an image in which other areas are hidden except for an area corresponding only to the specific building element, and may be an image for applying a specific effect only to an area corresponding to the specific building element.

[0569] Meanwhile, when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps with shading effects so that each mask map can display a different material texture.

[0570] Normally, when there is a curve in the surface of an object, the direction of the surface changes due to the curve, and light is reflected differently depending on the changed surface direction, and the curve of the surface of the object can be felt by the reflected light that is reflected differently. The normal map is used to express the curved surface using the reflected light of the curve rather than the actual curve, and may mean an image having a 3D vector value that represents the 3D direction of the surface of the object required for calculating the degree of shading corresponding to the reflected light reflected by the surface curve of the specific material when light incident on a specific material is incident on the surface of the specific material. When such a normal map is used, even if it is a 2D image that does not have a three-dimensional shape, the reflected light corresponding to the curve according to the specific material can be expressed for the light incident on the building, so the texture of the specific material can be formed.

[0571] Accordingly, the processor (1330) can blend normal maps corresponding to different materials with respect to the mask maps corresponding to each building element. Then, a 3D vector value according to the normal map can be applied to the building element corresponding to each mask map, so that a material texture according to the normal map can be expressed.

[0572] Here, in the case of an area covered by the mask map, the 3D vector value may not be applied even when the normal map is blended. Therefore, the normal map to which the mask map is blended may be textured with a specific material only in an area not covered by the blended mask map. In this case, the area not covered by the mask map may be an area corresponding to a specific building element specified by the mask map. Therefore, the normal map to which the mask map is blended may be an image to which a 3D vector value is applied so that a surface texture of a specific material is expressed only in a specific building element that is not masked by the mask map.

[0573] Meanwhile, the mask map may be performed for each distinguished building element in each surface area of ​​the building virtual object in which texture images are tiled. Accordingly, multiple mask maps may be generated for each building element in the surface area of ​​the building virtual object. Each mask map may be blended with a normal map of a different material. The processor (1330) may perform blending between normal maps to which the mask maps are blended.

[0574] Accordingly, the processor (1330) can generate a single normal map (hereinafter referred to as a multi-blended normal map) in which 3D vector values ​​corresponding to surface textures of different materials are applied to areas corresponding to different building elements, respectively, for the surface area of ​​the building virtual object through blending between normal maps to which the mask map is blended. Then, by synthesizing the generated multi-blended normal map to the surface area of ​​the building virtual object, the areas corresponding to each building element of the building virtual object can be textured to have different surface textures.

[0575] Here, the processor (1330) can generate a normal map in which 3D vectors corresponding to different surface textures are applied to areas corresponding to each building element included in the texture image for each texture image tiled on the surface area of ​​the building virtual object. Then, the generated normal map, i.e., the multi-blended normal map, can be synthesized for each texture image arranged on the surface area of ​​the building virtual object. In this case, since the texture image is synthesized in a tiling manner on the surface area of ​​the building virtual object, the multi-blended normal map synthesized for each texture image can also be synthesized in a tiling manner on the surface area of ​​the building virtual object.

[0576] Meanwhile, a normal map having 3D vector values ​​corresponding to the surface texture of the specific material applied may be blended with an albedo map for representing the color of the specific material. In this case, the albedo map may be an image having a color value unique to the specific material. Accordingly, a normal map having 3D vector values ​​corresponding to the texture of the specific surface material may form a pair with an albedo map having a color value unique to the specific surface material, and may be blended together.

[0577] Therefore, when an albedo map is blended with a normal map, the normal map may have a 3D vector value corresponding to the surface texture of a specific material as well as a color value unique to the specific material. Therefore, when a mask map corresponding to a specific building element and a normal map are blended, and the albedo map is further blended to the blended normal map, the specific building element corresponding to the mask map may have a color value unique to the specific material as well as a surface texture of the specific material. In this case, a multi-blended normal map corresponding to the entire surface area of ​​a building virtual object or a texture image tiled on the building virtual object may be an image in which 3D vector values ​​having different material textures are applied to different areas corresponding to a plurality of building elements, and each area is expressed with a color of a different material.

[0578] Meanwhile, when texturing of a surface area of ​​a building virtual object is completed through the multi-blending normal map, the processor (1330) can control the MR service device (900) to MR render the building virtual object for which texturing has been completed and display it on the display unit (251). In this case, the processor (1330) can control the MR service device (900) and the display unit (251) of the vehicle (100) through the interface unit (1320).

[0579] Meanwhile, the processor (1330) may determine at least one of the colors of the normal map corresponding to each building element based on an image captured from the camera of the vehicle (100). For example, the processor (1330) may obtain an image corresponding to a visible area of ​​a frustum shape according to the current position of the vehicle (100) and the driving direction of the vehicle (100) from the camera of the vehicle (100). Then, the objects included in the received image may be classified by type and a representative color may be determined for each type of the classified object. In this case, the representative color may be the RGB average value of the color of each object classified as a specific type, or the color having the highest ratio among the colors of each object classified as the specific type.

[0580] For example, the processor (1330) can detect colors corresponding to building objects from the acquired image, and determine a specific color having the highest RGB average value or ratio of the detected colors as a representative color of the building objects included in the image.

[0581] And the processor (1330) can determine at least one color among the colors of the normal map corresponding to each building element as the determined representative color. Accordingly, at least one color among the colors of the normal map corresponding to each building element of the building virtual object can be determined from an image acquired from the camera of the vehicle (100).

[0582] Meanwhile, the above description describes modeling building virtual objects corresponding to buildings included in the map area and texturing and rendering some of the modeled building virtual objects. However, it is also possible to model and render road virtual objects corresponding to road shapes included in the map area.

[0583] To model the above road virtual object, the processor (1330) may generate a plurality of nodes along a road shape included in the map area from map data. Then, the plurality of generated nodes may be defined as segments, and polygon objects having a specific width may be generated for each segment, and the generated polygon objects may be combined to model the road virtual object. In this case, the polygon objects may be round objects having a specific width.

[0584] Meanwhile, the processor (1330) may utilize a morphology operation to more smoothly express the edge of the road object. The morphology operation is one of the techniques for analyzing and transforming the shape and structure of an object within an image, and may include erosion, dilation, opening, and closing operations. Here, the processor (1330) may more smoothly express the road virtual object composed of the polygon objects through a closing operation used when filling a groove or cutting off a protruding part among the morphology operation techniques.

[0585] Meanwhile, FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view creation device (1300) and an MR service device (900) according to an embodiment of the present invention.

[0586] First, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can collect driving information of the vehicle (100). Here, the driving information of the vehicle (100) can include information on the speed, location, and driving direction of the vehicle (100) detected through the sensing unit (120). In addition, a map area for modeling virtual objects can be determined based on the collected driving information of the vehicle (100).

[0587] Meanwhile, when a map area is determined, the processor (1330) can detect building reference information of buildings included in the determined map area from the object reference information storage unit (1343). Then, when the building reference information is detected, building image setting values, which are information for modeling building virtual objects corresponding to each building, are determined, and building virtual objects corresponding to each building can be modeled based on the determined building image setting values ​​and the building reference information of each building.

[0588] And the processor (1330) can determine any texture image as a tile from the tile DB (1341) for each building or for each partial region (e.g., floor region, at least one middle region, top region) of each building. And the texture image determined as a tile can be tiled for each building or for each partial region of each building. And mask maps corresponding to different components of the building included in the tiled texture image are blended with normal maps corresponding to different materials, and each normal map blended with the mask map is blended with an albedo map having a color corresponding to each material, so that the different components of the building can be textured to have different materials and different surface textures. And by blending the textured normal maps, a multi-blended normal map corresponding to a specific surface region or partial region of the building virtual object or a specific texture image can be generated. And, by synthesizing the generated multi-blended normal map to a specific surface area or partial area of ​​the building virtual object, or by tiling a specific surface area or partial area of ​​the building virtual object along a tiled texture image, a surface texture for a surface area of ​​the building virtual object can be formed. That is, a textured virtual object can be generated.

[0589] Meanwhile, the MR service device (900) can model the location of the self-vehicle (100), pedestrians and other vehicles around the vehicle (100), POI (Position Of Interest) information, route information, and road information and terrain information around the vehicle (100) based on information provided from the vehicle system, i.e., GPS, ADAS, and navigation system of the vehicle (100). In addition, the MR service device (900) can configure a UX scene including the textured virtual objects using the modeled information and information about the modeled virtual objects provided from the 3D map view generation device (1300). In addition, the MR renderer can be controlled to generate a map image including a camera view, 3D effects, and GUI (graphics rendering), and the image can be displayed through the display unit (251).

[0590] Hereinafter, a method of determining a map area, modeling virtual objects included in the determined map area, and rendering and displaying the modeled virtual objects by a 3D map view generation device (1300) according to an embodiment of the present invention will be described in more detail with reference to the attached drawings.

[0591] FIG. 13 is a flowchart illustrating an operation process of a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention for determining a map area, modeling virtual objects included in the determined map area, blending the modeled virtual objects into a normal map having 3D vector values ​​of different materials to texture them, and rendering and displaying the textured virtual objects. FIG. 14 is an exemplary diagram illustrating an example of a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention for updating a map area according to vehicle driving information. FIG. 15 is a conceptual diagram illustrating map tile background modeling described in FIG. 14 in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0592] First, referring to FIG. 13, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can obtain driving information about the vehicle's location, speed, driving direction, destination, and path to the destination through the navigation system (770) and the vehicle's sensing unit (120) (S1300).

[0593] When driving information of a vehicle (100) is acquired, the processor (1330) can determine a map area based on a map tile corresponding to the location of the vehicle (100). The map area can be determined differently depending on the preset zoom level of the map.

[0594] For example, the map area may be an area formed by 25 map tiles centered on the map tile according to the location of the current vehicle (100) when the zoom level of the currently set map is 17. In this case, when the zoom level of the map is lowered, an area formed by a greater number of map tiles centered on the map tile according to the location of the current vehicle (100) may be determined as the map area, and when the zoom level of the map is higher, a map area formed by a smaller number of map tiles (e.g., 9) centered on the map tile according to the location of the current vehicle (100) may be determined. When the map area is determined, the processor (1330) may detect data of map tiles included in the determined map area from map data stored in the memory (1340).

[0595] Meanwhile, the above map area may be updated according to driving information of the vehicle (100). For example, when the vehicle (100) leaves the map tile where it is currently located due to driving of the vehicle (100), the processor (1330) may update the map area so that the map area includes map tiles within a range according to the zoom level of the currently set map based on the map tile into which the vehicle (100) newly enters.

[0596] FIG. 14 is an exemplary diagram showing an example in which a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention updates a map area according to driving information of a vehicle (100).

[0597] First, referring to (a) of FIG. 14, (a) of FIG. 14 is a drawing showing an example of a map area (1402) formed with 25 map tiles centered on a map tile according to the location of the current vehicle (100) according to the currently set zoom level '17' of the map. In this case, the map area (1402) may be formed centered on the first map tile (1400) in which the current vehicle (100) is located.

[0598] In this state, if the vehicle (100) moves from the first map tile (1400) to the second map tile (1410) according to the movement of the vehicle (100), the processor (1330) can update the map area (1402) including a number of map tiles (25) determined according to the preset zoom level of the map centered on the first map tile (1400), as shown in FIG. 14b, to a map area (1412) including a number of map tiles (25) determined according to the preset zoom level of the map centered on the second map tile (1410) according to the position of the moved vehicle (100). Accordingly, the processor (1330) can detect data of map tiles corresponding to the updated map area (1412) from map data stored in the memory (1340). In this case, the processor (1330) can detect at least one map tile (1420) that is connected to each other from the map data of the memory (1340), as shown in FIG. 14b.

[0599] Meanwhile, when a map area is determined from data of detected map tiles, the processor (1330) can model virtual objects corresponding to buildings and roads included in the determined map area (S1304).

[0600] In the above step S1304, the processor (1330) can detect buildings included in the detected map area. And, building reference information corresponding to the detected buildings can be detected from the building reference information storage unit (1343). And, the setting values ​​of each building image related to the detected building reference information can be determined. Here, the building image setting values ​​are information for modeling each building virtual object, and may include a setting value of a projection direction according to an angle or distance for generating the building virtual objects according to the driving direction of the vehicle (100) and / or an angle at which the MR view is displayed, a setting value of modeling according to a 2.5D virtual object or a setting value of modeling according to a 3D virtual object, for example, a vector value such as a coordinate value of a vertex or a normal line.

[0601] Once the building image setting values ​​are determined, the processor (1330) can calculate building setting values ​​corresponding to each building included in the map area based on the determined building image setting values ​​and building reference information of each building. Here, the building setting values ​​can include the height of the building virtual object to be created, the side area of ​​each side, the plane area, etc. Then, once the calculation of the building setting values ​​is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included in the map area based on the building setting values.

[0602] Meanwhile, as mentioned in step S1302, the processor (1330) may update the map area according to the driving information of the vehicle (100). In this case, if new map tile data is additionally detected as shown in FIG. 14b by updating the map area, the processor (1330) may additionally model virtual objects corresponding to buildings and roads included in the added map tiles. On the other hand, if the map area is updated as shown in FIG. 14b, map tiles (1430) may be excluded from the map area. In this case, the processor (1330) may delete virtual objects generated from map tiles (1430) excluded from the map area, thereby increasing the efficiency of the memory (1340).

[0603] Here, the processor (1330) may maintain at least some of the virtual objects generated from the excluded map tiles, depending on the building type, user-specified status, POI status, etc. In this case, the maintained virtual objects may be stored in the memory (1340) and may be reused when the corresponding map tile is detected again in the future. In other words, in the case of a building virtual object corresponding to a building frequently visited by the user or a building specified by the user, the virtual object may be stored and reused regardless of whether the map tile containing the building leaves the map area.

[0604] Meanwhile, the step S1304 may include a process of modeling a road virtual object corresponding to the road shape of each map tile included in the map area.

[0605] In this case, the processor (1330) can generate a plurality of nodes along the road shape included in the map area in step S1304. Then, the generated plurality of nodes can be defined as each segment, and the road virtual object can be modeled by generating polygon objects having a specific width for each segment and combining the generated polygon objects. In this case, the polygon objects can be round objects having a specific width.

[0606] Meanwhile, the processor (1330) may utilize morphological operations to more smoothly express the edges of the road virtual object. In particular, the processor (1330) may temporarily increase and then restore the radius of the round polygon objects to perform a closing operation among the morphological operations. In this case, the degree of smoothness of the edge portion of the road virtual object may vary depending on the radius of the polygon objects and the number of times the closing operation is performed.

[0607] Additionally, the processor (1330) can raise the generated road virtual object above the ground or form it lower than the ground level. In this case, if the height of the road virtual object is lower than the ground level, the ground around the road can exhibit a curb effect, forming a curb depending on the height difference between the ground level and the road virtual object. In this case, by varying the height difference between the road virtual object and the ground level depending on the width or length of the road or the characteristics of the road, the road virtual object can be expressed differently depending on the characteristics of the actual road.

[0608] In addition, the processor (1330) may also form an object corresponding to a river or stream similar to the method of generating the road virtual object. In this case, the height of the virtual object corresponding to the river or stream may be formed so as to be lower than the height of the ground by a preset depth. In this case, similar to the road virtual object, by making the height difference between the virtual object corresponding to the river or stream and the ground different depending on the width or length of the river or stream, the virtual object corresponding to the river or stream may be expressed differently depending on the characteristics of the actual river or stream.

[0609] Meanwhile, in step S1304, when virtual objects such as buildings and roads of each map tile included in the map area are modeled, the processor (1330) can determine an area according to driving information obtained from the vehicle (100) (S1306).

[0610] The area determined in the above step S1306 is an area determined based on the location of the vehicle (100) or the driving direction of the vehicle (100), and may be a map area determined based on the current driving information of the vehicle (100), that is, an area corresponding to at least a portion of at least one map tile detected from the map data storage unit (1342) of the memory (1340). For example, the area may be an area corresponding to the surroundings of the current location of the vehicle (100) within a first map tile according to the current location of the vehicle (100). Alternatively, the area may be an area corresponding to the front of the vehicle (100) in a first map tile including the current location of the vehicle (100) according to the driving direction of the vehicle (100). Alternatively, the area may be an area including a portion of the first map tile and an area corresponding to a portion of at least one other map tile.

[0611] Meanwhile, the area determined in the above step S1306 may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), i.e., a user field of view area according to the driving direction of the vehicle (100). In this case, the user field of view area has a form that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to perspective projection that can form a sense of perspective, and thus may have the shape of a frustum with the location of the vehicle (100) as a vanishing point.

[0612] Here, an area determined according to the driving information of the vehicle (100) in the step S1306 may be an area where virtual objects displayed as a digital twin map can be displayed on the display unit (251). That is, it may be an area displayed as a digital twin map to the user of the vehicle (100) through the MR view, and may be a map area shown to the user of the vehicle (100), i.e., a visible area. Hereinafter, among the map areas determined according to the driving information of the current vehicle (100), an area determined in the step S1306 that is displayed on the display unit according to the driving information of the vehicle (100) will be referred to as a visible area.

[0613] When the visible area is determined in step S1306, the processor (1330) may determine some of the virtual objects to be rendered among the virtual objects modeled in step S1304 according to the determined visible area (S1308). The objects determined in step S1308 may be only some of the virtual objects included in the visible area among the virtual objects modeled in step S1304, that is, the map tiles detected as the current map area. In this case, the modeled objects determined as the rendering target in step S1308, that is, the virtual objects, may include a plurality of building virtual objects and a plurality of road virtual objects or parts of road virtual objects.

[0614] Meanwhile, when the virtual objects to be rendered are determined in step S1308, the processor (1330) can tile a preset texture image, i.e., a tile, for each of the building virtual objects among the determined virtual objects (S1310). Here, the tiling can be performed for each side of each building virtual object determined as the rendering target, and can be performed for the side areas of each building virtual object excluding a preset offset area (or margin area) from the corner of each side.

[0615] Here, at least one of the above-described building virtual objects may have its side region divided into multiple sub-regions. For example, the side region may be divided into a bottom region, a middle region, and a top region, and at least one of the bottom region, the middle region, and the top region may be further divided into multiple regions. In this case, the processor (1330) may tile different texture images for each of the multiple divided regions.

[0616] In addition, the processor (1330) may tile an arbitrary texture image for each building virtual object or for each partial region of each building virtual object. In this case, texture images corresponding to each building virtual object or each partial region of each building virtual object may be determined in advance. In this case, the building reference information, which is information on an actual building corresponding to each building virtual object and stored in the memory (1340), may include information on a texture image corresponding to each building virtual object or each partial region of each building virtual object, and the processor (1330) may determine a texture image corresponding to each building virtual object or each partial region of each building virtual object when determining a building image setting value based on the driving information of the vehicle (100).

[0617] And, based on the height, area, side area of ​​the building included in the reference information of each building virtual object, and the offset area (or margin area), at least one of the horizontal length and the vertical length of the area in which the texture image is to be tiled in each building virtual object can be calculated, and the number of texture images to be tiled in the horizontal or vertical direction can be calculated according to the calculated length. That is, the processor (1330) can calculate information for modeling building virtual objects based on the reference information of the building including the building image setting value and the height of the actual building, and at the same time calculate a building setting value including information on the area of ​​each side area in which the texture images are to be tiled in each side area for each building virtual object and the number of texture images to be tiled in each side area.

[0618] And the processor (1330) can tile texture images in the side areas of each rendering target building virtual object based on information about texture images corresponding to each building virtual object or a partial area of ​​each building virtual object, and information about the number of tilings of the side areas of each virtual object and the texture images included in the calculated building setting value.

[0619] Meanwhile, when tiling of texture images for side areas of each building virtual object is completed in step S1310, the processor (1330) can generate a normal map including 3D vector values ​​that form different textures for each building element included in the texture image, which is a tiled 2D image (S1312).

[0620] In this case, the normal map may be a normal map (hereinafter referred to as a multi-blended normal map) obtained by blending multiple normal maps generated for each building element included in the texture image. In order to generate the multi-blended normal map, the processor (1330) may generate mask maps for masking building elements for each distinguished building element in each surface area of ​​the building virtual object in which the texture images are tiled.

[0621] In this case, the mask map is a map formed so that building elements other than one building element are hidden in the surface area of ​​the building virtual object on which the texture images are tiled, and may mean an image used to specify only one building element among each building element included in the texture image.

[0622] And when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps to which shading effects are applied so that the textures of different materials can be represented in each mask map. In this case, as described above, since the mask map is an image that covers the remaining area except for the corresponding specific building element, when the normal map is blended, a 3D vector value forming a texture corresponding to the specific material corresponding to the normal map can be blended only in an area not covered by the mask map and an area corresponding to the specific building element corresponding to the mask map.

[0623] Meanwhile, the processor (1330) may generate a mask map for each distinguished building element in each surface area of ​​the building virtual object in which texture images are tiled. Accordingly, multiple mask maps may be generated for each building element, and each mask map may be blended with a normal map of a different material. In addition, the processor (1330) may generate a multi-blended normal map in which 3D vector values ​​corresponding to surface textures of different materials are applied to areas corresponding to different building elements, respectively, with respect to the surface area of ​​the building virtual object, by blending normal maps in which the mask maps are blended with each other. In addition, by synthesizing the generated multi-blended normal map to the surface area of ​​the building virtual object, the processor (1330) may texture the surface areas of each building virtual object so that the areas corresponding to each building element of the building virtual object each have a different surface texture.

[0624] Here, when generating a mask map for each building element, the processor (1330) may further blend at least some of the plurality of mask maps generated for each building element with an albedo map, which is a map image having a color corresponding to a specific material. In this case, as at least some of the plurality of mask maps generated for each building element further have a color according to the albedo map, color mapping can be performed with a color corresponding to a specific material.

[0625] In this case, the processor (1330) may determine a color corresponding to at least some of the plurality of mask maps generated for each building element based on the results of analyzing an image captured from the camera of the vehicle (100). For example, the processor (1330) may detect objects corresponding to buildings among image objects included in an image captured from the camera of the vehicle (100). Then, colors may be detected from the detected building image objects, and based on the detected colors, a dominant color or an average color of actual buildings around the vehicle (100) may be determined as a representative color. And, by blending an albedo map corresponding to a color determined as a representative color with at least some of a plurality of mask maps generated for each building element, blending mask maps to which the albedo map is blended to generate the multi-blended normal map, and synthesizing the generated multi-blended normal map to each building virtual object, at least some of the building elements of the building virtual objects currently determined as rendering targets in step S1308 can be textured to have colors similar to those of actual buildings around the vehicle (100).

[0626] Meanwhile, when texturing is completed in step S1312, the processor (1330) may render the building virtual objects for which texturing has been completed to generate images corresponding to the building virtual objects. In addition, the processor (1330) may render other virtual objects, including at least one road virtual object included in the visible area, to generate images corresponding to each virtual object (S1314).

[0627] And the processor (1330) can control the vehicle interface unit (1320) to display virtual objects including each building virtual object and road virtual object generated through the above rendering on the display unit (251) (S1316).

[0628] As described above, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention has a configuration that detects map tiles within a certain area (map area) set centered on the vehicle (100), models virtual objects within the detected map tiles in advance, and renders and images only some of the objects in the driving direction of the vehicle (100) among the pre-modeled objects. In addition, the processor (1330) synthesizes a normal map so that a 3D vector value corresponding to a specific material for each building element is applied only to some of the objects to be rendered, so that a texture according to a specific material for each building element can be formed by the applied 3D vector value during the rendering. In this way, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can shorten the time required for modeling by rendering pre-modeled objects when displaying virtual objects as an MR view using a digital twin map. In addition, the number of virtual objects to be rendered is also limited to the visible area to minimize the number, and the surface area of ​​each building virtual object can be textured to have the three-dimensional effect of various materials using a texture image, which is a 2D image, and a normal map having 3D vector values ​​corresponding to different materials. In this way, by performing texturing using only 2D images, the amount of computation required for texturing can be greatly reduced, and as a result, the time required for rendering can be greatly shortened.

[0629] In this way, since the amount of computation and time required to display virtual objects in an MR view using the digital twin map can be significantly reduced, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can perform rendering of virtual objects to be displayed through the MR view during the time when the app for displaying the MR view using the digital twin map is running, i.e., during the run time. In other words, real-time run-time 3D map modeling can be possible.

[0630] Meanwhile, as described above, the present invention has a configuration for pre-modeling virtual objects included in a plurality of map tile data according to the location of the vehicle (100) and rendering some of the modeled virtual objects, including texturing. In this case, the map tile data can be updated according to the location of the vehicle (100), and the detection of such map tile data can be performed through background processing. In this case, since the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention pre-models virtual objects included in the map tile data when the map tile data is detected according to the determined map area, the modeling of the virtual objects can also be performed through the background processing. The modeling of virtual objects performed through background processing in this way will be referred to as background modeling.

[0631] Meanwhile, the processor (1330) according to an embodiment of the present invention has a configuration that models virtual objects in a map area in advance through background modeling, and renders a portion of virtual objects that have already been modeled based on a visible area determined according to driving information of the vehicle (100). Here, the modeling of the virtual objects is performed through background processing, and the rendering of the modeled virtual objects is performed through foreground processing according to the execution of the user's MR view app, so the 3D map view generation device (1300) according to an embodiment of the present invention has a configuration in which the modeling and rendering of virtual objects are performed separately. Therefore, when the 3D map view generation device (1300) according to an embodiment of the present invention has a plurality of processors, the modeling of the virtual objects and the rendering of the modeled virtual objects may be performed by different processors, respectively.

[0632] FIG. 15 is a block diagram illustrating a configuration in which modeling and rendering of virtual objects are performed by different processes in a 3D map view generation device (1300) according to an embodiment of the present invention having multiple processors.

[0633] Referring to FIG. 15, a 3D map view generation device (1300) according to an embodiment of the present invention, i.e., an embedded system, may include a first processor that loads a plurality of map tiles through an API based on the current location (GPS Position) of a vehicle (100) in conjunction with a navigation map, and a second processor that renders an image of a virtual object displayed on a display unit (251). Here, the first processor may be a central processing unit (CPU) of the 3D map view generation device (1300) according to the embodiment of the present invention, and the second processor may be a GPU (Graphics Processing Unit) that processes graphic operations on the display device and outputs a result value.

[0634] In this case, the first processor may load a plurality of map tiles according to the location of the vehicle (100) and perform modeling on virtual objects included in the loaded plurality of map tiles. In this case, the first processor may perform modeling of the virtual objects in a background modeling manner. In addition, the modeled virtual objects may be stored in a buffer. In this case, the modeled objects may be simple polygonal (2.5D) virtual objects without surface texturing.

[0635] In this case, at least one of the plurality of map tiles may be added when the map area is updated according to the location of the vehicle (100). Then, the first processor may additionally model virtual objects included in at least one additional map tile according to the updated map area. Then, the buffer may be updated so that the modeled virtual objects are added. Accordingly, virtual objects corresponding to each object, including buildings and roads included in the entire updated map area, may be stored in the buffer.

[0636] Meanwhile, when an app for displaying an MR view is executed by a user, the second processor can render the modeled virtual objects stored in the buffer. During the rendering process, the second processor can perform texturing by tiling a texture image to a side area of ​​each modeled building virtual object, and synthesizing a normal map to which a 3D vector corresponding to a different material is applied to each area of ​​each building element included in the tiled texture image to the side area of ​​each building virtual object. In this case, the 3D vector is for providing a shading effect that forms a texture of a specific material through a shading effect, and the second processor, as a custom shader, can perform rendering to generate an image of each building virtual object, including texturing that includes shading according to the 3D vector applied to each building element.

[0637] Therefore, the 3D map view generation device (1300) according to an embodiment of the present invention can process modeling of virtual objects and rendering of modeled virtual objects in parallel on different processors. Therefore, the computational load applied to the processor can be significantly reduced compared to a case where all processes are performed on a single processor, and by displaying virtual objects by performing rendering from pre-modeled objects, the time required until the virtual objects are displayed on the display unit can also be significantly shortened. Accordingly, when an app related to the MR view is executed, rendering of 3D virtual objects included in the digital twin map can be performed during the runtime time, which is the time when the app is executed, and real-time provision of a digital twin map including virtual objects with high-quality textures can be possible.

[0638] However, for convenience of explanation, the following description assumes that modeling and rendering are performed by a single processor (1330). However, it should be understood that the present invention is not limited thereto.

[0639] Meanwhile, in step S1316 of FIG. 13, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can simultaneously display a screen showing information collected around the vehicle according to the ADAS system and an MR view screen according to the digital twin map. FIG. 16A is an exemplary diagram illustrating such an example.

[0640] Referring to FIG. 16A, the processor (1330) of the 3D map view generation device (1300) can display information on surrounding vehicles and obstacles based on information collected through the vehicle interface unit (1320) and an ADAS (Advanced Driver Assistance Systems) view image based on driving information of the vehicle (100) in one area (1600) of the display unit (251). ADAS refers to an active safety device that detects a dangerous situation with a sensor or a camera, warns the driver of the risk of an accident, and helps the driver make a judgment and respond. The ADAS view image can include information on objects corresponding to the surrounding situation of the vehicle, for example, a vehicle object and other vehicles and obstacles detected around the vehicle, and can include route information such as the current driving speed or driving time of the vehicle (100) or the expected time required to reach the destination.

[0641] Meanwhile, the processor (1330) can display an MR view image in another area (1610) on the display unit (251). The MR view image is a view image including a building virtual object or a road virtual object created as a 3D object, and can display a digital twin map formed by the virtual objects.

[0642] Meanwhile, according to the above description, it has been mentioned that the processor (1330) of the 3D map view generation device (1300) according to the embodiment of the present invention can determine an area to be displayed by rendering virtual objects according to the driving information of the vehicle (100) in step S1306 of FIG. 13, i.e., a visible area. Here, the visible area may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), i.e., a user viewing area according to the driving direction of the vehicle (100).

[0643] In this case, the user's field of view has a shape that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to perspective projection that can form a sense of perspective, and thus may have the shape of a frustum with the position of the vehicle (100) as a vanishing point. Accordingly, the visible area may be formed in a frustum shape with the position of the vehicle (100) as a vanishing point.

[0644] FIG. 16b is an exemplary diagram illustrating a frustum-shaped visible area formed across a plurality of map tiles or one map tile, depending on the position of the vehicle and the driving direction of the vehicle, in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0645] First, referring to (a) of FIG. 16b, the processor (1330) may use the current location (2000) of the vehicle (100) as a vanishing point and form a frustum-shaped visible area (1650) formed along the driving direction of the vehicle (100). In this case, the processor (1330) may perform rendering only on virtual objects located within the formed visible area (1650). That is, even within one map tile, virtual objects located in areas other than the visible area (1650) may not be rendered.

[0646] Here, "rendering" can refer to the process or computational process of generating an image using a computer program. Therefore, even if a modeled virtual object is created, if rendering is not performed, an image corresponding to the virtual object may not be created. Furthermore, if an image is not created, it may not be displayed on the display unit (251).

[0647] That is, the processor (1330) can selectively determine objects to be displayed on the display unit (251) among the modeled virtual objects based on the visible area formed according to the driving direction of the vehicle (100) among the modeled virtual objects. Accordingly, as shown in (a) of FIG. 16b, images corresponding to each virtual object can be generated through a rendering process for the virtual objects within the area included in the visible area (1650).

[0648] However, images of virtual objects may not be generated for virtual objects within the area included in the above-mentioned visible area (1650). Therefore, the processor (1330) can reduce the amount of computation required for rendering and shorten the computation time by rendering only some of the virtual objects modeled based on the above-mentioned visible area.

[0649] Meanwhile, the visible area (1650) may be formed across a plurality of map tiles, as shown in (b) of FIG. 16B, depending on the zoom level of the set map. In addition, the plurality of map tiles may be a part of a map area (1660) set around the current location (2000) of the vehicle (100). In this case, since the processor (1330) models virtual objects based on the map area (1660), images of virtual objects in an area included in the visible area (1650) among the plurality of map tiles may be generated through a rendering process according to the visible area (1650) formed across the plurality of map tiles, and may be displayed on the display unit (251).

[0650] FIG. 17 is an exemplary diagram showing an example of a 3D map view generation device (1300) according to an embodiment of the present invention, in which texture images are tiled in a margin width and at least one preset texturing area.

[0651] When the building virtual objects determined as rendering targets (hereinafter referred to as rendering target virtual objects) in step S1308 of the above-described FIG. 13 are determined, the processor (1330) may tile a preset texture image in each side area for each of the building virtual objects among the determined virtual objects. In this case, the processor (1330) may detect a side area of ​​the building virtual object excluding a preset offset area (or margin area) inward from each side edge of the building virtual object, and may tile the texture image in the detected side area.

[0652] First, referring to (a) of FIG. 17, the processor (1330) can determine the left and right margins (2511, 2512) of the side area of ​​the building virtual object determined as the rendering target according to the preset left and right margin values. According to the left and right margin values, the coordinate range of the area (2510) where the texture image is to be tiled in the horizontal direction can be determined.

[0653] For example, if the left and right margin values ​​are 0.1, the lower left point of the side area of ​​the building virtual object (2520) can be defined as (0, 0) and the upper right point can be defined as (1, 1). Then, as shown in Fig. 17, the width of the tiling area (2510) excluding the left and right margins can be calculated as 0.8. In addition, if the building width of the actual building corresponding to the building virtual object (2520) is 14 m, the actual horizontal width of the tiling area (2510) can be calculated as 11.2 m.

[0654] Meanwhile, the texture image (2500) may be a normalized tile image and may have a preset width. In this case, if the horizontal width of the texture image (2500) is 4 m, dividing the actual horizontal width of the tiling area (2510), 11.2 m, by the horizontal width of the texture image (2500), 4 m, a value of 2.8 may be calculated. In this case, since the number of texture images (2500) is an integer, a value of 3 may be calculated when rounding is performed, and the calculated value 3 may be the number of texture images (2500) to be tiled in the tiling area (2510) of the side area of ​​the building virtual object (2520) set according to the preset left and right margin values. And the processor (1330) inputs (0.1, 0.9), which are the left and right X-axis coordinates of the tiling area (2510), as input values ​​of the remap function and inputs (0, 3) as a new range according to the remap function, so that up to three tiles (texture images (2500)) can be successively attached to the tiling area (2510) in the horizontal direction in the tiling area (2510).

[0655] In addition, the processor (1330) can set multiple partial regions based on the bottom layer, which is the first floor, according to preset upper and lower margin values. For example, the processor (1330) can set a floor divider corresponding to the space between the bottom layer and the upper layer of the partial region. In this case, the space between the set floor dividers based on the bottom layer can be determined as the partial region where texture images are to be tiled.

[0656] Here, the side region of the building virtual object may include multiple subregions. For example, the side region of the building virtual object may be divided into a bottom region (2552), a middle region (2550), and a top region (2551), and at least one of the bottom region (2552), the middle region (2550), and the top region (2551) may be further divided into multiple regions. In this case, the processor (1330) may tile different texture images for each of the multiple divided regions.

[0657] In this case, the processor (1330) can obtain a mask map corresponding to each sub-region based on the UV coordinate system corresponding to each sub-region on each building side. For example, the processor (1330) can define the lower left point of the side region of the building virtual object as (0, 0) and the upper right point as (1, 1). Then, by defining the Y-axis coordinate in the UV coordinate system according to the margin value of the sub-region in which the layer division is formed, the vertical width corresponding to the sub-region can be defined, and the mask map corresponding to the sub-region can be obtained according to the defined vertical width. In this case, the mask map corresponding to the sub-region can be a mask map in which a height lower than the Y-axis coordinate according to the margin value of the sub-region has a value of 0, and a height higher than the Y-axis coordinate has a value of 1. In this case, when blending with a normal map is performed, a part having a value of 0 maintains a value of 0, but a part having a value of 1 can have a 3D vector value of the normal map.

[0658] As an example of such a partial region, if the margin value of the partial region where the layer division is formed is 0.8, the processor (1330) can determine the coordinates of the upper right point of the partial region as (1, 0.8). Then, the processor (1330) can obtain a mask map in which an area corresponding to a height less than 80% of the total height of the building virtual object from the floor surface of the building virtual object has a value of 0, and an area corresponding to a height greater than that has a value of 1. That is, an area corresponding to the upper edge of the building virtual object from a height of 80% or more of the total height of the building virtual object can be formed as a partial region (e.g., a tower region).

[0659] Meanwhile, the processor (1330) can tile different texture images for each sub-region (2550, 2551, 2552) of the building virtual object (2560) in the same manner as described in (a) of FIG. 17. In this case, as shown in (b) of FIG. 17, the top region (2551) can be tiled with the first texture image (2501), the bottom region (2552) can be tiled with the second texture image (2502), and the middle region (2550) can be tiled with the third texture image (2500). Accordingly, the building virtual object (2650) can be tiled with different texture images for each different sub-region.

[0660] In this way, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can diversify the exterior shape (facade design) of the building virtual object by dividing each building virtual object into a plurality of different regions and tiling different texture images in each of the divided regions.

[0661] FIG. 18 illustrates examples of facade designs of a building virtual object expressed in various ways by using different partial areas and tiled texture images in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0662] First, Figure 18 (a) illustrates an example of a building virtual object formed by three sub-regions: a ground floor, a middle floor, and a top floor. Figures 18 (b) and (c) illustrate examples of a building virtual object formed by only the ground and middle floors. In this way, by varying the number of sub-regions, building virtual objects can be expressed with different exterior shapes.

[0663] In addition, the processor (1330) can vary the texture images tiled in each sub-region for each building virtual object. In this case, since not only the number of sub-regions but also the tiled texture images vary, the number of possible combinations can increase. Therefore, a more diverse exterior representation of the building virtual object can be achieved.

[0664] Meanwhile, the processor (1330) can blend normal maps having 3D vector values ​​corresponding to different materials for each building element in step S1312 of FIG. 13 so that the images can be displayed with different materials for each building element included in the tiled texture image of the building virtual object during rendering.

[0665] FIG. 19 is a flowchart illustrating in more detail the texturing process of step S1312 of FIG. 13, which forms different surface textures for each component of a building in a 3D map view generation device (1300) according to an embodiment of the present invention. FIG. 20 is an exemplary diagram illustrating an example of normal maps blended to mask maps corresponding to different building elements and a multi-blended normal map generated by blending normal maps together, according to the process described in FIG. 19.

[0666] First, referring to FIG. 19, the processor (1330) can identify and distinguish building elements included in each texture image for each texture image tiled in the tiling area of ​​the building virtual object (S1900).

[0667] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different areas of the texture image by each color value, and accordingly, different areas of the texture image, i.e., different building elements, can be distinguished by color channels. For example, if the texture image includes at least one window element, the window element may have different color values ​​from the rest of the building background. In addition, if the window element includes a glass element corresponding to a glass area, a window frame element corresponding to a window frame area, and a blind element corresponding to a blind area, the glass element, the window frame element, and the blind element may each have different color values. Therefore, the processor (1330) can distinguish and identify areas corresponding to a plurality of different building elements included in the texture image based on the color values. In this case, the texture image may be referred to as a labeling map, which is a map image that labels a plurality of different building elements.

[0668] Meanwhile, when each building element included in the texture image is distinguished and identified, the processor (1330) can generate mask maps corresponding to each building element (S1902).

[0669] In this case, the mask map corresponding to a specific building element may be an image having a value of 1 in an area corresponding to a specific building element identified by a specific color value in the texture image, and a value of 0 in the remaining areas.

[0670] For example, if the texture image includes at least one window element, the mask map corresponding to the window element may be a mask map that has a value of 1 only in the area corresponding to the window element and a value of 0 in the remaining area. On the other hand, the mask map corresponding to the building wall may be a mask map that has a value of 1 only in the building wall portion, i.e., in the remaining area of ​​the texture image excluding the window element, and a value of 0 in the remaining area.

[0671] In addition, if the window element includes the glass element, the window frame element, and the blind element, the processor (1330) can generate a glass mask map in which only the area corresponding to the glass element in each window element has a value of 1 and the rest has a value of 0, a window frame mask map in which only the area corresponding to the window frame element in each window element has a value of 1 and the rest has a value of 0, and a blind mask map in which only the area corresponding to the blind element in each window element has a value of 1 and the rest has a value of 0.

[0672] When mask maps corresponding to each building element are generated for each building element included in the tiled texture image in this way, the processor (1330) can blend each generated mask map with a normal map having a 3D vector value corresponding to a different material (S1904). In this case, the normal map means an image having a 3D vector value indicating a 3D direction of the surface of an object required for calculating the degree of shading corresponding to the reflected light reflected by the surface curvature of the specific material when light incident on the specific material is incident on the specific material, in a graphic processing device, and may be a 2D image capable of expressing the texture of the specific material through the degree of shading, i.e., the shading effect.

[0673] The blending of these normal maps and mask maps may be a process similar to a multiplication process. Therefore, when the normal map and the mask map are blended, the 3D vector value applied to the normal map may be applied only to the area that is not masked with a value of 0 in the mask map, i.e., only to the area corresponding to a specific building element. Therefore, only the area corresponding to a specific building element corresponding to the mask map may have a texture according to a specific material expressed. In this case, the processor (1330) may blend the mask map corresponding to each building element with a normal map having a 3D vector value of a different material, thereby allowing each building element to have a texture of a different material.

[0674] In the above step S1904, when each mask map is blended with each normal map, the processor (1330) may further blend an albedo map having a different color for each of the mask maps and the blended normal maps. In this case, the albedo map may be an image having a color value unique to a specific material. In addition, a pair may be formed with a normal map having a 3D vector value corresponding to the texture of a specific surface material.

[0675] Then, the processor (1330) can further blend the albedo map forming a pair with the normal map for each normal map blended with the mask map (S1906). In this case, blending with the albedo map can be applied similarly to the multiplication process as with blending with the normal map. Therefore, a color value can be applied only to a part having a value of 1 in the mask map, and accordingly, a normal map capable of expressing the texture of a specific material and the unique color of the material can be generated for a specific building element.

[0676] Meanwhile, in step S1906, when the mask maps and blended normal maps corresponding to each component of the building are blended with the albedo maps, the processor (1330) can blend the blended normal maps of the building elements with each other (S1908). In this case, the blending of the normal maps may be a union concept, which may mean a process of combining the areas to which 3D vector values ​​(and color values ​​according to the albedo map) are applied between the normal maps into a single map image, a multi-blended normal map. Therefore, when the blending process of step S1906 is completed, a normal map (multi-blended normal map) to which 3D vector values ​​(and albedo color values) of different normal maps are applied for each building element included in a single texture image may be generated.

[0677] And the processor (1330) can perform texturing on the side area of ​​the building virtual object by synthesizing the multi-blending normal map into the tiling area among the side areas of the building virtual object (S1910).

[0678] In this case, the multi-blending normal map may be generated for the entire tiled area in which texture images are tiled in the side area or partial area of ​​the building virtual object. In this case, the processor (1330) may synthesize the multi-blending normal map generated in a size corresponding to the entire side area or partial area of ​​the building virtual object onto the side area or partial area of ​​the building virtual object.

[0679] Alternatively, the multi-blending normal map may be generated for any one texture image tiled in a side area or partial area of ​​the building virtual object. In this case, the processor (1330) may synthesize the multi-blending normal map for the side area or partial area of ​​the building virtual object by tiling the multi-blending normal map for each of the texture images tiled in the side area or partial area of ​​the building virtual object.

[0680] Meanwhile, FIG. 20 illustrates an example of generating a multi-blending normal map including areas to which different 3D vector values ​​corresponding to each of a plurality of building elements included in a texture image are applied according to the operation process described in FIG. 19.

[0681] Referring to FIG. 20, the processor (1330) can distinguish and identify each building element from a texture image (2500) including a window element. In this case, if the window element includes a glass element and a window frame element (2702), the processor (1330) can identify the wall element (2701), the glass element, and the window frame element (2702) from the texture image (2500).

[0682] And the processor (1330) can generate a mask map corresponding to each identified building element. In this case, as shown in FIG. 20, a mask map (2711) corresponding to the wall element can be generated in which only the area corresponding to the wall element (2701) has a value of 1 and the remaining areas, for example, the areas corresponding to the glass element and the window frame element (2702), have a value of 0. In addition, a mask map (2712) corresponding to the window frame element can be generated in which only the area corresponding to the window frame element (2702) has a value of 1 and the remaining areas, for example, the areas corresponding to the glass element and the wall element (2701), have a value of 0.

[0683] And the processor (1330) can blend the first normal map (2721) and the second normal map (2722), which are map images to which 3D vector values ​​corresponding to different materials are applied, for each generated mask map (2711, 2712). In this case, the first normal map (2721) may be an image to which a 3D vector value capable of expressing the first material with a shading effect is applied, and the second normal map (2722) may be an image to which a 3D vector value capable of expressing the second material with a shading effect is applied.

[0684] Then, the 3D vector value of the blended normal map can be synthesized only for the area having the value 1, excluding the area having the value 0 in each mask map. Therefore, when the mask map (2711) corresponding to the wall element and the first normal map (2721) are blended, a normal map can be generated in which a 3D vector value that can express the first material only in the area corresponding to the wall element in the texture image (2500) is synthesized, as shown in FIG. 20. In addition, when the mask map (2712) corresponding to the window frame element and the second normal map (2722) are blended, a normal map can be generated in which a 3D vector value that can express the second material only in the area corresponding to the window frame element in the texture image (2500) is synthesized, as shown in FIG. 20.

[0685] Then, the processor (1330) can blend a normal map in which 3D vector values ​​that can express a first material only in an area corresponding to the wall element are synthesized, and a normal map in which 3D vector values ​​that can express a second material only in an area corresponding to the window frame element are synthesized. In this case, blending is performed between normal maps, and each normal map can be synthesized into a single multi-blended normal map. Therefore, as shown in FIG. 20, a multi-blended normal map (2750) can be generated in which 3D vector values ​​that can express a first material in an area corresponding to the wall element (2701) are synthesized, and 3D vector values ​​that can express a second material in an area corresponding to the window frame element (2702) are synthesized. That is, a normal map in which 3D vector values ​​that can express different materials in each area corresponding to each building element included in the texture image are synthesized can be generated.

[0686] Meanwhile, in the description of the above-described Fig. 19, it was explained that a color unique to each material is mapped to each building element by blending an albedo map having a color unique to the material to a normal map to which a mask map is blended (S1906). However, in contrast, the step S1906 may be a step of mapping the colors of virtual objects within the visible area to colors similar to the colors of buildings included in the actual visible area based on an image captured from a camera of the vehicle (100).

[0687] FIG. 21 is a flowchart illustrating an operation process of mapping the colors of virtual building objects based on the colors of actual buildings within a visible area in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0688] Referring to FIG. 21, when step S1904 of blending the mask maps generated for each building element in FIG. 19 into a normal map having 3D vector values ​​corresponding to different materials is performed, the processor (1330) can acquire an image of the surroundings of the vehicle (100) through the camera of the vehicle (100) (S2100). In this case, the camera may be a camera that captures an image of the front area of ​​the vehicle (100), and the acquired image may be an image of the front of the vehicle (100) acquired according to the driving direction of the vehicle (100), i.e., an image of the visible area.

[0689] And the processor (1330) can identify each object included in the acquired image and partition the acquired image into multiple regions according to the type of each identified object. That is, the processor (1330) can partition the regions where each object included in the acquired image is displayed on the image into different regions for each object (S2102).

[0690] For example, if the acquired image is an image including trees, buildings, the sky, and a road, the processor (1330) may, in step S2102, segment the image into an area where each tree is displayed, an area where each building is displayed, an area where the sky is displayed, and an area where the road is displayed. Then, a dominant color may be detected in each segmented area, and each area may be displayed with the detected dominant color. In this case, an image may be acquired in which the area where each object included in the image is displayed is segmented with the dominant color in each area.

[0691] To this end, the processor (1330) may utilize an artificial intelligence module (not shown) trained through preset learning data. In this case, the training may be training to identify each region on the image as a different object based on shape and color. Accordingly, the artificial intelligence module, upon completion of the training, may identify a region of the input image as belonging to one of the preset object types based on the pattern formed by the shape and color displayed in the region.

[0692] And the processor (1330) can classify the regions on the image divided by each object by type of the object. And a representative color can be determined from the regions on the image divided by type of each object (S2104).

[0693] In this case, one or more areas on the segmented image can be classified as having the same object type. For example, an area where each building is displayed on the image can be classified into areas corresponding to each building, and a dominant color can be detected for each segmented area. In this state, the processor (1330) can classify each area on the image classified by each object according to the type of each object. Accordingly, at least one area identified as a 'building' can be classified as an area corresponding to the object type 'building'. In addition, the processor (1330) can determine a representative color corresponding to the object type 'building' based on the color of each of the plurality of areas classified as areas corresponding to the object type 'building'.

[0694] For example, the processor (1330) may determine the color with the largest proportion among the colors of each of the areas corresponding to the object type 'building' as the representative color corresponding to the object type 'building'. Alternatively, the processor (1330) may calculate the average color of the colors of each of the areas corresponding to the object type 'building' and determine the calculated average color as the representative color corresponding to the object type 'building'.

[0695] When a representative color corresponding to the object type 'building' is determined, the processor (1330) can map at least some of the building elements of the building virtual object within the visible area to the determined representative color (S2106). For example, the processor (1330) can blend at least some of the normal maps blended to the mask maps corresponding to each of the different building elements with an albedo map having a representative color corresponding to the object type 'building'.

[0696] FIG. 22 is an exemplary diagram showing an example of determining colors to be mapped to texture images through images captured from a vehicle according to the operation process described in FIG. 21 in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0697] First, (a) of FIG. 22 illustrates an example of an image captured from a camera of a vehicle (100). In this case, the image may be an image captured from the front of the vehicle (100) along the driving direction of the vehicle (100).

[0698] Then, the processor (1330) can identify objects included in the acquired image. And, as shown in (b) of Fig. 22, the areas identified as each object can generate a color map image in which each area is displayed with a dominant color.

[0699] As shown in (b) of FIG. 22, when a color map image is generated, the processor (1330) can distinguish the area of ​​each object identified in the generated color map image by the type of each object. That is, as shown in (b) of FIG. 22, in the color map image, a first area (2901) corresponding to a first building object, a second area (2902) corresponding to a second building object, a third area (2903) corresponding to a third building object, a fourth area (2904) corresponding to a fourth building object, and a fifth area (2905) corresponding to a fifth building object can be distinguished as areas corresponding to the object type 'building'.

[0700] Then, the processor (1330) can calculate the color proportion of each area corresponding to the object type "building." In this case, areas with the same color can be considered as a single area. Accordingly, the first area (2901) and the fifth area (2905) with the same color can be considered as a single area.

[0701] In this case, as shown in (c) of Fig. 22, the color proportions determined from each area identified as a building object in the image can be calculated. In this case, similar to the building object, the color proportions of road objects or sky objects can also be calculated.

[0702] In this case, as shown in (c) of FIG. 22, if the first color, which is the dominant color of the first region (2901) and the fifth region (2905), has the largest proportion, the processor (1330) can determine the representative color of the building virtual object with the first color. Then, the processor (1330) can generate an albedo map having the first color, and blend at least some of the normal maps blended with the albedo map having the first color and the mask map corresponding to each different building element. In addition, in step S1908 of FIG. 19, at least one normal map to which the albedo map of the first color is blended can be blended to generate a multi-blended normal map. Therefore, texturing of the building virtual object can be performed so that at least some of the building elements of the building virtual object are displayed with the first color.

[0703] Meanwhile, in the description of FIG. 19 described above, it was described that an albedo map having a material-specific color is blended into a normal map to which a mask map is blended, thereby mapping a material-specific color to each building element (S1906). However, alternatively, at least one representative color may be determined based on the colors of objects identified as buildings among objects identified from the captured image, and the color of each building element may be mapped to the determined representative color. In this case, the representative color may be determined according to the density of each color of the building objects identified from the captured image. In addition, the processor (1330) may determine the priority of each color according to the density of colors detected from the building objects as well as one representative color, and may determine the color of each building element according to the determined priority.

[0704] FIG. 23 is a flowchart illustrating in more detail the operation process of dividing a captured image into a plurality of regions, determining representative colors of building elements from each of the divided regions, and mapping the colors of each of the building elements of building virtual objects corresponding to each of the regions divided into the plurality of regions with the determined representative colors in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0705] Referring to FIG. 23, first, the processor (1330) can acquire an image of the surroundings of the vehicle (100) obtained from the camera of the vehicle (100) (S2300). In this case, the camera may be a camera that captures an image of the front area of ​​the vehicle (100), and the acquired image may be an image of the front of the vehicle (100) obtained according to the driving direction of the vehicle (100), i.e., an image of the visible area.

[0706] Then, the processor (1330) can determine a reference point from the acquired image based on a preset standard and partition the acquired image into multiple regions centered on the reference point (S2302). For example, the processor (1330) can detect a vanishing point from the image and determine the vanishing point as a reference point.

[0707] Here, the vanishing point refers to the point where parallel straight lines meet when extended farther in a perspective drawing through perspective drawing. The processor (1330) can detect the vanishing point in the image based on road shapes, etc. detected from the image. In addition, the image can be divided into multiple regions based on the detected vanishing points.

[0708] For example, the processor (1330) can distinguish between the area occupied by the road area and other areas on the image based on the vanishing point. Alternatively, the area occupied by the sky area and other areas can be distinguished. Furthermore, the remaining area excluding the road area and / or the sky area can be distinguished as an area (object area) where objects, including buildings, exist. Furthermore, the object area where objects, including buildings, exist can be distinguished into one or more areas.

[0709] In this case, the processor (1330) may divide the object area into an area adjacent to the vanishing point and an area not adjacent to the vanishing point. In this case, the area adjacent to the vanishing point may be an area that is at a certain distance or more from the vehicle (100). And, the area not adjacent to the vanishing point may be an area that is at a certain distance or less from the vehicle (100). Alternatively, the processor (1330) may divide the object area into a plurality of areas based on saturation or brightness according to the color of the objects. Alternatively, the processor (1330) may divide the object area into a plurality of different areas based on the direction with respect to the vehicle (100).

[0710] Meanwhile, if the image captured in step S2302 is divided into multiple regions, the processor (1330) can identify the type of each image object included in each region for each region. Then, among the types of each identified object, objects identified as buildings can be determined (S2304).

[0711] The above step S2304 may be a process for identifying a building object from an image. In this case, the processor (1330) may utilize an artificial intelligence algorithm, such as a pre-trained deep learning network, to identify the building object. Alternatively, an area containing a shape in which a certain pattern is repeated at a preset level or higher may be detected from the image as an area corresponding to the building object.

[0712] When building objects are detected in step S2304, the processor (1330) can calculate the ratio of areas corresponding to the detected building objects for each of the plurality of divided areas. Furthermore, areas among the plurality of divided areas in which the ratio corresponding to building objects is above a certain level can be detected. Furthermore, for the detected areas, the colors of the building objects can be detected and the density of each detected color, i.e., the color density, can be calculated (S2306).

[0713] Here, detecting areas among the above-mentioned multiple areas where the proportion of building objects is above a certain level is to reduce the amount of computation, and may be to prevent unnecessary color-based cluster analysis from being performed in areas where the proportion of building objects is almost zero. Therefore, it goes without saying that the present invention is not limited thereto, and in step S2306, the processor (1330) may detect building objects for each of the multiple areas regardless of the proportion of the area occupied by the building objects and calculate the color density for the detected building objects.

[0714] Meanwhile, the color density is the ratio of the amount occupied by each color according to the color distribution of an area, and the density of a specific color may mean the ratio of the area of ​​the area where the specific color is distributed (occupied) in the area.

[0715] In this case, the processor (1330) can cluster colors that are similar to a certain level in each area into a single color. For example, the processor (1330) can detect colors with a color density of a certain level or higher among the colors of each detected building object (hereinafter, referred to as representative colors). In addition, colors with a density lower than a certain level can be considered as a single cluster with the most similar representative color. In this case, the density of each of the representative colors can be determined by adding up the densities of all other similar colors included in the cluster.

[0716] Here, the processor (1330) may change the color of each cluster based on the colors included in the cluster. That is, the representative color of each cluster may be determined based on the average of the RGB values ​​of all colors included in the cluster. In other words, the average color of the colors included in the cluster may be determined as the representative color of the cluster.

[0717] Meanwhile, for each of the multiple regions, once color clusters are determined and representative colors for each cluster are determined, the processor (1330) can determine the priority of the representative colors according to the color density of each cluster (S2308). In this case, the mapping priority of the representative color of each cluster can be set higher in order of the size of the area occupied by each cluster in each region, i.e., in order of higher density.

[0718] And the processor (1330) can group building virtual objects corresponding to buildings around the vehicle (100) into different groups according to the plurality of partitioned areas (S2310). That is, each of the building virtual objects around the vehicle (100) can be classified as a building virtual object corresponding to any one of the plurality of areas. In this case, the building virtual objects can be grouped into different groups according to the direction in which the objects are located with respect to the location of the vehicle (100) or the driving direction of the vehicle (100) (e.g., left or right with respect to the vehicle (100).

[0719] Alternatively, the building virtual objects surrounding the vehicle (100) may be objects included in a frustum-shaped visible area that becomes wider as the distance from the vehicle (100) increases, centered on the current location of the vehicle (100). In this case, the building virtual objects may be grouped into different groups according to the distance from the vehicle (100).

[0720] Then, the processor (1330) can determine the color of each building element of the building virtual objects for each group corresponding to each area according to the mapping priority of each representative color determined for each area (S2312).

[0721] Here, the processor (1330) may map a plurality of representative colors in order of high mapping priority for each region to building elements of each group of building virtual objects corresponding to each region. To this end, the processor (1330) may determine the area priority for each building virtual object according to the area of ​​each building element formed in the surface area. For example, if the building elements include a wall element corresponding to a building wall, a glass element corresponding to a window glass area, and a window frame element corresponding to a window frame area, and the area occupied by each building element is large in the order of the wall element, the glass element, and the window frame element, the processor (1330) may determine the area priority in the order of the wall element, the glass element, and the window frame element.

[0722] Once the area priority is determined in this way, the processor (1330) can determine the color of each building element in order of highest area priority and each representative color in order of highest mapping priority. For example, if the representative colors are determined as ivory, sky blue, and navy blue, and among them, ivory has the highest mapping priority and navy blue has the lowest mapping priority, the color with the highest mapping priority can be mapped to the color of the building element with the highest area priority. Then, the color with the next highest mapping priority can be mapped to the building element with the next highest area priority. Accordingly, the wall element can be mapped with ivory, the glass element with sky blue, and the window frame element with navy blue. In other words, the colors to be mapped to each building element can be determined based on the area priority according to the width of each building element and the mapping priority determined based on the density.

[0723] Here, if the number of types of building elements exceeds the number of representative colors, the processor (1330) may map arbitrary different colors to building elements that are not color-mapped. Conversely, if the number of types of building elements is fewer than the number of representative colors, colors that are not mapped to building elements may not be mapped to building elements.

[0724] In step S2312, when the color of each building object is determined, the processor (1330) can perform color mapping for each building element of each group of building virtual objects according to the determined colors (S2314). In this case, step S2312 can be performed for each group corresponding to each area of ​​the image. Accordingly, since the representative colors are determined for each of the plurality of divided areas, the representative colors can be determined differently for each area. Accordingly, in the case of building objects included in different areas of the image, at least one building element can be mapped to a different color in step S2314.

[0725] Meanwhile, the above description describes a configuration that detects areas with a certain percentage of building objects in each segmented area, performs color-based clustering analysis only within these areas, and determines representative colors and their priorities. However, this may be intended to reduce computational complexity and prevent unnecessary color-based clustering analysis in areas with a low percentage of building objects.

[0726] In this case, if the ratio condition of building objects is set higher than necessary to detect the area where the color-based cluster analysis is to be performed, for example, if it is set to 50%, the color-based cluster analysis may not be performed on areas where the ratio occupied by building objects is less than 50%. Accordingly, in order to prevent the color-based cluster analysis from not being performed even though building objects are included, the processor (1330) may set the ratio condition of building objects sufficiently low so that the color-based cluster analysis may not be performed only on areas where the ratio of the area occupied by building objects is negligibly small.

[0727] FIG. 24 is a drawing showing an example of colors extracted from each of a plurality of areas divided according to the operation process of FIG. 23 in the 3D map view generation device (1300) of the present invention.

[0728] First, Fig. 24 (a) illustrates an example of an image acquired from a camera of a vehicle (100). As shown in Fig. 24 (a), the image may include building objects corresponding to at least one building located around the vehicle (100) with the vehicle (100) as the center. In addition, the image may be an image in the forward direction according to the driving direction of the vehicle (100) projected with a vanishing point according to a perspective projection method as the center.

[0729] Meanwhile, the processor (1330) may divide the acquired image into multiple regions. In this case, the processor (1330) may divide the acquired image into multiple regions based on reference points determined according to preset criteria.

[0730] Here, the reference point can be determined in various ways. For example, it can be determined according to the shape of a specific object included in the image. For example, in the case of an image in which a road object is displayed horizontally, the processor (1330) can determine both end points of the horizontal road object as reference points. In this case, the processor (1330) can generate a straight line connecting the reference points and divide the image into multiple regions based on the generated straight line. Such reference points can be determined according to the characteristics of the acquired image (e.g., whether the image is based on a perspective projection method, etc.).

[0731] For example, if the image obtained is an image in which a vanishing point can be detected, that is, if the image obtained is an image according to a perspective projection method as shown in (a) of the above-described Figure 24, the reference point may be determined as the vanishing point of the image obtained.

[0732] In this case, the processor (1330) can detect the vanishing point from the acquired image. To this end, the processor (1330) can detect a point on the image where the extended lines of the road shape around the vehicle or the extended lines of the surrounding curb shape converge as the vanishing point. Alternatively, the point at the center of the image can be detected as the vanishing point.

[0733] When a vanishing point is detected, the processor (1330) can divide the acquired image into a plurality of regions based on the distribution of objects included in the image with respect to the vanishing point. For example, in the case of an image in which objects are distributed to the left and right with respect to the vanishing point as the center, as in (a) of FIG. 24, the processor (1330) can divide the image into a plurality of regions based on the distribution of objects included in the image and the direction in which each object is located with respect to the position of the vehicle (100). Accordingly, as shown in (a) of FIG. 24, the processor (1330) can divide the acquired image into four regions: a first region (3001) corresponding to the upper left of the vanishing point, a second region (3002) corresponding to the upper right, a third region (3003) corresponding to the lower right, and a fourth region (3004) corresponding to the lower left.

[0734] Then, the processor (1330) can detect areas among the above-mentioned divided areas in which the proportion of the area occupied by building objects is above a certain level. In this case, in the case of the image shown in (a) of FIG. 24, the first area (3001) and the second area (3002) may have building objects arranged above a certain level, but the third area (3003) and the fourth area (3004) may be almost entirely areas corresponding to roads. Therefore, the processor (1330) detects the first area (3001) and the second area (3002) among the first to fourth areas (3001, 3002, 3003, 3004) as areas corresponding to building objects, and through cluster analysis on the detected areas, can calculate at least one representative color for each area and the color density of the cluster corresponding to each representative color.

[0735] In this case, as shown in (b) of Fig. 24, representative colors (3011, 3012, 3013) of the first region (3001) and representative colors (3021, 3022, 3023) of the second region (3002) can be determined according to the density of color-specific clusters in the first region (3001) and the second region (3002). And, for each determined color, mapping priorities can be determined for each region according to the color density of each cluster. In addition, for each region, building virtual objects corresponding to each region can be grouped, and the colors of building elements can be mapped according to the determined representative colors for each group.

[0736] Meanwhile, in the description of Fig. 24 described above, an example of dividing an image into multiple regions according to different directions based on the position of the vehicle (100) based on a vanishing point was described. However, it is of course also possible to divide the image into multiple regions according to the distance from the vehicle (100).

[0737] Fig. 25 is an exemplary diagram showing an example in which multiple areas are distinguished from a captured image based on a vanishing point and a distance from a vehicle (100).

[0738] First, as in (a) of the above-described Fig. 24, if the acquired image is an image according to the perspective projection method, the closer an object is to the vanishing point detected from the image according to the principle of the perspective projection method, the further away the object may be from the vehicle (100). Conversely, the further away an object is positioned from the vanishing point, the closer the object may be from the vehicle (100).

[0739] Accordingly, the processor (1330) can divide the acquired image into a plurality of regions based on the distance from the vanishing point (3000), centered on the vanishing point (3000). In this case, the processor (1330) can divide the image into a first region (3101) whose distance from the vanishing point (3000) is less than or equal to a first distance, a second region (3102) whose distance from the vanishing point (3000) exceeds the first distance and is less than or equal to a second distance, and a third region (3103) whose distance from the vanishing point (3000) exceeds the second distance.

[0740] Meanwhile, as shown in Fig. 25, when areas are divided according to the distance from the vehicle (100) based on the vanishing point (3000), the processor (1330) can perform a color-based cluster analysis for each area (3101, 3012, 3103). Then, a representative color can be determined based on the cluster analysis result performed for each area, and a priority can be determined for each of the determined representative colors. In addition, for each area, building virtual objects corresponding to each area can be grouped, and the colors of building elements can be mapped according to the determined representative colors for each group.

[0741] Meanwhile, the above-described Fig. 25 illustrates an example of dividing the acquired image into multiple regions according to the distance from the vehicle (100) based on the vanishing point detected from the acquired image. However, it should be understood that the processor (1330) may, alternatively, divide the image into multiple regions based on the color analysis results of each building object detected from the image.

[0742] For example, the processor (1330) may divide each building object detected from the acquired image into multiple regions based on saturation or brightness. In this case, the acquired image may be divided into multiple regions based on regions with similar brightness or saturation.

[0743] FIG. 26 is an example diagram showing an example of a processor (1330) dividing a captured image into multiple regions based on saturation.

[0744] Referring to FIG. 26, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can first detect building objects included in an acquired image. Then, a specific color characteristic can be detected for each of the detected building objects. In this case, if the color characteristic is saturation, the processor (1330) can detect saturation for each building object and segment the acquired image into a plurality of regions according to the detected saturation.

[0745] For example, when an image as shown in (a) of the above-described Figure 24 is acquired, the processor (1330) can detect objects corresponding to a building from the acquired image. In addition, the processor can detect saturation from the color of each of the detected objects.

[0746] In this case, the saturation of the buildings adjacent to the left and right sides of the vehicle (100) may be similar to each other. Accordingly, as shown in FIG. 26, the processor (1330) may divide the areas occupied by the building objects (3201a, 3201b) adjacent to the left and right sides of the vehicle (100) into one area (first area) (3201).

[0747] Meanwhile, depending on the position of the sun, the saturation of building objects in front of the vehicle (100) at a certain distance away may be different. For example, if the sun is located at the upper right of the vehicle (100), the building object on the left side facing the sunlight may be clearer than the building object on the right side. In this case, as shown in FIG. 26, the areas occupied by the left and right objects among the building objects in front of the vehicle (100) at a certain distance away may be divided into different areas (second area (3202), third area (3203)).

[0748] In addition, in the case of building objects that are spaced apart from the vehicle (100) by a certain distance or more, the saturation may be lowered due to the distance. Therefore, as shown in FIG. 26, the processor (1330) may divide the area occupied by building objects with the lowest saturation into a fourth area (3204) that is distinct from the first to third areas (3201 to 3203). In addition, representative colors may be determined for each of the divided areas through color-based cluster analysis, and the priorities of each of the representative colors determined according to density may be determined. In addition, for each of the divided areas, building virtual objects corresponding to each area may be grouped, and the colors of building elements may be mapped for each group according to the determined representative colors.

[0749] The present invention described above can be implemented as computer-readable code on a medium having a program recorded thereon. Computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also include media implemented in the form of carrier waves (e.g., transmission via the Internet). Therefore, the above detailed description should not be construed as limiting in any respect, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are intended to be included in the scope of the present invention.

Claims

1. An interface unit that receives sensing information collected from at least one sensor equipped in a vehicle; Memory for storing map data, and Control the interface unit to receive vehicle driving information including the vehicle's location, speed, and direction of movement, Based on the above driving information, a map area is determined from the map data, and building virtual objects corresponding to each building included in the map area are modeled. A 3D map view generation device comprising a processor for determining some of the modeled building virtual objects as rendering targets based on the driving information, tiling each side area of ​​the building virtual objects to be rendered with a plurality of texture images including shapes of different building elements when a user requests, mapping at least one of the building elements formed in each side area of ​​the building virtual objects to be rendered with a building color extracted from an image captured by a camera of the vehicle, and controlling the interface unit to render the building virtual objects to be rendered with colors mapped to each building element.

2. In the first paragraph, the processor, A 3D map view generation device characterized in that it identifies building objects from the image, detects the colors of each of the identified building objects, calculates a color density, which is an area ratio on the image corresponding to each color, for each detected color, determines at least one representative color according to the calculated color density, and maps the determined at least one representative color to the at least one building element.

3. In the second paragraph, the processor, A 3D map view generation device characterized in that colors having a color density of a certain level or higher are detected as representative colors, each of the colors having a color density of less than a certain level is formed into a cluster with the most similar representative color, and the color density of at least one color included in one cluster is added up to determine the color density of the representative color corresponding to the one cluster.

4. In the third paragraph, the processor, A 3D map view generation device characterized in that, when the representative color of each cluster is determined, a color based on the average of the RGB values ​​of the colors included in each cluster is determined as the representative color of each cluster.

5. In the second paragraph, the processor, A 3D map view generation device characterized in that, when there are multiple representative colors determined, the priority of each representative color is determined based on color density, and each representative color is mapped to each building element in order of highest priority.

6. In paragraph 5, the processor, Determine the priority for mapping colors according to the area of ​​each building element formed in each side area of ​​the virtual objects of the building to be rendered above, A 3D map view generation device characterized by mapping high-priority building elements to a representative color with high priority.

7. In the second paragraph, the processor, A 3D map view generation device characterized in that it determines to divide the image into a plurality of regions, identify building objects in each region, detect colors of each of the identified building objects, calculate color densities for each region, and determine at least one representative color according to the calculated color density.

8. In the 7th paragraph, the processor, A 3D map view generation device characterized in that the virtual objects to be rendered are grouped into a plurality of groups according to each area on the image divided into the plurality of areas, and each building element is mapped to at least one representative color determined in the area on the image corresponding to each group for each group.

9. In paragraph 8, the processor, A 3D map view generation device characterized in that the image is divided into a plurality of regions based on a road image object among image objects included in the image or a vanishing point detected from the image.

10. In the 9th paragraph, the processor, A 3D map view generation device characterized in that it detects a point where the extension lines of a road object or a curb object included in the image converge or a central point of the image as the vanishing point.

11. In the 9th paragraph, the processor, A 3D map view generation device characterized in that the areas on the image where each object is displayed are divided into a plurality of areas according to the direction in which each object is located based on the driving direction of the vehicle, or according to the distance from the vehicle estimated based on the vanishing point.

12. In the 7th paragraph, the processor, A 3D map view generation device characterized in that it detects areas in which the ratio of the area occupied by building objects identified for each area is above a certain level, calculates the color densities for each area only for the detected areas, and determines at least one representative color according to the calculated color density.

13. In the 8th paragraph, the processor, A 3D map view generation device characterized in that the image is divided into a plurality of regions based on at least one of the saturation or brightness of the colors of each building object detected from the image.

14. In the first paragraph, the processor, For each building element identified from the tiled texture images, a normal map having vector values ​​applied to form a texture of a different material is blended to form a texture of a different material for each building element, A 3D map view generation device characterized in that it maps building colors extracted from the image to each building element on which the texture is formed.

15. In the first paragraph, the processor, Based on the location of the vehicle included in the driving information, a frustum-shaped visible area that becomes wider as it gets farther away from the vehicle is determined according to the driving direction of the vehicle, and among the modeled building virtual objects, building virtual objects included in the visible area are determined as rendering targets. The above captured image is, A 3D map view generation device characterized in that the image is in front of the vehicle corresponding to the above visible area.

16. In paragraph 1, The above user's request is, Execution of a program or application related to the above map view image, The above processor, A 3D map view generation device characterized in that, during the runtime in which the above program or application is executed, the building virtual objects determined as the rendering target among the pre-modeled building virtual objects are rendered.

17. In paragraph 1, The above processor is composed of multiple units, Some of the above multiple processors, Modeling a plurality of building virtual objects based on the map area determined according to the driving information received through background processing, Other some of the above multiple processors, A 3D map view generation device characterized in that, in response to a request from the user, it renders some of the modeled building virtual objects.

18. Step of receiving vehicle driving information; A step of determining a map area from map data based on the above driving information and modeling building virtual objects corresponding to each building included in the map area; A step of determining some of the modeled building virtual objects as rendering targets based on the driving information; A step of receiving a user request to display a map view image including 3D virtual objects; In response to the user request, a step of tiling each side area of ​​the rendering target building virtual objects with a plurality of texture images containing shapes of different building elements; A step of acquiring an image around the vehicle; A step of detecting areas corresponding to a building from an acquired image, detecting colors of the detected building areas, and determining at least one representative color based on the colors of the detected building areas; A step of mapping at least one building element formed on each side of the rendering target building virtual objects with at least one representative color determined above; and, A control method of a 3D map view generation device, characterized in that it comprises the step of performing rendering on virtual objects of a rendering target building including at least one building element to which the representative color is mapped, and displaying a map view image including an image of the rendered virtual objects of the building on a display unit of the vehicle.

19. In paragraph 18, the step of determining at least one representative color is: A step of dividing the acquired image into multiple regions, detecting building regions for each region, and calculating the densities of the detected colors according to the area ratio of the region occupied by each region for each detected color; A step of determining a preset number of representative colors in order of increasing density; and A control method for a 3D map view generation device, characterized in that it includes a step of determining the priority of the determined representative colors in the order of the generated density.

20. In paragraph 19, the step of mapping the representative color to the building element is: A step of grouping the virtual objects of the rendering target building into a plurality of groups according to each area of ​​the image divided into a plurality of sections; A step of matching each area on the image corresponding to each group and matching representative colors determined from each matched area; For each group, a step of determining the priority of each building element according to the area of ​​each building element formed in each building virtual object; and, A control method for a 3D map view generation device, characterized in that it includes a step of mapping a representative color for each building element according to the priority of determined building elements and the priority of each representative color.

Images