Device for generating 3D map view based on mixed reality and control method of same device

Abstract

The present invention comprises: an interface unit for receiving sensing information collected from at least one sensor provided in a vehicle; a memory for storing map data for a plurality of map tiles including a map tile including a landmark object; and a processor for controlling the interface unit to receive vehicle driving information including a position, a speed, and a moving direction of the vehicle, modelling building objects included in map data of a plurality of map tiles determined on the basis of the driving information, generating a map view image including building objects that are rendered by tiling a plurality of texture images, which include shapes of different building elements, on respective sides of the modeled building objects, modeling and rendering a landmark object with priority over the building objects when a map tile including the landmark object is present among the plurality of map tiles, and performing modeling and rendering of the building objects after the landmark object is rendered.

Description

A device for generating a 3D map view based on mixed reality and a method for controlling the device

[0001] The present invention relates to a device for generating a 3D map view based on mixed reality.

[0002] Recently, following Augmented Reality (AR), which outputs graphic objects through a vehicle's windshield or HUD (Head Up Display), or uses images captured by a camera to additionally output graphic objects in the real world, technology development related to Mixed Reality (MR), which can provide various simulation information related to vehicles by applying Digital Twin (DT) technology, is actively underway.

[0003] As part of the technological development related to this mixed reality, active research is being conducted on methods that utilize mixed reality to provide drivers with route guidance-related information. Route guidance using mixed reality, for example, utilizes digital twin technology to display graphical objects corresponding to vehicles on a digitized 3D map. This map and graphical objects can be used to provide information on driving routes the driver has not yet driven, or provide a bird's-eye view, offering various information that the driver in the cockpit cannot view.

[0004] Meanwhile, for drivers to obtain accurate information through route guidance using mixed reality, the digital twin technology used to create the map—the digital twin map—must have a high degree of consistency with the real world. This requires the provision of a 3D map, not a 2D one. Therefore, service providers currently offering digital twin maps are offering 3D maps, which are essentially 2D maps mapped into 3D.

[0005] However, current service providers rely on building modeling using satellite or aerial photography to provide 3D mapping. However, the high angles at which satellite or aerial photography is used make it difficult to capture images of various aspects of a building, making it difficult to provide accurate textures. Therefore, conventional 3D maps are provided in the form of polygon data containing the building's floor coordinates (footprint) and height.

[0006] Meanwhile, 3D maps containing models provided in the form of polygon data like this have the problem of not including accurate building textures. These polygonal maps only represent the location and 3D volume of buildings, differing from the actual building shapes. This discrepancy creates a gap between the digital twin map and the real world, making it difficult to provide accurate information to drivers.

[0007] In addition, in order to provide a 3D map that includes 3D virtual objects with high-quality textures similar to actual buildings, a texture model with a three-dimensional shape similar to a high-quality real-world building or road must be mapped to each side of the modeling provided in the form of polygon data. However, since the texture model with such a three-dimensional shape is large in size and has a large number of vertices, there is a problem in that the amount of computation required for mapping the texture model is very large. Accordingly, there is a problem that a delay may occur when providing a high-quality digital twin map.

[0008] The present invention aims to solve the above-mentioned problems and other problems.

[0009] One object of the present invention is to provide a 3D map view generation device capable of displaying virtual objects having high-quality texture materials similar to real-world buildings in real time without delay according to changes in the driving condition of a vehicle, and a method for controlling the device.

[0010] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof capable of providing a more realistic mixed reality-based route guidance service through a digital twin map including virtual objects with more realistic textures corresponding to the shape of each side of a building.

[0011] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof that can provide a mixed reality-based route guidance service through a digital twin map that is more similar to the real world at a lower cost.

[0012] According to an embodiment of the present invention, a 3D map view generation device includes an interface unit that receives sensing information collected from at least one sensor provided in a vehicle, a memory that stores map data for a plurality of map tiles including map tiles including landmark objects, and a processor that controls the interface unit to receive driving information of a vehicle including a location, a speed, and a moving direction of the vehicle, models building objects included in the map data of the plurality of map tiles determined based on the driving information, and generates a map view image including building objects rendered by tiling a plurality of texture images including shapes of different building elements on each side of the modeled building objects, and, when there is a map tile including a landmark object among the plurality of map tiles, models and renders the landmark object with priority over the building objects, and performs modeling and rendering of the building objects after the landmark object is rendered.

[0013] In one embodiment, the processor is characterized in that it first models and renders road and terrain objects included in map data of a plurality of map tiles determined based on the driving information, and then models and renders the landmark object after the road and terrain objects are rendered.

[0014] In one embodiment, the processor is characterized in that, when the vehicle approaches within a predetermined distance from the vehicle, some of the building objects included in the map data of the plurality of map tiles are modeled and rendered while landmark objects included in the map data of the plurality of map tiles are rendered based on the driving information, and when the vehicle moves away from the vehicle beyond a predetermined distance, the rendering of some of the rendered building objects is deactivated while the landmark objects remain rendered.

[0015] In one embodiment, the processor is characterized in that it places the modeled landmark object and building object under a portion of the rendered road and terrain object corresponding to a foot print area of ​​each of the landmark object and building object, moves the modeled landmark object or building object toward the sky under the road and terrain object, and sequentially renders the modeled landmark object or building object from a portion exposed above the rendered road and terrain object according to the movement.

[0016] In one embodiment, the processor is characterized by at least partially limiting the height of at least one building object located between the 3D map view camera providing the 3D map view field of view and the landmark object.

[0017] In one embodiment, the processor is characterized in that it determines a height to be exposed and rendered above the road and terrain objects based on a height of the building object, a height of the landmark object, and a height of the 3D map view camera, for at least one building object located between the 3D map view camera and the landmark object.

[0018] In one embodiment, the processor is characterized in that it limits the height at which at least one building object located between the 3D map view camera and the landmark object, which covers at least a portion of the landmark object within the exposure range, is exposed and rendered above the road and terrain objects so that the landmark object is displayed on the screen beyond a preset exposure range set from the top of the landmark object.

[0019] In one embodiment, the processor is characterized in that it designates at least one building object among building objects included in a plurality of map tiles determined based on the driving information as a custom landmark object, and if there is a map tile including the custom landmark object among the plurality of map tiles determined based on the driving information, it models and renders the custom landmark object with priority over other building objects, and performs modeling and rendering for the other building objects after the landmark object is rendered.

[0020] In one embodiment, the processor designates a building object of a user's selection, a waypoint or destination on a driving route of the vehicle, or a building set as a guide indicator for the driving route, and a building object corresponding to a specific POI (Point Of Interest) as the custom landmark object, and when the custom landmark object is located within a preset distance from the vehicle, changes the rendering of at least one of the building objects around the custom landmark object and the custom landmark object so that the building objects around the custom landmark object are displayed differently from the custom landmark.

[0021] In one embodiment, the processor is characterized in that, when the vehicle approaches within the preset distance, the processor highlights at least one of the sides of the building object designated as the custom landmark, changes one of the colors or brightnesses of the texture images tiled on the side of the building object designated as the custom landmark object, or limits the tiling of the texture images for at least some of the sides of the building objects around the custom landmark object, or limits the height at which the building objects around the custom landmark object are exposed and rendered above the road and terrain objects.

[0022] In one embodiment, the memory includes map data of map tiles of different sizes according to different zoom levels, and the processor is characterized in that it detects map data including object data for generating the map view image from map tiles having different zoom levels according to a distance from the vehicle.

[0023] In one embodiment, the processor detects the map data from map tiles having a lower zoom level as the distance from the vehicle increases, and the map data includes object data for modeling objects having different resolutions depending on the zoom level of the corresponding map tile, or the types of object data included are different.

[0024] In one embodiment, the processor is characterized in that it adopts a first zoom level for a first map area having a distance from the vehicle less than or equal to a first distance, and detects first map data from first map tiles having the first zoom level, adopts a second zoom level for a second map area exceeding the first distance but less than or equal to a second distance, and detects second map data from second map tiles having the second zoom level, adopts a third zoom level for a third map area exceeding the second distance, and detects third map data from third map tiles having the third zoom level, and the second map tiles or the third map tiles include data for a footprint area of ​​each of the building objects instead of building object data for modeling the building object.

[0025] In one embodiment, the processor is characterized in that, when a zoom level of a map tile for a map area around the vehicle changes as the distance from the vehicle changes, the processor displays building objects of different resolutions according to map data of map tiles corresponding to the changed zoom level, or displays each building object in the shape of a footprint area representing an area occupied by the building object.

[0026] In one embodiment, the landmark object is included in map data of a map tile corresponding to a zoom level of some of the first to third zoom levels, and the processor is characterized in that, when the landmark object is modeled and rendered from a map tile including the landmark object among a plurality of map tiles determined based on the driving information of the vehicle, the rendering state of the landmark object is maintained regardless of the zoom level of the changed map tile.

[0027] In one embodiment, the map data of a plurality of map tiles determined based on the driving information further include geometric data of a plurality of roof shapes having different shapes, and the processor further determines a roof shape corresponding to each building object, and when modeling and rendering the building objects, models and renders the roof shape determined for each building object together.

[0028] In one embodiment, the processor is characterized in that it detects roofs of a plurality of buildings from satellite images of an area in which the vehicle is driving or images obtained from a camera of the vehicle, determines the roof shape based on the shape of the detected roofs, and determines a texture image and color to be tiled to the roof shape based on the surface shape and color of the detected roofs.

[0029] In one embodiment, the memory includes at least one roof shape, at least one texture image, and at least one color information for a plurality of different regions, and the processor detects at least one roof shape, at least one texture image, and at least one color information corresponding to information of a region in which the vehicle is driving, and performs modeling and rendering of the roof shape of each building object based on the detected at least one roof shape, texture image, and color information.

[0030] In one embodiment, the memory includes a plurality of color palettes including different colors and adoption probabilities for each color according to time, season, or weather for each region, and the processor determines one of the plurality of color palettes provided for the region in which the vehicle is driven based on at least one of the time or season in which the vehicle is driven or the weather around the vehicle, and determines the colors of the roof shape of each building object according to the adoption probability for each color included in the determined color palette.

[0031] In addition, a control method of a 3D map view generation device according to an embodiment of the present invention comprises the steps of: receiving driving information of a vehicle; determining at least one map tile included in a field of view based on the driving information, detecting map data of the determined map tile; checking whether a landmark object is included in the determined map tile based on the detected map data; modeling and rendering the landmark object with priority over other building objects of the determined map tile based on the check result; modeling other building objects of the determined map tile after the landmark object is modeled and rendered; determining some of the modeled building objects as rendering targets based on the driving information; receiving a user request for displaying a map view image; in response to the user request, tiling each side area of ​​the determined rendering target building objects with a plurality of texture images including shapes of different building elements; blending normal maps to which vector values ​​for forming textures of different materials are applied for each building element identified from the tiled texture images to generate the determined rendering target building objects. It is characterized by including a texturing step and a step of displaying a rendered map view image of the landmark objects and the textured building objects on a display unit of the vehicle.

[0032] The effects of a 3D map view generation device and a control method thereof according to an embodiment of the present invention are described as follows.

[0033] First, the present invention can pre-load map data of a region corresponding to an area around a vehicle based on the vehicle's location according to vehicle driving information from map data of a wide area, and model virtual objects corresponding to buildings and roads included in the loaded region in advance. In addition, by rendering only at least a portion of the modeled virtual objects according to the vehicle's driving information, it has the advantage of providing a digital twin map containing high-quality virtual objects similar to real-world buildings and roads in real time according to the vehicle's driving information.

[0034] Second, for at least some of the modeled building objects, at least one mask map corresponding to different parts of a texture image that shapes each part of the building, and a normal map, which is an image that is superimposed on the mask map and has 3D vector values ​​that form different textures, are superimposed on the texture image so that each component of the building shaped by the texture image has different materials, and the texture image in which the mask map and the normal map are superimposed to form a surface material can be tiled on each face of at least some of the building objects. Then, by rendering and displaying on a display unit at least some of the building objects in which the texture image is tiled, it is possible to perform texturing of the building objects with a small amount of computation while including building objects having high-quality texture materials similar to the actual building, and there is an advantage in that a digital twin map can be provided that includes a building model in which a high-quality texture similar to an actual building is synthesized.

[0035] In addition, the present invention has the effect of providing a digital twin map including higher quality textures by rendering a high quality model pre-made for a well-known landmark and displaying the high quality landmark model by aligning it with general building objects.

[0036] In addition, the present invention has the effect of reducing the amount of computation required for modeling an object by making the zoom levels of map tiles, from which modeling of building objects included in a currently set visible area is selected, different from each other depending on the distance from a vehicle, and limiting the modeling of objects included in a map tile according to the zoom level.

[0037] FIG. 1 is a drawing showing the exterior of a vehicle according to an embodiment of the present invention.

[0038] Figure 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles.

[0039] Figures 3 and 4 are drawings showing the interior of a vehicle according to an embodiment of the present invention.

[0040] Figures 5 and 6 are drawings for reference in explaining an object according to an embodiment of the present invention.

[0041] FIG. 7 is a block diagram for reference in explaining a vehicle according to an embodiment of the present invention.

[0042] Figure 8a is a conceptual diagram for explaining the AR service platform of the present invention.

[0043] Figure 8b is a conceptual diagram for explaining an MR service platform for providing the MR service of the present invention.

[0044] Figure 8c is a conceptual diagram for explaining the MR AMS client of the present invention.

[0045] Figure 8d is a conceptual diagram for explaining the MR AMS server of the present invention.

[0046] Figure 9 is a conceptual diagram for explaining the DTaaS server of the present invention.

[0047] FIG. 10 is a block diagram illustrating the structure of a 3D map view generation device according to an embodiment of the present invention.

[0048] FIG. 11 is a block diagram illustrating in more detail the structure of a 3D map view generation device according to an embodiment of the present invention.

[0049] FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view generation device and an MR service device according to an embodiment of the present invention.

[0050] FIG. 13 is a flowchart illustrating an operation process of a processor of a 3D map view generation device according to an embodiment of the present invention, modeling virtual objects included in a map area and rendering and displaying the modeled virtual objects.

[0051] FIG. 14 is an exemplary diagram illustrating an example in which a processor of a 3D map view generation device according to an embodiment of the present invention updates a map area according to driving information of a vehicle.

[0052] FIG. 15 is a block diagram illustrating a configuration in which modeling and rendering of virtual objects are performed by different processes in a 3D map view generation device according to an embodiment of the present invention having multiple processors.

[0053] FIG. 16 is an exemplary diagram showing an example of simultaneously displaying a screen showing information collected around a vehicle according to an ADAS system and an MR view screen according to a digital twin map in a 3D map view generation device according to an embodiment of the present invention.

[0054] FIG. 17a is an exemplary diagram illustrating a frustum-shaped visible area formed across a plurality of map tiles or one map tile, depending on the position of the vehicle and the driving direction of the vehicle, in a 3D map view generation device according to an embodiment of the present invention.

[0055] Figure 17b is an exemplary diagram showing an example in which the visible area of ​​the frustum shape illustrated in Figure 17a includes a plurality of map tiles.

[0056] FIG. 18 is a flowchart illustrating an operation process of preferentially modeling and rendering a landmark model in a 3D map view generation device according to an embodiment of the present invention.

[0057] FIG. 19 is an exemplary diagram showing an example in which each object included in a map tile is rendered in a preset order in a 3D map view generation device according to an embodiment of the present invention.

[0058] FIG. 20 is an exemplary diagram showing an example of rendering a landmark model and a general building object model in a 3D map view generation device according to an embodiment of the present invention.

[0059] FIG. 21 is an exemplary diagram showing an example in which the maximum height at which a general building object model can be rendered is determined based on the height of a landmark in a 3D map view generation device according to an embodiment of the present invention.

[0060] FIG. 22 is an exemplary diagram showing an example in which a specific building is set as a landmark according to a user's selection in a 3D map view generation device according to an embodiment of the present invention.

[0061] Figure 23 is an exemplary diagram showing various examples in which objects around a building object set as a landmark in Figure 22 are displayed.

[0062] FIG. 24 is an exemplary diagram showing an example of limiting modeling of building objects included in a map tile according to a zoom level based on the distance from a vehicle in a 3D map view generation device according to an embodiment of the present invention.

[0063] FIG. 25a is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention, in which texture images are tiled in a margin width and at least one preset texturing area.

[0064] FIG. 25b is an exemplary diagram showing examples of facade designs of a building virtual object expressed in various ways by varying partial areas and tiled texture images in a 3D map view generation device according to an embodiment of the present invention.

[0065] FIG. 26 is a flowchart illustrating a texturing process for forming different surface textures on each component of a building in a 3D map view generation device according to an embodiment of the present invention.

[0066] FIG. 27 is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention, in which normal maps are blended to mask maps corresponding to different building elements, and a multi-blended normal map is generated by blending normal maps with each other.

[0067] FIG. 28 is a flowchart illustrating an operation process of mapping the colors of virtual building objects based on the colors of actual buildings within a visible area in a 3D map view generation device according to an embodiment of the present invention.

[0068] FIGS. 29 and 30 illustrate examples of roof objects for extending roof modeling of building objects and examples of textures for the roof objects in a 3D map view generation device according to an embodiment of the present invention.

[0069] FIG. 31 is an exemplary diagram illustrating an example of determining a roof object to be added to building objects in an area for which a 3D map view is to be provided, based on an image of the area in which a 3D map view is to be provided, in a 3D map view generation device according to an embodiment of the present invention.

[0070] FIG. 32 is an exemplary diagram showing an example of determining colors mapped to texture images through images captured from a vehicle in a 3D map view generation device according to an embodiment of the present invention.

[0071] FIG. 33 is an exemplary diagram illustrating an example of determining a color for displaying a building object corresponding to a building based on an area occupied by a building and a satellite image of the actual building in a 3D map view generation device according to an embodiment of the present invention.

[0072] Figure 34 is an example diagram showing an example in which a color palette for displaying building objects in a specific area is determined based on satellite imagery for the specific area.

[0073] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0074] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0075] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0076] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0077] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0078] The vehicle described in this specification may include a concept that includes automobiles and motorcycles. In the following, the vehicle will be described primarily with automobiles.

[0079] The vehicle described in this specification may be a concept that includes all types of vehicles, including internal combustion engine vehicles equipped with an engine as a power source, hybrid vehicles equipped with an engine and an electric motor as power sources, and electric vehicles equipped with an electric motor as a power source.

[0080] In the following description, the left side of the vehicle means the left side of the vehicle's driving direction, and the right side of the vehicle means the right side of the vehicle's driving direction.

[0081] FIG. 1 is a drawing showing the exterior of a vehicle according to an embodiment of the present invention.

[0082] Figure 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles.

[0083] Figures 3 and 4 are drawings showing the interior of a vehicle according to an embodiment of the present invention.

[0084] Figures 5 and 6 are drawings for reference in explaining an object according to an embodiment of the present invention.

[0085] FIG. 7 is a block diagram for reference in explaining a vehicle according to an embodiment of the present invention.

[0086] Referring to FIGS. 1 to 7, the vehicle (100) may include wheels that rotate by a power source and a steering input device (510) for controlling the direction of travel of the vehicle (100).

[0087] The vehicle (100) may be an autonomous vehicle.

[0088] The vehicle (100) can be switched to autonomous driving mode or manual mode based on user input.

[0089] For example, the vehicle (100) may be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on user input received through the user interface device (200).

[0090] The vehicle (100) can be switched to autonomous driving mode or manual mode based on driving situation information. The driving situation information can be generated based on object information provided by the object detection device (300).

[0091] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information generated by the object detection device (300).

[0092] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information received through the communication device (400).

[0093] The vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on information, data, and signals provided from an external device.

[0094] When the vehicle (100) is operated in autonomous driving mode, the autonomous vehicle (100) can be operated based on the driving system (700).

[0095] For example, an autonomous vehicle (100) may be driven based on information, data, or signals generated from a driving system (710), an exit system (740), or a parking system (750).

[0096] When the vehicle (100) is driven in manual mode, the autonomous vehicle (100) can receive user input for driving through the driving control device (500). Based on the user input received through the driving control device (500), the vehicle (100) can be driven.

[0097] The overall length refers to the length from the front to the rear of the vehicle (100), the overall width refers to the width of the vehicle (100), and the overall height refers to the length from the bottom of the wheel to the roof. In the following description, the overall length direction (L) may refer to the direction that serves as a reference for measuring the overall length of the vehicle (100), the overall width direction (W) may refer to the direction that serves as a reference for measuring the overall width of the vehicle (100), and the overall height direction (H) may refer to the direction that serves as a reference for measuring the overall height of the vehicle (100).

[0098] As illustrated in FIG. 7, the vehicle (100) may include a user interface device (200), an object detection device (300), a communication device (400), a driving operation device (500), a vehicle driving device (600), a driving system (700), a navigation system (770), a sensing unit (120), a vehicle interface unit (130), a memory (140), a control unit (170), and a power supply unit (190).

[0099] Depending on the embodiment, the vehicle (100) may include additional components other than the components described herein, or may not include some of the components described herein.

[0100] The user interface device (200) is a device for communication between a vehicle (100) and a user. The user interface device (200) can receive user input and provide information generated in the vehicle (100) to the user. The vehicle (100) can implement a UI (User Interface) or UX (User Experience) through the user interface device (200).

[0101] The user interface device (200) may include an input unit (210), an internal camera (220), a biometric detection unit (230), an output unit (250), and a processor (270).

[0102] Depending on the embodiment, the user interface device (200) may include additional components other than the described components, or may not include some of the described components.

[0103] The input unit (200) is for receiving information from a user, and data collected from the input unit (120) can be analyzed by a processor (270) and processed into a user's control command.

[0104] The input unit (200) may be placed inside the vehicle. For example, the input unit (200) may be placed in an area of ​​a steering wheel, an area of ​​an instrument panel, an area of ​​a seat, an area of ​​each pillar, an area of ​​a door, an area of ​​a center console, an area of ​​a head lining, an area of ​​a sun visor, an area of ​​a windshield, or an area of ​​a window.

[0105] The input unit (200) may include a voice input unit (211), a gesture input unit (212), a touch input unit (213), and a mechanical input unit (214).

[0106] The voice input unit (211) can convert a user's voice input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0107] The voice input unit (211) may include one or more microphones.

[0108] The gesture input unit (212) can convert a user's gesture input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0109] The gesture input unit (212) may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input.

[0110] According to an embodiment, the gesture input unit (212) can detect a user's three-dimensional gesture input. To this end, the gesture input unit (212) can include a light output unit that outputs a plurality of infrared lights or a plurality of image sensors.

[0111] The gesture input unit (212) can detect a user's 3D gesture input through a TOF (Time of Flight) method, a structured light method, or a disparity method.

[0112] The touch input unit (213) can convert a user's touch input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0113] The touch input unit (213) may include a touch sensor for detecting a user's touch input.

[0114] According to an embodiment, the touch input unit (213) may be formed integrally with the display unit (251), thereby implementing a touch screen. Such a touch screen may provide both an input interface and an output interface between the vehicle (100) and the user.

[0115] The mechanical input unit (214) may include at least one of a button, a dome switch, a jog wheel, and a jog switch. An electrical signal generated by the mechanical input unit (214) may be provided to a processor (270) or a control unit (170).

[0116] The mechanical input unit (214) can be placed on a steering wheel, center fascia, center console, cockpit module, door, etc.

[0117] The internal camera (220) can capture images of the vehicle interior. The processor (270) can detect the user's status based on the images of the vehicle interior. The processor (270) can obtain information about the user's gaze from the images of the vehicle interior. The processor (270) can detect the user's gestures from the images of the vehicle interior.

[0118] The biometric detection unit (230) can obtain the user's biometric information. The biometric detection unit (230) includes a sensor capable of obtaining the user's biometric information, and can use the sensor to obtain the user's fingerprint information, heartbeat information, etc. The biometric information can be used for user authentication.

[0119] The output unit (250) is for generating output related to vision, hearing, or touch.

[0120] The output unit (250) may include at least one of a display unit (251), an audio output unit (252), and a haptic output unit (253).

[0121] The display unit (251) can display graphic objects corresponding to various information.

[0122] The display unit (251) may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0123] The display unit (251) can implement a touch screen by forming a mutual layer structure with the touch input unit (213) or forming it as an integral part.

[0124] The display unit (251) may be implemented as a HUD (Head Up Display). When the display unit (251) is implemented as a HUD, the display unit (251) may be equipped with a projection module to output information through an image projected onto a windshield or window.

[0125] The display unit (251) may include a transparent display. The transparent display may be attached to a windshield or window.

[0126] A transparent display can display a predetermined screen while having a predetermined transparency. To have transparency, the transparent display may include at least one of a transparent TFEL (Thin Film Electroluminescent), a transparent OLED (Organic Light-Emitting Diode), a transparent LCD (Liquid Crystal Display), a transmissive transparent display, and a transparent LED (Light Emitting Diode) display. The transparency of the transparent display can be adjusted.

[0127] Meanwhile, the user interface device (200) may include a plurality of display units (251a to 251g).

[0128] The display unit (251) may be arranged in one area of ​​the steering wheel, one area of ​​the instrument panel (521a, 251b, 251e), one area of ​​the seat (251d), one area of ​​each pillar (251f), one area of ​​the door (251g), one area of ​​the center console, one area of ​​the head lining, one area of ​​the sun visor, or may be implemented in one area of ​​the windshield (251c), one area of ​​the window (251h).

[0129] The audio output unit (252) converts an electric signal provided from the processor (270) or the control unit (170) into an audio signal and outputs the converted signal. To this end, the audio output unit (252) may include one or more speakers.

[0130] The haptic output unit (253) generates a tactile output. For example, the haptic output unit (253) can operate by vibrating a steering wheel, a seat belt, or a seat (110FL, 110FR, 110RL, 110RR) so that the user can perceive the output.

[0131] The processor (270) can control the overall operation of each unit of the user interface device (200).

[0132] Depending on the embodiment, the user interface device (200) may include a plurality of processors (270) or may not include a processor (270).

[0133] If the user interface device (200) does not include a processor (270), the user interface device (200) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0134] Meanwhile, the user interface device (200) may be referred to as a vehicle display device.

[0135] The user interface device (200) can be operated under the control of the control unit (170).

[0136] The object detection device (300) is a device for detecting an object located outside a vehicle (100).

[0137] Objects may be various objects related to the operation of the vehicle (100).

[0138] Referring to FIGS. 5 and 6, objects may include lanes (OB10), other vehicles (OB11), pedestrians (OB12), two-wheeled vehicles (OB13), traffic signals (OB14, OB15), lights, roads, structures, speed bumps, terrain, animals, etc.

[0139] A lane (OB10) may be a driving lane, a lane adjacent to a driving lane, or a lane in which opposing vehicles drive. A lane (OB10) may be a concept that includes lines on the left and right sides that form a lane.

[0140] Another vehicle (OB11) may be a vehicle driving around the vehicle (100). The other vehicle may be a vehicle located within a predetermined distance from the vehicle (100). For example, the other vehicle (OB11) may be a vehicle preceding or following the vehicle (100).

[0141] A pedestrian (OB12) may be a person located around a vehicle (100). A pedestrian (OB12) may be a person located within a predetermined distance from a vehicle (100). For example, a pedestrian (OB12) may be a person located on a sidewalk or roadway.

[0142] A two-wheeled vehicle (OB12) may refer to a vehicle that is positioned around a vehicle (100) and moves using two wheels. The two-wheeled vehicle (OB12) may be a vehicle with two wheels that is positioned within a predetermined distance from the vehicle (100). For example, the two-wheeled vehicle (OB13) may be a motorcycle or bicycle positioned on a sidewalk or road.

[0143] Traffic signals may include traffic lights (OB15), traffic signs (OB14), and patterns or text painted on the road surface.

[0144] The light may be generated from a lamp installed in another vehicle. The light may be generated from a streetlight. The light may be sunlight.

[0145] A road may include slopes such as road surfaces, curves, uphill and downhill slopes, etc.

[0146] Structures may be objects located along roads and fixed to the ground. For example, structures may include streetlights, street trees, buildings, utility poles, traffic lights, and bridges.

[0147] Landforms may include mountains, hills, etc.

[0148] Meanwhile, objects can be classified into moving objects and fixed objects. For example, moving objects may include concepts such as other vehicles and pedestrians. For example, fixed objects may include concepts such as traffic signals, roads, and structures.

[0149] The object detection device (300) may include a camera (310), a radar (320), a lidar (330), an ultrasonic sensor (340), an infrared sensor (350), and a processor (370).

[0150] Depending on the embodiment, the object detection device (300) may include additional components other than the described components, or may not include some of the described components.

[0151] The camera (310) may be positioned at an appropriate location outside the vehicle to capture images of the vehicle's exterior. The camera (310) may be a mono camera, a stereo camera (310a), an AVM (Around View Monitoring) camera (310b), or a 360-degree camera.

[0152] For example, the camera (310) may be positioned inside the vehicle, close to the front windshield, to capture an image of the front of the vehicle. Alternatively, the camera (310) may be positioned around the front bumper or radiator grill.

[0153] For example, the camera (310) may be positioned inside the vehicle, close to the rear glass, to capture images of the rear of the vehicle. Alternatively, the camera (310) may be positioned around the rear bumper, trunk, or tailgate.

[0154] For example, the camera (310) may be positioned close to at least one of the side windows inside the vehicle to obtain an image of the side of the vehicle. Alternatively, the camera (310) may be positioned around a side mirror, fender, or door.

[0155] The camera (310) can provide the acquired image to the processor (370).

[0156] The radar (320) may include an electromagnetic wave transmitter and receiver. The radar (320) may be implemented in a pulse radar or continuous wave radar manner based on the principle of radio wave emission. Among continuous wave radar methods, the radar (320) may be implemented in an FMCW (Frequency Modulated Continuous Wave) manner or an FSK (Frequency Shift Keying) manner depending on the signal waveform.

[0157] The radar (320) can detect an object using electromagnetic waves, based on a TOF (Time of Flight) method or a phase-shift method, and can detect the location of the detected object, the distance to the detected object, and the relative speed.

[0158] The radar (320) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0159] The lidar (330) may include a laser transmitter and receiver. The lidar (330) may be implemented using a TOF (Time of Flight) method or a phase-shift method.

[0160] The lidar (330) can be implemented as a driven or non-driven type.

[0161] When implemented as a drive type, the lidar (330) is rotated by a motor and can detect objects around the vehicle (100).

[0162] When implemented in a non-driven manner, the lidar (330) can detect an object located within a predetermined range relative to the vehicle (100) through optical steering. The vehicle (100) can include a plurality of non-driven lidars (330).

[0163] Lidar (330) can detect an object based on a time-of-flight (TOF) method or a phase-shift method using laser light as a parameter, and can detect the position of the detected object, the distance to the detected object, and the relative speed.

[0164] The lidar (330) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.

[0165] The ultrasonic sensor (340) may include an ultrasonic transmitter and a receiver. The ultrasonic sensor (340) may detect an object based on ultrasonic waves, and may detect the location of the detected object, the distance from the detected object, and the relative speed.

[0166] The ultrasonic sensor (340) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0167] The infrared sensor (350) may include an infrared transmitter and a receiver. The infrared sensor (340) may detect an object based on infrared light, and may detect the location of the detected object, the distance to the detected object, and the relative speed.

[0168] The infrared sensor (350) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0169] The processor (370) can control the overall operation of each unit of the object detection device (300).

[0170] The processor (370) can detect and track an object based on the acquired image. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object through an image processing algorithm.

[0171] The processor (370) can detect and track an object based on the reflected electromagnetic waves that are returned when the transmitted electromagnetic waves are reflected by the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the electromagnetic waves.

[0172] The processor (370) can detect and track an object based on the reflected laser light that is reflected back by the transmitted laser beam from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the laser light.

[0173] The processor (370) can detect and track an object based on the reflected ultrasonic waves that are returned when the transmitted ultrasonic waves are reflected off the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the ultrasonic waves.

[0174] The processor (370) can detect and track an object based on the reflected infrared light that is reflected back by the transmitted infrared light from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the infrared light.

[0175] Depending on the embodiment, the object detection device (300) may include a plurality of processors (370) or may not include a processor (370). For example, each of the camera (310), radar (320), lidar (330), ultrasonic sensor (340), and infrared sensor (350) may individually include a processor.

[0176] If the object detection device (300) does not include a processor (370), the object detection device (300) can be operated under the control of the processor or control unit (170) of the device in the vehicle (100).

[0177] The object detection device (400) can be operated under the control of the control unit (170).

[0178] The communication device (400) is a device for communicating with an external device. Here, the external device may be another vehicle, a mobile terminal, or a server.

[0179] The communication device (400) may include at least one of a transmitting antenna, a receiving antenna, an RF (Radio Frequency) circuit capable of implementing various communication protocols, and an RF element to perform communication.

[0180] The communication device (400) may include a short-range communication unit (410), a location information unit (420), a V2X communication unit (430), an optical communication unit (440), a broadcast transmission / reception unit (450), and a processor (470).

[0181] Depending on the embodiment, the communication device (400) may include additional components other than the described components, or may not include some of the described components.

[0182] The short-range communication unit (410) is a unit for short-range communication. The short-range communication unit (410) can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0183] The short-range communication unit (410) can form a short-range wireless communication network (Wireless Area Network) to perform short-range communication between the vehicle (100) and at least one external device.

[0184] The location information unit (420) is a unit for obtaining location information of a vehicle (100). For example, the location information unit (420) may include a GPS (Global Positioning System) module or a DGPS (Differential Global Positioning System) module.

[0185] The V2X communication unit (430) is a unit for performing wireless communication with a server (V2I: Vehicle to Infrastructure), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). The V2X communication unit (430) may include an RF circuit capable of implementing protocols for communication with infrastructure (V2I), communication between vehicles (V2V), and communication with pedestrians (V2P).

[0186] The optical communication unit (440) is a unit for communicating with an external device via light. The optical communication unit (440) may include an optical transmission unit that converts an electrical signal into an optical signal and transmits it to the outside, and an optical reception unit that converts a received optical signal into an electrical signal.

[0187] According to an embodiment, the light transmitting unit may be formed to be integrated with a lamp included in the vehicle (100).

[0188] The broadcast transmitter / receiver (450) is a unit for receiving broadcast signals from an external broadcast management server via a broadcast channel, or transmitting broadcast signals to the broadcast management server. The broadcast channels may include satellite channels and terrestrial channels. The broadcast signals may include TV broadcast signals, radio broadcast signals, and data broadcast signals.

[0189] The processor (470) can control the overall operation of each unit of the communication device (400).

[0190] Depending on the embodiment, the communication device (400) may include a plurality of processors (470) or may not include a processor (470).

[0191] If the communication device (400) does not include a processor (470), the communication device (400) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0192] Meanwhile, the communication device (400) may implement a vehicle display device together with the user interface device (200). In this case, the vehicle display device may be referred to as a telematics device or an AVN (Audio Video Navigation) device.

[0193] The communication device (400) can be operated under the control of the control unit (170).

[0194] The driving control device (500) is a device that receives user input for driving.

[0195] In manual mode, the vehicle (100) can be driven based on signals provided by the driving control device (500).

[0196] The driving control device (500) may include a steering input device (510), an acceleration input device (530), and a brake input device (570).

[0197] The steering input device (510) can receive input for the direction of travel of the vehicle (100) from the user. The steering input device (510) is preferably formed in the form of a wheel so that steering input can be provided by rotation. Depending on the embodiment, the steering input device may be formed in the form of a touch screen, a touch pad, or a button.

[0198] The acceleration input device (530) can receive an input from a user for accelerating the vehicle (100). The brake input device (570) can receive an input from a user for decelerating the vehicle (100). The acceleration input device (530) and the brake input device (570) are preferably formed in the form of a pedal. Depending on the embodiment, the acceleration input device or the brake input device may also be formed in the form of a touch screen, a touch pad, or a button.

[0199] The driving operation device (500) can be operated under the control of the control unit (170).

[0200] The vehicle driving device (600) is a device that electrically controls the driving of various devices in the vehicle (100).

[0201] The vehicle driving device (600) may include a power train driving unit (610), a chassis driving unit (620), a door / window driving unit (630), a safety device driving unit (640), a lamp driving unit (650), and an air conditioning driving unit (660).

[0202] Depending on the embodiment, the vehicle drive device (600) may include additional components other than the described components, or may not include some of the described components.

[0203] Meanwhile, the vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0204] The power train drive unit (610) can control the operation of the power train device.

[0205] The power train drive unit (610) may include a power source drive unit (611) and a transmission drive unit (612).

[0206] The power source driving unit (611) can perform control over the power source of the vehicle (100).

[0207] For example, if a fossil fuel-based engine is the power source, the power source drive unit (610) can perform electronic control of the engine. This can control the engine output torque, etc. The power source drive unit (611) can adjust the engine output torque according to the control of the control unit (170).

[0208] For example, if an electric energy-based motor is the power source, the power source driving unit (610) can perform control over the motor. The power source driving unit (610) can adjust the rotation speed, torque, etc. of the motor according to the control of the control unit (170).

[0209] The transmission drive unit (612) can perform control over the transmission.

[0210] The transmission drive unit (612) can adjust the state of the transmission. The transmission drive unit (612) can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or parking (P).

[0211] Meanwhile, when the engine is the power source, the transmission drive unit (612) can adjust the gear engagement state in the forward (D) state.

[0212] The chassis drive unit (620) can control the operation of the chassis device.

[0213] The chassis drive unit (620) may include a steering drive unit (621), a brake drive unit (622), and a suspension drive unit (623).

[0214] The steering drive unit (621) can perform electronic control of the steering apparatus within the vehicle (100). The steering drive unit (621) can change the direction of travel of the vehicle.

[0215] The brake drive unit (622) can perform electronic control of the brake apparatus within the vehicle (100). For example, the speed of the vehicle (100) can be reduced by controlling the operation of the brakes placed on the wheels.

[0216] Meanwhile, the brake driving unit (622) can individually control each of the plurality of brakes. The brake driving unit (622) can control the braking force applied to the plurality of wheels differently.

[0217] The suspension drive unit (623) can perform electronic control of the suspension apparatus within the vehicle (100). For example, when there is a curve in the road surface, the suspension drive unit (623) can control the suspension apparatus to reduce vibration of the vehicle (100).

[0218] Meanwhile, the suspension drive unit (623) can individually control each of the plurality of suspensions.

[0219] The door / window actuator (630) can perform electronic control of a door apparatus or window apparatus within a vehicle (100).

[0220] The door / window driving unit (630) may include a door driving unit (631) and a window driving unit (632).

[0221] The door driving unit (631) can control the door device. The door driving unit (631) can control the opening and closing of a plurality of doors included in the vehicle (100). The door driving unit (631) can control the opening or closing of a trunk or tail gate. The door driving unit (631) can control the opening or closing of a sunroof.

[0222] The window driving unit (632) can perform electronic control of a window apparatus. It can control the opening or closing of a plurality of windows included in a vehicle (100).

[0223] The safety device driving unit (640) can perform electronic control of various safety devices in the vehicle (100).

[0224] The safety device drive unit (640) may include an airbag drive unit (641), a seat belt drive unit (642), and a pedestrian protection device drive unit (643).

[0225] The airbag driving unit (641) can perform electronic control of the airbag apparatus within the vehicle (100). For example, the airbag driving unit (641) can control the airbag to deploy when a danger is detected.

[0226] The seat belt drive unit (642) can perform electronic control of the seat belt apparatus within the vehicle (100). For example, the seat belt drive unit (642) can control the passenger to be secured to the seat (110FL, 110FR, 110RL, 110RR) using the seat belt when a danger is detected.

[0227] The pedestrian protection device drive unit (643) can perform electronic control of the hood lift and pedestrian airbag. For example, the pedestrian protection device drive unit (643) can control the hood lift up and the pedestrian airbag to deploy when a collision with a pedestrian is detected.

[0228] The lamp driving unit (650) can perform electronic control of various lamp apparatuses within the vehicle (100).

[0229] The air conditioning drive unit (660) can perform electronic control of the air conditioning device (air conditioner) within the vehicle (100). For example, the air conditioning drive unit (660) can control the air conditioning device to operate and supply cool air to the vehicle when the temperature inside the vehicle is high.

[0230] The vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0231] The vehicle driving device (600) can be operated under the control of the control unit (170).

[0232] The driving system (700) is a system that controls various operations of the vehicle (100). The driving system (700) can be operated in autonomous driving mode.

[0233] The driving system (700) may include a driving system (710), an exiting system (740), and a parking system (750).

[0234] Depending on the embodiment, the driving system (700) may include additional components other than the described components, or may not include some of the described components.

[0235] Meanwhile, the driving system (700) may include a processor. Each unit of the driving system (700) may individually include a processor.

[0236] Meanwhile, depending on the embodiment, if the operation system (700) is implemented in software, it may be a sub-concept of the control unit (170).

[0237] Meanwhile, according to an embodiment, the driving system (700) may be a concept including at least one of a user interface device (200), an object detection device (300), a communication device (400), a vehicle driving device (600), and a control unit (170).

[0238] The driving system (710) can drive the vehicle (100).

[0239] The driving system (710) receives navigation information from the navigation system (770) and provides a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0240] The driving system (710) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to enable driving of the vehicle (100).

[0241] The driving system (710) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0242] The exit system (740) can perform exit of a vehicle (100).

[0243] The exit system (740) receives navigation information from the navigation system (770) and provides a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0244] The exit system (740) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0245] The exit system (740) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0246] The parking system (750) can perform parking of a vehicle (100).

[0247] The parking system (750) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0248] The parking system (750) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0249] The parking system (750) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0250] A navigation system (770) can provide navigation information. The navigation information can include at least one of map information, set destination information, route information based on the set destination, information on various objects along the route, lane information, and current vehicle location information.

[0251] The navigation system (770) may include memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system (770).

[0252] According to an embodiment, the navigation system (770) may receive information from an external device through a communication device (400) and update previously stored information.

[0253] Depending on the embodiment, the navigation system (770) may be classified as a sub-component of the user interface device (200).

[0254] The sensing unit (120) can sense the status of the vehicle. The sensing unit (120) can include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight detection sensor, a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0255] The sensing unit (120) can obtain sensing signals for vehicle attitude information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle inclination information, vehicle forward / backward information, battery information, fuel information, tire information, vehicle lamp information, vehicle internal temperature information, vehicle internal humidity information, steering wheel rotation angle, vehicle external illumination, pressure applied to an accelerator pedal, pressure applied to a brake pedal, etc.

[0256] The sensing unit (120) may further include, in addition, an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0257] The vehicle interface unit (130) can serve as a conduit for various types of external devices connected to the vehicle (100). For example, the vehicle interface unit (130) may be equipped with a port capable of connecting to a mobile terminal, and may be connected to the mobile terminal through the port. In this case, the vehicle interface unit (130) can exchange data with the mobile terminal.

[0258] Meanwhile, the vehicle interface unit (130) may serve as a conduit for supplying electrical energy to a connected mobile terminal. When the mobile terminal is electrically connected to the vehicle interface unit (130), the vehicle interface unit (130) may provide the mobile terminal with electrical energy supplied from the power supply unit (190) under the control of the control unit (170).

[0259] The memory (140) is electrically connected to the control unit (170). The memory (140) can store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory (140) can be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware. The memory (140) can store various data for the overall operation of the vehicle (100), such as programs for processing or controlling the control unit (170).

[0260] Depending on the embodiment, the memory (140) may be formed integrally with the control unit (170) or implemented as a sub-component of the control unit (170).

[0261] The control unit (170) can control the overall operation of each unit within the vehicle (100). The control unit (170) can be referred to as an ECU (Electronic Control Unit).

[0262] The power supply unit (190) can supply power required for the operation of each component under the control of the control unit (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0263] One or more processors and control units (170) included in the vehicle (100) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0264] Figure 8a is a conceptual diagram for explaining the AR service platform of the present invention.

[0265] The AR service platform that provides the AR service of the present invention may be called an AR service system.

[0266] The above AR service platform may include a server (850) installed outside the vehicle, which collects and processes information required for AR service and transmits it to the vehicle, and an AR service device (800) installed in the vehicle, which provides AR service using information transmitted from the server.

[0267] The fact that the server (850) collects and processes information required for the AR service and transmits it to the vehicle may include the meaning that the server (850) collects and processes information required for the AR service and transmits it to the AR service device (800) equipped in the vehicle.

[0268] The above AR service device (800) can change the information provided as an AR service based on the situation of the vehicle.

[0269] That is, the AR service device (800) of the present invention can dynamically adjust (variably) the information and amount of information to be displayed in AR according to the situation of the vehicle, and select information to be emphasized.

[0270] In addition, the AR service platform of the present invention can control the AR service provided in the vehicle to vary depending on specific conditions such as vehicle conditions and advertisement exposure conditions.

[0271] In the case of conventional AR navigation, when displaying destinations or major POIs (Points of Interest) in AR navigation, it is difficult to reflect the latest information because it uses information stored in map data, and there are limitations in that POIs with real-time attributes such as fueling / parking cannot be provided.

[0272] On the other hand, the AR service platform of the present invention can display vehicle location information, map data, multiple sensor data, real-time POI information, advertisement / event information, etc., by integrating them in AR navigation.

[0273] For example, to display AR information, the AR service device (800) of the present invention may receive AR service information from a server based on the current location of the vehicle and navigation route / guidance information, and process the information into a form for displaying on an AR navigation screen.

[0274] For example, the AR service device (800) of the present invention can reconstruct real-time AR display information. The AR service device (800) can reconstruct service data received from a server to display on an AR navigation screen by determining the display format, size, location, exposure method, etc. of AR content by considering the driving situation (e.g., varying the POI exposure location and size according to driving speed, changing the service information exposure location according to traffic conditions, adjusting the AR Wall display location and exposure time, etc.).

[0275] In addition, the AR service device (800) of the present invention can analyze the frequency of exposure of AR display information through user feedback.

[0276] The server (850) collects user input information (input information such as touch, order, etc.) for AR service content, performs content exposure frequency analysis, and can adjust the service content exposure policy based on the information.

[0277] Through this configuration, the present invention can express various external service contents in AR navigation by integrating them, and can provide various services through POI information including real-time properties.

[0278] In addition, the present invention enables display of various forms of AR content, such as POI information, advertisements, events, and major landmark information.

[0279] In addition, a new user experience of AR navigation can be presented through an embodiment based on the UX scenario proposed in the present invention.

[0280] The present invention can provide a service platform structure and an AR information display method (UX) that dynamically adjusts the amount of information (POI data, advertisement) to be displayed in AR according to vehicle conditions and advertisement exposure conditions, a module that collects POI information and commerce service information for AR expression and processes them into a form that is easy to render in an AR engine, a module that processes specific POI information to be emphasized according to the inside / outside vehicle conditions, a module that collects vehicle situation information and applies a UX policy according to the situation, and an AR engine module that renders AR objects (Group POI, Mini POI, 3D Object, Event wall, etc.) according to the UX policy.

[0281] In addition, the present invention can provide a client module for transmitting and receiving interaction and data between the front and rear displays of a vehicle, a Service App module for exposing commerce service information linked to POI, a client module for collecting user actions for advertisements such as AR advertisement object exposure results and clicks, and a cloud module for collecting / analyzing user actions for advertisements such as AR advertisement object exposure results and clicks.

[0282] Referring to FIG. 8a, the AR service platform of the present invention may include a server (850) that is a configuration existing outside the vehicle (off-board) and an AR service device (800) that is a configuration equipped in the vehicle (on-board).

[0283] First, the server (850) may include a POI data aggregator (851), an Ads manager (852), an Ads monitoring unit (853), a Service & Ads Manager (854), a Commerce Manager (855), a DB Connector (856), and a Dashboard (857).

[0284] The POI Data Aggregator (851) can receive information required for an AR service from multiple external servers and convert / integrate it into a message format of the AR service platform.

[0285] The Ads manager (852) can manage advertising information / content and advertising campaigns (advertisement exposure conditions).

[0286] The Ads Monitoring Department (853) can collect / save ad exposure and click results.

[0287] The Service & Ads Manager (854) can insert advertising information that meets the exposure conditions into service information and provide it to the client.

[0288] The Commerce Manager (855) can collect commerce service linkage / payment information.

[0289] The database connector (856) can store / query advertising content, advertising exposure result information, and commerce payment information.

[0290] The Dashboard (857) can display real-time AR service status visualized with advertisement exposure results / payment history results.

[0291] In addition, the server (850) may further include an AR service cloud API (or data conversion unit) for converting information transmitted from the vehicle's AR service device (800) into a data format available to the server, and for converting information processed / generated by the server into a data format available to the AR service device (800).

[0292] Meanwhile, the AR service device (800) may include a client (810) including a cloud interface, a commerce app, a CID-RSE interaction manager, a policy manager, advertisement monitoring, a driving context, personalized recommendations, etc., and an AR engine (820) including a POI renderer, a display manager, a touch manager, etc.

[0293] The client (810) can receive POI information, advertisements, etc. from the server.

[0294] Additionally, the client (810) can transmit and receive order / payment information to and from the server (850) and transmit advertisement exposure results to the server (850).

[0295] The AR engine (820) can transmit data such as the number of times an AR object output as AR is touched and the number of times it is exposed to the client (810).

[0296] Additionally, the AR engine (820) can transmit and receive front / rear seat (CID, RSE) linkage data to and from the client (810), and output an AR object according to the AR display policy received from the client (810).

[0297] In addition, the AR engine (820) can determine the type, output location, POI type, output size, etc. of AR objects provided through the AR service based on data collected from a gyro sensor, camera, communication unit, navigation, ADAS (Advanced Driver Assistance Systems), GPS, etc. equipped in the vehicle.

[0298] The AR service device (800) installed in the vehicle can AR render service content to display data transmitted from a cloud server in AR on the front camera image.

[0299] Additionally, the AR service device (800) can mediate data transmission between the server and the AR engine, such as collecting advertisement posting result data from the AR engine and transmitting it to the server.

[0300] Additionally, the AR service device (800) can link data generated by AR between CID-RSE (i.e., front / rear seats).

[0301] In addition, the AR service device (800) can perform data management for AR display policies, and specifically, can provide AR display policy data according to driving conditions to the AR engine.

[0302] In addition, the AR service device (800) can provide situational awareness and personalized services, and specifically, can provide AR objects to the AR engine according to driving situations (speed, TBT (Turn-by-Turn), etc.) using in-vehicle data.

[0303] In this specification, an example of providing an AR service by overlapping and outputting AR information (or AR object, AR content, POI information, etc.) on an image captured (received, processed) by a camera installed in a vehicle will be described.

[0304] However, the AR service described in this specification is not limited thereto, and can be applied in a similar or identical manner to various methods of implementing augmented reality, such as outputting AR information directly on the windshield of a vehicle to overlap with the real world space for a driver or passenger, or outputting AR information through a HUD (Head-up Display).

[0305] The input data (input information) used to provide AR services and the output data (output information) provided through the AR service platform are as follows.

[0306] First, the types of input data may include map data (navigation information), service content information (POI, advertisement, etc.), dynamic information, vehicle sensor information, historical information, and driving-related information.

[0307] Map data (navigation information) may include information such as route information to the destination (navigation route), guidance information (turn-by-turn), road shape ahead (road / lane), multiple map attribute information (road type / attribute, road and lane width, curvature, slope, speed limit, etc.), and localization object information (road marking, traffic sign, etc.).

[0308] Service content information (POI, advertisements, etc.) may include POI information received from multiple service providers, advertisement information available at the current location, and real-time information for reservation / payment services such as gas stations, charging stations, and parking lots.

[0309] Dynamic information may include traffic information (road-level traffic, lane-level traffic), event information (accidents, hazard warnings, etc.), weather information, V2X (V2V, V2I) (Vehicle to Everything, Vehicle to Vehicle, Vehicle to Infra), etc.

[0310] Vehicle sensor information may include current location information (GPS / DR), camera input information (ADAS information, object recognition information), and V2X (real-time surrounding situation information that can be collected through V2V and V2I).

[0311] Historical information may include information about past driving routes, traffic history (e.g., traffic volume by time of day), and communication speeds by zone and time of day.

[0312] Driving-related information may include driving mode (manual, autonomous driving, semi-autonomous driving, ADAS function operation, etc.), whether the vehicle has entered a destination or waypoint, and whether the vehicle has entered a parking lot.

[0313] Output information that can be provided through the AR service platform may include AR service display data based on current location / route.

[0314] Current location / route-based AR service display data may include AR advertisement display points along the route (AR Wall, POI building highlights), selectable AR building information (selectable main building information such as landmarks), general POI information (POI summary information such as icons or speech bubbles), remote POI information (distance / direction of important POI information that does not exist on the route but is helpful for driving), display information displayed when multiple POIs exist in the same building, destination building and real-time parking lot status information, real-time status information of gas stations / charging stations, and location-based advertisement / event information.

[0315] The AR service platform of the present invention can filter AR service information according to real-time information and determine a display method.

[0316] Specifically, the AR service platform can determine the number of real-time POI exposures based on driving speed, POI overlap removal, size adjustment, exposure time, etc.

[0317] In addition, the AR service platform can determine the POI exposure method according to risk information recognition, and specifically, can dynamically change the POI display method according to situations such as accidents, construction, and recognition of multiple moving objects.

[0318] Additionally, the AR service platform can dynamically change the POI display location when AR display visibility is reduced due to traffic.

[0319] In addition, the AR service platform can reconfigure front / rear seat AR display data. For example, the AR service information can be minimized on the front seat display and the rear seat display can be reconfigured to display all displayable information by considering driving speed, hazard information, weather information, etc.

[0320] The operation / function / control method of such an AR service platform may be implemented by a server or AR service device included in the AR service platform, or by organic interaction between the server and the AR service device.

[0321] Referring to FIG. 8a, the configuration of the server (850) of the AR service platform is described in more detail as follows.

[0322] The service and advertisement manager (854) can perform a client linkage (request) function, a POI information and advertisement information integration (data processing & aggregation) function, and a client linkage (respond) function.

[0323] Specifically, the client connection (request) function may include requesting / receiving POI information (location, category) through Unified API, or requesting / receiving destination entrance location information (select one of destination coordinates / address / id) through Unified API.

[0324] Here, Unified API refers to an API defined in the AR service cloud that is independent of a specific data provider (to minimize client changes).

[0325] The POI information and advertisement information integration (data processing & aggregation) function may include integrating POI information and advertisement information within a radius of 000m from the location requested by the client (from data manager, ads manager) or integrating the entrance location of the destination requested by the client and POI advertisement information (from data manager, ads manager).

[0326] Specifically, the POI information and advertisement information integration function may include a function to align advertisement information including building wall and event wall information with POI information, or, if there are multiple POIs in the same building, to filter them by priority on the server (e.g., leaving only partner companies and excluding other POI information).

[0327] Here, the filtering criteria may include assigning priority scores to each POI and comparing them.

[0328] Client response functionality may include transmitting POI information and advertisement information via Unified API, or transmitting destination entrance location and advertisement information via Unified API.

[0329] The data manager (not shown) included in the server (850) may include a POI information collection / transmission function, a building shape (polygon) information collection / transmission function, and a destination entrance information collection / transmission function.

[0330] The POI information collection / transfer function can perform the function of requesting POI information from a 3rd party API or transmitting (to Service & Ads Aggregator) POI information received from a 3rd party API (by converting it into a Unified API response format).

[0331] The building shape (polygon) information collection / transfer function can request building exterior shape information from a 3rd party API / data set, or perform the function of transmitting (to Service & Ads Aggregator) POI information received from a 3rd party API (by converting it into a Unified API response format).

[0332] The destination entrance information collection / transfer function can perform the function of requesting destination entrance information from a 3rd party API or transmitting destination entrance information received from a 3rd party API (by converting it into a Unified API response format) (to Service & Ads Aggregator).

[0333] The Ads Manager (852) can provide a partner (advertising) company management interface, an advertising campaign management interface, and an advertising content management interface.

[0334] The Ads Monitoring Department (853) can perform the function of receiving advertising effectiveness measurement feedback and the function of transmitting advertising information.

[0335] The partner (advertiser) management interface can perform POI advertiser management (add / edit / delete advertiser information) and general advertiser management (add / delete advertiser information).

[0336] POI supported ad formats may include Brand poi pin, Building wall, 3D rendering, Event wall, etc., and the supported ad format (general supported ad format) for brand advertisements that are not related to actual POI / location (e.g. Coca-Cola advertisements) may be Event wall.

[0337] The ad campaign management interface allows you to add / edit / delete ad campaigns (ad location, type, time, etc.).

[0338] The advertising content management interface can add / edit / view / delete content by advertising format (POI brand icon image, building wall image, event wall image / video, 3D rendering image).

[0339] The advertising effectiveness measurement feedback reception function may include a function for receiving advertising exposure feedback sent by the client and transmitting it to the DB Manager (CPC / CMP / CPT&P).

[0340] The advertising information delivery function may include a function that searches for and delivers advertising campaign information that should be displayed within a radius of 000m from the location requested by the Service & Ads Aggregator (in the case of CPT&P, only deliver advertisements that meet the time conditions).

[0341] The Commerce Manager (855) can perform client linkage functions, external commerce service linkage functions, and payment information management functions.

[0342] The client linkage function may include receiving requests by linking the client with Unified API, converting the request contents received with Unified API into external commerce API specifications, converting data received from the external API into the Unified API message format, and transmitting data to the client.

[0343] The Commerce Manager can convert the request content received through the Unified API into an external commerce API specification and then perform external service linkage functions using the converted content.

[0344] Converting data received from an external API into the message format of the Unified API may mean converting data received from an external service connection into the Unified API.

[0345] The external commerce service linkage function may include functions for requesting a list of nearby stores and meta information from the current location and receiving results, requesting detailed information about a specific store from the list and receiving results, requesting reservations / orders and receiving results, requesting service usage status and receiving results, and linking member information of commerce services and receiving results.

[0346] Here, the service usage status request & result reception can be used for sequence management and AR message pop-up according to the service usage status (reservation completed / parking lot entry / parking in progress / parking lot exit / reservation cancellation).

[0347] Service member information linking & result reception can be used to link commerce service member ↔ AR service member (OEM Connectivity service member) information.

[0348] The payment information management function may include a function to collect payment details (contents, amount) from external commerce services and charge fees to external commerce companies based on the payment details.

[0349] The database connector (856) can perform advertising effectiveness measurement data management functions, commerce data management functions, advertiser data management functions, advertising content data management functions, and advertising location data management functions.

[0350] The advertising effectiveness measurement data management function can save / delete CPC / CPM / CPT&P related log data and view data (by POI, brand, time, and ad type).

[0351] The commerce data management function can save / delete payment history (content, amount) from external commerce services and perform data search (by POI, brand, time, and ad type).

[0352] The advertiser data management function can save / edit / delete / view advertiser information and advertising campaign settings for each advertiser.

[0353] The advertising content data management function can save / edit / delete / view advertising content in conjunction with advertiser information.

[0354] The advertising location data management function can manage the Event wall section coordinates and Building wall coordinates (by brand) for displaying AR advertisements, and can be distinguished by coordinates directly registered by the user or specific coordinates obtained through company API linkage.

[0355] The Service Dashboard (857) can perform advertising effectiveness measurement data visualization functions and commerce service data visualization functions.

[0356] The advertising effectiveness measurement data visualization function can provide the following charts: CPC: Total number of ad clicks by company / brand (viewable by period), CPC: Aggregated chart of total number of ad clicks (viewable by period), CPM: Total number of ad impressions by company / brand (viewable by period), CPM: Aggregated chart of total number of ad impressions (viewable by period), CPT&P: Chart of number of ad clicks by company / brand (viewable by period), CPT&P: Chart of number of ad impressions by company / brand (viewable by period).

[0357] These charts can be presented in a variety of formats, including Bar graph, Line graph, Pie Chart, Word graph, and Geospatial graph.

[0358] Although CPT&P is billed per hour rather than per click or impression, it can be used as data for measuring the effectiveness of exposure.

[0359] The commerce service data visualization feature can provide a chart of the cumulative payment amount by company (viewable by period) and a chart of the total cumulative payment amount (viewable by period).

[0360] Figure 8b is a conceptual diagram illustrating an MR service platform for providing the MR service of the present invention.

[0361] The present invention can provide an MR service platform capable of providing a mixed reality automotive meta service (MR AMS) (hereinafter, used interchangeably with MR service).

[0362] The MR service platform may be named as an MR service system, an MR navigation system, an MR platform, an MR system, etc.

[0363] An MR service platform refers to a platform capable of providing services based on mixed reality, and may include multiple independent components.

[0364] For example, the MR service platform may include an MR service device (900) (or referred to as an MR navigation engine) equipped on the vehicle, an MR AMS (hereinafter referred to as an MR AMS server) (1100) equipped on a server (or cloud) outside the vehicle (offboard), and a DTaaS (Digital Twin as a Service) server (1200).

[0365] The above MR service device (900) may include an MR AMS client (910) and an MR renderer (920).

[0366] The MR service described herein can be understood as a mixed reality navigation service for vehicles. That is, the MR service platform of the present invention can provide a vehicle interface implemented in mixed reality to a user riding in a vehicle.

[0367] The MR service provided by the MR service platform of the present invention can provide an experience of the digital world through a display inside the vehicle while driving in the real world.

[0368] Specifically, MR services can interactively provide drivers with navigation, safe driving, POI (Point of Interest), and entertainment user experiences in a virtual 3D space that combines real-world information with the digital world.

[0369] Through this, the MR service platform of the present invention can provide various UX (User Experience) that are free from the spatial and temporal constraints compared to existing camera-based (or HUD (Head up Display)-based) AR (Augmented Reality).

[0370] Here, the digital world refers to a digital twin or a digital twin smart city, and real-world information may include infrastructure data such as V2X (Vehicle to Everything) and C-ITS (Cooperative-Intelligent transport Systems) and / or surrounding perception data sensed by sensors equipped in autonomous vehicles.

[0371] In addition, the fusion described above may include the concept of fusion of vehicle and infrastructure sensor data as well as the MR service cloud (1000) (or MR AMS cloud) and MR service device (900) for implementing the present MR service platform.

[0372] Additionally, interactivity should be understood as a concept that includes not only mixed reality navigation, but also high-quality 3D image rendering and user interaction.

[0373] Meanwhile, mixed reality (MR) described in this specification may mean an environment in which virtual reality is grafted onto the real world, allowing real physical objects and virtual objects to interact.

[0374] Mixed reality (MR) can include augmented reality (AR), which adds virtual information based on reality, and augmented virtuality (AV), which adds real information to a virtual environment.

[0375] In other words, it provides a smart environment where the real and virtual are seamlessly connected, allowing users to experience richer experiences. For example, users can interact with a virtual pet in the palm of their hand, create a virtual game environment within their real-world room, play games, virtually rearrange furniture in their home, or create an environment where remote people can gather and work together.

[0376] A mixed reality vehicle meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention can help users prepare road shapes and actions in advance by displaying a preview of a future driving path.

[0377] In addition, a mixed reality automobile meta service (MR AMS) using mixed reality (MR) according to an embodiment of the present invention can improve advertising effectiveness or induce increased service usage by focusing on a specific POI (Point of Interest).

[0378] In addition, the mixed reality automobile meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention is not dependent on a specific map company and can also merge data from various map companies.

[0379] The present invention can provide an MR navigation function as one of the mixed reality automotive meta services.

[0380] In the MR navigation function, since it is implemented in the virtual world rather than overlapping the augmented reality object on the real world, it can solve the difficulties in achieving matching quality and the problem of forward occlusion that occurs in AR navigation.

[0381] Accordingly, the present invention can improve user UX by effectively expressing various contexts that were difficult to express in existing navigation through the MR navigation function.

[0382] To this end, the MR service platform of the present invention can provide a method for managing an MR context, and a method and system for obtaining metadata and 3D assets for providing an MR AMS.

[0383] The MR service platform of the present invention can render related service metadata and 3D assets to MR navigation by defining the MR context required in the digital world and modeling service metadata for obtaining the same.

[0384] Accordingly, the present invention provides a digital world experience through a display within a vehicle while driving in the real world, and can recommend and provide various additional HMI (Human Machine Interface) services to the user by utilizing the improved visibility and visibility of MR navigation.

[0385] Hereinafter, an MR service platform according to an embodiment of the present invention for providing the MR service described above will be described.

[0386] Referring to FIG. 8b, the MR service platform (or MR service system) of the present invention may include an MR service cloud (1000) provided outside the vehicle and an MR service device (900) provided in the vehicle.

[0387] The MR service cloud (1100) installed outside the vehicle (Offboard) may include at least one of an MR AMS (Mixed Reality Automotive Meta Service) server (1100) and a DTaaS (Digital Twin as a Service) server (1200).

[0388] The MR service device (900) equipped on the above vehicle may include an MR AMS client (910) and an MR renderer (920).

[0389] The above MR service device (900) can provide a navigation function (or navigation application) by interacting with the AR service device (800) and the navigation system (770).

[0390] Information required for the above navigation function can be received through user input (or user request) input through a camera (310), sensing unit (120), and user input unit (210) provided in the vehicle.

[0391] In addition, information required for the above navigation function can be received through an MR service cloud (1000) provided offboard the vehicle, and in order to receive the information required for each vehicle, information sensed or processed in the vehicle can be transmitted to the MR service cloud (1000).

[0392] The MR AMS (Mixed Reality Automotive Meta Service) server (1100) can be connected to various service providers (1300a, 1300b, 1300c) that provide online map services such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP, as illustrated in FIG. 8d. Then, based on the results of aggregating map data provided from the connected service providers (1300a, 1300b, 1300c), the server can aggregate shape information (e.g., building floor coordinates (footprint) information and height information) of each building included in the map and provide the aggregated information to the DTaaS server (1200). Here, the DTaaS server (1200) may refer to a server or device that provides a service using DTaaS, i.e., a digital twin map.

[0393] The above DTaaS may mean Digital Twin as a Service or Digital Transformation as a Service.

[0394] Meanwhile, the DTaaS server (1200) may be connected to a POI database in which POI service data for each building or each area included in the map data is stored. In addition, the DTaaS server (1200) may be connected to a 3D model database in which data of a 3D polygon model (or a 3D polygon map) for each building included in the map data is stored. Here, the 3D polygon model is a polygon model that can provide a building volume and may be a polygon model in which no texture is formed on the surface. The DTaaS server (1200) may receive service data related to POI from the connected POI database, and may receive data of 3D polygon models of each building included in the map data of an area from the connected 3D model database.

[0395] The processor (not shown) of the MR service device (900) can receive various information related to the driving of the vehicle from the object detection device (300), the sensing unit (120), and the navigation system (770). For example, the processor can receive information on an object detected in front, behind, or to the side of the vehicle from the camera (310) of the object detection device (300).

[0396] Additionally, the processor may receive information related to the vehicle's speed, driving direction, current vehicle location (GPS), etc. from a sensing unit (120) including sensors connected to each component of the vehicle, including the driving system (710). Additionally, the processor may receive information related to the vehicle's driving path from a navigation system (770).

[0397] Meanwhile, the MR service device (900) and the DTaaS server (1200) may be connected via the interface (Interface APIs) of the MR AMS. Here, the connection between the MR service device (900) and the interface (Interface APIs) of the MR AMS may be established via a wireless network connection. In this case, the MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the MR service device (900).

[0398] In this way, when connected to the MR AMS server (1100), the MR service device (900) can provide at least some of the information received from the connected components (e.g., the vehicle's camera (310), the sensing unit (120), or the user input input through the user input unit (210)) to the MR AMS server (1100) via a network connection. Then, the MR AMS server (1100) can provide 3D map data for providing mixed reality to the MR service device (900) in response to the provided information.

[0399] For example, the MR service device (900) can transmit information about objects detected around the vehicle, as well as information about the vehicle's speed, direction, and current location. In addition, it can provide information about the driving path to the MR AMS server (1100). Then, the MR AMS server (1100) can provide 3D map data of an area according to the current location of the vehicle to the MR service device (900) based on the information provided by the MR service device (900).

[0400] In this case, the MR AMS server (1100) may determine POI information based on the current vehicle location, vehicle speed, and vehicle driving path, and may provide 3D map data that further includes the determined POI information to the 3D building map. In addition, the MR AMS server (1100) may provide 3D map data that further includes information on the situation around the vehicle to the MR service device (900) based on the information on objects around the vehicle provided.

[0401] Meanwhile, the MR service device (900) can render a mixed reality image based on the 3D map data provided from the MR AMS server (1100). For example, the MR service device (900) can control the MR renderer (920) to display a 3D map screen including models of buildings around the vehicle based on the provided 3D map data. In addition, the MR renderer (920) can display a graphic object corresponding to the vehicle on the 3D map screen, and display graphic objects corresponding to the provided POI data and situation information around the vehicle on the 3D map screen.

[0402] Accordingly, an image of a virtual environment (in other words, a mixed reality (MR) image, or an MR navigation screen, or an MR navigation interface) including a three-dimensional building model similar to the shape of the current vehicle and buildings around the vehicle and graphic objects corresponding to the vehicle can be output to a display (251) equipped in the vehicle, for example, a Center Information Display (CID), a Head Up Display (HUD), a Rear Sheet Information (RSI), or a Rear Sheet Entertainment (RSE).

[0403] In this case, information related to the vehicle's driving and the surroundings of the vehicle can be provided to the driver through the virtual environment. Through the 3D map data, i.e., the digital twin map (hereinafter referred to as the "DT map"), the MR service device (900) according to an embodiment of the present invention can provide the driver with a mixed reality service.

[0404] Meanwhile, the MR AMS server (1100) can determine 3D map data and POI information or situation information around each vehicle that can be provided together with the 3D map data based on information collected from not only the MR service device (900) equipped in one vehicle but also the MR service devices (900) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect from multiple vehicles in the form of a cloud server and generate 3D map data for mixed reality based on the collected information. In addition, it can be implemented to provide mixed reality service to at least one MR service device (900) equipped in different vehicles based on the generated 3D map data.

[0405] Therefore, for convenience of explanation, in the following, a cloud or server that includes the MR AMS server (1100) and the DTaaS server (1200), and provides information such as metadata (e.g., service metadata, 3D assets), 3D polygon maps, and digital twin maps for providing mixed reality services, i.e., a digital twin map (DT map), is referred to as an MR service cloud (1000).

[0406] As illustrated in FIG. 8b, the MR service device (900) (or MR navigation engine) may include an MR AMS client (910) and an MR renderer (920).

[0407] In addition, in order to implement the MR navigation function, which is one of the MR services, the MR service device (900) can transmit and receive data with the AR service device (800) (or AR engine) and the navigation system (770) equipped in the vehicle.

[0408] The MR AMS client (910) may include a context manager (911), a scene manager (913), and a UX scenario database (914).

[0409] Additionally, the MR renderer (920) may include a DTaaS client (921), an MR visualization unit (922), and a 3D HMI framework (923).

[0410] The MR AMS client (910) can collect vehicle location information, user input, user feedback information, payment information, etc. and transmit them to the MR AMS server (1100) located outside the vehicle.

[0411] The MR AMS server (1100) can transmit at least one of metadata, service metadata, and 3D assets required to provide MR service to the MR AMS client (910) based on information received from the MR AMS client.

[0412] The MR AMS client (910) can transmit data received from the MR AMS server (910) to the MR renderer (920).

[0413] The MR renderer (920) can create a digital twin map using a 3D polygon map received from a DTaaS server (1200) and an image received from an MR AMS client (910) or a camera (310) installed in a vehicle.

[0414] Additionally, the MR renderer (920) can render data received from the MR AMS client (920) into an MR object that can be overlapped on a digital twin map, and can create a mixed reality (MR) image by overlapping the rendered MR object on the digital twin map.

[0415] Thereafter, the MR renderer (920) can output the generated mixed reality image to a display (251) equipped in the vehicle.

[0416] All components described in this specification can be implemented as separate hardware modules, and can be understood as components implemented in software block units as needed.

[0417] Below, each component that constitutes the MR service platform will be described in more detail with reference to the attached drawings.

[0418] Figure 8c is a conceptual diagram for explaining the MR AMS client of the present invention.

[0419] The MR AMS client (910) is equipped in a vehicle and can provide a mixed reality automotive meta service (MR AMS).

[0420] The MR AMS client (910) may include a context manager (911) that requests a context corresponding to a user request (or user input) from an MR AMS server (1100) installed outside the vehicle, a scene manager (913) that manages MR scene information provided to a display (251) installed in the vehicle, and a UX scenario database (914) that provides a UX rule to at least one of the context manager (911) and the scene manager (913).

[0421] Additionally, the MR AMS client (910) may further include an interface API (912) that calls a function for communicating with the MR AMS server (1100) provided outside the vehicle.

[0422] The above interface API (912) may be configured with one or more functions configured to perform communication with the MR AMS server (1100), and may use these functions to convert a data format or message format to transmit data to the MR AMS server (1100), or to convert the format of data received from the MR AMS server (1100).

[0423] The interface API (921) can transmit a context request output from the context manager (911) to the MR AMS server (1100) and receive a 3D asset corresponding to the requested context from the MR AMS server (912).

[0424] Here, the context may refer to situational information and information corresponding to the situation the vehicle is in. Furthermore, the context may also include the meaning of content.

[0425] The above 3D asset may refer to 3D object data corresponding to the requested context. Furthermore, the 3D asset may refer to a 3D graphic object that overlaps or can be updated on a digital twin image (or digital twin map).

[0426] The MR AMS client (910) may be included in the MR service device (900).

[0427] The MR service device (900) may include a user interaction handler (901) that, when a user input is received through an input unit (210) provided in the vehicle, generates an action corresponding to the user input and transmits the action to the context manager.

[0428] The above user interaction handler (901) may be included in the MR service device (900) or may be included in the MR AMS client (910).

[0429] For example, when a user input of “Find nearby Starbucks” is received through the vehicle input unit (210), the user interaction handler (901) can generate an action corresponding to the user input (e.g., “Search POI”) and transmit it to the context manager (911) provided in the MR AMS client (910).

[0430] For example, the action may be determined by a motion matching a word included in the user input, and the action may be named a command or control command.

[0431] The above context manager (911) can generate a command to request a context corresponding to an action received from a user interaction handler (901) and transmit the command to the MR AMS server (1100) via an interface API (912).

[0432] The above command may be generated based on an action (e.g., “Search POI”) received from the user interaction handler (901), and may be formed to include, for example, the current vehicle location, the type of POI to be found, and radius information (e.g., GET “Starbucks” (type of POI) WITHIN “500m” (radius) FROM “37.7795, -122.4201” (current vehicle location (latitude, longitude)).

[0433] The context manager (911) can receive current scene information (current scene) currently being output from the vehicle from the scene manager (913) and receive UX rules from the UX scenario database (914).

[0434] Additionally, the context manager (911) can receive navigation information including the current route and current location from the navigation handler (902) that handles information of the navigation system (770).

[0435] The navigation handler (902) may be provided in the MR service device (900) or may be provided in the MR AMS client (910).

[0436] The context manager (911) can generate a command to request the context based on at least one of the current scene information, the UX rule, and the navigation information.

[0437] The current scene information may include screen information currently being output on the vehicle's display (251). For example, the current scene information may include information about a mixed reality image in which an MR object and an MR interface overlap on a digital twin map.

[0438] Additionally, at least one of the context manager (911) and the scene manager (913) of the present invention can receive sensor data processed through a sensor data adapter (903) that processes information sensed through the vehicle's sensing unit (120).

[0439] The above sensor data adapter (903) may be provided in the MR service device (900) or may be provided in the MR AMS client (910). The sensor data adapter (903) may transmit processed sensor data to an AR engine handler (904) that handles data transmitted to the AR engine (or AR service device) (800).

[0440] The interface API (912) can receive metadata of a context corresponding to the command and / or a 3D asset corresponding to the context from the MR AMS server (1100).

[0441] Thereafter, the interface API (912) can transmit the received metadata and / or 3D assets to the scene manager (913).

[0442] The scene manager (913) can generate UI data using the UX rules received from the UX scenario database (914) and the metadata and 3D assets received from the interface API (912).

[0443] Thereafter, the scene manager (913) can transmit the generated UI data to the MR renderer (920) that renders the generated UI data to be output as mixed reality (MR) or a mixed reality image on the display (251) provided in the vehicle.

[0444] Additionally, the scene manager (913) can further transmit the generated UI data to an AR engine handler (904) configured to handle an AR service device (800) equipped in the vehicle.

[0445] The UX rule stored in the UX scenario database (914) may refer to information about the rules, forms, formats or templates for creating a screen, UX or user interface provided by the MR service device, and these UX rules may be predefined for each type of data.

[0446] Additionally, UX rules can be updated or modified by users or administrators.

[0447] Figure 8d is a conceptual diagram for explaining the MR AMS server of the present invention.

[0448] Referring to FIG. 8d, an MR AMS server (1100) provided offboard in a vehicle and providing a Mixed Reality Automotive Meta Service (MR AMS) may include an interface API (1101) for calling a function for communicating with an MR AMS client provided in the vehicle, a service aggregation manager (1110) for requesting and receiving a context corresponding to a request received from the MR AMS client from a service provider, and a data integration manager (1120) for loading a 3D asset corresponding to the received context from a database (3D Assets for MR Navigation Database) (1130).

[0449] The above interface API (1101) may be named a server interface API (1101) to distinguish it from the interface API (912) of the MR AMS client (910) equipped in the vehicle.

[0450] Additionally, the interface API (912) of the MR AMS client (910) may be named a vehicle interface API or an MR AMS client interface API.

[0451] The interface API (1101) provided in the MR AMS server (1100) can transmit a user request (or context request) received from the MR AMS client to the service aggregation manager (1110).

[0452] The above interface API may include a first interface API (1101) that calls a function for performing communication with the MR AMS client (910) and a second interface API (1102a, 1102b, 1102c) that calls a function for performing communication with the service aggregation manager (1110) and the service provider (1300a, 1300b, 1300c).

[0453] The above second interface API (1102a, 1102b, 1102c) can receive service data and / or map data through the interface API provided to the service provider (1300a, 1300b, 1300c).

[0454] The interface APIs provided in the second interface API (1102a, 1102b, 1102c) and the service provider (1300a, 1300b, 1300c) may include functions configured to perform mutual data transmission and reception and to convert data formats or message formats, and these functions may be used to convert data formats or message formats to enable mutual data transmission and reception.

[0455] The service aggregation manager (1110) can request the requested context from different service providers based on the type of context requested from the MR AMS client (910) equipped in the vehicle.

[0456] Specifically, the service aggregation manager (1110) may, if the type of the requested context is a first type of context, request the first type of context from a first service provider (1300a) that provides the first type of context, and if the type of the requested context is a second type of context different from the first type, request the second type of context from a second service provider (1300b) that provides the second type of context.

[0457] For example, if the type of requested context is about a POI (e.g., “Starbucks”), the service aggregation manager (1110) can request and receive context (or POI data) about the POI from the first service provider (1300a) that provides information about the POI.

[0458] Additionally, if the type of the requested context is a view of a street, the service aggregation manager (1110) can request and receive context (or imagery data) for the view of a street from a second service provider (1300b) that provides information about the view of the street.

[0459] Additionally, if the type of the requested context is a service, the service aggregation manager (1110) may request and receive context related to the service (or data about the service (e.g., service ratings or prices)) from a third-party service provider (1300c) that provides information related to the service.

[0460] Additionally, the interface API (1101) can request a confirmed service API (Expand service API calls) to the service aggregation manager (1110) based on a service (or context request) requested from the MR AMS client (910).

[0461] The service aggregation manager (1110) can request and receive information corresponding to a confirmed service from a service provider (1300a, 1300b, 1300c) based on the extended service API request, and use the information to create a service API and output it to the data integration manager (1120).

[0462] The above data integration manager (1120) can perform data enhancement based on the service API received from the service aggregation manager (1110), generate a metadata package for the requested context, and transmit it to the vehicle's MR AMS client (910) through the interface API (1101).

[0463] The above metadata package may include the 3D asset and service metadata described above. Here, the service metadata may refer to metadata for providing a service corresponding to the requested context.

[0464] Meanwhile, the interface API (1101) can transmit a 3D asset loaded from the data integration manager (1120) to the MR AMS client (910).

[0465] Meanwhile, the MR AMS server (1100) of the present invention may further include the context manager (911) described above.

[0466] That is, the context manager (911) may be included in the MR AMS client (910) and installed in the vehicle unit, may be included in the MR AMS server (1100) and installed in the server (cloud) unit, or may be installed in both units.

[0467] The above context manager (911), when equipped in the MR AMS server (1100), can be configured to manage a context corresponding to a request received from the MR AMS client (910).

[0468] The above context manager (911) may include a context handler (911a) that handles and parses a context request, a context interpreter (911b) that manages a session for interpreting a context request and creates a context set using a data model, and a context graph database (Context Graph DB or MR Context DB) (911c) that stores the data model.

[0469] Here, the context handler (911a) can receive a user request input to the MR AMS client through the interface API (1101), parse the received user request, and transmit it to the context interpreter (911b).

[0470] The above context interpreter (911b) can create a query for a context request corresponding to the user request after creating a session, and request and receive a context data model corresponding to the query from the context graph database (911c).

[0471] The context interpreter (911b) requests a context corresponding to the context data model from the service aggregation manager (1110), and the service aggregation manager (1110) can request and receive context data corresponding to the context data model from the service provider (1300a, 1300b, 1300c).

[0472] The above service aggregation manager (1110) can request and receive a 3D asset (and / or service metadata) corresponding to the requested context from the data integration manager (1120), and transmit the context data received from the service provider and the 3D asset (and / or service metadata) received from the data integration manager to the context interpreter (911b).

[0473] The context interpreter (911b) can transmit the received context data and the 3D asset to the MR AMS client (910) equipped in the vehicle through the context handler (911a) and the interface API (1101).

[0474] Meanwhile, the context manager (911) may further include a context recorder (911d) that extracts a recommended context based on the above-mentioned generated context set, and a context controller (911e) (or context tracker) that manages contexts that must be periodically acquired.

[0475] The above context recorder (911d) may request the context interpreter (911b) to generate a query to recommend a service that can replace the specific service when the completed context data contains information that a specific service is unavailable.

[0476] Figure 9 is a conceptual diagram for explaining the DTaaS server of the present invention.

[0477] Referring to FIG. 9, the DTaaS (Digital Twin as a Service, or Digital Transformation as a Service) server (1200) of the present invention is installed outside a vehicle and can provide a mixed reality automotive meta service (MR AMS). Specifically, the DTaaS server (1200) can provide a digital twin map or data necessary for creating a digital twin map (e.g., all types of information regarding a 3D polygon map or an object overlapping on a digital twin).

[0478] The DTaaS server (1200) may include a DTaaS API (1210) that calls a function for communicating with an MR service device (900) equipped in a vehicle, a database (Digital Twins Maps DB) (1220) that stores a digital twin map and a renderable three-dimensional polygon map provided to the MR service device, and a processor (1280) that transmits a three-dimensional polygon map corresponding to the location information of the vehicle received from the MR service device to the MR service device through the DTaaS API.

[0479] In addition, the DTaaS server (1200) may further include a communication unit (TeleCommunicatino Unit, TCU) (1290) that performs communication with the MR AMS server (1100) that is installed outside the vehicle and provides MR AMS service.

[0480] In addition, the DTaaS server (1200) may further include a digital twin map generation unit (Digital Twin Representation and Update Unit) (1230) that creates a digital twin map by matching an actual captured image to a 3D polygon map stored in the database (1220).

[0481] In addition, the DTaaS server (1200) may further include a dynamic model database (Dynamics Modeling DB) (1240) that stores dynamic information about moving objects received from at least one of the MR service device (900) and the MR AMS server (1100) and a scenario database (Scenarios DB) (1250) that stores information related to scenarios that can be implemented in a digital twin.

[0482] In addition, the DTaaS server (1200) may further include a simulation unit (1260) that performs a simulation corresponding to a user request on the digital twin and a visualization unit (1270) that visualizes information to be implemented on the digital twin.

[0483] All of the components described above can be implemented as independent hardware (e.g., chips or modules), and can also be implemented as software-blocked components as needed.

[0484] The above DTaaS server (1200) can transmit and receive data through the DTaaS API (1210) not only with the vehicle (100), but also with a server (FMS Server) (1280) that provides a fleet management service (or vehicle fleet management service) and a server (1290) that provides a city planning service.

[0485] For example, the DTaaS server (1200) can collect log information collected from each server from at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290).

[0486] Afterwards, the DTaaS server (1200) can store the collected log information in a log database.

[0487] The DTaaS server (1200) can provide a digital twin map for visualization in at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290) based on the collected log information.

[0488] Additionally, the DTaaS server (1200) may transmit at least one of event notification information, simulation information, and visualization information to at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290) based on the received log information.

[0489] FIG. 10 is a block diagram illustrating the structure of a 3D map view generation device (1300) connected to a cloud server (1350) according to an embodiment of the present invention.

[0490] Referring to FIG. 10, the MR AMS (Mixed Reality Automotive Meta Service) server (1100) described in FIG. 8d can be connected to various service providers (1351) that provide online map services, such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP. Furthermore, based on the results of aggregating map data provided from the connected service providers (1351), shape information of each building included in the map, for example, floor coordinate (footprint) information and height information of a building, can be aggregated and the aggregated information can be provided to DTaaS (1352). Here, DTaaS (1352) may refer to a DTaaS server (1200), i.e., a server or device that provides a service using a digital twin map.

[0491] Meanwhile, DTaaS (1352) may be connected to a POI database in which POI service data for each building or each area included in the map data is stored. In addition, it may be connected to a 3D model database in which data of a 2.5D polygon model for each building included in the map data is stored. Here, the 2.5D polygon model is a polygon model that can provide a building volume and may be a polygon model without a texture formed on the surface. The DTaaS (1352) may receive service data related to POI from the connected POI database, and may receive data of 2.5D polygon models of each building included in the map data of an area from the connected 3D model database.

[0492] The processor (1330) of the 3D map view generation device (1300) can receive various information related to the driving of the vehicle from the camera (310), the sensing unit (120), and the navigation system (770). For example, the processor (1330) can receive information on objects detected in front, behind, or to the side of the vehicle from the camera (310). In addition, the processor (1330) can receive information related to the speed, driving direction, and current location (GPS) of the vehicle from the sensing unit (120) including sensors connected to each component of the vehicle including the driving system (710). In addition, the processor (1330) can receive information related to the driving path of the vehicle from the navigation system (770).

[0493] Meanwhile, the 3D map view creation device (1300) and DTaaS (1352) may be connected through the interface (Interface APIs) of the MR AMS server (1100). In this case, the 3D map view creation device (1300) may be configured to correspond to the MR AMS client (910).

[0494] Here, the interface (Interface APIs) of the 3D map view creation device (1300) and the MR AMS server (1100) can be connected via a wireless network connection. In this case, the MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the 3D map view creation device (1300).

[0495] In this way, when connected to the MR AMS server (1100), the 3D map view generation device (1300) can provide at least some of the information received from the connected components to the MR AMS server (1100) via a network connection. Then, the MR AMS server (1100) can provide 3D map data for providing mixed reality to the 3D map view generation device (1300) in response to the provided information.

[0496] The 3D map data provided here is map data including a polygon model provided in the 3D model DB, and may include 2.5D map data providing a simple polygon model or 3D map data providing a three-dimensional virtual object.

[0497] For example, the 3D map view generation device (1300) can provide information on objects detected around the vehicle, information on the speed and direction of the vehicle, information on the current location of the vehicle, and information on the driving path of the vehicle to the MR AMS server (1100). Then, the MR AMS server (1100) can provide 2.5D map data of an area according to the current location of the vehicle to the 3D map view generation device (1300) based on the information provided from the 3D map view generation device (1300).

[0498] In this case, the MR AMS server (1100) may determine POI information based on the current vehicle location, direction, speed, and driving path, and may provide 3D map data that further includes the determined POI information to the 3D building map. In addition, the MR AMS server (1100) may provide 3D map data that further includes information on the situation around the vehicle to the 3D map view generation device (1300) based on the information on objects around the vehicle provided.

[0499] Meanwhile, the 3D map view generation device (1300) can render a mixed reality image based on the 3D map data provided from the MR AMS server (1100). For example, the 3D map view generation device (1300) can control an MR renderer to display a 3D map screen including models of buildings around the vehicle based on the provided 3D map data. In addition, the device can display a graphic object corresponding to the vehicle on the 3D map screen, and display graphic objects corresponding to the provided POI data and situation information around the vehicle on the 3D map screen.

[0500] Accordingly, an image of a virtual environment including a three-dimensional building model similar to the shape of the current vehicle and the buildings around the vehicle and graphic objects corresponding to the vehicle can be output to a display unit (251) such as a CID (Center Information Display), HUD (Head Up Display), RSI (Rear Sheet Information), or RSE (Rear Sheet Entertainment).

[0501] In this case, information related to the vehicle's driving and the surroundings of the vehicle can be provided to the driver through the virtual environment. Through the 3D map data, i.e., the digital twin map (hereinafter referred to as the "DT map"), the 3D map view generation device (1300) according to an embodiment of the present invention can provide the driver with a mixed reality service.

[0502] Meanwhile, the MR AMS server (1100) can determine 3D map data and POI information or situation information around each vehicle that can be provided together with the 3D map data based on information collected from not only a 3D map view generation device (1300) equipped in one vehicle but also 3D map view generation devices (1300) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect information from multiple vehicles in the form of a cloud server and generate 3D map data for mixed reality based on the collected information. In addition, it can be implemented to transmit MR information for providing a mixed reality service to at least one 3D map view generation device (1300) equipped in different vehicles based on the generated 3D map data.

[0503] For convenience of explanation, in the following, the MR AMS server (1100) that provides 3D map data, i.e., a digital twin map (DT map), for providing a mixed reality service by being connected to the DTaaS (1352) and the DTaaS (1352) are collectively referred to as a cloud server (1350).

[0504] Meanwhile, the 3D map view generation device (1300) may be a device that controls a display equipped in a vehicle through an interface unit. Alternatively, the 3D map view generation device (1300) may of course be a display device equipped in the vehicle. In this case, the 3D map view generation device (1300), that is, the 3D map view generation device (1300), may be equipped with a display unit, and may receive the updated DT map on which the real-world texturing has been performed from a cloud server, and directly display an MR view image including the received DT map on the display unit equipped in the 3D map view generation device (1300).

[0505] In the following description, for convenience of explanation, it is referred to as a 3D map view generation device (1300).

[0506] Meanwhile, a vehicle (100) related to the present invention may include a 3D map view generation device (1300).

[0507] The 3D map view generation device (1300) is capable of controlling at least one of the components described in FIG. 7. From this perspective, the 3D map view generation device (1300) may be a control unit (170).

[0508] Without being limited thereto, the 3D map view generation device (1300) may be a separate configuration independent from the control unit (170). When the 3D map view generation device (1300) is implemented as a component independent from the control unit (170), the 3D map view generation device (1300) may be provided in a part of the vehicle (100).

[0509] Hereinafter, for convenience of explanation, the 3D map view generation device (1300) will be described as a separate component independent from the control unit (170). The functions (operations) and control methods described for the 3D map view generation device (1300) in this specification can be performed by the control unit (170) of the vehicle. That is, all contents described in relation to the 3D map view generation device (1300) can be analogically applied to the control unit (170) in the same / similar manner.

[0510] In addition, the 3D map view generation device (1300) described in this specification may include some of the components described in FIG. 7 and various components provided in the vehicle. In this specification, for convenience of explanation, the components described in FIG. 7 and various components provided in the vehicle will be described by assigning separate names and drawing numbers.

[0511] Figure 11 is a conceptual diagram for explaining the above 3D map view generation device (1300).

[0512] A 3D map view generation device (1300) according to one embodiment of the present invention may include a communication unit (1310), an interface unit (1320), a memory (1340), and a processor (1330).

[0513] The above communication unit (1310) may be configured to perform wireless communication with at least one of the electrical components equipped in the vehicle (for example, the electrical components equipped in the vehicle as shown in FIG. 7).

[0514] In addition, the communication unit (1310) may be configured to perform communication with devices other than the vehicle, such as a mobile terminal, a server, another vehicle, infrastructure installed on the road, etc.

[0515] The above communication unit (1310) may be the communication device (400) described above, and may include at least one of the components included in the communication device (400).

[0516] The interface unit (1320) can communicate with at least one of the components provided in the vehicle.

[0517] Specifically, the interface unit (1320) can be configured to perform wired communication with at least one of the components provided in the vehicle as shown in FIG. 7.

[0518] The above interface unit (1320) receives sensing information from one or more sensors provided in the vehicle (100).

[0519] The above interface unit (1320) may be referred to as a sensor data collector.

[0520] The above interface unit (1320) collects (receives) information sensed through sensors provided in the vehicle (e.g., sensors for detecting vehicle operation (V.Sensors) (e.g., heading, throttle, break, wheel, etc.) and sensors for sensing vehicle surrounding information (S.Sensors) (e.g., Camera, Radar, LiDAR, Sonar, etc.)).

[0521] The above interface unit (1320) can transmit information sensed through a sensor equipped in the vehicle to the communication unit (1310) (or processor (1330)) so that it is reflected in a high-precision map.

[0522] The interface unit (1320) may, for example, serve as a passageway with electrical components installed in the vehicle through the vehicle interface unit (130).

[0523] The interface unit (1320) can exchange data with the vehicle interface unit (130).

[0524] The interface unit (1320) can be connected to a vehicle and serve as a passage for supplying electric energy.

[0525] For example, the route guidance device can be powered on by receiving electric energy from the vehicle's power supply unit (190) through the interface unit (1320).

[0526] Meanwhile, the present invention may include a memory (1340) that stores data supporting various functions of the 3D map view creation device (1300). For example, the memory (1340) may store a plurality of application programs (or applications) that can be executed by the processor (1330), data for the operation of the 3D map view creation device (1300), and commands.

[0527] As an example of such data, the memory (1340) may store building reference information, which is information on actual buildings corresponding to each of the building virtual objects included in the map data. The building reference information may store information on the height of the actual building, planar shape information (e.g., footprint information) which is the shape of the land occupied by the actual building, and information on the actual location of each building, i.e., the latitude and longitude of each building. In addition, the building reference information may include information on the shape of each side of each building or various three-dimensional shapes of the building. In addition, the building reference information may include information related to the use, type, and age of each building corresponding to each building virtual object.

[0528] The above memory (1340) can store building reference information corresponding to at least one of the building virtual objects included in the map data. Hereinafter, an area of ​​the memory (1340) where the building reference information is stored will be referred to as a building reference information storage unit (1343).

[0529] In addition, the memory (1340) may store building image setting values ​​including setting values ​​for angles and projection directions for generating building virtual objects, modeling setting values ​​for 2.5D building virtual objects formed of polygons, or modeling setting values ​​for 3D building virtual objects having a three-dimensional shape, as information for modeling the building virtual objects. In this way, based on the building reference information and building image setting values ​​stored in the memory (1340), the processor (1330) may calculate various building setting values ​​such as height, area, size, area of ​​each side, margin value, and offset value of the building virtual object to be generated, and may generate each building virtual object based on the calculated building setting values. Here, the building setting value may include a value related to at least one of the height, location, planar shape, three-dimensional shape, and shape of each side of the building included in the building reference information.

[0530] In addition, the memory (1340) can further store data for generating surface materials, i.e., surface textures, of each building virtual object. For example, the memory (1340) can store images of a portion of a building including shapes of various building elements. The images are images for forming the surface texture of the building (hereinafter, texture images) and can be formed as 2D images. In addition, the memory (1340) can store data of images (hereinafter, normal maps) corresponding to normal maps having different vector values ​​according to different materials so as to provide texture to the texture images. In addition, images (hereinafter, albedo maps) corresponding to albedo maps having material-specific color values ​​for each different material can be stored.

[0531] Meanwhile, the memory (1340) may store 2D images including shapes corresponding to building components. The 2D image may be an image including images corresponding to at least one building element for describing the texture of the building. In addition, each 2D image may include different building elements depending on the type, and the shapes of the included building elements may also be different. The 2D image may be a texture image that can form a surface texture (material) of the building virtual object in a form that is tiled on the surface of the building virtual object.

[0532] Meanwhile, the texture image may be synthesized with the outer surface of the building virtual object in a tiled form as described above. Here, the tiling may refer to a process of filling the surface area of ​​the building virtual object with tiles normalized to a certain size in a horizontal and vertical direction without overlapping each other. Accordingly, the texture image may refer to a tile, which is a unit image synthesized on the outer surface of the building virtual object through the tiling. Therefore, in the following description, a texture image texturing the outer surface of the building virtual object in a tiling manner may also be referred to as a tile. Here, the texturing may refer to an operation process of adding a texture representing a surface texture (material) to the surface area of ​​the building virtual object and synthesizing a texture on the surface area of ​​the building virtual object using a tile.

[0533] Additionally, the surface area of ​​the virtual object may be composed of a plurality of different sub-areas. In this case, each of the plurality of sub-areas may be matched with a different tile, and each of the plurality of areas may be tiled using different matching tiles.

[0534] Meanwhile, the above tiles may be grouped into different groups according to the characteristics of the building to which the tiles can be applied, such as the type, purpose, or size of the building. For example, the tiles may be grouped into residential buildings, commercial buildings, or officetel buildings. In this case, the tiles grouped into different groups, i.e., the tiles grouped into the residential building group, the tiles grouped into the commercial building group, and the tiles grouped into the officetel building group, may be tiles normalized to different sizes according to each group.

[0535] In addition, some of the tiles may be associated with at least one other tile. For example, the associated tiles may be tiles that are similar in shape and color. That is, the associated tiles may be tiles that differ only in at least a portion of their shape or in color. Alternatively, the associated tiles may be tiles that are similar to a certain degree in at least one of their shape and color. These associated tiles may reflect changes over time. For example, tiles that have the same shape but differ only in color may be used to reflect changes over time in a portion of a specific building.

[0536] Meanwhile, memory (1340) may contain association information regarding tiles that are interconnected in this manner. Furthermore, through association information, a specific tile can be associated with another tile. Therefore, by utilizing this association information, even tiles with different shapes or colors can be associated with each other.

[0537] The tiles stored in the above memory (1340) may form a database including each tile and tile information corresponding to each tile. In this case, the tile information stored in the database may include address information of each tile, information on a group to which each tile is included (group information), and, if there is another tile associated with the corresponding tile, association information including information on the other associated tile. Hereinafter, a database including tiles and tile information corresponding to each tile will be referred to as a tile database (DB) (hereinafter referred to as a tile DB) (1341).

[0538] Meanwhile, the memory (1340) may store map data including virtual objects. Here, the map data may be formed by connecting map data from multiple different regions in tile form. In this case, each map tile may represent a logical rectangular section corresponding to a certain area, and may be used to subdivide a wide map area. In other words, the map data may be formed by data from map tiles each corresponding to a different region.

[0539] Meanwhile, the above map data may be map data including data of a so-called 2.5D polygon model. In addition, the map data may include high-definition (HD) map data having high accuracy. The map data may be linked to a navigation system (770) and a driving system (700) provided in a vehicle (100), and the memory (1340) may provide the map data to various linked systems and components under the control of the processor (1330). Such map data may be stored in a map data storage unit (1342).

[0540] Meanwhile, a 3D map view generation device (1300) according to an embodiment of the present invention may include a processor (1330) that generates a 3D map as a digital twin using at least one of an image captured by a camera (310) provided in a vehicle (100), 2D map data (e.g., HD map data), and 3D map data (e.g., map data including data of a 2.5D polygon model). The processor (1330) controls each connected component and may control the overall operation of the 3D map view generation device (1300).

[0541] Additionally, the processor (1330) can overlap (or superimpose, output) graphic objects related to route guidance on a digital twinned 3D map.

[0542] Here, the graphic object related to route guidance refers to an object output as mixed reality (MR), and may include various types of objects (e.g., POI objects, carpet-shaped objects, 3D objects, etc.) required to perform route guidance. Here, the graphic object related to the route guidance may also be referred to as an MR object.

[0543] When the destination of the vehicle (100) is set, the processor (1330) can determine a route for the vehicle (100) to drive through the navigation system (770). And, when the driving route of the vehicle (100) is determined, the processor (1330) can determine at least one virtual object (e.g., an object output as the mixed reality (MR)) to perform texturing on a digital twin 3D map for route guidance.

[0544] Meanwhile, in order to determine virtual objects to be displayed on the digital twinned 3D map, the processor (1330) may first determine a map area in which to generate the virtual objects, i.e., to model. To this end, the processor (1330) may first detect, from map data stored in the memory (1340), at least one map tile around a map tile corresponding to an area in which the vehicle (100) is located, according to the current location of the vehicle (100) and the driving direction of the vehicle (100). For example, the processor (1330) may determine an area corresponding to 5 horizontal and 5 vertical maps centered on the map tile in which the vehicle (100) is located, i.e., a total of 25 map tile data, as a map area, according to a zoom level of a specified map, and may detect map tile data corresponding to the determined map area from the map data.

[0545] And the processor (1330) can detect buildings included in the detected map area. And it can detect building reference information corresponding to the detected buildings from the building reference information storage unit (1343). And the setting values ​​of each building image related to the detected building reference information can be determined. Here, the building image setting values ​​are information for modeling each building virtual object, and may include a setting value of a projection direction according to an angle or distance for generating the building virtual objects according to the driving direction of the vehicle (100) and / or the angle at which the MR view is displayed, a setting value of modeling according to a 2.5D virtual object or a setting value of modeling according to a 3D virtual object, for example, a vector value such as a coordinate value of a vertex or a normal line.

[0546] Once the building image setting values ​​are determined, the processor (1330) can calculate building setting values ​​corresponding to each building included in the map area based on the determined building image setting values ​​and building reference information of each building. Here, the building setting values ​​can include the height of the building virtual object to be created, the side area of ​​each side, the plane area, etc. Then, once the calculation of the building setting values ​​is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included in the map area based on the building setting values.

[0547] And when the modeling of virtual objects included in the above map area is completed, the processor (1330) can determine virtual objects to be displayed on the display unit (251) through rendering among the modeled virtual objects. To this end, the processor (1330) can determine a visible area of ​​a frustum shape formed according to the driving direction of the vehicle (100) based on the current position of the vehicle (100), determine building virtual objects included in the visible area of ​​the frustum shape as objects to be rendered, and perform texturing on the determined building virtual objects.

[0548] Meanwhile, the processor (1330) may pre-model building virtual objects corresponding to each building included in the map area, and render and display only at least some of the modeled building virtual objects on the display unit according to the visible area of ​​the frustum shape determined according to the position and driving direction of the vehicle (100). In this case, the texturing may be performed during the rendering process. Therefore, by pre-modeling objects to be rendered and selectively rendering only some of the modeled objects as needed, texturing of virtual objects that are not displayed on the display unit can be prevented. Therefore, the amount of computation required for the texturing can be reduced, and the time for virtual objects to be displayed on the display unit can be significantly reduced.

[0549] Moreover, the processor (1330) may be implemented as a plurality of processors. For example, the processor (1330) may be implemented as a processor that models the building virtual objects and a processor that renders some of the modeled virtual objects as different processors. For example, the processor that models the virtual objects may be a central processing unit (Central Processing Unit) that interprets various program commands of the 3D map view generation device (1300) according to an embodiment of the present invention and performs data calculations and processing. On the other hand, the processor that renders some of the pre-modeled virtual objects may be a GPU (Graphics Processing Unit) that processes graphic calculations in the 3D map view generation device (1300) according to an embodiment of the present invention and outputs result values. In this case, since the modeling of the virtual object and the rendering of the modeled virtual object can be processed in parallel on different processors, the computational load on the processor can be significantly reduced compared to the case where all processes are performed on a single processor, and by displaying the virtual object by performing rendering from objects that have been modeled in advance, the time required until the virtual object is displayed on the display unit can also be significantly shortened. Furthermore, the 3D map view generation device (1300) according to the embodiment of the present invention described above generates modeling of building virtual objects corresponding to each building included in the map area in advance, and renders only some virtual objects according to some areas (e.g., the visible area of ​​the frustum shape) among the generated building virtual objects, thereby further shortening the time for displaying the building virtual objects, and thus real-time provision of an MR view according to the driving of the vehicle (100) can be made possible.

[0550] In this way, since the present invention has a configuration that pre-generates modeling of building virtual objects corresponding to each building included in the map area, when the map area needs to be updated according to the location and driving direction of the vehicle (100), the processor (1330) can additionally detect at least one map tile data from the map data through the update. In addition, the virtual objects of the buildings included in the area of ​​the additionally detected map tile can be further modeled. On the other hand, the processor (1330) can delete virtual objects of the buildings included in the area of ​​the map tile that has deviated from the map area according to the update of the map area for efficient memory utilization.

[0551] Meanwhile, when the destination of the vehicle (100) or the route to the destination is determined, the processor (1330) may additionally detect at least one map tile data from the map data based on the determined destination or the route to the destination. In this case, the additionally detected at least one map tile data may be map tile data that are connected to each other. In addition, when the additionally detected at least one map tile data is additionally detected, the processor (1330) may model virtual objects corresponding to each of the buildings included in the additionally detected map tile data.

[0552] In this case, virtual objects modeled from the additionally detected map tile data may be rendered and displayed on the display unit (251) regardless of the driving information of the vehicle (100) when a preset condition is met. For example, the preset condition may be met when a user requests a search for a route to a destination or a check of the surroundings of the destination. In this case, regardless of the current driving status of the vehicle (100), building virtual objects modeled from the additionally detected map tile data may be rendered and displayed along the shortest path to the set destination or the searched driving path to the destination. In this case, texturing of the building virtual objects may be performed during the rendering process.

[0553] Meanwhile, texturing of the above-mentioned modeled virtual objects can be performed in a manner in which texture images are tiled, and can be performed by sequentially combining multiple sub-regions constituting the virtual object.

[0554] For example, the virtual object corresponding to the target building may have a surface area divided into multiple layers according to the exterior appearance of the target building, and a partial area may be formed for each of the divided surface areas. For example, the processor (1330) may divide the target building into a bottom layer, a top layer, and at least one middle layer, and different tiles may be matched to each partial area.

[0555] In this case, the top floor is distinguished when it has a shape or form that is distinct from the middle floor, such as a roof, an antenna, or a spire. If the target building does not have a structure placed on the roof of the building, such as a roof, an antenna, or a spire, the processor (1330) may distinguish the target building only by the bottom floor and at least one middle floor.

[0556] Meanwhile, in the case of an intermediate layer, the processor (1330) can be divided into at least one layer according to a preset floor height or window arrangement, etc. For example, if the intermediate layers have different colors or shapes (e.g., different window arrangements), the layers with the different colors or shapes can be distinguished as different intermediate layers. That is, in the case of floors designated as commercial spaces and floors designated as residential spaces, such as in a mixed-use building, the floors designated as commercial spaces and the floors designated as residential spaces can each be distinguished as different intermediate layers.

[0557] In this case, different layers, for example, the ground floor, at least one intermediate layer, and the top layer, can be matched with different tiles. And based on the different matching tiles, tiling can be performed for each layer, i.e., each sub-region (e.g., the ground floor, at least one intermediate layer, and the top layer). Then, each sub-region for which tiling has been performed can be sequentially combined in a preset order based on combination information specified in the building reference information corresponding to the target building.

[0558] Meanwhile, the tiling may be sequentially performed based on a tiling order specified in the building reference information. For example, each of the sub-regions may be sequentially combined starting from a sub-region corresponding to a floor close to the ground. In this case, tiling may be performed first for a sub-region corresponding to an area in contact with the ground, i.e., the bottom floor, and then tiling may be performed for a sub-region corresponding to the next order (e.g., the first middle floor) on top of the sub-region already tiled. Finally, tiling may be performed for a sub-region corresponding to the uppermost floor (e.g., the top floor). In this way, by sequentially performing tiling for each sub-region in a preset order for the virtual object, texturing may be performed for the entire surface area of ​​the virtual object.

[0559] In this way, texturing of a virtual object according to an embodiment of the present invention can be performed based on the sequential combination of each tiled sub-region. Therefore, texturing of a virtual object according to an embodiment of the present invention can also be referred to as procedural texturing.

[0560] Meanwhile, the tile DB (1341) may be provided from a preset server that is communicatively connected to the 3D map view generation device (1300). For example, the preset server may be a cloud server. In this case, the cloud server may be the same server as the cloud server that provides map data (e.g., map data including 2.5D polygon objects), but of course, they may be different servers. In this case, that is, if they are different servers, the cloud server that provides the map data will be described as a first cloud server, and the cloud server that provides the tile DB (1341) and building reference information will be described as a second cloud server.

[0561] When the above tile DB (1341) is provided, the processor (1330) can determine tiles to be tiled on the virtual object surface area of ​​each building included in an area defined based on driving information of the vehicle (100) based on information collected around the vehicle (100).

[0562] For example, the processor (1330) can segment an image of a building around a vehicle (100) acquired through a camera (310) into a plurality of partial regions and detect a tile matching each of the segmented images (i.e., partial region images of the target building). To this end, the processor (1330) can calculate a similarity for each tile included in the tile DB (1341) for each of the partial region images, and determine a tile matching each of the partial region images based on the calculated similarity.

[0563] For example, the processor (1330) can detect feature information for a certain partial region image. And, among the tiles included in the tile DB (1341), at least one tile having a feature similar to a feature detected from the certain partial region image can be detected. And, for each of the at least one detected tiles, a similarity to the certain partial region image can be calculated, and a certain tile having the highest calculated similarity can be matched as a tile corresponding to the certain partial region image.

[0564] For this tile matching, the processor (1330) may utilize a pre-trained artificial intelligence (AI) model. In this case, the AI ​​model may recognize a partial region image from features detected in the partial region image, and detect a tile having features most similar to the feature points of the recognized partial region image.

[0565] Here, the feature points of the partial region image can be detected in various ways. For example, the shape, form, or color pattern of the image can be detected as the feature point. In this case, the more distinct the shape, form, or color pattern of the partial region image, the more clearly the AI ​​model can detect the pattern from the partial region image, resulting in a higher recognition rate.

[0566] On the other hand, if it is difficult to detect a pattern of shape, form or color from the partial region image due to low image quality, brightness difference, shaking, etc., the artificial intelligence model may not be able to detect the feature points of the partial region image, in which case a low recognition rate may be produced. Then, based on the recognition rate, the artificial intelligence model may not detect a tile corresponding to the partial region image. In other words, the artificial intelligence model may determine whether to detect a tile corresponding to any one of the partial region images based on the produced recognition rate.

[0567] Here, the artificial intelligence model may be included in the 3D map view generation device (1300) according to an embodiment of the present invention as a separate component from the processor (1330) (not shown). Alternatively, the artificial intelligence model may be part of the processor (1330), or the processor (1330) may perform the function of the artificial intelligence model. In this case, the artificial intelligence model may be understood as the same component as the processor (1330).

[0568] Meanwhile, when the tiles corresponding to each partial region image are determined, the processor (1330) can generate combination information related to the combination of each partial region image based on the acquired building image. For example, the combination information may be information about the location of each partial region image or the order in which tiling is performed for the acquired building image.

[0569] And the processor (1330) can model virtual objects of buildings located in an area defined according to the location of the vehicle (100) from map data. And, among the area, building reference information of buildings located in at least a part of the area (hereinafter referred to as a map area) defined according to the driving information of the vehicle (100) can be detected. And, the processor (1330) can perform texturing on virtual objects corresponding to the detected building reference information among the modeled virtual objects.

[0570] In more detail, the processor (1330) can sequentially tile surface areas of virtual objects corresponding to the detected building reference information using tiles pre-designated for each sub-area.

[0571] For example, if the target building is formed of three layers, including a ground floor, a first intermediate floor, and a second intermediate floor, the processor (1330) may sequentially tile tiles matching each sub-area in a preset order onto the surface area of ​​a virtual object corresponding to the target building. Here, the preset order may be an order from the ground.

[0572] Accordingly, tiles matching the bottom area, i.e., the partial area (bottom layer) included in the building reference information, can be tiled on the surface of a virtual object in an area in contact with the ground in the virtual object in the map data corresponding to the target building. Then, when tiling for the bottom layer is completed, the processor (1330) can tile tiles matching the next partial area (first intermediate layer) in a preset order on the partial area (bottom layer) for which tiling has been completed. Then, when tiling for the first intermediate layer is completed, the processor (1330) can tile tiles matching the next partial area (second intermediate layer) in a preset order on the partial area (first intermediate layer) for which tiling has been completed. In this way, by sequentially tiling the tiles matching each partial area on the surface area of ​​the target building, texturing can be performed on the virtual object in the map data corresponding to the target building.

[0573] Here, the height between each floor of the target building may correspond to an integer multiple of the tile. For example, if the height between each floor of the target building is equal to the height of the tile, the processor (1330) may perform tiling for each sub-region by arranging tiles matching each sub-region in a non-overlapping manner.

[0574] To this end, the processor (1330) may determine an area in which the tiling is to be performed based on the left and right margin values ​​preset from the corners of each side area of ​​each virtual object (building virtual object) and the horizontal and vertical lengths of each side area of ​​the building according to the building reference information. In this case, each building surface may be divided into a plurality of sub-areas based on a floor divider that divides the top and bottom, the top layer, and at least one middle layer. In addition, the processor (1330) may configure the left and right margin values ​​and the top and bottom margin values ​​or the width of the floor divider as margins for each of the plurality of sub-areas, and determine the remaining area excluding the margins as an area in which the tiling is to be performed. In addition, in this case, the inter-floor height or inter-floor width of the bottom or top layer may be formed differently from the inter-floor height or inter-floor width of the middle layer. Additionally, among the tiles stored in the tile DB (1341), the tiles corresponding to the middle layer may be tiles that differ in at least one of width or height from the tiles corresponding to the bottom layer and the top layer.

[0575] And the processor (1330) can determine the number of tiles to be tiled horizontally or the number of tiles to be tiled vertically (tiling offset) in each sub-area where the tiling is to be performed according to the width or height of the determined area. And the processor (1330) can distinguish tiles that are tiled continuously in the surface area of ​​the building virtual object, i.e., each building element included in the texture image.

[0576] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different areas of the texture image by each color value, and accordingly, different areas of the texture image, i.e., different building elements, can be distinguished by color channel.

[0577] And the processor (1330) can generate mask maps for masking building elements for each distinguished building element in each surface area of ​​the tiled building virtual object. The mask map is a map formed so that building elements other than a certain building element are hidden in the surface area of ​​the tiled building virtual object, and may mean an image used to specify only one building element among each building element included in the texture image. In other words, the mask map corresponding to a specific building element may be an image in which other areas are hidden except for an area corresponding only to the specific building element, and may be an image for applying a specific effect only to an area corresponding to the specific building element.

[0578] Meanwhile, when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps with shading effects so that each mask map can display a different material texture.

[0579] Normally, when there is a curve in the surface of an object, the direction of the surface changes due to the curve, and light is reflected differently depending on the changed surface direction, and the curve of the surface of the object can be felt by the reflected light that is reflected differently. The normal map is used to express the curved surface using the reflected light of the curve rather than the actual curve, and may mean an image having a 3D vector value that represents the 3D direction of the surface of the object required for calculating the degree of shading corresponding to the reflected light reflected by the surface curve of the specific material when light incident on a specific material is incident on the surface of the specific material. When such a normal map is used, even if it is a 2D image that does not have a three-dimensional shape, the reflected light corresponding to the curve according to the specific material can be expressed for the light incident on the building, so the texture of the specific material can be formed.

[0580] Accordingly, the processor (1330) can blend normal maps corresponding to different materials with respect to the mask maps corresponding to each building element. Then, a 3D vector value according to the normal map can be applied to the building element corresponding to each mask map, so that a material texture according to the normal map can be expressed.

[0581] Here, in the case of an area covered by the mask map, the 3D vector value may not be applied even when the normal map is blended. Therefore, the normal map to which the mask map is blended may be textured with a specific material only in an area not covered by the blended mask map. In this case, the area not covered by the mask map may be an area corresponding to a specific building element specified by the mask map. Therefore, the normal map to which the mask map is blended may be an image to which a 3D vector value is applied so that a surface texture of a specific material is expressed only in a specific building element that is not masked by the mask map.

[0582] Meanwhile, the mask map may be performed for each distinguished building element in each surface area of ​​the building virtual object in which texture images are tiled. Accordingly, multiple mask maps may be generated for each building element in the surface area of ​​the building virtual object. Each mask map may be blended with a normal map of a different material. The processor (1330) may perform blending between normal maps to which the mask maps are blended.

[0583] Accordingly, the processor (1330) can generate a single normal map (hereinafter referred to as a multi-blended normal map) in which 3D vector values ​​corresponding to surface textures of different materials are applied to areas corresponding to different building elements, respectively, for the surface area of ​​the building virtual object through blending between normal maps to which the mask map is blended. Then, by synthesizing the generated multi-blended normal map to the surface area of ​​the building virtual object, the areas corresponding to each building element of the building virtual object can be textured to have different surface textures.

[0584] Here, the processor (1330) can generate a normal map in which 3D vectors corresponding to different surface textures are applied to areas corresponding to each building element included in the texture image for each texture image tiled on the surface area of ​​the building virtual object. Then, the generated normal map, i.e., the multi-blended normal map, can be synthesized for each texture image arranged on the surface area of ​​the building virtual object. In this case, since the texture image is synthesized in a tiling manner on the surface area of ​​the building virtual object, the multi-blended normal map synthesized for each texture image can also be synthesized in a tiling manner on the surface area of ​​the building virtual object.

[0585] Meanwhile, a normal map having 3D vector values ​​corresponding to the surface texture of the specific material applied may be blended with an albedo map for representing the color of the specific material. In this case, the albedo map may be an image having a color value unique to the specific material. Accordingly, a normal map having 3D vector values ​​corresponding to the texture of the specific surface material may form a pair with an albedo map having a color value unique to the specific surface material, and may be blended together.

[0586] Therefore, when an albedo map is blended with a normal map, the normal map may have a 3D vector value corresponding to the surface texture of a specific material as well as a color value unique to the specific material. Therefore, when a mask map corresponding to a specific building element and a normal map are blended, and the albedo map is further blended to the blended normal map, the specific building element corresponding to the mask map may have a color value unique to the specific material as well as a surface texture of the specific material. In this case, a multi-blended normal map corresponding to the entire surface area of ​​a building virtual object or a texture image tiled on the building virtual object may be an image in which 3D vector values ​​having different material textures are applied to different areas corresponding to a plurality of building elements, and each area is expressed with a color of a different material.

[0587] Meanwhile, when texturing of a surface area of ​​a building virtual object is completed through the multi-blending normal map, the processor (1330) can control the MR service device (900) to MR render the building virtual object for which texturing has been completed and display it on the display unit (251). In this case, the processor (1330) can control the MR service device (900) and the display unit (251) of the vehicle (100) through the interface unit (1320).

[0588] Meanwhile, the processor (1330) may determine at least one of the colors of the normal map corresponding to each building element based on an image captured from the camera of the vehicle (100). For example, the processor (1330) may obtain an image corresponding to a visible area of ​​a frustum shape according to the current position of the vehicle (100) and the driving direction of the vehicle (100) from the camera of the vehicle (100). Then, the objects included in the received image may be classified by type and a representative color may be determined for each type of the classified object. In this case, the representative color may be the RGB average value of the color of each object classified as a specific type, or the color having the highest ratio among the colors of each object classified as the specific type.

[0589] For example, the processor (1330) can detect colors corresponding to building objects from the acquired image, and determine a specific color having the highest RGB average value or ratio of the detected colors as a representative color of the building objects included in the image.

[0590] And the processor (1330) can determine at least one color among the colors of the normal map corresponding to each building element as the determined representative color. Accordingly, at least one color among the colors of the normal map corresponding to each building element of the building virtual object can be determined from an image acquired from the camera of the vehicle (100).

[0591] Meanwhile, the above description describes modeling building virtual objects corresponding to buildings included in the map area and texturing and rendering some of the modeled building virtual objects. However, it is also possible to model and render road virtual objects corresponding to road shapes included in the map area.

[0592] To model the above road virtual object, the processor (1330) may generate a plurality of nodes along a road shape included in the map area from map data. Then, the plurality of generated nodes may be defined as segments, and polygon objects having a specific width may be generated for each segment, and the generated polygon objects may be combined to model the road virtual object. In this case, the polygon objects may be round objects having a specific width.

[0593] Meanwhile, the processor (1330) may utilize a morphology operation to more smoothly express the edge of the road object. The morphology operation is one of the techniques for analyzing and transforming the shape and structure of an object within an image, and may include erosion, dilation, opening, and closing operations. Here, the processor (1330) may more smoothly express the road virtual object composed of the polygon objects through a closing operation used when filling a groove or cutting off a protruding part among the morphology operation techniques.

[0594] Meanwhile, FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view creation device (1300) and an MR service device (900) according to an embodiment of the present invention.

[0595] First, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can collect driving information of the vehicle (100). Here, the driving information of the vehicle (100) can include information on the speed, location, and driving direction of the vehicle (100) detected through the sensing unit (120). In addition, a map area for modeling virtual objects can be determined based on the collected driving information of the vehicle (100).

[0596] Meanwhile, when a map area is determined, the processor (1330) can detect building reference information of buildings included in the determined map area from the object reference information storage unit (1343). Then, when the building reference information is detected, building image setting values, which are information for modeling building virtual objects corresponding to each building, are determined, and building virtual objects corresponding to each building can be modeled based on the determined building image setting values ​​and the building reference information of each building.

[0597] And the processor (1330) can determine any texture image as a tile from the tile DB (1341) for each building or for each partial region (e.g., floor region, at least one middle region, top region) of each building. And the texture image determined as a tile can be tiled for each building or for each partial region of each building. And mask maps corresponding to different components of the building included in the tiled texture image are blended with normal maps corresponding to different materials, and each normal map blended with the mask map is blended with an albedo map having a color corresponding to each material, so that the different components of the building can be textured to have different materials and different surface textures. And by blending the textured normal maps, a multi-blended normal map corresponding to a specific surface region or partial region of the building virtual object or a specific texture image can be generated. And, by synthesizing the generated multi-blended normal map to a specific surface area or partial area of ​​the building virtual object, or by tiling a specific surface area or partial area of ​​the building virtual object along a tiled texture image, a surface texture for a surface area of ​​the building virtual object can be formed. That is, a textured virtual object can be generated.

[0598] Meanwhile, the MR service device (900) can model the location of the self-vehicle (100), pedestrians and other vehicles around the vehicle (100), POI (Position Of Interest) information, route information, and road information and terrain information around the vehicle (100) based on information provided from the vehicle system, i.e., GPS, ADAS, and navigation system of the vehicle (100). In addition, the MR service device (900) can configure a UX scene including the textured virtual objects using the modeled information and information about the modeled virtual objects provided from the 3D map view generation device (1300). In addition, the MR renderer can be controlled to generate a map image including a camera view, 3D effects, and GUI (graphics rendering), and the image can be displayed through the display unit (251).

[0599] Hereinafter, a method of determining a map area, modeling virtual objects included in the determined map area, and rendering and displaying the modeled virtual objects by a 3D map view generation device (1300) according to an embodiment of the present invention will be described in more detail with reference to the attached drawings.

[0600] FIG. 13 is a flowchart illustrating an operation process of a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention for determining a map area, modeling virtual objects included in the determined map area, blending the modeled virtual objects into a normal map having 3D vector values ​​of different materials to texture them, and rendering and displaying the textured virtual objects. FIG. 14 is an exemplary diagram illustrating an example of a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention for updating a map area according to vehicle driving information. FIG. 15 is a conceptual diagram illustrating map tile background modeling described in FIG. 14 in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0601] First, referring to FIG. 13, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can obtain driving information about the vehicle's location, speed, driving direction, destination, and path to the destination through the navigation system (770) and the vehicle's sensing unit (120) (S1300).

[0602] When driving information of a vehicle (100) is acquired, the processor (1330) can determine a map area based on a map tile corresponding to the location of the vehicle (100). The map area can be determined differently depending on the preset zoom level of the map.

[0603] For example, the map area may be an area formed by 25 map tiles centered on the map tile according to the location of the current vehicle (100) when the zoom level of the currently set map is 17. In this case, when the zoom level of the map is lowered, an area formed by a greater number of map tiles centered on the map tile according to the location of the current vehicle (100) may be determined as the map area, and when the zoom level of the map is higher, a map area formed by a smaller number of map tiles (e.g., 9) centered on the map tile according to the location of the current vehicle (100) may be determined. When the map area is determined, the processor (1330) may detect data of map tiles included in the determined map area from map data stored in the memory (1340).

[0604] Meanwhile, the above map area may be updated according to driving information of the vehicle (100). For example, when the vehicle (100) leaves the map tile where it is currently located due to driving of the vehicle (100), the processor (1330) may update the map area so that the map area includes map tiles within a range according to the zoom level of the currently set map based on the map tile into which the vehicle (100) newly enters.

[0605] FIG. 14 is an exemplary diagram showing an example in which a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention updates a map area according to driving information of a vehicle (100).

[0606] First, referring to (a) of FIG. 14, (a) of FIG. 14 is a drawing showing an example of a map area (1402) formed with 25 map tiles centered on a map tile according to the location of the current vehicle (100) according to the currently set zoom level '17' of the map. In this case, the map area (1402) may be formed centered on the first map tile (1400) in which the current vehicle (100) is located.

[0607] In this state, if the vehicle (100) moves from the first map tile (1400) to the second map tile (1410) according to the movement of the vehicle (100), the processor (1330) can update the map area (1402) including a number of map tiles (25) determined according to the preset zoom level of the map centered on the first map tile (1400), as shown in FIG. 14b, to a map area (1412) including a number of map tiles (25) determined according to the preset zoom level of the map centered on the second map tile (1410) according to the position of the moved vehicle (100). Accordingly, the processor (1330) can detect data of map tiles corresponding to the updated map area (1412) from map data stored in the memory (1340). In this case, the processor (1330) can detect at least one map tile (1420) that is connected to each other from the map data of the memory (1340), as shown in FIG. 14b.

[0608] Meanwhile, when a map area is determined from data of detected map tiles, the processor (1330) can model virtual objects corresponding to buildings and roads included in the determined map area (S1304).

[0609] In the above step S1304, the processor (1330) can detect buildings included in the detected map area. And, building reference information corresponding to the detected buildings can be detected from the building reference information storage unit (1343). And, the setting values ​​of each building image related to the detected building reference information can be determined. Here, the building image setting values ​​are information for modeling each building virtual object, and may include a setting value of a projection direction according to an angle or distance for generating the building virtual objects according to the driving direction of the vehicle (100) and / or an angle at which the MR view is displayed, a setting value of modeling according to a 2.5D virtual object or a setting value of modeling according to a 3D virtual object, for example, a vector value such as a coordinate value of a vertex or a normal line.

[0610] Once the building image setting values ​​are determined, the processor (1330) can calculate building setting values ​​corresponding to each building included in the map area based on the determined building image setting values ​​and building reference information of each building. Here, the building setting values ​​can include the height of the building virtual object to be created, the side area of ​​each side, the plane area, etc. Then, once the calculation of the building setting values ​​is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included in the map area based on the building setting values.

[0611] Meanwhile, as mentioned in step S1302, the processor (1330) may update the map area according to the driving information of the vehicle (100). In this case, if new map tile data is additionally detected as shown in FIG. 14b by updating the map area, the processor (1330) may additionally model virtual objects corresponding to buildings and roads included in the added map tiles. On the other hand, if the map area is updated as shown in FIG. 14b, map tiles (1430) may be excluded from the map area. In this case, the processor (1330) may delete virtual objects generated from map tiles (1430) excluded from the map area, thereby increasing the efficiency of the memory (1340).

[0612] Here, the processor (1330) may maintain at least some of the virtual objects generated from the excluded map tiles, depending on the building type, user-specified status, POI status, etc. In this case, the maintained virtual objects may be stored in the memory (1340) and may be reused when the corresponding map tile is detected again in the future. In other words, in the case of a building virtual object corresponding to a building frequently visited by the user or a building specified by the user, the virtual object may be stored and reused regardless of whether the map tile containing the building leaves the map area.

[0613] Meanwhile, the step S1304 may include a process of modeling a road virtual object corresponding to the road shape of each map tile included in the map area.

[0614] In this case, the processor (1330) can generate a plurality of nodes along the road shape included in the map area in step S1304. Then, the generated plurality of nodes can be defined as each segment, and the road virtual object can be modeled by generating polygon objects having a specific width for each segment and combining the generated polygon objects. In this case, the polygon objects can be round objects having a specific width.

[0615] Meanwhile, the processor (1330) may utilize morphological operations to more smoothly express the edges of the road virtual object. In particular, the processor (1330) may temporarily increase the radius of the round polygon objects and then restore it to its original state in order to perform a closing operation among the morphological operations. In this case, the degree of smoothness of the edge portion of the road virtual object may vary depending on the radius of the polygon objects and the number of times the closing operation is performed.

[0616] Additionally, the processor (1330) can raise the generated road virtual object above the ground or form it lower than the ground level. In this case, if the height of the road virtual object is lower than the ground level, the ground around the road can exhibit a curb effect, forming a curb depending on the height difference between the ground level and the road virtual object. In this case, by varying the height difference between the road virtual object and the ground level depending on the width or length of the road or the characteristics of the road, the road virtual object can be expressed differently depending on the characteristics of the actual road.

[0617] In addition, the processor (1330) may also form an object corresponding to a river or stream similar to the method of generating the road virtual object. In this case, the height of the virtual object corresponding to the river or stream may be formed so as to be lower than the height of the ground by a preset depth. In this case, similar to the road virtual object, by making the height difference between the virtual object corresponding to the river or stream and the ground different depending on the width or length of the river or stream, the virtual object corresponding to the river or stream may be expressed differently depending on the characteristics of the actual river or stream.

[0618] Meanwhile, in step S1304, when virtual objects such as buildings and roads of each map tile included in the map area are modeled, the processor (1330) can determine an area according to driving information obtained from the vehicle (100) (S1306).

[0619] The area determined in the above step S1306 is an area determined based on the location of the vehicle (100) or the driving direction of the vehicle (100), and may be a map area determined based on the current driving information of the vehicle (100), that is, an area corresponding to at least a portion of at least one map tile detected from the map data storage unit (1342) of the memory (1340). For example, the area may be an area corresponding to the surroundings of the current location of the vehicle (100) within a first map tile according to the current location of the vehicle (100). Alternatively, the area may be an area corresponding to the front of the vehicle (100) in a first map tile including the current location of the vehicle (100) according to the driving direction of the vehicle (100). Alternatively, the area may be an area including a portion of the first map tile and an area corresponding to a portion of at least one other map tile.

[0620] Meanwhile, the area determined in the above step S1306 may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), i.e., a user field of view area according to the driving direction of the vehicle (100). In this case, the user field of view area has a form that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to perspective projection that can form a sense of perspective, and thus may have the shape of a frustum with the location of the vehicle (100) as a vanishing point.

[0621] Here, an area determined according to the driving information of the vehicle (100) in the step S1306 may be an area where virtual objects displayed as a digital twin map can be displayed on the display unit (251). That is, it may be an area displayed as a digital twin map to the user of the vehicle (100) through the MR view, and may be a map area shown to the user of the vehicle (100), i.e., a visible area. Hereinafter, among the map areas determined according to the driving information of the current vehicle (100), an area determined in the step S1306 that is displayed on the display unit according to the driving information of the vehicle (100) will be referred to as a visible area.

[0622] When the visible area is determined in step S1306, the processor (1330) may determine some of the virtual objects to be rendered among the virtual objects modeled in step S1304 according to the determined visible area (S1308). The objects determined in step S1308 may be only some of the virtual objects included in the visible area among the virtual objects modeled in step S1304, that is, the map tiles detected as the current map area. In this case, the modeled objects determined as the rendering target in step S1308, that is, the virtual objects, may include a plurality of building virtual objects and a plurality of road virtual objects or parts of road virtual objects.

[0623] Meanwhile, when the virtual objects to be rendered are determined in step S1308, the processor (1330) can tile a preset texture image, i.e., a tile, for each of the building virtual objects among the determined virtual objects (S1310). Here, the tiling can be performed for each side of each building virtual object determined as the rendering target, and can be performed for the side areas of each building virtual object excluding a preset offset area (or margin area) from the corner of each side.

[0624] Here, at least one of the above-described building virtual objects may have its side region divided into multiple sub-regions. For example, the side region may be divided into a bottom region, a middle region, and a top region, and at least one of the bottom region, the middle region, and the top region may be further divided into multiple regions. In this case, the processor (1330) may tile different texture images for each of the multiple divided regions.

[0625] In addition, the processor (1330) may tile an arbitrary texture image for each building virtual object or for each partial region of each building virtual object. In this case, texture images corresponding to each building virtual object or each partial region of each building virtual object may be determined in advance. In this case, the building reference information, which is information on an actual building corresponding to each building virtual object and stored in the memory (1340), may include information on a texture image corresponding to each building virtual object or each partial region of each building virtual object, and the processor (1330) may determine a texture image corresponding to each building virtual object or each partial region of each building virtual object when determining a building image setting value based on the driving information of the vehicle (100).

[0626] And, based on the height, area, side area of ​​the building included in the reference information of each building virtual object, and the offset area (or margin area), at least one of the horizontal length and the vertical length of the area in which the texture image is to be tiled in each building virtual object can be calculated, and the number of texture images to be tiled in the horizontal or vertical direction can be calculated according to the calculated length. That is, the processor (1330) can calculate information for modeling building virtual objects based on the reference information of the building including the building image setting value and the height of the actual building, and at the same time calculate a building setting value including information on the area of ​​each side area in which the texture images are to be tiled in each side area for each building virtual object and the number of texture images to be tiled in each side area.

[0627] And the processor (1330) can tile texture images in the side areas of each rendering target building virtual object based on information about texture images corresponding to each building virtual object or a partial area of ​​each building virtual object, and information about the number of tilings of the side areas of each virtual object and the texture images included in the calculated building setting value.

[0628] Meanwhile, when tiling of texture images for side areas of each building virtual object is completed in step S1310, the processor (1330) can generate a normal map including 3D vector values ​​that form different textures for each building element included in the texture image, which is a tiled 2D image (S1312).

[0629] In this case, the normal map may be a normal map (hereinafter referred to as a multi-blended normal map) obtained by blending multiple normal maps generated for each building element included in the texture image. In order to generate the multi-blended normal map, the processor (1330) may generate mask maps for masking building elements for each distinguished building element in each surface area of ​​the building virtual object in which the texture images are tiled.

[0630] In this case, the mask map is a map formed so that building elements other than one building element are hidden in the surface area of ​​the building virtual object on which the texture images are tiled, and may mean an image used to specify only one building element among each building element included in the texture image.

[0631] And when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps to which shading effects are applied so that the textures of different materials can be represented in each mask map. In this case, as described above, since the mask map is an image that covers the remaining area except for the corresponding specific building element, when the normal map is blended, a 3D vector value forming a texture corresponding to the specific material corresponding to the normal map can be blended only in an area not covered by the mask map and an area corresponding to the specific building element corresponding to the mask map.

[0632] Meanwhile, the processor (1330) may generate a mask map for each distinguished building element in each surface area of ​​the building virtual object in which texture images are tiled. Accordingly, multiple mask maps may be generated for each building element, and each mask map may be blended with a normal map of a different material. In addition, the processor (1330) may generate a multi-blended normal map in which 3D vector values ​​corresponding to surface textures of different materials are applied to areas corresponding to different building elements, respectively, with respect to the surface area of ​​the building virtual object, by blending normal maps in which the mask maps are blended with each other. In addition, by synthesizing the generated multi-blended normal map to the surface area of ​​the building virtual object, the processor (1330) may texture the surface areas of each building virtual object so that the areas corresponding to each building element of the building virtual object each have a different surface texture.

[0633] Here, when generating a mask map for each building element, the processor (1330) may further blend at least some of the plurality of mask maps generated for each building element with an albedo map, which is a map image having a color corresponding to a specific material. In this case, as at least some of the plurality of mask maps generated for each building element further have a color according to the albedo map, color mapping can be performed with a color corresponding to a specific material.

[0634] In this case, the processor (1330) may determine a color corresponding to at least some of the plurality of mask maps generated for each building element based on the results of analyzing an image captured from the camera of the vehicle (100). For example, the processor (1330) may detect objects corresponding to buildings among image objects included in an image captured from the camera of the vehicle (100). Then, colors may be detected from the detected building image objects, and based on the detected colors, a dominant color or an average color of actual buildings around the vehicle (100) may be determined as a representative color. And, by blending an albedo map corresponding to a color determined as a representative color with at least some of a plurality of mask maps generated for each building element, blending mask maps to which the albedo map is blended to generate the multi-blended normal map, and synthesizing the generated multi-blended normal map to each building virtual object, at least some of the building elements of the building virtual objects currently determined as rendering targets in step S1308 can be textured to have colors similar to those of actual buildings around the vehicle (100).

[0635] Meanwhile, when texturing is completed in step S1312, the processor (1330) may render the building virtual objects for which texturing has been completed to generate images corresponding to the building virtual objects. In addition, the processor (1330) may render other virtual objects, including at least one road virtual object included in the visible area, to generate images corresponding to each virtual object (S1314).

[0636] And the processor (1330) can control the vehicle interface unit (1320) to display virtual objects including each building virtual object and road virtual object generated through the above rendering on the display unit (251) (S1316).

[0637] As described above, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention has a configuration that detects map tiles within a certain area (map area) set centered on the vehicle (100), models virtual objects within the detected map tiles in advance, and renders and images only some of the objects in the driving direction of the vehicle (100) among the pre-modeled objects. In addition, the processor (1330) synthesizes a normal map so that a 3D vector value corresponding to a specific material for each building element is applied only to some of the objects to be rendered, so that a texture according to a specific material for each building element can be formed by the applied 3D vector value during the rendering. In this way, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can shorten the time required for modeling by rendering pre-modeled objects when displaying virtual objects as an MR view using a digital twin map. In addition, the number of virtual objects to be rendered is also limited to the visible area to minimize the number, and the surface area of ​​each building virtual object can be textured to have the three-dimensional effect of various materials using a texture image, which is a 2D image, and a normal map having 3D vector values ​​corresponding to different materials. In this way, by performing texturing using only 2D images, the amount of computation required for texturing can be greatly reduced, and as a result, the time required for rendering can be greatly shortened.

[0638] In this way, since the amount of computation and time required to display virtual objects in an MR view using the digital twin map can be significantly reduced, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can perform rendering of virtual objects to be displayed through the MR view during the time when the app for displaying the MR view using the digital twin map is running, i.e., during the run time. In other words, real-time run-time 3D map modeling can be possible.

[0639] Meanwhile, as described above, the present invention has a configuration for pre-modeling virtual objects included in a plurality of map tile data according to the location of the vehicle (100) and rendering some of the modeled virtual objects, including texturing. In this case, the map tile data can be updated according to the location of the vehicle (100), and the detection of such map tile data can be performed through background processing. In this case, since the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention pre-models virtual objects included in the map tile data when the map tile data is detected according to the determined map area, the modeling of the virtual objects can also be performed through the background processing. The modeling of virtual objects performed through background processing in this way will be referred to as background modeling.

[0640] Meanwhile, the processor (1330) according to an embodiment of the present invention has a configuration that models virtual objects in a map area in advance through background modeling, and renders a portion of virtual objects that have already been modeled based on a visible area determined according to driving information of the vehicle (100). Here, the modeling of the virtual objects is performed through background processing, and the rendering of the modeled virtual objects is performed through foreground processing according to the execution of the user's MR view app, so the 3D map view generation device (1300) according to an embodiment of the present invention has a configuration in which the modeling and rendering of virtual objects are performed separately. Therefore, when the 3D map view generation device (1300) according to an embodiment of the present invention has a plurality of processors, the modeling of the virtual objects and the rendering of the modeled virtual objects may be performed by different processors, respectively.

[0641] FIG. 15 is a block diagram illustrating a configuration in which modeling and rendering of virtual objects are performed by different processes in a 3D map view generation device (1300) according to an embodiment of the present invention having multiple processors.

[0642] Referring to FIG. 15, a 3D map view generation device (1300) according to an embodiment of the present invention, i.e., an embedded system, may include a first processor that loads a plurality of map tiles through an API based on the current location (GPS Position) of a vehicle (100) in conjunction with a navigation map, and a second processor that renders an image of a virtual object displayed on a display unit (251). Here, the first processor may be a central processing unit (CPU) of the 3D map view generation device (1300) according to the embodiment of the present invention, and the second processor may be a GPU (Graphics Processing Unit) that processes graphic operations on the display device and outputs a result value.

[0643] In this case, the first processor may load a plurality of map tiles according to the location of the vehicle (100) and perform modeling on virtual objects included in the loaded plurality of map tiles. In this case, the first processor may perform modeling of the virtual objects in a background modeling manner. In addition, the modeled virtual objects may be stored in a buffer. In this case, the modeled objects may be simple polygonal (2.5D) virtual objects without surface texturing.

[0644] In this case, at least one of the plurality of map tiles may be added when the map area is updated according to the location of the vehicle (100). Then, the first processor may additionally model virtual objects included in at least one additional map tile according to the updated map area. Then, the buffer may be updated so that the modeled virtual objects are added. Accordingly, virtual objects corresponding to each object, including buildings and roads included in the entire updated map area, may be stored in the buffer.

[0645] Meanwhile, when an app for displaying an MR view is executed by a user, the second processor can render the modeled virtual objects stored in the buffer. During the rendering process, the second processor can perform texturing by tiling a texture image to a side area of ​​each modeled building virtual object, and synthesizing a normal map to which a 3D vector corresponding to a different material is applied to each area of ​​each building element included in the tiled texture image to the side area of ​​each building virtual object. In this case, the 3D vector is for providing a shading effect that forms a texture of a specific material through a shading effect, and the second processor, as a custom shader, can perform rendering to generate an image of each building virtual object, including texturing that includes shading according to the 3D vector applied to each building element.

[0646] Therefore, the 3D map view generation device (1300) according to an embodiment of the present invention can process modeling of virtual objects and rendering of modeled virtual objects in parallel on different processors. Therefore, the computational load applied to the processor can be significantly reduced compared to a case where all processes are performed on a single processor, and by displaying virtual objects by performing rendering from pre-modeled objects, the time required until the virtual objects are displayed on the display unit can also be significantly shortened. Accordingly, when an app related to the MR view is executed, rendering of 3D virtual objects included in the digital twin map can be performed during the runtime time, which is the time when the app is executed, and real-time provision of a digital twin map including virtual objects with high-quality textures can be possible.

[0647] However, for convenience of explanation, the following description assumes that modeling and rendering are performed by a single processor (1330). However, it should be understood that the present invention is not limited thereto.

[0648] Meanwhile, in step S1316 of FIG. 13, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can simultaneously display a screen showing information collected around the vehicle according to the ADAS system and an MR view screen according to the digital twin map. FIG. 16 is an exemplary diagram illustrating such an example.

[0649] Referring to FIG. 16, the processor (1330) of the 3D map view generation device (1300) can display information on surrounding vehicles and obstacles based on information collected through the vehicle interface unit (1320) and an ADAS (Advanced Driver Assistance Systems) view image based on driving information of the vehicle (100) in one area (1600) of the display unit (251). ADAS refers to an active safety device that detects a dangerous situation with a sensor or a camera, warns the driver of the risk of an accident, and helps the driver make a judgment and respond. The ADAS view image can include information on objects corresponding to the surrounding situation of the vehicle, for example, a vehicle object and other vehicles and obstacles detected around the vehicle, and can include route information such as the current driving speed or driving time of the vehicle (100) or the expected time required to reach the destination.

[0650] Meanwhile, the processor (1330) can display an MR view image in another area (1610) on the display unit (251). The MR view image is a view image including a building virtual object or a road virtual object created as a 3D object, and can display a digital twin map formed by the virtual objects.

[0651] Meanwhile, according to the above description, it has been mentioned that the processor (1330) of the 3D map view generation device (1300) according to the embodiment of the present invention can determine an area to display virtual objects by rendering them according to the driving information of the vehicle (100), i.e., a visible area, in step S1306 of FIG. 13. Here, the visible area may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), i.e., a user viewing area according to the driving direction of the vehicle (100). In this case, the user viewing area has a form that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to perspective projection that can form a sense of perspective, and thus may have the shape of a frustum with the location of the vehicle (100) as a vanishing point. Therefore, the above-mentioned visible area can be formed in a frustum shape with the position of the vehicle (100) as the vanishing point.

[0652] FIG. 17a is an exemplary diagram illustrating a frustum-shaped visible area formed across a plurality of map tiles or a single map tile, depending on the position of the vehicle and the driving direction of the vehicle, in a 3D map view generation device (1300) according to an embodiment of the present invention. FIG. 17b illustrates an example of a visible area formed to include a plurality of map tiles.

[0653] First, referring to (a) of FIG. 17A, the processor (1330) can form a frustum-shaped visible area (1700) formed along the driving direction of the vehicle (100) with the current location (2000) of the vehicle (100) as a reference point. In this case, the processor (1330) can perform rendering only on virtual objects located within the formed visible area (1700). That is, even within one map tile, virtual objects located in areas other than the visible area (1700) may not be rendered.

[0654] Here, rendering may refer to a process or calculation process of generating an image using a computer program. Therefore, even if it is a modeled virtual object, that is, even if a virtual object is created, if the rendering is not performed, an image corresponding to the virtual object may not be created. In addition, if an image is not created, it may not be displayed on the display unit (251). That is, the processor (1330) may selectively determine objects to be displayed on the display unit (251) among the modeled virtual objects based on a visible area formed according to the driving direction of the vehicle (100).

[0655] Accordingly, as shown in (a) of FIG. 17A, images corresponding to each virtual object can be generated through a rendering process for virtual objects within an area included in the visible area (1700). However, images of virtual objects within an area included in the visible area (1700) may not be generated. Therefore, the processor (1330) can reduce the amount of computation required for rendering and shorten the computation time by rendering only some of the virtual objects modeled based on the visible area.

[0656] Meanwhile, the visible area (1700) may be formed across a plurality of map tiles, as shown in (b) of FIG. 17, depending on the zoom level of the set map. In addition, the plurality of map tiles may be a part of a map area set around the current location (2000) of the vehicle (100). In this case, since the processor (1330) models virtual objects based on the map area, images of virtual objects in an area included in the visible area (1700) among the plurality of map tiles may be generated through a rendering process according to the visible area (1700) formed across the plurality of map tiles, and may be displayed on the display unit (251).

[0657] Meanwhile, Fig. 17b is an example diagram showing an example in which a visible area formed in a frustum shape includes multiple map tiles.

[0658] As shown in Fig. 17b, when a plurality of map tiles are included in the visible area (1700), the processor (1330) can perform modeling on virtual objects included in each of the plurality of map tiles included in the visible area (1700). In addition, the virtual objects for which modeling has been performed can be rendered in a preset order or by selecting only some of the virtual objects.

[0659] For example, the processor (1330) may perform modeling and rendering in order of proximity to the vehicle based on the current location (2000) of the vehicle. Accordingly, objects included in the first map tiles (1751) that are close to the current location (2000) of the vehicle may be modeled and rendered before objects included in the second map tiles (1752) that are further away from the current location (2000) of the vehicle.

[0660] Alternatively, the processor (1330) may model and render only a portion of the objects included in each of the map tiles included within the visible area (1700) based on a distance from the current location (2000) of the vehicle.

[0661] For example, the processor (1330) may perform modeling and rendering for all objects included in each map tile of the first map tiles (1751), but may perform modeling and rendering for only some of the objects included in each map tile of the second map tiles (1752). In this case, the processor (1330) may model and render only objects corresponding to roads and terrain among the objects included in the second map tiles (1752), and may not perform modeling and rendering for general building objects.

[0662] In this way, the modeling and rendering priorities for objects included in each map tile can be determined based on their type. These object type-specific priorities can be included in the data for the objects for the map tile (hereinafter referred to as "map tile data"), along with geometry data containing the shape information of each object, as attribute data corresponding to each object.

[0663] In this way, the order in which objects included in a map tile are modeled and rendered according to their type is determined by attribute information, and the processor (1330) can sequentially model (procedural modeling) and render (procedural rendering) objects included in a map tile according to the order, i.e., priority.

[0664] Meanwhile, by using the above priority, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention may model and render a specific object with priority over other objects. For example, the processor (1330) may give a higher priority to objects such as landmarks than to general building objects. In this case, the building object corresponding to the landmark may be modeled and rendered with priority over or earlier than other general building objects. For example, if the second map tiles (1752) include a building object corresponding to a landmark, the processor (1330) may perform modeling and rendering for the landmark building object together with the road objects and terrain objects included in the second map tiles (1752).

[0665] Meanwhile, a 3D map view generation device (1300) according to an embodiment of the present invention may be configured to receive model data for modeling a building object model (hereinafter referred to as a landmark model) corresponding to a preset landmark building separately from model data for modeling a general building object.

[0666] For example, the memory (1340) of the 3D map view generation device (1300) may store model data of a plurality of landmark models. Alternatively, the 3D map view generation device (1300) may receive landmark model data of landmarks located around the vehicle based on the current vehicle location information from a preset external server (e.g., a map information provider) having a plurality of landmark model data.

[0667] In this case, the landmark model may be a higher-quality 3D model compared to typical building models. For example, the landmark model may be a model textured with a higher-quality image than typical building models, or a model composed of a three-dimensional shape similar to the shape of an actual landmark building object that is the target of the landmark model.

[0668] Meanwhile, when landmark model data is provided separately in this way, data of objects included in the map tile, i.e., map tile data, may include information on the landmark (hereinafter referred to as landmark information). For example, the map tile data may include geometric data for modeling the geometric shape of each road, terrain, and building object, and attribute data including the name, height, size, and location of each building object. In this case, the landmark information may be included in the map tile data as the attribute information, and may include the size and location of an area corresponding to the landmark, and the name and height of the landmark.

[0669] Accordingly, the processor (1330) may check whether the attribute data includes the landmark attribute information while parsing the map tile data of each of the map tiles included in the visible area (1700) to model and render objects from each of the map tiles. If the landmark attribute information is included, the landmark model corresponding to the landmark attribute information may be modeled and rendered with priority over other building objects.

[0670] Figure 18 illustrates the operation process in which a landmark model is modeled and rendered preferentially.

[0671] Referring to FIG. 18, the processor (1330) can parse each map tile data included within the visible area of ​​the vehicle camera, i.e., the field of view, to detect geometric data and attribute data for objects included in each map tile (S1800). Here, the geometric data may be data related to the geometric shape of the objects, and the attribute data may include data such as the name, height, size, and location of the objects. In addition, the attribute data may include flag information indicating the status of the map tile data.

[0672] And the processor (1330) can perform modeling on the objects with the highest priority among the objects included in each of the map tiles within the field of view based on the detected attribute data. In this case, the objects with the highest priority may be objects related to roads and terrain (S1802). Here, the processor (1330) can perform modeling on the road and terrain objects in the order of distance from the vehicle among the map tiles within the field of view. That is, even if they have the same priority, objects in map tiles that are closer to the vehicle can be modeled with greater priority.

[0673] In the step S1802, when the mesh of the road and terrain objects is modeled, the processor (1330) can check whether there is landmark attribute information among the attribute information detected from the map tiles within the field of view (S1804). For example, the processor (1330) can directly detect the landmark attribute information from the attribute information. Alternatively, the processor (1330) can determine whether the corresponding map tile includes landmark information based on any one of the flags indicating the status of the map tile data (e.g., landmark flag).

[0674] In this case, the processor (1330) can determine that the corresponding map tile includes landmark attribute information when the landmark flag has a specific value (e.g., a true value). Conversely, when the landmark flag does not have the specific value (e.g., a false value), the corresponding map tile can be determined not to include landmark attribute information. In this way, when flag information is used, whether or not the landmark attribute information is included can be detected more quickly.

[0675] Meanwhile, if the result of the check in step S1804 shows that landmark attribute information is included among the parsed attribute data, the processor (1330) can detect model data of a landmark model corresponding to the landmark attribute information included in the attribute data and parse the detected model data, i.e., landmark model data (S1806).

[0676] In this case, the landmark model data, unlike the model data of general buildings, may be separately provided or pre-saved model data. Furthermore, compared to the model data of general buildings, the landmark model data may be textured with a higher resolution texture image or may be of higher quality, such as having a more detailed three-dimensional shape.

[0677] And when parsing of landmark model data is completed, the processor (1330) can generate a model mesh of the parsed landmark model data (landmark model modeling) and render the model mesh of the generated landmark model data (S1808).

[0678] In the step S1808, when rendering of the model mesh of the landmark model data is completed, the processor (1330) can parse the model data of the general building objects included in each map tile based on the geometric data detected in the step S1800, and generate a model mesh of the model data of the general building objects (general building object model modeling). Then, the generated building mesh can be rendered (S1810). In this case, the processor (1330) can model and render the general building object models only in the area where the landmark model is not rendered.

[0679] Accordingly, if the attribute data detected in step S1800 includes landmark attribute information, the processor (1330) may first model and render roads and terrain objects, and then model and render landmark models. Finally, the processor (1330) may model and render general building object models. In this way, the 3D map view generation device (1300) according to an embodiment of the present invention may perform procedural modeling and rendering according to the type of object (road, terrain object, landmark model, general building model), thereby allowing the landmark model to be modeled and rendered preferentially with respect to other building objects.

[0680] Accordingly, the 3D map view generation device (1300) according to an embodiment of the present invention can provide a user UX that allows the user to intuitively identify the area where the current vehicle is located by more quickly displaying a 3D building virtual object corresponding to a landmark that can represent the map area. In addition, it can have the effect of allowing the user to more easily identify the area where the user is currently located by providing a landmark representing a specific area in a higher quality model.

[0681] Meanwhile, in step S1810, the processor (1330) can sequentially model and render objects included in each map tile within the field of view according to the distance from the vehicle. That is, the 3D map view generation device (1300) according to an embodiment of the present invention can perform procedural modeling and rendering according to the type of object (road, terrain object, landmark model, general building model) as well as the distance from the vehicle or camera.

[0682] Meanwhile, if, as a result of the check in step S1804, there is no map tile including landmark attribute information among the plurality of map tiles included in the field of view, the processor (1330) can proceed directly to step S1810 to perform modeling and rendering of general building objects. Accordingly, if a landmark model is not included, if the road object and terrain object are modeled and rendered in step S1802, modeling and rendering of general building objects can be performed immediately. In this case, the processor (1330) can perform procedural modeling and rendering according to the distance of the vehicle or camera.

[0683] When modeling and rendering of general building objects are performed in step S1810, the processor (1330) can check whether there is a map tile in which general building objects are modeled, that is, a general building area overlapping with a landmark model among the rendered building objects (S1812). If, as a result of the check in step S1812, there is a map tile in the general building area overlapping with the landmark model, a landmark attribute can be assigned and managed for the map tile. In this case, even if the attribute data of the map tile does not include landmark attribute information, the processor (1330) can regard the map tile as a map tile including landmark attribute information based on the landmark attribute information assigned in step S1812 when parsing the map tile later.

[0684] Meanwhile, FIGS. 19 and 20 are exemplary diagrams showing examples of procedural modeling and rendering according to the type of object in a 3D map view generation device (1300) according to an embodiment of the present invention as described above.

[0685] As described in the above-described Fig. 18, the 3D map view generation device (1300) can preferentially model and render terrain and road objects from map tile data for each map tile included in the viewing area. In addition, if there is a map tile including landmark attribute information among the map tiles within the viewing area, the landmark model corresponding to the landmark attribute information can be modeled and rendered with priority over models of general building objects.

[0686] Therefore, when a landmark model is included, the processor (1330) can model and render terrain and road objects (1900) as a first priority, as shown in (a) of FIG. 19. And, as a second priority, modeling and rendering can be performed for landmark models (1910). And, after all landmark models (1910) included in map tiles included within the viewing area are modeled and rendered, general building objects (1920) can be modeled and rendered last, as shown in (c) of FIG. 19.

[0687] In addition, when a vehicle enters a specific area, the landmark models (1910) included in the area can be rendered regardless of whether the vehicle is approaching. Here, the specific area can be a map tile corresponding to the location of the vehicle or a plurality of map tiles corresponding to the surroundings of the vehicle within the field of view. In this case, since terrain and road objects (1900) are preferentially modeled and rendered for map tiles included within the field of view, among the map tiles in which the terrain and road objects (1900) are rendered (map tiles within the field of view), landmark models (1910) located within the specific area can be rendered simultaneously with the vehicle entering.

[0688] Additionally, the modeled and rendered landmark model can remain in the rendered state until the map tile containing the landmark model leaves the viewing area. That is, once the landmark model is rendered, the rendered state can be maintained until the landmark model becomes invisible due to the movement of the 3D map view camera (1950) or the vehicle.

[0689] On the other hand, the general building objects (1920) may be rendered only when a vehicle approaches a certain distance or more. That is, when a vehicle enters the specific area, the landmark model (1910) may be rendered first as soon as the vehicle enters, and accordingly, general building objects (1920) may be rendered in an area within a certain distance from the vehicle, but only landmark models (1910) may be rendered in an area outside a certain distance from the vehicle.

[0690] In this way, the 3D map view generation device (1300) according to an embodiment of the present invention can model and render the general building objects (1920) in a state where the terrain and road objects (1900) and landmark models (1910) are modeled and rendered. Accordingly, the 3D map view generation device (1300) can provide a user UX in which the landmark models (1910) are displayed and then the general building objects (1920) are displayed. Therefore, the user can more easily, quickly, and intuitively recognize the area where the vehicle is currently located.

[0691] Meanwhile, the 3D map view creation device (1300) according to an embodiment of the present invention can first model and render terrain and road objects (1900) as described above, and then place the modeled objects below the rendered terrain and road objects (1900) corresponding to the positions where each modeled object is to be displayed, i.e., below the ground object. Then, while raising the modeled objects, rendering can start from the portion protruding above the terrain and road objects (1900). Fig. 20 illustrates examples of objects rendered in this manner.

[0692] First, referring to (a) of Fig. 20, (a) of Fig. 20 illustrates an example of a landmark model (1910) being rendered.

[0693] As described above, the landmark model (1910) can be placed under the ground object once parsing and modeling are completed. Then, when rendering begins, the processor (1330) can smoothly move the ground object from below toward the sky for a predetermined period of time and sequentially render the landmark model (1910) from the portion exposed above the ground object. By rendering the pre-modeled objects so that they rise from below the ground and are sequentially displayed, the sense of time axis segmentation of the objects newly displayed on the display screen can be eliminated.

[0694] Meanwhile, as described above, the landmark model (1910) is modeled and rendered as a second priority, following the terrain and road objects (1900) that are modeled and rendered as a first priority, i.e., ground objects, and the general building object (1920) can be modeled and rendered as a third priority, following the landmark model (1910). Accordingly, the general building object (1920) can be rendered in a state where the landmark models (1910) are rendered. Fig. 20 (b) shows such an example.

[0695] Likewise, a general building object (1920) can be placed beneath a ground object once parsing and modeling are complete. Once rendering begins, the processor (1330) can smoothly move the ground object from below toward the sky. Rendering can then be performed sequentially, starting from the portion exposed above the ground object. Accordingly, the sense of temporal fragmentation of the building objects (1920) can be eliminated.

[0696] Meanwhile, the processor (1330) may restrict the rendering of at least some general building objects (1920) to ensure the visibility of the landmark model (1910). For example, the processor (1330) may restrict the rendering of at least one general building object (1920) located between the viewing direction of the camera, i.e., the 3D map view, and the landmark model (1910). To this end, the processor (1330) may generate a virtual viewing line connecting each of the currently rendered landmark models (1910) from the current 3D map view camera (1950) or the position of the vehicle, and may restrict the rendering of the general building objects (1920) in at least one area (e.g., an area corresponding to each building object - e.g., a footprint area) through which the viewing line passes. In this case, the models of the general building objects (1920) whose rendering is restricted may be placed below the ground object (1900) and may not be elevated above the ground object (1900).

[0697] In this case, if the camera's position and viewpoint change due to vehicle movement or the user's screen manipulation, and thus the gaze line changes, the area where rendering is restricted may change. Then, depending on the changed gaze line, rendering of general building objects may be restricted in other areas, and in areas where the gaze line previously passed, rendering restrictions may be lifted, allowing rendering of general building objects to begin.

[0698] In this case, if the rendering of the general building object is restricted, the processor (1330) can move the model of the building object in the corresponding area toward the ground below, and can release the rendering from the part adjacent to the ground. Accordingly, the rendering of the building objects in the corresponding area can be released in a manner of sinking below the ground. On the other hand, in the area where the rendering starts, the processor (1330) can move the model of the building object in the corresponding area toward the ground above, and can sequentially render from the part protruding from the ground. Accordingly, the building objects can be displayed on the display unit of the vehicle or the 3D map view generation device (1300) as if they are rising from the ground.

[0699] Alternatively, the processor (1330) may at least partially limit the rendering of general building objects in the viewing direction, i.e., the area through which the viewing line passes, depending on the range in which the landmark model (1910) can be viewed.

[0700] In this case, the processor (1330) can determine a landmark exposure range that allows a certain portion of the landmark model (1910) to be displayed on the display unit. Here, the landmark exposure range can be set to a range that includes at least a portion of the landmark model (1910) based on the top of the landmark model (1910). In addition, the processor (1330) can determine portions corresponding to the set landmark exposure range for each landmark, and determine general building objects that cover the portions corresponding to the determined landmark exposure range as objects whose rendering is restricted.

[0701] Figure 21 illustrates an example in which the rendering of daily building objects is limited based on a preset landmark exposure range.

[0702] For example, the processor (1330) can detect the range in which the landmark model (1910) is visible beyond a general building object (2110), i.e., the landmark exposure range (2100), as shown in (a) of FIG. 21.

[0703] In this case, the landmark exposure range (2100) may be determined based on the height of the current 3D map view camera (1950) viewpoint, the height of the landmark model (1910), and the height of at least one general building object (2110) located on a virtual line of sight connecting the center of the 3D map view camera (1950) and the landmark model (1910). In addition, it may be further determined based on the distance between the 3D map view camera (1950) and the landmark model (1910).

[0704] And the processor (1330) can check whether the detected landmark exposure range (2100) is larger than or equal to a preset size. And if the detected landmark exposure range (2100) is smaller than or equal to a preset size, the rendering of at least one general building object (2110) located on the virtual line of sight can be restricted. That is, as shown in (a) of FIG. 21, among the building objects located on the virtual line of sight, a building object having a height larger than a certain height may not be rendered.

[0705] On the other hand, if the detected landmark exposure range (2100) is larger than a preset size as shown in (b) of Fig. 21, rendering of building objects located on the virtual line of sight may not be restricted. Accordingly, rendering of at least one general building object (2110) located on the virtual line of sight may be performed.

[0706] Meanwhile, if the detected landmark exposure range (2100) is less than a preset size, the processor (1330) may limit rendering of at least one general building object (2110) located on the virtual line of sight to at least a portion.

[0707] For example, the height from the top of the landmark model (1910) to the ground object adjacent to the landmark model (1910), i.e., the entire height of the landmark model (1910), is assumed to be 100% of the landmark exposure range, and the landmark exposure range can be determined according to a ratio set by a user, etc. In this case, if the landmark exposure range is 50%, a portion corresponding to half of the height of the landmark model (1910) from the top of the landmark model (1910) can be determined as the landmark exposure range.

[0708] Then, the processor (1330) can detect a portion corresponding to the lower limit of the exposure range of the landmark model (1910) from the landmark model (1910) based on the height of the current 3D map view camera (1950) viewpoint, the height of the landmark model (1910), and the distance between the landmark model (1910) and the 3D map view camera (1950). Then, a virtual gaze limit line (2101) connecting the portion corresponding to the lower limit of the exposure range of the landmark model (1910) from the center of the 3D map view camera (1950) can be formed, and general building objects having a height higher than the virtual gaze limit line (2101) can be determined as building objects whose rendering is restricted.

[0709] When the building objects whose rendering is restricted are determined in this way, the processor (1330) can determine the rendering restriction height of the general building objects whose rendering is restricted based on the virtual gaze limit line (2101). In this case, the virtual gaze limit line (2101) can be restricted to the maximum height of each general building object located on the virtual gaze line connecting the center of the 3D map view camera (1950) to the landmark model (1910).

[0710] Then, when rendering the general building objects whose rendering is restricted, the processor (1330) can restrict the top of the building object from rising above the virtual viewing limit line (2101) from the ground object (1900). Accordingly, among the plurality of building objects located between the center of the 3D map view camera (1950) and the landmark model (1910), only a portion of the building objects having a height above the viewing limit line (2101) can be exposed above the ground object (1900), and only the exposed portion can be rendered.

[0711] That is, as shown in (a) of FIG. 21, if there is a building object (2110) having a height that obscures the landmark model (1910), the processor (1330) can render the building object (2110) by exposing it above the ground object (1900) only up to a height corresponding to the viewing limit line (2101). Accordingly, as shown in (b) of FIG. 21, the upper part of the building object (2110) exposed above the ground object (1900) is rendered, but the lower part may not be rendered as it remains below the ground object (1900). Accordingly, as shown in (b) of FIG. 21, the 3D map view generation device (1300) according to an embodiment of the present invention can prevent the landmark model (1910) from being obscured by another building object above a certain level.

[0712] Meanwhile, the above description assumes that the landmark model is provided by a preset provider (e.g., a map information provider), but it is of course possible for a user to directly set the landmark.

[0713] Figure 22 is an example diagram illustrating a case where a user designates an arbitrary building object as a landmark.

[0714] Referring to Fig. 22, Fig. 22 shows an example of a landmark building (custom landmark) (2200) arbitrarily designated by a user.

[0715] Users can designate any building object as the custom landmark (2200) based on their needs or preferences. For example, a transit point or destination considered important by the user in a specific region or situation may be designated as the custom landmark (2300). Accordingly, any building object can be designated as the custom landmark (2300), even if it is not a landmark pre-provided or pre-set by the provider.

[0716] Meanwhile, the custom landmark may be designated based on specific conditions, in addition to the user designating a specific building object. For example, if the user presets a destination and multiple waypoints while driving, the destination and multiple waypoints set by the user may be designated as the custom landmark. Alternatively, a building object set as an indicator for guidance during route guidance (e.g., a building called out during route guidance as an indicator for human-like guidance), or a building object corresponding to a specific brand POI (e.g., Starbucks) through a premium service contract with a specific business operator may be designated as the custom landmark.

[0717] In this way, when a specific building object is designated as a custom landmark (2300), the processor (1330) can detect attribute information about the building object designated as the custom landmark (2300) from map tile data. Then, the processor (1330) can store the detected information as landmark attribute information. Accordingly, a map tile including a building designated as the custom landmark (2300) can be changed to a map tile including landmark attribute information. In this case, the processor (1330) can also set flag information including landmark information for the map tile data.

[0718] By storing the property information of the building object designated as the custom landmark (2300) as landmark property information in this way, the building designated as the custom landmark (2300) can be processed in the same way as an actual landmark. That is, when a road and terrain object (1900) is modeled and rendered, the processor (1330) can perform modeling and rendering of the building object designated as the custom landmark (2300) prior to a general building object.

[0719] Additionally, the building object designated as the custom landmark (2300) may be modeled and rendered simultaneously with the vehicle entering an area (2210) containing the building designated as the custom landmark (2300). In this case, the building object designated as the custom landmark (2300) may be modeled and rendered first, regardless of whether the vehicle is approaching it.

[0720] Accordingly, as shown in Fig. 22, only the building object designated as the custom landmark (2300) may be displayed first while other building objects are not displayed. In this case, general building objects around the building object designated as the custom landmark (2300) may be rendered only when the vehicle approaches within a certain distance, depending on the distance from the vehicle.

[0721] Meanwhile, in the case of a building designated as the custom landmark (2300), since the user can arbitrarily designate it, unlike a typical landmark, a building with a height similar to or smaller than a typical building object can be designated as the custom landmark. Accordingly, when the rendering of typical building objects located between the camera or vehicle and the custom landmark (2300) is restricted according to the exposure range of the landmark as shown in FIG. 21 described above, there is a problem that the number of building objects whose rendering is restricted may become too large.

[0722] Accordingly, the 3D map view generation device (1300) according to an embodiment of the present invention can increase the identifiability of the custom landmark building object by changing the rendering so that the building objects around the building object designated as the custom landmark (2300) are displayed differently from the custom landmark.

[0723] Figure 23 is an example diagram showing various examples of changes in the rendering of objects around a building object set as a custom landmark.

[0724] Referring to (a) of Fig. 23, (a) of Fig. 23 shows an example of a 3D map view in which general building objects (2300, 2301, 2302, 2303) are rendered. In this case, a user can designate any one of the building objects (e.g., building object (2300)) as a custom landmark building object.

[0725] In this way, when there is a building object (2300) designated as the custom landmark around the vehicle (100), the processor (1330) can render other building objects located within an area around the custom landmark differently from the custom landmark building object (2300), as shown in (b) of FIG. 23.

[0726] For example, the processor (1330) may texture the surface of a custom landmark building object (2300) using a preset texture image. In this case, the processor (1330) may perform the texturing by tiling the preset texture image on each side area of ​​the landmark building object (2300). On the other hand, the processor (1330) may not perform texturing on the surfaces of other building objects located within an area from the custom landmark building object (2300) when the vehicle (100) approaches within a certain distance from the custom landmark building object (2300).

[0727] Accordingly, as shown in (b) of FIG. 23, only the custom landmark building object (2300) is textured with a high-quality texture image around the custom landmark building object (2300), and the textures of other building objects are removed, thereby improving the user's intuitiveness regarding the custom landmark building object (2300).

[0728] In this case, the processor (1330) may display additional information while displaying the custom landmark building object (2300). For example, the processor (1330) may designate a landmark building for route guidance as the custom landmark building object (2300). For example, the processor (1330) may call out a specific building object while outputting turn section guidance as route guidance. In addition, the specific building object may be designated as the custom landmark building object (2300).

[0729] And, when outputting the rotation section guidance, the processor (1330) may highlight at least one of the building sides of the custom landmark building object (2300) or change and display at least one of the surface color or brightness of texture images tiled on the side of the custom landmark building object (2300). For example, the processor (1330) may display each side of the custom landmark building object (2300) as if it were blinking, thereby further enhancing the user's intuitiveness with respect to the custom landmark building object (2300).

[0730] Alternatively, the processor (1330) can increase the user's intuitiveness with respect to the custom landmark building object (2300) by adjusting the height of building objects around the custom landmark building object, as shown in (c) of FIG. 23.

[0731] In this case, when the vehicle (100) approaches within a certain distance from the custom landmark building object (2300), the processor (1330) can lower other building object models located within an area from the custom landmark building object (2300) below the ground so that they protrude only a certain height from the ground. Accordingly, as shown in (c) of FIG. 23, since only the custom landmark building object (2300) is displayed above a certain height from the ground, the user's intuitiveness with respect to the custom landmark building object (2300) can be further enhanced.

[0732] Meanwhile, FIG. 24 is an exemplary diagram showing an example of limiting modeling of building objects included in a map tile according to a zoom level based on the distance from a vehicle in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0733] Figure 24 (a) shows an example in which the zoom level set for each map tile is fixed to '17', an example of the camera field of view range (2410), and an example of the maximum distance (2420) at which map tiles can be identified within the field of view range.

[0734] According to the above description, the processor (1330) detects map tiles included within the field of view of the 3D map view camera (1950), parses map tile data of the detected map tiles in order of proximity to the 3D map view camera (1950), and performs procedural modeling and rendering according to the distance from the 3D map view camera (1950). In this case, as shown in (a) of the above-described FIG. 24, the zoom levels of the map tiles may all be the same.

[0735] Meanwhile, as the zoom level increases, the map area corresponding to the map tile becomes narrower, so data of more detailed objects (e.g., building objects, detailed road objects) can be included as map tile data. On the other hand, as the zoom level decreases, the area corresponding to one map tile becomes wider, so the size of included objects, such as building objects, can become smaller. In addition, when the zoom level decreases below a certain level, building objects become so small that they are difficult to identify with the naked eye, so data of building objects may not be included in the map tile data. For example, map tile data of map tiles with a zoom level of 17 or lower may only include data related to roads and terrain, and may not include data of building objects.

[0736] In addition, the 3D map view generation device (1300) according to an embodiment of the present invention can provide a 3D map view screen in which each object is rendered in a perspective projection manner based on the position of the 3D map view camera (1950). Accordingly, the rendered building objects can be displayed in different sizes depending on the distance from the 3D map view camera (1950). That is, the farther away the object is from the 3D map view camera (1950), the smaller the object can be displayed, and when the distance from the 3D map view camera (1950) is a certain distance or more, the size may become so small that it may disappear from the screen.

[0737] Therefore, for map tiles that are located at a distance greater than the predetermined distance from the 3D map view camera (1950), modeling and rendering of building objects are performed as they are included in the field of view, but the rendered objects may not be displayed on the screen in a manner that can be visually identified. Therefore, modeling of objects included in map tiles that are located at a distance greater than the predetermined distance from the 3D map view camera (1950) may be unnecessary.

[0738] In addition, since the 3D map view camera (1950) provides the 3D map view screen in the perspective projection manner, the areas included in the 3D map view screen may expand as the distance from the 3D map view camera (1950) increases. That is, the greater the distance from the 3D map view camera (1950), the greater the number of map tiles may be included in the field of view, and accordingly, the number of map tiles on which unnecessary modeling is performed may increase.

[0739] Therefore, as shown in (a) of the above Fig. 24, if the zoom level of each map tile is fixed, there is a problem that the amount of computation required for unnecessary modeling and rendering increases. Accordingly, the 3D map view generation device (1300) according to an embodiment of the present invention can vary the zoom levels of map tiles differently depending on the distance from the 3D map view camera (1950), as shown in (b) of Fig. 24. That is, the processor (1330) can divide the map area within the field of view of the 3D map view camera (1950) into a plurality of areas depending on the distance from the 3D map view camera (1950), and can vary and adopt the zoom levels of map tiles covering each area.

[0740] That is, in (b) of FIG. 24, within the field of view (2410) of the 3D map view camera (1950), a first map area within a first distance (2451) from the 3D map view camera (1950) may adopt map tiles (first map tiles (2401)) of a first zoom level, and a second map area within a second distance (2452) that exceeds the first distance (2451) may adopt map tiles (second map tiles (2402)) of a second zoom level that is a lower zoom level than the first zoom level. And, a third map area that exceeds the second distance (2452) may adopt map tiles (third map tiles (2403)) of a third zoom level that is a lower zoom level than the second zoom level.

[0741] Here, the map area covered by a map tile with a lower zoom level can cover four times the area of ​​the map area covered by a map tile with a higher zoom level, twice in width and length. In addition, as the zoom level increases, the resolution of the included objects increases. Therefore, even for the same building object, the higher the zoom level of the map tile, the larger the building object can be and the more detailed its structure can be. On the other hand, the lower the zoom level of the map tile, the smaller the object contained in the map tile and the lower the resolution.

[0742] Additionally, when the above zoom level is changed, the types of map tile data included in the map tile may vary depending on the changed zoom level. That is, as described above, when the zoom level is lower than level 17, the map tile data of the map tile may not include data for general building objects.

[0743] In this case, as shown in (b) of FIG. 24, the first map tiles (2401) located within the first distance from the 3D map view camera (1950) may have the highest zoom level. And the first map tiles (2401) may cover the smallest map area.

[0744] On the other hand, the second map tiles (2402) located within the second distance but exceeding the first distance may have a lower zoom level than the zoom level of the first map tile (2401). Accordingly, the second map tile (2402) may cover a map area corresponding to a preset multiple of the first map tile (2401) (e.g., if the zoom level is 1 level lower, 2 times each horizontally and vertically).

[0745] Meanwhile, the third map tiles (2403) located at a distance exceeding the second distance may have a lower zoom level than the zoom level of the second map tile (2402). Accordingly, the third map tile (2403) may cover a map area corresponding to a preset multiple of the second map tile (2402) (e.g., if the zoom level is one level lower, twice the width and twice the height).

[0746] In this case, if the first map tile (2401) is a map tile having a zoom level of 17 or higher that includes data of a building object, modeling of the building object may not be performed for an area covered by the second map tile (2402) and the third map tile (2403) having a zoom level less than 17.

[0747] In addition, when adopting different zoom levels of map tiles according to the distance from the 3D map view camera (1950) in this way, although it is a map area of ​​the same size, multiple map tiles in some map areas are replaced with a single map tile having a lower zoom level (four map tiles of zoom level 17 are replaced with one map tile of zoom level 16), thereby reducing the number of map tiles included in the field of view of the 3D map view camera (1950) compared to using a fixed zoom level. That is, in Fig. 24 (a), modeling of objects can be performed from map tile data of 25 map tiles, but as shown in Fig. 24 (b), when adopting different zoom levels of map tiles according to the distance from the 3D map view camera (1950), modeling of objects can be performed with only map tile data of 14 map tiles. Accordingly, the number of map tile data required to generate a 3D map view can be reduced.

[0748] Meanwhile, map tiles with different zoom levels may contain different object data. For example, map tiles with a zoom level of 17 or higher may contain object data corresponding to general building objects and detailed roads (roads with a road width of less than a certain width). However, map tiles with a zoom level of less than 15 may only contain objects for main roads and terrain, and may not contain object data corresponding to the general building object data and detailed roads. In addition, the footprint area representing the area occupied by each building object may only be provided to map tiles with a zoom level of 15 or higher.

[0749] Accordingly, for areas where the distance from the 3D map view camera (1950) is greater than a certain level, the processor (1330) may select a map tile of a zoom level that does not include building objects. Then, modeling of the building objects may not be performed for areas where the distance from the 3D map view camera (1950) is greater than a certain level.

[0750] However, for areas located far from the 3D map view camera (1950), the building objects displayed on the screen become smaller due to perspective projection. Accordingly, by adopting map tiles with a low zoom level for areas far from the 3D map view camera (1950), the building objects displayed smaller can be modeled as objects with a lower resolution. This reduces the amount of computation required for object modeling.

[0751] In addition, in the case of an area located at a certain distance or more from the 3D map view camera (1950), building objects in the area are lost due to the distance from the 3D map view camera (1950). In this way, in the case of an area located at a certain distance or more from the 3D map view camera (1950), the 3D map view generation device (1300) according to an embodiment of the present invention can prevent modeling of objects that are not visible on the screen by adopting a map tile of a zoom level that does not include building objects. Accordingly, the amount of computation required for object modeling can be reduced.

[0752] Meanwhile, in the above description, it has been mentioned that the 3D map view generation device (1300) according to an embodiment of the present invention can provide a high-quality 3D landmark model. For example, the model data of the 3D landmark model may be included only in the map tile data of map tiles having zoom levels 15 and 16.

[0753] Accordingly, for map tiles with zoom levels of 15 and 16, the map tiles may include data of main roads and terrain objects and model data of the landmark model. Therefore, when included in the field of view of the 3D map view camera (1950), the model data of the landmark model may be detected from map tiles with zoom levels of 15 and 16 among the map tiles detected as being included in the field of view, and modeling and rendering of the landmark model may be performed following modeling and rendering of the road and terrain objects.

[0754] However, the zoom level of the map tile in which the landmark model is provided may be different from the zoom level of the map tile in which the model of the general building object is provided. In this case, the landmark model may be provided only for map tiles having a lower zoom level than the zoom level of the map tile in which the model of the general building object is provided. For example, the zoom level of the map tile in which the model of the general building object is provided may be level 17, and the zoom levels of the map tile in which the landmark model is provided may be levels 15 and 16. In this case, when the zoom level of the map tile in which the landmark model is provided and the zoom level of the map tile in which the model of the general building object is provided are different, as described above, when the zoom level of the map tile is varied according to the distance from the 3D map view camera (1950), the landmark model may not be rendered for an area in which the distance from the 3D map view camera (1950) is short.

[0755] Accordingly, if there is a map tile including a landmark model among the map tiles included in the field of view of the 3D map view camera (1950) according to an embodiment of the present invention, the processor (1330) of the 3D map view generation device (1300) can maintain the rendered state of the landmark model extracted from the corresponding map tile regardless of the change in the zoom level of the map tile. That is, by separating the landmark model and models of other general building objects from the corresponding map tile, the models of general building objects are modeled and rendered at different resolutions according to the change in the zoom level of the map tile covering the corresponding area (e.g., the zoom level increases when a vehicle or camera approaches, and decreases when the zoom level decreases when the vehicle or camera moves away), but the modeled and rendered state of the landmark model can be maintained regardless of the change in the zoom level of the map tile covering the corresponding area.

[0756] Accordingly, the landmark model can be continuously displayed on the display unit regardless of changes in distance due to movement of the vehicle or 3D map view camera (1950).

[0757] That is, as described above, when the processor (1330) lowers the zoom level of the map tile as the distance from the vehicle or the 3D map view camera (1950) increases, the zoom level of the map tile may be lowered in an area where the distance from the vehicle or the 3D map view camera (1950) is a certain distance or more, so that the zoom level of the map tile in which the model of the building object is supported may be lower than the zoom level of the map tile. Then, modeling and rendering of the building object may not be performed in the area.

[0758] Meanwhile, as described above, when the landmark model is supported only in map tiles that are less than the zoom level of the map tile in which the model of the building object is supported, the processor (1330) may perform modeling and rendering of the landmark model even if the modeling and rendering of the building object is not performed in an area that is a certain distance or more away from the vehicle or the 3D map view camera (1950). In addition, the rendered landmark model may be displayed based on a perspective projection method according to the distance away from the vehicle or the 3D map view camera (1950). In addition, the processor (1330) may consider the zoom level of the map tile for the rendered landmark model to be fixed, separate from other objects of the corresponding map tile.

[0759] Therefore, when a vehicle or a 3D map view camera (1950) approaches, for example, even if the zoom level of a map tile including a landmark model changes to the zoom level of a map tile that does not support the landmark model, the zoom level of the map tile may not change for the rendered landmark model. Therefore, even if the zoom level of a map tile covering an area including the landmark model increases to a zoom level that a model of a building object is supported due to the proximity of the vehicle or the 3D map view camera (1950), the rendered landmark model may remain rendered. Accordingly, a 3D map view in which the landmark model is aligned on a map object together with other surrounding building objects that are modeled and rendered can be provided.

[0760] Meanwhile, (b) of the above-described FIG. 24 has been described as an example in which the zoom ratio of the second map tiles (2402) is one higher than the zoom ratio of the third map tiles (2403), and the zoom ratio of the second map tiles (2402) is one higher than the zoom ratio of the first map tiles (2401), but this is merely an example of the present invention, and the present invention is not limited thereto. That is, the processor (1330) can select a zoom level that covers a map area of ​​a distance between the first street (2451) and the second street (2452), i.e., a map area of ​​a size that best matches the size of the second area, as the zoom level of the second map tile (2402).

[0761] For example, the processor (1330) can detect a zoom level having a largest value that exceeds the distance between the first distance (2451) and the second distance (2452) in the horizontal or vertical length of the covered map area, and can adopt a zoom level having a value that is one greater than the detected zoom level as the zoom level of the second map tile (2402).

[0762] Referring to (b) of FIG. 24, the zoom level having the largest value among the map tiles whose horizontal or vertical length of the covered map area exceeds the distance between the first distance (2451) and the second distance (2452) may be 15 (when the zoom level is 16, the horizontal or vertical length of the covered map area is less than or equal to the distance between the first distance (2451) and the second distance (2452)). Therefore, the processor (1330) may select a zoom level that is one greater than the detected zoom level of 15 as the zoom level of the second map tile (2402). Accordingly, the processor (1330) may select the zoom level of the map tile having the largest size of the covered map area without exceeding the second area between the first distance (2451) and the second distance (2452), and may model virtual objects by parsing the map tile data according to the selected zoom level.

[0763] Meanwhile, FIG. 25a is an exemplary diagram showing an example of texture images being tiled on modeled building virtual objects in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0764] When the building virtual objects determined as rendering targets (hereinafter referred to as rendering target virtual objects) in step S1308 of the above-described FIG. 13 are determined, the processor (1330) may tile a preset texture image in each side area for each of the building virtual objects among the determined virtual objects. In this case, the processor (1330) may detect a side area of ​​the building virtual object excluding a preset offset area (or margin area) inward from each side edge of the building virtual object, and may tile the texture image in the detected side area.

[0765] First, referring to (a) of FIG. 25a, the processor (1330) can determine the left and right margins (2511, 2512) of the side area of ​​the building virtual object determined as the rendering target according to the preset left and right margin values. According to the left and right margin values, the coordinate range of the area (2510) where the texture image is to be tiled in the horizontal direction can be determined.

[0766] For example, if the left and right margin values ​​are 0.1, the lower left point of the side area of ​​the building virtual object (2520) can be defined as (0, 0) and the upper right point can be defined as (1, 1). Then, as shown in Fig. 25a, the width of the tiling area (2510) excluding the left and right margins can be calculated as 0.8. In addition, if the building width of the actual building corresponding to the building virtual object (2520) is 14 m, the actual horizontal width of the tiling area (2510) can be calculated as 11.2 m.

[0767] Meanwhile, the texture image (2500) may be a normalized tile image and may have a preset width. In this case, if the horizontal width of the texture image (2500) is 4 m, dividing the actual horizontal width of the tiling area (2510), 11.2 m, by the horizontal width of the texture image (2500), 4 m, a value of 2.8 may be calculated. In this case, since the number of texture images (2500) is an integer, a value of 3 may be calculated when rounding is performed, and the calculated value 3 may be the number of texture images (2500) to be tiled in the tiling area (2510) of the side area of ​​the building virtual object (2520) set according to the preset left and right margin values. And the processor (1330) inputs (0.1, 0.9), which are the left and right X-axis coordinates of the tiling area (2510), as input values ​​of the remap function and inputs (0, 3) as a new range according to the remap function, so that up to three tiles (texture images (2500)) can be successively attached to the tiling area (2510) in the horizontal direction in the tiling area (2510).

[0768] In addition, the processor (1330) can set multiple partial regions based on the bottom layer, which is the first floor, according to preset upper and lower margin values. For example, the processor (1330) can set a floor divider corresponding to the space between the bottom layer and the upper layer of the partial region. In this case, the space between the set floor dividers based on the bottom layer can be determined as the partial region where texture images are to be tiled.

[0769] Here, the side region of the building virtual object may include multiple subregions. For example, the side region of the building virtual object may be divided into a bottom region (2552), a middle region (2550), and a top region (2551), and at least one of the bottom region (2552), the middle region (2550), and the top region (2551) may be further divided into multiple regions. In this case, the processor (1330) may tile different texture images for each of the multiple divided regions.

[0770] In this case, the processor (1330) can obtain a mask map corresponding to each sub-region based on the UV coordinate system corresponding to each sub-region on each building side. For example, the processor (1330) can define the lower left point of the side region of the building virtual object as (0, 0) and the upper right point as (1, 1). Then, by defining the Y-axis coordinate in the UV coordinate system according to the margin value of the sub-region in which the layer division is formed, the vertical width corresponding to the sub-region can be defined, and the mask map corresponding to the sub-region can be obtained according to the defined vertical width. In this case, the mask map corresponding to the sub-region can be a mask map in which a height lower than the Y-axis coordinate according to the margin value of the sub-region has a value of 0, and a height higher than the Y-axis coordinate has a value of 1. In this case, when blending with a normal map is performed, a part having a value of 0 maintains a value of 0, but a part having a value of 1 can have a 3D vector value of the normal map.

[0771] As an example of such a partial region, if the margin value of the partial region where the layer division is formed is 0.8, the processor (1330) can determine the coordinates of the upper right point of the partial region as (1, 0.8). Then, the processor (1330) can obtain a mask map in which an area corresponding to a height less than 80% of the total height of the building virtual object from the floor surface of the building virtual object has a value of 0, and an area corresponding to a height greater than that has a value of 1. That is, an area corresponding to the upper edge of the building virtual object from a height of 80% or more of the total height of the building virtual object can be formed as a partial region (e.g., a tower region).

[0772] Meanwhile, the processor (1330) can tile different texture images for each sub-region (2550, 2551, 2552) of the building virtual object (2560) in the same manner as the method described in (a) of FIG. 25A. In this case, as shown in (b) of FIG. 25A, the top region (2551) can be tiled with the first texture image (2501), the bottom region (2552) can be tiled with the second texture image (2502), and the middle region (2550) can be tiled with the third texture image (2500). Accordingly, the building virtual object (2650) can be tiled with different texture images for each different sub-region.

[0773] In this way, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can diversify the exterior shape (facade design) of the building virtual object by dividing each building virtual object into a plurality of different regions and tiling different texture images in each of the divided regions.

[0774] FIG. 25b illustrates examples of facade designs of a building virtual object expressed in various ways by using different partial regions and tiled texture images in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0775] First, Figure 25b (a) illustrates an example of a building virtual object formed by three sub-regions: a ground floor, a middle floor, and a top floor. Figures 25b (b) and (c) illustrate examples of a building virtual object formed by only the ground and middle floors. In this way, by varying the number of sub-regions, the building virtual object can be expressed to have different exterior shapes.

[0776] In addition, the processor (1330) can vary the texture images tiled in each sub-region for each building virtual object. In this case, since not only the number of sub-regions but also the tiled texture images vary, the number of possible combinations can increase. Therefore, a more diverse exterior representation of the building virtual object can be achieved.

[0777] Meanwhile, the processor (1330) can blend normal maps having 3D vector values ​​corresponding to different materials for each building element in step S1312 of FIG. 13 so that the images can be displayed with different materials for each building element included in the tiled texture image of the building virtual object during rendering.

[0778] FIG. 26 is a flowchart illustrating in more detail the texturing process of step S1312 of FIG. 13, which forms different surface textures for each component of a building in a 3D map view generation device (1300) according to an embodiment of the present invention. FIG. 27 is an exemplary diagram illustrating an example of normal maps blended to mask maps corresponding to different building elements and a multi-blended normal map generated by blending normal maps together, according to the process described in FIG. 26.

[0779] First, referring to FIG. 26, the processor (1330) can identify and distinguish building elements included in each texture image for each texture image tiled in the tiling area of ​​the building virtual object (S2600).

[0780] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different areas of the texture image by each color value, and accordingly, different areas of the texture image, i.e., different building elements, can be distinguished by color channels. For example, if the texture image includes at least one window element, the window element may have different color values ​​from the rest of the building background. In addition, if the window element includes a glass element corresponding to a glass area, a window frame element corresponding to a window frame area, and a blind element corresponding to a blind area, the glass element, the window frame element, and the blind element may each have different color values. Therefore, the processor (1330) can distinguish and identify areas corresponding to a plurality of different building elements included in the texture image based on the color values. In this case, the texture image may be referred to as a labeling map, which is a map image that labels a plurality of different building elements.

[0781] Meanwhile, when each building element included in the texture image is distinguished and identified, the processor (1330) can generate mask maps corresponding to each building element (S2602).

[0782] In this case, the mask map corresponding to a specific building element may be an image having a value of 1 in an area corresponding to a specific building element identified by a specific color value in the texture image, and a value of 0 in the remaining areas.

[0783] For example, if the texture image includes at least one window element, the mask map corresponding to the window element may be a mask map that has a value of 1 only in the area corresponding to the window element and a value of 0 in the remaining area. On the other hand, the mask map corresponding to the building wall may be a mask map that has a value of 1 only in the building wall portion, i.e., in the remaining area of ​​the texture image excluding the window element, and a value of 0 in the remaining area.

[0784] In addition, if the window element includes the glass element, the window frame element, and the blind element, the processor (1330) can generate a glass mask map in which only the area corresponding to the glass element in each window element has a value of 1 and the rest has a value of 0, a window frame mask map in which only the area corresponding to the window frame element in each window element has a value of 1 and the rest has a value of 0, and a blind mask map in which only the area corresponding to the blind element in each window element has a value of 1 and the rest has a value of 0.

[0785] When mask maps corresponding to each building element are generated for each building element included in the tiled texture image in this way, the processor (1330) can blend each generated mask map with a normal map having a 3D vector value corresponding to a different material (S2604). In this case, the normal map means an image having a 3D vector value indicating a 3D direction of the surface of an object required for calculating the degree of shading corresponding to the reflected light reflected by the surface curvature of the specific material when light incident on the specific material is incident on the specific material, in a graphic processing device, and may be a 2D image capable of expressing the texture of the specific material through the degree of shading, i.e., the shading effect.

[0786] The blending of these normal maps and mask maps may be a process similar to a multiplication process. Therefore, when the normal map and the mask map are blended, the 3D vector value applied to the normal map may be applied only to the area that is not masked with a value of 0 in the mask map, i.e., only to the area corresponding to a specific building element. Therefore, only the area corresponding to a specific building element corresponding to the mask map may have a texture according to a specific material expressed. In this case, the processor (1330) may blend the mask map corresponding to each building element with a normal map having a 3D vector value of a different material, thereby allowing each building element to have a texture of a different material.

[0787] In the above step S2604, when each mask map is blended with each normal map, the processor (1330) can further blend an albedo map having a different color for each mask map and the blended normal map. In this case, the albedo map may be an image having a color value unique to a specific material. In addition, a pair may be formed with a normal map having a 3D vector value corresponding to the texture of a specific surface material.

[0788] Then, the processor (1330) can further blend the albedo map forming a pair with the normal map for each normal map blended with the mask map (S2606). In this case, blending with the albedo map can be applied similarly to the multiplication process as with blending with the normal map. Therefore, a color value can be applied only to a part having a value of 1 in the mask map, and accordingly, a normal map capable of expressing the texture of a specific material and the unique color of the material can be generated for a specific building element.

[0789] Meanwhile, in step S2606, when the mask maps and blended normal maps corresponding to each component of the building are blended with the albedo maps, the processor (1330) can blend the blended normal maps of the building elements with each other (S2608). In this case, the blending of the normal maps may be a union concept, which may mean a process of combining the areas to which 3D vector values ​​(and color values ​​according to the albedo map) are applied between the normal maps into a single map image, a multi-blended normal map. Therefore, when the blending process of step S2606 is completed, a normal map (multi-blended normal map) to which 3D vector values ​​(and albedo color values) of different normal maps are applied for each building element included in a single texture image may be generated.

[0790] And the processor (1330) can perform texturing on the side area of ​​the building virtual object by synthesizing the multi-blending normal map into the tiling area among the side areas of the building virtual object (S2610).

[0791] In this case, the multi-blending normal map may be generated for the entire tiled area in which texture images are tiled in the side area or partial area of ​​the building virtual object. In this case, the processor (1330) may synthesize the multi-blending normal map generated in a size corresponding to the entire side area or partial area of ​​the building virtual object onto the side area or partial area of ​​the building virtual object.

[0792] Alternatively, the multi-blending normal map may be generated for any one texture image tiled in a side area or partial area of ​​the building virtual object. In this case, the processor (1330) may synthesize the multi-blending normal map for the side area or partial area of ​​the building virtual object by tiling the multi-blending normal map for each of the texture images tiled in the side area or partial area of ​​the building virtual object.

[0793] Meanwhile, FIG. 27 illustrates an example of generating a multi-blending normal map including areas to which different 3D vector values ​​corresponding to each of a plurality of building elements included in a texture image are applied according to the operation process described in FIG. 26.

[0794] Referring to FIG. 27, the processor (1330) can distinguish and identify each building element from a texture image (2500) including a window element. In this case, if the window element includes a glass element and a window frame element (2702), the processor (1330) can identify the wall element (2701), the glass element, and the window frame element (2702) from the texture image (2500).

[0795] And the processor (1330) can generate a mask map corresponding to each identified building element. In this case, as shown in FIG. 27, a mask map (2711) corresponding to the wall element can be generated in which only the area corresponding to the wall element (2701) has a value of 1 and the remaining areas, for example, the areas corresponding to the glass element and the window frame element (2702), have a value of 0. In addition, a mask map (2712) corresponding to the window frame element can be generated in which only the area corresponding to the window frame element (2702) has a value of 1 and the remaining areas, for example, the areas corresponding to the glass element and the wall element (2701), have a value of 0.

[0796] And the processor (1330) can blend the first normal map (2721) and the second normal map (2722), which are map images to which 3D vector values ​​corresponding to different materials are applied, for each generated mask map (2711, 2712). In this case, the first normal map (2721) may be an image to which a 3D vector value capable of expressing the first material with a shading effect is applied, and the second normal map (2722) may be an image to which a 3D vector value capable of expressing the second material with a shading effect is applied.

[0797] Then, the 3D vector value of the blended normal map can be synthesized only for the area having the value 1, excluding the area having the value 0 in each mask map. Therefore, when the mask map (2711) corresponding to the wall element and the first normal map (2721) are blended, a normal map can be generated in which a 3D vector value that can express the first material only in the area corresponding to the wall element in the texture image (2500) is synthesized, as shown in FIG. 27. In addition, when the mask map (2712) corresponding to the window frame element and the second normal map (2722) are blended, a normal map can be generated in which a 3D vector value that can express the second material only in the area corresponding to the window frame element in the texture image (2500) is synthesized, as shown in FIG. 27.

[0798] Then, the processor (1330) can blend a normal map in which 3D vector values ​​that can express a first material only in an area corresponding to the wall element are synthesized, and a normal map in which 3D vector values ​​that can express a second material only in an area corresponding to the window frame element are synthesized. In this case, blending is performed between normal maps, and each normal map can be synthesized into a single multi-blended normal map. Therefore, as shown in FIG. 27, a multi-blended normal map (2750) can be generated in which 3D vector values ​​that can express a first material in an area corresponding to the wall element (2701) are synthesized, and 3D vector values ​​that can express a second material in an area corresponding to the window frame element (2702) are synthesized. That is, a normal map in which 3D vector values ​​that can express different materials in each area corresponding to each building element included in the texture image are synthesized can be generated.

[0799] Meanwhile, in the description of the above-described Fig. 26, it was described that the albedo map having the material's own color is blended into the normal map to which the mask map is blended, thereby mapping the material's own color to each building element. However, in contrast to this, the colors of virtual objects within the visible area can also be determined as colors similar to the colors of buildings within the actual visible area based on the image captured from the camera of the vehicle (100).

[0800] FIG. 28 is a flowchart illustrating an operation process of mapping the colors of virtual building objects based on the colors of actual buildings within a visible area in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0801] Referring to FIG. 28, when step S2604 of blending the mask maps generated for each building element in FIG. 26 into a normal map having 3D vector values ​​corresponding to different materials is performed, the processor (1330) can acquire an image of the surroundings of the vehicle (100) through the camera of the vehicle (100) (S2800). In this case, the camera may be a camera that captures an image of the front area of ​​the vehicle (100), and the acquired image may be an image of the front of the vehicle (100) acquired according to the driving direction of the vehicle (100), i.e., an image of the visible area.

[0802] And the processor (1330) can identify each object included in the acquired image and partition the acquired image into multiple regions according to the type of each identified object. That is, the processor (1330) can partition the regions where each object included in the acquired image is displayed on the image into different regions for each object (S2802).

[0803] For example, if the acquired image is an image including trees, buildings, the sky, and a road, the processor (1330) may, in step S2802, segment the image into an area where each tree is displayed, an area where each building is displayed, an area where the sky is displayed, and an area where the road is displayed. Then, a dominant color may be detected in each segmented area, and each area may be displayed with the detected dominant color. In this case, an image may be acquired in which the area where each object included in the image is displayed is segmented with the dominant color in each area.

[0804] To this end, the processor (1330) may utilize an artificial intelligence module (not shown) trained through preset learning data. In this case, the training may be training to identify each region on the image as a different object based on shape and color. Accordingly, the artificial intelligence module, upon completion of the training, may identify a region of the input image as belonging to one of the preset object types based on the pattern formed by the shape and color displayed in the region.

[0805] And the processor (1330) can classify the areas on the image divided by each object by type of the object. And a representative color can be determined from the areas on the image divided by type of each object (S2804).

[0806] In this case, one or more areas on the segmented image can be classified as having the same object type. For example, an area where each building is displayed on the image can be classified into areas corresponding to each building, and a dominant color can be detected for each segmented area. In this state, the processor (1330) can classify each area on the image classified by each object according to the type of each object. Accordingly, at least one area identified as a 'building' can be classified as an area corresponding to the object type 'building'. In addition, the processor (1330) can determine a representative color corresponding to the object type 'building' based on the color of each of the plurality of areas classified as areas corresponding to the object type 'building'.

[0807] For example, the processor (1330) may determine the color with the largest proportion among the colors of each of the areas corresponding to the object type 'building' as the representative color corresponding to the object type 'building'. Alternatively, the processor (1330) may calculate the average color of the colors of each of the areas corresponding to the object type 'building' and determine the calculated average color as the representative color corresponding to the object type 'building'.

[0808] When a representative color corresponding to the object type 'building' is determined, the processor (1330) can map at least some of the building elements of the building virtual object within the visible area to the determined representative color (S2806). For example, the processor (1330) can blend at least some of the normal maps blended to the mask maps corresponding to each of the different building elements with an albedo map having a representative color corresponding to the object type 'building'.

[0809] Meanwhile, the processor (1330) can parse geometric data and attribute data of a building object from data of map tiles included in the viewing area. In this case, the attribute data may include the area on the ground occupied by the building object, i.e., footprint information and height information, and the processor (1330) can model the building object based on the footprint information and height information, render the modeled object, and display it on the display unit.

[0810] Meanwhile, when a building object is formed based on this footprint information, the building object can be formed in a form that is extruded from the ground according to the shape of the footprint area. In this case, since the ground has a flat shape, the building object can be modeled with a flat roof shape.

[0811] Accordingly, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention further defines roof shapes as attribute information for various building objects, and further includes roof shapes corresponding to various building objects as geometric data, thereby adding roof shapes to building objects modeled in a flat form to generate a building object model more similar to an actual building, thereby enabling the 3D map view to be expressed more realistically.

[0812] FIG. 29 and FIG. 30 illustrate examples of roof objects for expanding roof modeling of building objects in the 3D map view generation device (1300) and examples of textures for the roof objects.

[0813] Referring to Fig. 29, Fig. 29 shows examples of various roof shapes that have been modeled in advance. In order to model a building object including such roof shapes, information on a roof shape corresponding to the building object may be further defined in the attribute information corresponding to the building object. To this end, the processor (1330) may detect the attribute information of the building object by parsing map tile data, determine the roof shape corresponding to the building object, and detect the roof shape corresponding to the determined roof shape from geometric data. For example, the processor (1330) may detect location information of an area in which a vehicle is currently driving and determine a specific roof shape based on the detected location information. In this case, the detected roof shape may be any one of the roof shapes illustrated in Fig. 29.

[0814] For example, if the cur...

Claims

In the 3D map view generation device, An interface unit that receives sensing information collected from at least one sensor equipped in a vehicle; A memory storing map data for a plurality of map tiles including map tiles containing landmark objects; and Controlling the interface unit to receive driving information of the vehicle including the location, speed, and direction of movement of the vehicle, modeling building objects included in map data of a plurality of map tiles determined based on the driving information, and generating a map view image including building objects rendered by tiling a plurality of texture images including shapes of different building elements on each side of the modeled building objects, A 3D map view generation device characterized in that it includes a processor that, when there is a map tile including a landmark object among the plurality of map tiles, models and renders the landmark object with priority over the building objects, and performs modeling and rendering for the building objects after the landmark object is rendered. In the first paragraph, the processor, A 3D map view generation device characterized in that it first models and renders road and terrain objects included in map data of a plurality of map tiles determined based on the above driving information, and then models and renders the landmark object after the road and terrain objects are rendered. In the second paragraph, the processor, A 3D map view generation device characterized in that, when the vehicle approaches within a certain distance or less while landmark objects included in map data of a plurality of map tiles are rendered based on the driving information, some of the building objects included in the map data of the plurality of map tiles are modeled and rendered, and when the vehicle moves away beyond a certain distance, the landmark objects are maintained in a rendered state and the rendering of some of the rendered building objects is deactivated. In the third paragraph, the processor, A 3D map view generation device characterized in that it places the modeled landmark object and building object under a portion of the rendered road and terrain object corresponding to the footprint area of ​​each of the landmark object and building object, moves the modeled landmark object or building object in the sky direction under the road and terrain object, and sequentially renders the modeled landmark object or building object from a portion exposed above the rendered road and terrain object according to the movement. In the fourth paragraph, the processor, A 3D map view generation device characterized in that the height of at least one building object located between a 3D map view camera providing the 3D map view field and the landmark object is at least partially limited. In the fifth paragraph, the processor, A 3D map view generation device characterized in that, for at least one building object located between the 3D map view camera and the landmark object, a height to be exposed and rendered above the road and terrain objects is determined based on the height of the building object, the height of the landmark object, and the height of the 3D map view camera. In the sixth paragraph, the processor, A 3D map view generation device characterized in that the height at which at least one building object, located between the 3D map view camera and the landmark object, covers at least a portion of the landmark object within the exposure range, among the building objects positioned between the 3D map view camera and the landmark object, is exposed above the road and terrain objects and is rendered, so that the landmark object is displayed on the screen beyond a preset exposure range set from the top of the landmark object, is limited. In the first paragraph, the processor, Designate at least one building object among the building objects included in the plurality of map tiles determined based on the above driving information as a custom landmark object, A 3D map view generation device characterized in that, among a plurality of map tiles determined based on the driving information, if there is a map tile including the custom landmark object, the custom landmark object is modeled and rendered with priority over other building objects, and modeling and rendering of the other building objects are performed after the landmark object is rendered. In the 8th paragraph, the processor, The user's selection, the building object of the building set as a waypoint or destination on the driving route of the vehicle or a guide indicator for the driving route, and the building object corresponding to a specific POI (Point Of Interest) are designated as the custom landmark object. A 3D map view generation device characterized in that, when the custom landmark object is located within a preset distance from the vehicle, the rendering of at least one of the building objects around the custom landmark object and the custom landmark object is changed so that the building objects around the custom landmark object are displayed differently from the custom landmark. In the 9th paragraph, the processor, If the above vehicle approaches within the above preset distance, Highlighting at least one of the sides of the building object designated as the custom landmark or changing any of the colors or brightness of the texture images tiled on the sides of the building object designated as the custom landmark object, Alternatively, a 3D map view generation device characterized in that it limits the tiling of texture images for at least some of the sides of building objects around the custom landmark object, or limits the height at which building objects around the custom landmark object are exposed and rendered above the road and terrain objects. In the first paragraph, The above memory is, Contains map data of map tiles of different sizes according to different zoom levels, The above processor, A 3D map view generation device characterized in that map data including object data for generation of the above map view image are detected from map tiles having different zoom levels according to the distance from the vehicle. In Article 11, The above processor, The map data is detected from map tiles having a lower zoom level as the distance from the vehicle increases, The above map data are, A 3D map view generation device characterized in that it includes object data for modeling objects having different resolutions depending on the zoom level of the corresponding map tile, or the types of object data it includes are different. In the 12th paragraph, the processor, Adopting a first zoom level for a first map area having a distance from the vehicle less than or equal to a first distance, and detecting first map data from first map tiles having the first zoom level, A second zoom level is adopted for a second map area that exceeds the first distance and is less than or equal to the second distance, and second map data is detected from second map tiles having the second zoom level. For a third map area exceeding the second distance, a third zoom level is adopted, and third map data is detected from third map tiles having the third zoom level. The above second map tiles or third map tiles, A 3D map view generation device characterized in that it includes data for a footprint area of ​​each building object instead of building object data for modeling the building object. In the 13th paragraph, the processor, A 3D map view generation device characterized in that when the zoom level of a map tile for a map area around the vehicle changes as the distance from the vehicle changes, building objects with different resolutions are displayed according to map data of map tiles corresponding to the changed zoom level, or each building object is displayed in the shape of a footprint area representing the area occupied by the building object. In Article 13, The above landmark object is, It is included in the map data of the map tile corresponding to some of the zoom levels among the first to third zoom levels, The above processor, A 3D map view generation device characterized in that, when the landmark object is modeled and rendered from a map tile including the landmark object among a plurality of map tiles determined based on the driving information of the vehicle, the rendering state of the landmark object is maintained regardless of the zoom level of the changed map tile. In the first paragraph, The map data of multiple map tiles determined based on the above driving information are: It further includes geometric data of multiple roof shapes with different shapes, The above processor, A 3D map view generation device characterized in that a roof shape corresponding to each building object is further determined, and when modeling and rendering the building objects, the roof shape determined for each building object is modeled and rendered together. In the 16th paragraph, the processor, A 3D map view generation device characterized in that it detects the roofs of a plurality of buildings from satellite images of an area in which the vehicle is driving or images obtained from a camera of the vehicle, determines the roof shape based on the shape of the detected roofs, and determines a texture image and color to be tiled to the roof shape based on the surface shape and color of the detected roofs. In Article 17, The above memory is, Containing at least one roof shape, at least one texture image, and at least one color information for a plurality of different regions, The above processor, A 3D map view generation device characterized in that it detects at least one roof shape, at least one texture image, and at least one color information corresponding to information of an area in which the vehicle is driving, and performs modeling and rendering of the roof shape of each building object based on the detected at least one roof shape, texture image, and color information. In Article 18, The above memory is, Each region contains multiple color palettes, each containing different colors and adoption probabilities for each color depending on the time, season or weather. The above processor, A 3D map view generation device characterized in that it determines one of a plurality of color palettes provided for an area in which the vehicle is driven based on at least one of the time or season in which the vehicle is driven or the weather around the vehicle, and determines the colors of the roof shape of each building object based on the probability of adoption of each color included in the determined color palette. In a control method of a 3D map view generation device, A step of receiving driving information of a vehicle; A step of determining at least one map tile included in a field of view based on the driving information and detecting map data of the determined map tile; A step of checking whether a landmark object is included in the determined map tile based on the detected map data; A step of modeling and rendering a landmark object based on the above check result, giving priority to other building objects of the determined map tile; After the landmark object is modeled and rendered, a step of modeling other building objects of the determined map tile; A step of determining some of the modeled building objects as rendering targets based on the driving information; A step of receiving a user request to display a map view image; In response to the user request, a step of tiling each side area of ​​the determined rendering target building objects with a plurality of texture images including shapes of different building elements; A step of texturing the determined rendering target building objects by blending normal maps to which vector values ​​for forming textures of different materials are applied for each building element identified from the above tiled texture images; and, A control method of a 3D map view generation device, characterized in that it includes a step of displaying a rendered map view image of the landmark objects and the textured building objects on a display unit of the vehicle.

Images