Device for generating 3D map view on basis of mixed reality, and control method of device

Abstract

The present invention comprises: an interface unit that receives sensing information collected from a sensor provided in a vehicle; a memory storing map data for map tiles; and a processor that controls the interface unit to receive driving information of the vehicle, models building objects included in the map data determined on the basis of the driving information, generates texture images including the shapes of different building elements, and generates a 3D map view image including rendered building objects by tiling the texture images onto respective side surfaces of models of the modeled building objects, wherein the processor analyzes detailed features from multimedia content reproduced in the vehicle, and displays each building object such that at least one of the detailed features is linked to at least one of the brightness, color, and material of a specific building element and varies according to the reproduction of the multimedia content.

Description

A device for generating a 3D map view based on mixed reality and a method for controlling the device

[0001] The present invention relates to a device for generating a 3D map view based on mixed reality.

[0002] Recently, following Augmented Reality (AR), which outputs graphic objects through a vehicle's windshield or HUD (Head Up Display), or uses images captured by a camera to additionally output graphic objects in the real world, technology development related to Mixed Reality (MR), which can provide various simulation information related to vehicles by applying Digital Twin (DT) technology, is actively underway.

[0003] As part of the technological development related to this mixed reality, active research is being conducted on methods that utilize mixed reality to provide drivers with route guidance-related information. Route guidance using mixed reality, for example, utilizes digital twin technology to display graphical objects corresponding to vehicles on a digitized 3D map. This map and graphical objects can be used to provide information on driving routes the driver has not yet driven, or provide a bird's-eye view, offering various information that the driver in the cockpit cannot view.

[0004] Meanwhile, for drivers to obtain accurate information through route guidance using mixed reality, the digital twin technology used to create the map—the digital twin map—must have a high degree of consistency with the real world. This requires the provision of a 3D map, not a 2D one. Therefore, service providers currently offering digital twin maps are offering 3D maps, which are essentially 2D maps mapped into 3D.

[0005] However, current service providers rely on building modeling using satellite or aerial photography to provide 3D mapping. However, the high angles at which satellite or aerial photography is used make it difficult to capture images of various aspects of a building, making it difficult to provide accurate textures. Therefore, conventional 3D maps are provided in the form of polygon data containing the building's floor coordinates (footprint) and height.

[0006] Meanwhile, 3D maps containing models provided in the form of polygon data like this have the problem of not including accurate building textures. These polygonal maps only represent the location and 3D volume of buildings, differing from the actual building shapes. This discrepancy creates a gap between the digital twin map and the real world, making it difficult to provide accurate information to drivers.

[0007] The present invention aims to solve the above-mentioned problems and other problems.

[0008] One object of the present invention is to provide a 3D map view generation device capable of displaying virtual objects having high-quality texture materials similar to real-world buildings in real time without delay according to changes in the driving condition of a vehicle, and a method for controlling the device.

[0009] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof capable of providing a more realistic mixed reality-based route guidance service through a digital twin map including virtual objects with more realistic textures corresponding to the shape of each side of a building.

[0010] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof that provide a user UX that can visually provide enjoyment to the user by utilizing each side of a building virtual object as a media facade according to a multimedia environment within a vehicle.

[0011] In addition, another object of the present invention is to provide a 3D map view generation device and a control method thereof that can provide information for displaying a vehicle or the environment around the vehicle more intuitively or guiding route information by utilizing each aspect of a building virtual object.

[0012] According to an embodiment of the present invention, a 3D map view generation device includes an interface unit that receives sensing information collected from at least one sensor provided in a vehicle, a memory that stores map data for a plurality of map tiles, and a processor that controls the interface unit to receive driving information of the vehicle, models building objects included in the map data of a plurality of map tiles determined based on the driving information, generates a plurality of texture images including shapes of different building elements, and generates a 3D map view image including rendered building objects by tiling the texture images on each side of the modeled building object models, wherein the processor analyzes a plurality of detailed features from multimedia content played back in the vehicle, and displays each building object so that the specific building element changes for each building object according to the playback of the multimedia content by linking at least one of the analyzed detailed features to at least one of brightness, color, and material of a specific building element.

[0013] In one embodiment, the processor is characterized in that it determines which of the analyzed detailed features is associated with any one of the brightness, color, and material of the specific building element, based on a random number generated based on the unique coordinates of each building object.

[0014] In one embodiment, the processor is characterized in that it divides the frequency band of the multimedia content being played into a plurality of bands, and links the amplitude of each divided frequency band to one of the brightness, color, and material of the specific building element as the detailed characteristic.

[0015] In one embodiment, the processor searches for an image related to the multimedia content being played, and associates a plurality of representative colors detected from the searched image with one of the brightness, color, and material of the specific building element as the detailed feature, wherein the image related to the multimedia content is an album art image if the multimedia content is audio content, and is a screen on which the video content is played if the multimedia content is video content.

[0016] In one embodiment, the processor is characterized in that it changes at least one of the brightness or color of the specific building element by adding a brightness value according to at least one of the analyzed detailed features to a basic brightness set for the specific building element, or by blending a color according to at least one of the analyzed detailed features to a basic color set for the specific building element.

[0017] In one embodiment, the processor is characterized in that, for each building object, at least one of brightness, color, and material is to be linked to the detailed feature, and the specific building element is differently determined.

[0018] In one embodiment, the processor is characterized in that it displays video content on a portion of one side of a specific building object among the building object models, and corrects the color of an area surrounding a video display area where the video content is displayed among the one side of the specific building object to a representative color analyzed from the video content being played.

[0019] In one embodiment, the processor is characterized in that it divides the video display area into a plurality of areas based on specific building elements constituting one side of the specific building object, detects representative colors for each of the divided areas, and corrects the color of the specific building elements around each of the divided areas to the detected representative colors.

[0020] In one embodiment, the processor models a display area object in which tiling of the texture image is restricted and inserts the modeled display area object into a specific building object model, and inserts a 3D animation object in which at least one of movement, color, and shape changes according to detailed features analyzed from multimedia content played in the vehicle or a driving state of the vehicle, inside the inserted display area object, wherein at least one side of the display area object is exposed to one side of the specific building object model.

[0021] According to one embodiment of the present invention, a 3D map view generation device includes an interface unit that receives sensing information collected from at least one sensor provided in a vehicle, a memory that stores map data for a plurality of map tiles, and a processor that controls the interface unit to receive driving information of the vehicle, models building objects included in the map data of a plurality of map tiles determined based on the driving information, generates a plurality of texture images including shapes of different building elements, and generates a 3D map view image including rendered building objects by tiling the texture images to each side area of ​​the modeled building object models, wherein the processor is characterized in that it changes the brightness or color of at least some of specific building elements constituting one side of at least one of the building objects based on the collected sensing information or the driving information of the vehicle.

[0022] In one embodiment, the processor is characterized in that it detects an event occurring in the real world based on the collected sensing information, determines a graphic object related to the detected event, detects at least some of specific building elements constituting one side of one of the building objects according to the shape of the determined graphic object, and displays the detected building elements in a color or brightness that is distinct from other building elements, thereby displaying the graphic object related to the detected event on one side of one of the building objects.

[0023] In one embodiment, the processor is characterized in that it determines a media façade area where the graphic object can be displayed by excluding a building margin value from one side of the one building object, and substitutes a mask image corresponding to the graphic object into the determined media façade area to produce a shape of the graphic object with an aspect ratio changed according to an aspect ratio of the determined media façade area.

[0024] In one embodiment, the processor is characterized in that it detects at least some of the specific building elements constituting one side of the one building object along the shape of the graphic object included in the mask image assigned to the media facade area, and changes the color or brightness of at least some of the detected building elements to form a visual pattern corresponding to the shape of the graphic object.

[0025] In one embodiment, the processor checks whether a size of a media facade area determined from one side of the one building object is greater than or equal to a size according to a minimum scale value of the graphic object, and displays a graphic object related to the detected event on one side of the one building object according to a result of the check, wherein the minimum scale value of the graphic object is determined according to one of a type of the graphic object, a distance between a 3D map view camera providing the 3D map view and the one building object, or a display magnification of the 3D map view.

[0026] In one embodiment, the processor further detects at least one other building object adjacent to the one building object based on the check result, substitutes a mask image corresponding to the graphic object into a display area formed by one side of the one building object and at least one side of the at least one other building object, and changes the color or brightness of specific building elements detected from the side surfaces of the plurality of building objects based on the substituted mask image to form a visual pattern corresponding to the shape of the graphic object.

[0027] In one embodiment, the processor detects environmental information around an actual building corresponding to a specific building object, and displays brightness or color of at least one side of the specific building object differently according to the detected environmental information, wherein the specific building object is a preset landmark building object.

[0028] In one embodiment, the processor detects status information of the vehicle, and based on the detected status information of the vehicle, changes the number or ratio of building elements among specific building elements constituting one side of at least one building object, wherein at least one of brightness or color is displayed differently, and the at least one building object is characterized as a building object surrounding a vehicle object corresponding to the vehicle on the 3D map view.

[0029] In one embodiment, the processor detects route guidance information of the vehicle, and further displays additional information related to the detected route guidance information on at least one side of building objects surrounding a vehicle object corresponding to the vehicle on the 3D map view, wherein the additional information is characterized in that it is displayed as a pattern formed by some of the building elements forming one side of the building object that are formed with different colors or brightnesses.

[0030] In one embodiment, the processor is characterized in that, when the 3D map view is changed to a top view in which building objects are displayed in the shape of a footprint area, the processor displays the additional information in a footprint area corresponding to at least one of the building objects around the vehicle object.

[0031] In addition, a control method of a 3D map view generation device according to an embodiment of the present invention is characterized by including the steps of: receiving driving information of a vehicle; determining at least one map tile included in a field of view based on the driving information, detecting map data of the determined map tile; modeling building objects included in the detected map data; generating a plurality of texture images including shapes of different building elements; generating a 3D map view image including rendered building objects by tiling the generated texture images on each side of the modeled building object models; analyzing a plurality of detailed features from multimedia content played back in the vehicle; linking at least one of the analyzed detailed features to at least one of brightness, color, and material of a specific building element based on unique coordinates of each building object; and, in response to playback of the multimedia content, displaying at least one of the brightness, color, and material of the specific building element linked to the analyzed detailed feature differently for each building object.

[0032] The effects of a 3D map view generation device and a control method thereof according to an embodiment of the present invention are described as follows.

[0033] First, the present invention can pre-load map data of a region corresponding to an area around a vehicle based on the vehicle's location according to vehicle driving information from map data of a wide area, and model virtual objects corresponding to buildings and roads included in the loaded region in advance. In addition, by rendering only at least a portion of the modeled virtual objects according to the vehicle's driving information, it has the advantage of providing a digital twin map containing high-quality virtual objects similar to real-world buildings and roads in real time according to the vehicle's driving information.

[0034] Second, for at least some of the modeled building objects, at least one mask map corresponding to different parts of a texture image that shapes each part of the building, and a normal map, which is an image that is superimposed on the mask map and has 3D vector values ​​that form different textures, are superimposed on the texture image so that each component of the building shaped by the texture image has different materials, and the texture image in which the mask map and the normal map are superimposed to form a surface material can be tiled on each face of at least some of the building objects. Then, by rendering and displaying on a display unit at least some of the building objects in which the texture image is tiled, it is possible to perform texturing of the building objects with a small amount of computation while including building objects having high-quality texture materials similar to the actual building, and there is an advantage in that a digital twin map can be provided that includes a building model in which a high-quality texture similar to an actual building is synthesized.

[0035] Furthermore, the present invention allows each side of the building virtual object to be utilized as a media facade, where each side changes in response to changes in the multimedia environment by allowing the size and color of at least one of the building elements constituting each side of the building virtual object to change according to the multimedia environment within the vehicle. This has the effect of providing visual enjoyment to the user.

[0036] Furthermore, the present invention allows information regarding the vehicle's status, the surrounding environment, or route guidance to be displayed using at least one of the building elements constituting each side of a virtual building object surrounding the vehicle. This effectively conveys information regarding the vehicle's status, the surrounding environment, or route information to the user more intuitively and naturally.

[0037] FIG. 1 is a drawing showing the exterior of a vehicle according to an embodiment of the present invention.

[0038] Figure 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles.

[0039] Figures 3 and 4 are drawings showing the interior of a vehicle according to an embodiment of the present invention.

[0040] Figures 5 and 6 are drawings for reference in explaining an object according to an embodiment of the present invention.

[0041] FIG. 7 is a block diagram for reference in explaining a vehicle according to an embodiment of the present invention.

[0042] Figure 8a is a conceptual diagram for explaining the AR service platform of the present invention.

[0043] Figure 8b is a conceptual diagram for explaining an MR service platform for providing the MR service of the present invention.

[0044] Figure 8c is a conceptual diagram for explaining the MR AMS client of the present invention.

[0045] Figure 8d is a conceptual diagram for explaining the MR AMS server of the present invention.

[0046] Figure 9 is a conceptual diagram for explaining the DTaaS server of the present invention.

[0047] FIG. 10 is a block diagram illustrating the structure of a 3D map view generation device according to an embodiment of the present invention.

[0048] FIG. 11 is a block diagram illustrating in more detail the structure of a 3D map view generation device according to an embodiment of the present invention.

[0049] FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view generation device and an MR service device according to an embodiment of the present invention.

[0050] FIG. 13 is a flowchart illustrating an operation process of a processor of a 3D map view generation device according to an embodiment of the present invention, modeling virtual objects included in a map area and rendering and displaying the modeled virtual objects.

[0051] FIG. 14 is an exemplary diagram illustrating an example in which a processor of a 3D map view generation device according to an embodiment of the present invention updates a map area according to driving information of a vehicle.

[0052] FIG. 15 is a block diagram illustrating a configuration in which modeling and rendering of virtual objects are performed by different processes in a 3D map view generation device according to an embodiment of the present invention having multiple processors.

[0053] FIG. 16 is an exemplary diagram showing an example of simultaneously displaying a screen showing information collected around a vehicle according to an ADAS system and an MR view screen according to a digital twin map in a 3D map view generation device according to an embodiment of the present invention.

[0054] FIG. 17a is an exemplary diagram illustrating a frustum-shaped visible area formed across a plurality of map tiles or one map tile, depending on the position of the vehicle and the driving direction of the vehicle, in a 3D map view generation device according to an embodiment of the present invention.

[0055] Figure 17b is an exemplary diagram showing an example in which the visible area of ​​the frustum shape illustrated in Figure 17a includes a plurality of map tiles.

[0056] FIG. 18a is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention, in which texture images are tiled in a margin width and at least one preset texturing area.

[0057] FIG. 18b is an exemplary diagram showing examples of facade designs of a building virtual object expressed in various ways by varying partial areas and tiled texture images in a 3D map view generation device according to an embodiment of the present invention.

[0058] FIG. 19 is a flowchart illustrating a texturing process for forming different surface textures on each component of a building in a 3D map view generation device according to an embodiment of the present invention.

[0059] FIG. 20 is an exemplary diagram showing an example of normal maps blended to mask maps corresponding to different building elements and a multi-blended normal map generated by blending normal maps together in a 3D map view generation device according to an embodiment of the present invention.

[0060] FIG. 21 is a flowchart illustrating an operation process of mapping the colors of virtual building objects based on the colors of actual buildings within a visible area in a 3D map view generation device according to an embodiment of the present invention.

[0061] FIG. 22 is an exemplary diagram showing an example of determining colors mapped to texture images through images captured from a vehicle in a 3D map view generation device according to an embodiment of the present invention.

[0062] FIG. 23 is an exemplary diagram illustrating an example of a 3D map view generation device according to an embodiment of the present invention grouping window elements formed in a building virtual object into one or more groups and displaying the building virtual object in various ways using the grouped window elements.

[0063] FIG. 24 is an exemplary diagram showing an example in which a 3D map view generation device according to an embodiment of the present invention displays building virtual objects differently depending on the driving time zone.

[0064] FIG. 25 is a flowchart illustrating an operation process of a 3D map view generation device according to an embodiment of the present invention, in which at least one of the characteristics defining a building element formed in a building virtual object is changed in conjunction with multimedia content.

[0065] Fig. 26 is an exemplary diagram showing an example in which window elements formed in each building virtual object are changed in conjunction with the characteristics of multimedia content according to the operation process of Fig. 25.

[0066] FIG. 27 is an exemplary diagram illustrating an example of a 3D map view generation device according to an embodiment of the present invention changing a tiling area of ​​a building virtual object according to multimedia content played in a vehicle.

[0067] FIG. 28 is a conceptual diagram illustrating a process in which a 3D map view generation device according to an embodiment of the present invention determines the brightness of window elements for each building object based on analyzed media characteristics.

[0068] FIG. 29 is a conceptual diagram illustrating a process in which a 3D map view generation device according to an embodiment of the present invention determines the color of a building object based on an album cover image of media being played.

[0069] FIG. 30 is a conceptual diagram illustrating a process in which a 3D map view generation device according to an embodiment of the present invention determines different material values ​​for each building object based on a plurality of characteristics detected from multimedia being played.

[0070] FIG. 31 is an exemplary diagram illustrating an example of a 3D map view generation device according to an embodiment of the present invention determining a color of a surrounding area based on the color of image media displayed on a side area of ​​a building object.

[0071] FIG. 32 is an exemplary diagram showing an example of a 3D map view generation device (1300) according to an embodiment of the present invention displaying event information by utilizing a side area of ​​a building object as a media facade.

[0072] FIG. 33 is an exemplary diagram illustrating an example of a 3D map view generation device according to an embodiment of the present invention displaying event information by utilizing the side surfaces of multiple building objects as media facades.

[0073] FIG. 34 is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention displaying information related to the environment around a building object using the side color of the displayed building object.

[0074] FIG. 35 is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention dividing a panoramic image into sides of a plurality of building objects and displaying them.

[0075] FIG. 36 is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention displaying information related to a vehicle status by utilizing side components of surrounding building objects.

[0076] FIG. 37 is an exemplary diagram showing an example of a 3D map view generation device according to an embodiment of the present invention displaying path information by utilizing side components of surrounding building objects.

[0077] FIG. 38 is an exemplary diagram illustrating a process in which a 3D map view generation device according to an embodiment of the present invention displays a building object including a 3D display area in which a 3D object is displayed.

[0078] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0079] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0080] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0081] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0082] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0083] The vehicle described in this specification may include a concept that includes automobiles and motorcycles. In the following, the vehicle will be described primarily with automobiles.

[0084] The vehicle described in this specification may be a concept that includes all types of vehicles, including internal combustion engine vehicles equipped with an engine as a power source, hybrid vehicles equipped with an engine and an electric motor as power sources, and electric vehicles equipped with an electric motor as a power source.

[0085] In the following description, the left side of the vehicle means the left side of the vehicle's driving direction, and the right side of the vehicle means the right side of the vehicle's driving direction.

[0086] FIG. 1 is a drawing showing the exterior of a vehicle according to an embodiment of the present invention.

[0087] Figure 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles.

[0088] Figures 3 and 4 are drawings showing the interior of a vehicle according to an embodiment of the present invention.

[0089] Figures 5 and 6 are drawings for reference in explaining an object according to an embodiment of the present invention.

[0090] FIG. 7 is a block diagram for reference in explaining a vehicle according to an embodiment of the present invention.

[0091] Referring to FIGS. 1 to 7, the vehicle (100) may include wheels that rotate by a power source and a steering input device (510) for controlling the direction of travel of the vehicle (100).

[0092] The vehicle (100) may be an autonomous vehicle.

[0093] The vehicle (100) can be switched to autonomous driving mode or manual mode based on user input.

[0094] For example, the vehicle (100) may be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on user input received through the user interface device (200).

[0095] The vehicle (100) can be switched to autonomous driving mode or manual mode based on driving situation information. The driving situation information can be generated based on object information provided by the object detection device (300).

[0096] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information generated by the object detection device (300).

[0097] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information received through the communication device (400).

[0098] The vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on information, data, and signals provided from an external device.

[0099] When the vehicle (100) is operated in autonomous driving mode, the autonomous vehicle (100) can be operated based on the driving system (700).

[0100] For example, an autonomous vehicle (100) may be driven based on information, data, or signals generated from a driving system (710), an exit system (740), or a parking system (750).

[0101] When the vehicle (100) is driven in manual mode, the autonomous vehicle (100) can receive user input for driving through the driving control device (500). Based on the user input received through the driving control device (500), the vehicle (100) can be driven.

[0102] The overall length refers to the length from the front to the rear of the vehicle (100), the overall width refers to the width of the vehicle (100), and the overall height refers to the length from the bottom of the wheel to the roof. In the following description, the overall length direction (L) may refer to the direction that serves as a reference for measuring the overall length of the vehicle (100), the overall width direction (W) may refer to the direction that serves as a reference for measuring the overall width of the vehicle (100), and the overall height direction (H) may refer to the direction that serves as a reference for measuring the overall height of the vehicle (100).

[0103] As illustrated in FIG. 7, the vehicle (100) may include a user interface device (200), an object detection device (300), a communication device (400), a driving operation device (500), a vehicle driving device (600), a driving system (700), a navigation system (770), a sensing unit (120), a vehicle interface unit (130), a memory (140), a control unit (170), and a power supply unit (190).

[0104] Depending on the embodiment, the vehicle (100) may include additional components other than the components described herein, or may not include some of the components described herein.

[0105] The user interface device (200) is a device for communication between a vehicle (100) and a user. The user interface device (200) can receive user input and provide information generated in the vehicle (100) to the user. The vehicle (100) can implement a UI (User Interface) or UX (User Experience) through the user interface device (200).

[0106] The user interface device (200) may include an input unit (210), an internal camera (220), a biometric detection unit (230), an output unit (250), and a processor (270).

[0107] Depending on the embodiment, the user interface device (200) may include additional components other than the described components, or may not include some of the described components.

[0108] The input unit (200) is for receiving information from a user, and data collected from the input unit (120) can be analyzed by a processor (270) and processed into a user's control command.

[0109] The input unit (200) may be placed inside the vehicle. For example, the input unit (200) may be placed in an area of ​​a steering wheel, an area of ​​an instrument panel, an area of ​​a seat, an area of ​​each pillar, an area of ​​a door, an area of ​​a center console, an area of ​​a head lining, an area of ​​a sun visor, an area of ​​a windshield, or an area of ​​a window.

[0110] The input unit (200) may include a voice input unit (211), a gesture input unit (212), a touch input unit (213), and a mechanical input unit (214).

[0111] The voice input unit (211) can convert a user's voice input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0112] The voice input unit (211) may include one or more microphones.

[0113] The gesture input unit (212) can convert a user's gesture input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0114] The gesture input unit (212) may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input.

[0115] According to an embodiment, the gesture input unit (212) can detect a user's three-dimensional gesture input. To this end, the gesture input unit (212) can include a light output unit that outputs a plurality of infrared lights or a plurality of image sensors.

[0116] The gesture input unit (212) can detect a user's 3D gesture input through a TOF (Time of Flight) method, a structured light method, or a disparity method.

[0117] The touch input unit (213) can convert a user's touch input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0118] The touch input unit (213) may include a touch sensor for detecting a user's touch input.

[0119] According to an embodiment, the touch input unit (213) may be formed integrally with the display unit (251), thereby implementing a touch screen. Such a touch screen may provide both an input interface and an output interface between the vehicle (100) and the user.

[0120] The mechanical input unit (214) may include at least one of a button, a dome switch, a jog wheel, and a jog switch. An electrical signal generated by the mechanical input unit (214) may be provided to a processor (270) or a control unit (170).

[0121] The mechanical input unit (214) can be placed on a steering wheel, center fascia, center console, cockpit module, door, etc.

[0122] The internal camera (220) can capture images of the vehicle interior. The processor (270) can detect the user's status based on the images of the vehicle interior. The processor (270) can obtain information about the user's gaze from the images of the vehicle interior. The processor (270) can detect the user's gestures from the images of the vehicle interior.

[0123] The biometric detection unit (230) can obtain the user's biometric information. The biometric detection unit (230) includes a sensor capable of obtaining the user's biometric information, and can use the sensor to obtain the user's fingerprint information, heartbeat information, etc. The biometric information can be used for user authentication.

[0124] The output unit (250) is for generating output related to vision, hearing, or touch.

[0125] The output unit (250) may include at least one of a display unit (251), an audio output unit (252), and a haptic output unit (253).

[0126] The display unit (251) can display graphic objects corresponding to various information.

[0127] The display unit (251) may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0128] The display unit (251) can implement a touch screen by forming a mutual layer structure with the touch input unit (213) or forming it as an integral part.

[0129] The display unit (251) may be implemented as a HUD (Head Up Display). When the display unit (251) is implemented as a HUD, the display unit (251) may be equipped with a projection module to output information through an image projected onto a windshield or window.

[0130] The display unit (251) may include a transparent display. The transparent display may be attached to a windshield or window.

[0131] A transparent display can display a predetermined screen while having a predetermined transparency. To have transparency, the transparent display may include at least one of a transparent TFEL (Thin Film Electroluminescent), a transparent OLED (Organic Light-Emitting Diode), a transparent LCD (Liquid Crystal Display), a transmissive transparent display, and a transparent LED (Light Emitting Diode) display. The transparency of the transparent display can be adjusted.

[0132] Meanwhile, the user interface device (200) may include a plurality of display units (251a to 251g).

[0133] The display unit (251) may be arranged in one area of ​​the steering wheel, one area of ​​the instrument panel (521a, 251b, 251e), one area of ​​the seat (251d), one area of ​​each pillar (251f), one area of ​​the door (251g), one area of ​​the center console, one area of ​​the head lining, one area of ​​the sun visor, or may be implemented in one area of ​​the windshield (251c), one area of ​​the window (251h).

[0134] The audio output unit (252) converts an electric signal provided from the processor (270) or the control unit (170) into an audio signal and outputs the converted signal. To this end, the audio output unit (252) may include one or more speakers.

[0135] The haptic output unit (253) generates a tactile output. For example, the haptic output unit (253) can operate by vibrating a steering wheel, a seat belt, or a seat (110FL, 110FR, 110RL, 110RR) so that the user can perceive the output.

[0136] The processor (270) can control the overall operation of each unit of the user interface device (200).

[0137] Depending on the embodiment, the user interface device (200) may include a plurality of processors (270) or may not include a processor (270).

[0138] If the user interface device (200) does not include a processor (270), the user interface device (200) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0139] Meanwhile, the user interface device (200) may be referred to as a vehicle display device.

[0140] The user interface device (200) can be operated under the control of the control unit (170).

[0141] The object detection device (300) is a device for detecting an object located outside a vehicle (100).

[0142] Objects may be various objects related to the operation of the vehicle (100).

[0143] Referring to FIGS. 5 and 6, objects may include lanes (OB10), other vehicles (OB11), pedestrians (OB12), two-wheeled vehicles (OB13), traffic signals (OB14, OB15), lights, roads, structures, speed bumps, terrain, animals, etc.

[0144] A lane (OB10) may be a driving lane, a lane adjacent to a driving lane, or a lane in which opposing vehicles drive. A lane (OB10) may be a concept that includes lines on the left and right sides that form a lane.

[0145] Another vehicle (OB11) may be a vehicle driving around the vehicle (100). The other vehicle may be a vehicle located within a predetermined distance from the vehicle (100). For example, the other vehicle (OB11) may be a vehicle preceding or following the vehicle (100).

[0146] A pedestrian (OB12) may be a person located around a vehicle (100). A pedestrian (OB12) may be a person located within a predetermined distance from a vehicle (100). For example, a pedestrian (OB12) may be a person located on a sidewalk or roadway.

[0147] A two-wheeled vehicle (OB12) may refer to a vehicle that is positioned around a vehicle (100) and moves using two wheels. The two-wheeled vehicle (OB12) may be a vehicle with two wheels that is positioned within a predetermined distance from the vehicle (100). For example, the two-wheeled vehicle (OB13) may be a motorcycle or bicycle positioned on a sidewalk or road.

[0148] Traffic signals may include traffic lights (OB15), traffic signs (OB14), and patterns or text painted on the road surface.

[0149] The light may be generated from a lamp installed in another vehicle. The light may be generated from a streetlight. The light may be sunlight.

[0150] A road may include slopes such as road surfaces, curves, uphill and downhill slopes, etc.

[0151] Structures may be objects located along roads and fixed to the ground. For example, structures may include streetlights, street trees, buildings, utility poles, traffic lights, and bridges.

[0152] Landforms may include mountains, hills, etc.

[0153] Meanwhile, objects can be classified into moving objects and fixed objects. For example, moving objects may include concepts such as other vehicles and pedestrians. For example, fixed objects may include concepts such as traffic signals, roads, and structures.

[0154] The object detection device (300) may include a camera (310), a radar (320), a lidar (330), an ultrasonic sensor (340), an infrared sensor (350), and a processor (370).

[0155] Depending on the embodiment, the object detection device (300) may include additional components other than the described components, or may not include some of the described components.

[0156] The camera (310) may be positioned at an appropriate location outside the vehicle to capture images of the vehicle's exterior. The camera (310) may be a mono camera, a stereo camera (310a), an AVM (Around View Monitoring) camera (310b), or a 360-degree camera.

[0157] For example, the camera (310) may be positioned inside the vehicle, close to the front windshield, to capture an image of the front of the vehicle. Alternatively, the camera (310) may be positioned around the front bumper or radiator grill.

[0158] For example, the camera (310) may be positioned inside the vehicle, close to the rear glass, to capture images of the rear of the vehicle. Alternatively, the camera (310) may be positioned around the rear bumper, trunk, or tailgate.

[0159] For example, the camera (310) may be positioned close to at least one of the side windows inside the vehicle to obtain an image of the side of the vehicle. Alternatively, the camera (310) may be positioned around a side mirror, fender, or door.

[0160] The camera (310) can provide the acquired image to the processor (370).

[0161] The radar (320) may include an electromagnetic wave transmitter and receiver. The radar (320) may be implemented in a pulse radar or continuous wave radar manner based on the principle of radio wave emission. Among continuous wave radar methods, the radar (320) may be implemented in an FMCW (Frequency Modulated Continuous Wave) manner or an FSK (Frequency Shift Keying) manner depending on the signal waveform.

[0162] The radar (320) can detect an object using electromagnetic waves, based on a TOF (Time of Flight) method or a phase-shift method, and can detect the location of the detected object, the distance to the detected object, and the relative speed.

[0163] The radar (320) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0164] The lidar (330) may include a laser transmitter and receiver. The lidar (330) may be implemented using a TOF (Time of Flight) method or a phase-shift method.

[0165] The lidar (330) can be implemented as a driven or non-driven type.

[0166] When implemented as a drive type, the lidar (330) is rotated by a motor and can detect objects around the vehicle (100).

[0167] When implemented in a non-driven manner, the lidar (330) can detect an object located within a predetermined range relative to the vehicle (100) through optical steering. The vehicle (100) can include a plurality of non-driven lidars (330).

[0168] Lidar (330) can detect an object based on a time-of-flight (TOF) method or a phase-shift method using laser light as a parameter, and can detect the position of the detected object, the distance to the detected object, and the relative speed.

[0169] The lidar (330) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.

[0170] The ultrasonic sensor (340) may include an ultrasonic transmitter and a receiver. The ultrasonic sensor (340) may detect an object based on ultrasonic waves, and may detect the location of the detected object, the distance from the detected object, and the relative speed.

[0171] The ultrasonic sensor (340) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0172] The infrared sensor (350) may include an infrared transmitter and a receiver. The infrared sensor (340) may detect an object based on infrared light, and may detect the location of the detected object, the distance to the detected object, and the relative speed.

[0173] The infrared sensor (350) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0174] The processor (370) can control the overall operation of each unit of the object detection device (300).

[0175] The processor (370) can detect and track an object based on the acquired image. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object through an image processing algorithm.

[0176] The processor (370) can detect and track an object based on the reflected electromagnetic waves that are returned when the transmitted electromagnetic waves are reflected by the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the electromagnetic waves.

[0177] The processor (370) can detect and track an object based on the reflected laser light that is reflected back by the transmitted laser beam from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the laser light.

[0178] The processor (370) can detect and track an object based on the reflected ultrasonic waves that are returned when the transmitted ultrasonic waves are reflected off the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the ultrasonic waves.

[0179] The processor (370) can detect and track an object based on the reflected infrared light that is reflected back by the transmitted infrared light from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the infrared light.

[0180] Depending on the embodiment, the object detection device (300) may include a plurality of processors (370) or may not include a processor (370). For example, each of the camera (310), radar (320), lidar (330), ultrasonic sensor (340), and infrared sensor (350) may individually include a processor.

[0181] If the object detection device (300) does not include a processor (370), the object detection device (300) can be operated under the control of the processor or control unit (170) of the device in the vehicle (100).

[0182] The object detection device (400) can be operated under the control of the control unit (170).

[0183] The communication device (400) is a device for communicating with an external device. Here, the external device may be another vehicle, a mobile terminal, or a server.

[0184] The communication device (400) may include at least one of a transmitting antenna, a receiving antenna, an RF (Radio Frequency) circuit capable of implementing various communication protocols, and an RF element to perform communication.

[0185] The communication device (400) may include a short-range communication unit (410), a location information unit (420), a V2X communication unit (430), an optical communication unit (440), a broadcast transmission / reception unit (450), and a processor (470).

[0186] Depending on the embodiment, the communication device (400) may include additional components other than the described components, or may not include some of the described components.

[0187] The short-range communication unit (410) is a unit for short-range communication. The short-range communication unit (410) can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0188] The short-range communication unit (410) can form a short-range wireless communication network (Wireless Area Network) to perform short-range communication between the vehicle (100) and at least one external device.

[0189] The location information unit (420) is a unit for obtaining location information of a vehicle (100). For example, the location information unit (420) may include a GPS (Global Positioning System) module or a DGPS (Differential Global Positioning System) module.

[0190] The V2X communication unit (430) is a unit for performing wireless communication with a server (V2I: Vehicle to Infrastructure), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). The V2X communication unit (430) may include an RF circuit capable of implementing protocols for communication with infrastructure (V2I), communication between vehicles (V2V), and communication with pedestrians (V2P).

[0191] The optical communication unit (440) is a unit for communicating with an external device via light. The optical communication unit (440) may include an optical transmission unit that converts an electrical signal into an optical signal and transmits it to the outside, and an optical reception unit that converts a received optical signal into an electrical signal.

[0192] According to an embodiment, the light transmitting unit may be formed to be integrated with a lamp included in the vehicle (100).

[0193] The broadcast transmitter / receiver (450) is a unit for receiving broadcast signals from an external broadcast management server via a broadcast channel, or transmitting broadcast signals to the broadcast management server. The broadcast channels may include satellite channels and terrestrial channels. The broadcast signals may include TV broadcast signals, radio broadcast signals, and data broadcast signals.

[0194] The processor (470) can control the overall operation of each unit of the communication device (400).

[0195] Depending on the embodiment, the communication device (400) may include a plurality of processors (470) or may not include a processor (470).

[0196] If the communication device (400) does not include a processor (470), the communication device (400) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0197] Meanwhile, the communication device (400) may implement a vehicle display device together with the user interface device (200). In this case, the vehicle display device may be referred to as a telematics device or an AVN (Audio Video Navigation) device.

[0198] The communication device (400) can be operated under the control of the control unit (170).

[0199] The driving control device (500) is a device that receives user input for driving.

[0200] In manual mode, the vehicle (100) can be driven based on signals provided by the driving control device (500).

[0201] The driving control device (500) may include a steering input device (510), an acceleration input device (530), and a brake input device (570).

[0202] The steering input device (510) can receive input for the direction of travel of the vehicle (100) from the user. The steering input device (510) is preferably formed in the form of a wheel so that steering input can be provided by rotation. Depending on the embodiment, the steering input device may be formed in the form of a touch screen, a touch pad, or a button.

[0203] The acceleration input device (530) can receive an input from a user for accelerating the vehicle (100). The brake input device (570) can receive an input from a user for decelerating the vehicle (100). The acceleration input device (530) and the brake input device (570) are preferably formed in the form of a pedal. Depending on the embodiment, the acceleration input device or the brake input device may also be formed in the form of a touch screen, a touch pad, or a button.

[0204] The driving operation device (500) can be operated under the control of the control unit (170).

[0205] The vehicle driving device (600) is a device that electrically controls the driving of various devices in the vehicle (100).

[0206] The vehicle driving device (600) may include a power train driving unit (610), a chassis driving unit (620), a door / window driving unit (630), a safety device driving unit (640), a lamp driving unit (650), and an air conditioning driving unit (660).

[0207] Depending on the embodiment, the vehicle drive device (600) may include other components in addition to the described components, or may not include some of the described components.

[0208] Meanwhile, the vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0209] The power train drive unit (610) can control the operation of the power train device.

[0210] The power train drive unit (610) may include a power source drive unit (611) and a transmission drive unit (612).

[0211] The power source driving unit (611) can perform control over the power source of the vehicle (100).

[0212] For example, if a fossil fuel-based engine is the power source, the power source drive unit (610) can perform electronic control of the engine. This can control the engine output torque, etc. The power source drive unit (611) can adjust the engine output torque according to the control of the control unit (170).

[0213] For example, if an electric energy-based motor is the power source, the power source driving unit (610) can perform control over the motor. The power source driving unit (610) can adjust the rotation speed, torque, etc. of the motor according to the control of the control unit (170).

[0214] The transmission drive unit (612) can perform control over the transmission.

[0215] The transmission drive unit (612) can adjust the state of the transmission. The transmission drive unit (612) can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or parking (P).

[0216] Meanwhile, when the engine is the power source, the transmission drive unit (612) can adjust the gear engagement state in the forward (D) state.

[0217] The chassis drive unit (620) can control the operation of the chassis device.

[0218] The chassis drive unit (620) may include a steering drive unit (621), a brake drive unit (622), and a suspension drive unit (623).

[0219] The steering drive unit (621) can perform electronic control of the steering apparatus within the vehicle (100). The steering drive unit (621) can change the direction of travel of the vehicle.

[0220] The brake drive unit (622) can perform electronic control of the brake apparatus within the vehicle (100). For example, the speed of the vehicle (100) can be reduced by controlling the operation of the brakes placed on the wheels.

[0221] Meanwhile, the brake driving unit (622) can individually control each of the plurality of brakes. The brake driving unit (622) can control the braking force applied to the plurality of wheels differently.

[0222] The suspension drive unit (623) can perform electronic control of the suspension apparatus within the vehicle (100). For example, when there is a curve in the road surface, the suspension drive unit (623) can control the suspension apparatus to reduce vibration of the vehicle (100).

[0223] Meanwhile, the suspension drive unit (623) can individually control each of the plurality of suspensions.

[0224] The door / window actuator (630) can perform electronic control of a door apparatus or window apparatus within a vehicle (100).

[0225] The door / window driving unit (630) may include a door driving unit (631) and a window driving unit (632).

[0226] The door driving unit (631) can control the door device. The door driving unit (631) can control the opening and closing of a plurality of doors included in the vehicle (100). The door driving unit (631) can control the opening or closing of a trunk or tail gate. The door driving unit (631) can control the opening or closing of a sunroof.

[0227] The window driving unit (632) can perform electronic control of a window apparatus. It can control the opening or closing of a plurality of windows included in a vehicle (100).

[0228] The safety device driving unit (640) can perform electronic control of various safety devices in the vehicle (100).

[0229] The safety device drive unit (640) may include an airbag drive unit (641), a seat belt drive unit (642), and a pedestrian protection device drive unit (643).

[0230] The airbag driving unit (641) can perform electronic control of the airbag apparatus within the vehicle (100). For example, the airbag driving unit (641) can control the airbag to deploy when a danger is detected.

[0231] The seat belt drive unit (642) can perform electronic control of the seat belt apparatus within the vehicle (100). For example, the seat belt drive unit (642) can control the passenger to be secured to the seat (110FL, 110FR, 110RL, 110RR) using the seat belt when a danger is detected.

[0232] The pedestrian protection device drive unit (643) can perform electronic control of the hood lift and pedestrian airbag. For example, the pedestrian protection device drive unit (643) can control the hood lift up and the pedestrian airbag to deploy when a collision with a pedestrian is detected.

[0233] The lamp driving unit (650) can perform electronic control of various lamp apparatuses within the vehicle (100).

[0234] The air conditioning drive unit (660) can perform electronic control of the air conditioning device (air conditioner) within the vehicle (100). For example, the air conditioning drive unit (660) can control the air conditioning device to operate and supply cool air to the vehicle when the temperature inside the vehicle is high.

[0235] The vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0236] The vehicle driving device (600) can be operated under the control of the control unit (170).

[0237] The driving system (700) is a system that controls various operations of the vehicle (100). The driving system (700) can be operated in autonomous driving mode.

[0238] The driving system (700) may include a driving system (710), an exiting system (740), and a parking system (750).

[0239] Depending on the embodiment, the driving system (700) may include additional components other than the described components, or may not include some of the described components.

[0240] Meanwhile, the driving system (700) may include a processor. Each unit of the driving system (700) may individually include a processor.

[0241] Meanwhile, depending on the embodiment, if the operation system (700) is implemented in software, it may be a sub-concept of the control unit (170).

[0242] Meanwhile, according to an embodiment, the driving system (700) may be a concept including at least one of a user interface device (200), an object detection device (300), a communication device (400), a vehicle driving device (600), and a control unit (170).

[0243] The driving system (710) can drive the vehicle (100).

[0244] The driving system (710) receives navigation information from the navigation system (770) and provides a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0245] The driving system (710) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to enable driving of the vehicle (100).

[0246] The driving system (710) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0247] The exit system (740) can perform exit of a vehicle (100).

[0248] The exit system (740) receives navigation information from the navigation system (770) and provides a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0249] The exit system (740) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0250] The exit system (740) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0251] The parking system (750) can perform parking of a vehicle (100).

[0252] The parking system (750) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0253] The parking system (750) receives object information from the object detection device (300) and provides a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0254] The parking system (750) can receive a signal from an external device through a communication device (400) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0255] A navigation system (770) can provide navigation information. The navigation information can include at least one of map information, set destination information, route information based on the set destination, information on various objects along the route, lane information, and current vehicle location information.

[0256] The navigation system (770) may include memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system (770).

[0257] According to an embodiment, the navigation system (770) may receive information from an external device through a communication device (400) and update previously stored information.

[0258] Depending on the embodiment, the navigation system (770) may be classified as a sub-component of the user interface device (200).

[0259] The sensing unit (120) can sense the status of the vehicle. The sensing unit (120) can include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight detection sensor, a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0260] The sensing unit (120) can obtain sensing signals for vehicle attitude information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle inclination information, vehicle forward / backward information, battery information, fuel information, tire information, vehicle lamp information, vehicle internal temperature information, vehicle internal humidity information, steering wheel rotation angle, vehicle external illumination, pressure applied to an accelerator pedal, pressure applied to a brake pedal, etc.

[0261] The sensing unit (120) may further include, in addition, an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0262] The vehicle interface unit (130) can serve as a conduit for various types of external devices connected to the vehicle (100). For example, the vehicle interface unit (130) may be equipped with a port capable of connecting to a mobile terminal, and may be connected to the mobile terminal through the port. In this case, the vehicle interface unit (130) can exchange data with the mobile terminal.

[0263] Meanwhile, the vehicle interface unit (130) may serve as a conduit for supplying electrical energy to a connected mobile terminal. When the mobile terminal is electrically connected to the vehicle interface unit (130), the vehicle interface unit (130) may provide the mobile terminal with electrical energy supplied from the power supply unit (190) under the control of the control unit (170).

[0264] The memory (140) is electrically connected to the control unit (170). The memory (140) can store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory (140) can be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware. The memory (140) can store various data for the overall operation of the vehicle (100), such as programs for processing or controlling the control unit (170).

[0265] Depending on the embodiment, the memory (140) may be formed integrally with the control unit (170) or implemented as a sub-component of the control unit (170).

[0266] The control unit (170) can control the overall operation of each unit within the vehicle (100). The control unit (170) can be referred to as an ECU (Electronic Control Unit).

[0267] The power supply unit (190) can supply power required for the operation of each component under the control of the control unit (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0268] One or more processors and control units (170) included in the vehicle (100) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0269] Figure 8a is a conceptual diagram for explaining the AR service platform of the present invention.

[0270] The AR service platform that provides the AR service of the present invention may be called an AR service system.

[0271] The above AR service platform may include a server (850) installed outside the vehicle, which collects and processes information required for AR service and transmits it to the vehicle, and an AR service device (800) installed in the vehicle, which provides AR service using information transmitted from the server.

[0272] The fact that the server (850) collects and processes information required for the AR service and transmits it to the vehicle may include the meaning that the server (850) collects and processes information required for the AR service and transmits it to the AR service device (800) equipped in the vehicle.

[0273] The above AR service device (800) can change the information provided as an AR service based on the situation of the vehicle.

[0274] That is, the AR service device (800) of the present invention can dynamically adjust (variably) the information and amount of information to be displayed in AR according to the situation of the vehicle, and select information to be emphasized.

[0275] In addition, the AR service platform of the present invention can control the AR service provided in the vehicle to vary depending on specific conditions such as vehicle conditions and advertisement exposure conditions.

[0276] In the case of conventional AR navigation, when displaying destinations or major POIs (Points of Interest) in AR navigation, it is difficult to reflect the latest information because it uses information stored in map data, and there are limitations in that POIs with real-time attributes such as fueling / parking cannot be provided.

[0277] On the other hand, the AR service platform of the present invention can display vehicle location information, map data, multiple sensor data, real-time POI information, advertisement / event information, etc., by integrating them in AR navigation.

[0278] For example, to display AR information, the AR service device (800) of the present invention may receive AR service information from a server based on the current location of the vehicle and navigation route / guidance information, and process the information into a form for displaying on an AR navigation screen.

[0279] For example, the AR service device (800) of the present invention can reconstruct real-time AR display information. The AR service device (800) can reconstruct service data received from a server to display on an AR navigation screen by determining the display format, size, location, exposure method, etc. of AR content by considering the driving situation (e.g., varying the POI exposure location and size according to driving speed, changing the service information exposure location according to traffic conditions, adjusting the AR Wall display location and exposure time, etc.).

[0280] In addition, the AR service device (800) of the present invention can analyze the frequency of exposure of AR display information through user feedback.

[0281] The server (850) collects user input information (input information such as touch, order, etc.) for AR service content, performs content exposure frequency analysis, and can adjust the service content exposure policy based on the information.

[0282] Through this configuration, the present invention can express various external service contents in AR navigation by integrating them, and can provide various services through POI information including real-time properties.

[0283] In addition, the present invention enables display of various forms of AR content, such as POI information, advertisements, events, and major landmark information.

[0284] In addition, a new user experience of AR navigation can be presented through an embodiment based on the UX scenario proposed in the present invention.

[0285] The present invention can provide a service platform structure and an AR information display method (UX) that dynamically adjusts the amount of information (POI data, advertisement) to be displayed in AR according to vehicle conditions and advertisement exposure conditions, a module that collects POI information and commerce service information for AR expression and processes them into a form that is easy to render in an AR engine, a module that processes specific POI information to be emphasized according to the inside / outside vehicle conditions, a module that collects vehicle situation information and applies a UX policy according to the situation, and an AR engine module that renders AR objects (Group POI, Mini POI, 3D Object, Event wall, etc.) according to the UX policy.

[0286] In addition, the present invention can provide a client module for transmitting and receiving interaction and data between the front and rear displays of a vehicle, a Service App module for exposing commerce service information linked to POI, a client module for collecting user actions for advertisements such as AR advertisement object exposure results and clicks, and a cloud module for collecting / analyzing user actions for advertisements such as AR advertisement object exposure results and clicks.

[0287] Referring to FIG. 8a, the AR service platform of the present invention may include a server (850) that is a configuration existing outside the vehicle (off-board) and an AR service device (800) that is a configuration equipped in the vehicle (on-board).

[0288] First, the server (850) may include a POI data aggregator (851), an Ads manager (852), an Ads monitoring unit (853), a Service & Ads Manager (854), a Commerce Manager (855), a DB Connector (856), and a Dashboard (857).

[0289] The POI Data Aggregator (851) can receive information required for an AR service from multiple external servers and convert / integrate it into a message format of the AR service platform.

[0290] The Ads manager (852) can manage advertising information / content and advertising campaigns (advertisement exposure conditions).

[0291] The Ads Monitoring Department (853) can collect / save ad exposure and click results.

[0292] The Service & Ads Manager (854) can insert advertising information that meets the exposure conditions into service information and provide it to the client.

[0293] The Commerce Manager (855) can collect commerce service linkage / payment information.

[0294] The database connector (856) can store / query advertising content, advertising exposure result information, and commerce payment information.

[0295] The Dashboard (857) can display real-time AR service status visualized with advertisement exposure results / payment history results.

[0296] In addition, the server (850) may further include an AR service cloud API (or data conversion unit) for converting information transmitted from the vehicle's AR service device (800) into a data format available to the server, and for converting information processed / generated by the server into a data format available to the AR service device (800).

[0297] Meanwhile, the AR service device (800) may include a client (810) including a cloud interface, a commerce app, a CID-RSE interaction manager, a policy manager, advertisement monitoring, a driving context, personalized recommendations, etc., and an AR engine (820) including a POI renderer, a display manager, a touch manager, etc.

[0298] The client (810) can receive POI information, advertisements, etc. from the server.

[0299] Additionally, the client (810) can transmit and receive order / payment information to and from the server (850) and transmit advertisement exposure results to the server (850).

[0300] The AR engine (820) can transmit data such as the number of times an AR object output as AR is touched and the number of times it is exposed to the client (810).

[0301] Additionally, the AR engine (820) can transmit and receive front / rear seat (CID, RSE) linkage data to and from the client (810), and output an AR object according to the AR display policy received from the client (810).

[0302] In addition, the AR engine (820) can determine the type, output location, POI type, output size, etc. of AR objects provided through the AR service based on data collected from a gyro sensor, camera, communication unit, navigation, ADAS (Advanced Driver Assistance Systems), GPS, etc. equipped in the vehicle.

[0303] The AR service device (800) installed in the vehicle can AR render service content to display data transmitted from a cloud server in AR on the front camera image.

[0304] Additionally, the AR service device (800) can mediate data transmission between the server and the AR engine, such as collecting advertisement posting result data from the AR engine and transmitting it to the server.

[0305] Additionally, the AR service device (800) can link data generated by AR between CID-RSE (i.e., front / rear seats).

[0306] In addition, the AR service device (800) can perform data management for AR display policies, and specifically, can provide AR display policy data according to driving conditions to the AR engine.

[0307] In addition, the AR service device (800) can provide situational awareness and personalized services, and specifically, can provide AR objects to the AR engine according to driving situations (speed, TBT (Turn-by-Turn), etc.) using in-vehicle data.

[0308] In this specification, an example of providing an AR service by overlapping and outputting AR information (or AR object, AR content, POI information, etc.) on an image captured (received, processed) by a camera installed in a vehicle will be described.

[0309] However, the AR service described in this specification is not limited thereto, and can be applied in a similar or identical manner to various methods of implementing augmented reality, such as outputting AR information directly on the windshield of a vehicle to overlap with the real world space for a driver or passenger, or outputting AR information through a HUD (Head-up Display).

[0310] The input data (input information) used to provide AR services and the output data (output information) provided through the AR service platform are as follows.

[0311] First, the types of input data may include map data (navigation information), service content information (POI, advertisement, etc.), dynamic information, vehicle sensor information, historical information, and driving-related information.

[0312] Map data (navigation information) may include information such as route information to the destination (navigation route), guidance information (turn-by-turn), road shape ahead (road / lane), multiple map attribute information (road type / attribute, road and lane width, curvature, slope, speed limit, etc.), and localization object information (road marking, traffic sign, etc.).

[0313] Service content information (POI, advertisements, etc.) may include POI information received from multiple service providers, advertisement information available at the current location, and real-time information for reservation / payment services such as gas stations, charging stations, and parking lots.

[0314] Dynamic information may include traffic information (road-level traffic, lane-level traffic), event information (accidents, hazard warnings, etc.), weather information, V2X (V2V, V2I) (Vehicle to Everything, Vehicle to Vehicle, Vehicle to Infra), etc.

[0315] Vehicle sensor information may include current location information (GPS / DR), camera input information (ADAS information, object recognition information), and V2X (real-time surrounding situation information that can be collected through V2V and V2I).

[0316] Historical information may include information about past driving routes, traffic history (e.g., traffic volume by time of day), and communication speeds by zone and time of day.

[0317] Driving-related information may include driving mode (manual, autonomous driving, semi-autonomous driving, ADAS function operation, etc.), whether the vehicle has entered a destination or waypoint, and whether the vehicle has entered a parking lot.

[0318] Output information that can be provided through the AR service platform may include AR service display data based on current location / route.

[0319] Current location / route-based AR service display data may include AR advertisement display points along the route (AR Wall, POI building highlights), selectable AR building information (selectable main building information such as landmarks), general POI information (POI summary information such as icons or speech bubbles), remote POI information (distance / direction of important POI information that does not exist on the route but is helpful for driving), display information displayed when multiple POIs exist in the same building, destination building and real-time parking lot status information, real-time status information of gas stations / charging stations, and location-based advertisement / event information.

[0320] The AR service platform of the present invention can filter AR service information according to real-time information and determine a display method.

[0321] Specifically, the AR service platform can determine the number of real-time POI exposures based on driving speed, POI overlap removal, size adjustment, exposure time, etc.

[0322] In addition, the AR service platform can determine the POI exposure method according to risk information recognition, and specifically, can dynamically change the POI display method according to situations such as accidents, construction, and recognition of multiple moving objects.

[0323] Additionally, the AR service platform can dynamically change the POI display location when AR display visibility is reduced due to traffic.

[0324] In addition, the AR service platform can reconfigure front / rear seat AR display data. For example, the AR service information can be minimized on the front seat display and the rear seat display can be reconfigured to display all displayable information by considering driving speed, hazard information, weather information, etc.

[0325] The operation / function / control method of such an AR service platform may be implemented by a server or AR service device included in the AR service platform, or by organic interaction between the server and the AR service device.

[0326] Referring to FIG. 8a, the configuration of the server (850) of the AR service platform is described in more detail as follows.

[0327] The service and advertisement manager (854) can perform a client linkage (request) function, a POI information and advertisement information integration (data processing & aggregation) function, and a client linkage (respond) function.

[0328] Specifically, the client connection (request) function may include requesting / receiving POI information (location, category) through Unified API, or requesting / receiving destination entrance location information (select one of destination coordinates / address / id) through Unified API.

[0329] Here, Unified API refers to an API defined in the AR service cloud that is independent of a specific data provider (to minimize client changes).

[0330] The POI information and advertisement information integration (data processing & aggregation) function may include integrating POI information and advertisement information within a radius of 000m from the location requested by the client (from data manager, ads manager) or integrating the entrance location of the destination requested by the client and POI advertisement information (from data manager, ads manager).

[0331] Specifically, the POI information and advertisement information integration function may include a function to align advertisement information including building wall and event wall information with POI information, or, if there are multiple POIs in the same building, to filter them by priority on the server (e.g., leaving only partner companies and excluding other POI information).

[0332] Here, the filtering criteria may include assigning priority scores to each POI and comparing them.

[0333] Client response functionality may include transmitting POI information and advertisement information via Unified API, or transmitting destination entrance location and advertisement information via Unified API.

[0334] The data manager (not shown) included in the server (850) may include a POI information collection / transmission function, a building shape (polygon) information collection / transmission function, and a destination entrance information collection / transmission function.

[0335] The POI information collection / transfer function can perform the function of requesting POI information from a 3rd party API or transmitting (to Service & Ads Aggregator) POI information received from a 3rd party API (by converting it into a Unified API response format).

[0336] The building shape (polygon) information collection / transfer function can request building exterior shape information from a 3rd party API / data set, or perform the function of transmitting (to Service & Ads Aggregator) POI information received from a 3rd party API (by converting it into a Unified API response format).

[0337] The destination entrance information collection / transfer function can request destination entrance information from a 3rd party API or transmit destination entrance information received from a 3rd party API (by converting it into a Unified API response format) (to Service & Ads Aggregator).

[0338] The Ads Manager (852) can provide a partner (advertising) company management interface, an advertising campaign management interface, and an advertising content management interface.

[0339] The Ads Monitoring Department (853) can perform the function of receiving advertising effectiveness measurement feedback and the function of transmitting advertising information.

[0340] The partner (advertiser) management interface can perform POI advertiser management (add / edit / delete advertiser information) and general advertiser management (add / delete advertiser information).

[0341] POI supported ad formats may include Brand poi pin, Building wall, 3D rendering, Event wall, etc., and the supported ad format (general supported ad format) for brand advertisements that are not related to actual POI / location (e.g. Coca-Cola advertisements) may be Event wall.

[0342] The ad campaign management interface allows you to add / edit / delete ad campaigns (ad location, type, time, etc.).

[0343] The advertising content management interface can add / edit / view / delete content by advertising format (POI brand icon image, building wall image, event wall image / video, 3D rendering image).

[0344] The advertising effectiveness measurement feedback reception function may include a function for receiving advertising exposure feedback sent by the client and transmitting it to the DB Manager (CPC / CMP / CPT&P).

[0345] The advertising information delivery function may include a function that searches for and delivers advertising campaign information that should be displayed within a radius of 000m from the location requested by the Service & Ads Aggregator (in the case of CPT&P, only deliver advertisements that meet the time conditions).

[0346] The Commerce Manager (855) can perform client linkage functions, external commerce service linkage functions, and payment information management functions.

[0347] The client linkage function may include receiving requests by linking the client with Unified API, converting the request content received with Unified API into external commerce API specifications, converting data received from the external API into the Unified API message format, and transmitting data to the client.

[0348] The Commerce Manager can convert the request content received through the Unified API into an external commerce API specification and then perform external service linkage functions using the converted content.

[0349] Converting data received from an external API into the message format of the Unified API may mean converting data received from an external service connection into the Unified API.

[0350] The external commerce service linkage function may include functions for requesting a list of nearby stores and meta information from the current location and receiving results, requesting detailed information about a specific store from the list and receiving results, requesting reservations / orders and receiving results, requesting service usage status and receiving results, and linking member information of commerce services and receiving results.

[0351] Here, the service usage status request & result reception can be used for sequence management and AR message pop-up according to the service usage status (reservation completed / parking lot entry / parking in progress / parking lot exit / reservation cancellation).

[0352] Service member information linking & result reception can be used to link commerce service member ↔ AR service member (OEM Connectivity service member) information.

[0353] The payment information management function may include a function to collect payment details (contents, amount) from external commerce services and charge fees to external commerce companies based on the payment details.

[0354] The database connector (856) can perform advertising effectiveness measurement data management functions, commerce data management functions, advertiser data management functions, advertising content data management functions, and advertising location data management functions.

[0355] The advertising effectiveness measurement data management function can save / delete CPC / CPM / CPT&P related log data and view data (by POI, brand, time, and ad type).

[0356] The commerce data management function can save / delete payment history (content, amount) from external commerce services and perform data search (by POI, brand, time, and ad type).

[0357] The advertiser data management function can save / edit / delete / view advertiser information and advertising campaign settings for each advertiser.

[0358] The advertising content data management function can save / edit / delete / view advertising content in conjunction with advertiser information.

[0359] The advertising location data management function can manage the Event wall section coordinates and Building wall coordinates (by brand) for displaying AR advertisements, and can be distinguished by coordinates directly registered by the user or specific coordinates obtained through company API linkage.

[0360] The Service Dashboard (857) can perform advertising effectiveness measurement data visualization functions and commerce service data visualization functions.

[0361] The advertising effectiveness measurement data visualization function can provide the following charts: CPC: Total number of ad clicks by company / brand (viewable by period), CPC: Aggregated chart of total number of ad clicks (viewable by period), CPM: Total number of ad impressions by company / brand (viewable by period), CPM: Aggregated chart of total number of ad impressions (viewable by period), CPT&P: Chart of number of ad clicks by company / brand (viewable by period), CPT&P: Chart of number of ad impressions by company / brand (viewable by period).

[0362] These charts can be presented in a variety of formats, including Bar graph, Line graph, Pie Chart, Word graph, and Geospatial graph.

[0363] Although CPT&P is billed per hour rather than per click or impression, it can be used as data for measuring the effectiveness of exposure.

[0364] The commerce service data visualization feature can provide a chart of the cumulative payment amount by company (viewable by period) and a chart of the total cumulative payment amount (viewable by period).

[0365] Figure 8b is a conceptual diagram illustrating an MR service platform for providing the MR service of the present invention.

[0366] The present invention can provide an MR service platform capable of providing a mixed reality automotive meta service (MR AMS) (hereinafter, used interchangeably with MR service).

[0367] The MR service platform may be named as an MR service system, an MR navigation system, an MR platform, an MR system, etc.

[0368] An MR service platform refers to a platform capable of providing services based on mixed reality, and may include multiple independent components.

[0369] For example, the MR service platform may include an MR service device (900) (or referred to as an MR navigation engine) equipped on the vehicle, an MR AMS (hereinafter referred to as an MR AMS server) (1100) equipped on a server (or cloud) outside the vehicle (offboard), and a DTaaS (Digital Twin as a Service) server (1200).

[0370] The above MR service device (900) may include an MR AMS client (910) and an MR renderer (920).

[0371] The MR service described herein can be understood as a mixed reality navigation service for vehicles. That is, the MR service platform of the present invention can provide a vehicle interface implemented in mixed reality to a user riding in a vehicle.

[0372] The MR service provided by the MR service platform of the present invention can provide an experience of the digital world through a display inside the vehicle while driving in the real world.

[0373] Specifically, MR services can interactively provide drivers with navigation, safe driving, POI (Point of Interest), and entertainment user experiences in a virtual 3D space that combines real-world information with the digital world.

[0374] Through this, the MR service platform of the present invention can provide various UX (User Experience) that are free from the spatial and temporal constraints compared to existing camera-based (or HUD (Head up Display)-based) AR (Augmented Reality).

[0375] Here, the digital world refers to a digital twin or a digital twin smart city, and real-world information may include infrastructure data such as V2X (Vehicle to Everything) and C-ITS (Cooperative-Intelligent transport Systems) and / or surrounding perception data sensed by sensors equipped in autonomous vehicles.

[0376] In addition, the fusion described above may include the concept of fusion of vehicle and infrastructure sensor data as well as the MR service cloud (1000) (or MR AMS cloud) and MR service device (900) for implementing the present MR service platform.

[0377] Additionally, interactivity should be understood as a concept that includes not only mixed reality navigation, but also high-quality 3D image rendering and user interaction.

[0378] Meanwhile, mixed reality (MR) described in this specification may mean an environment in which virtual reality is grafted onto the real world, allowing real physical objects and virtual objects to interact.

[0379] Mixed reality (MR) can include augmented reality (AR), which adds virtual information based on reality, and augmented virtuality (AV), which adds real information to a virtual environment.

[0380] In other words, it provides a smart environment where the real and virtual are seamlessly connected, allowing users to experience richer experiences. For example, users can interact with a virtual pet in the palm of their hand, create a virtual game environment within their real-world room, play games, virtually rearrange furniture in their home, or create an environment where remote people can gather and work together.

[0381] A mixed reality vehicle meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention can help users prepare road shapes and actions in advance by displaying a preview of a future driving path.

[0382] In addition, a mixed reality automobile meta service (MR AMS) using mixed reality (MR) according to an embodiment of the present invention can improve advertising effectiveness or induce increased service usage by focusing on a specific POI (Point of Interest).

[0383] In addition, the mixed reality automobile meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention is not dependent on a specific map company and can also merge data from various map companies.

[0384] The present invention can provide an MR navigation function as one of the mixed reality automotive meta services.

[0385] In the MR navigation function, since it is implemented in the virtual world rather than overlapping augmented reality objects on the real world, it can solve difficulties such as forward occlusion problems and achieving matching quality that occur in AR navigation.

[0386] Accordingly, the present invention can improve user UX by effectively expressing various contexts that were difficult to express in existing navigation through the MR navigation function.

[0387] To this end, the MR service platform of the present invention can provide a method for managing an MR context, and a method and system for obtaining metadata and 3D assets for providing an MR AMS.

[0388] The MR service platform of the present invention can render related service metadata and 3D assets to MR navigation by defining the MR context required in the digital world and modeling service metadata for obtaining the same.

[0389] Accordingly, the present invention provides a digital world experience through a display within a vehicle while driving in the real world, and can recommend and provide various additional HMI (Human Machine Interface) services to the user by utilizing the improved visibility and visibility of MR navigation.

[0390] Hereinafter, an MR service platform according to an embodiment of the present invention for providing the MR service described above will be described.

[0391] Referring to FIG. 8b, the MR service platform (or MR service system) of the present invention may include an MR service cloud (1000) provided outside the vehicle and an MR service device (900) provided in the vehicle.

[0392] The MR service cloud (1100) installed outside the vehicle (Offboard) may include at least one of an MR AMS (Mixed Reality Automotive Meta Service) server (1100) and a DTaaS (Digital Twin as a Service) server (1200).

[0393] The MR service device (900) equipped on the above vehicle may include an MR AMS client (910) and an MR renderer (920).

[0394] The above MR service device (900) can provide a navigation function (or navigation application) by interacting with the AR service device (800) and the navigation system (770).

[0395] Information required for the above navigation function can be received through user input (or user request) input through a camera (310), sensing unit (120), and user input unit (210) provided in the vehicle.

[0396] In addition, information required for the above navigation function can be received through an MR service cloud (1000) provided offboard the vehicle, and in order to receive the information required for each vehicle, information sensed or processed in the vehicle can be transmitted to the MR service cloud (1000).

[0397] The MR AMS (Mixed Reality Automotive Meta Service) server (1100) can be connected to various service providers (1300a, 1300b, 1300c) that provide online map services such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP, as illustrated in FIG. 8d. Then, based on the results of aggregating map data provided from the connected service providers (1300a, 1300b, 1300c), the server can aggregate shape information (e.g., building floor coordinates (footprint) information and height information) of each building included in the map and provide the aggregated information to the DTaaS server (1200). Here, the DTaaS server (1200) may refer to a server or device that provides a service using DTaaS, i.e., a digital twin map.

[0398] The above DTaaS may mean Digital Twin as a Service or Digital Transformation as a Service.

[0399] Meanwhile, the DTaaS server (1200) may be connected to a POI database in which POI service data for each building or each area included in the map data is stored. In addition, the DTaaS server (1200) may be connected to a 3D model database in which data of a 3D polygon model (or a 3D polygon map) for each building included in the map data is stored. Here, the 3D polygon model is a polygon model that can provide a building volume and may be a polygon model in which no texture is formed on the surface. The DTaaS server (1200) may receive service data related to POI from the connected POI database, and may receive data of 3D polygon models of each building included in the map data of an area from the connected 3D model database.

[0400] The processor (not shown) of the MR service device (900) can receive various information related to the driving of the vehicle from the object detection device (300), the sensing unit (120), and the navigation system (770). For example, the processor can receive information on an object detected in front, behind, or to the side of the vehicle from the camera (310) of the object detection device (300).

[0401] Additionally, the processor may receive information related to the vehicle's speed, driving direction, current vehicle location (GPS), etc. from a sensing unit (120) including sensors connected to each component of the vehicle, including the driving system (710). Additionally, the processor may receive information related to the vehicle's driving path from a navigation system (770).

[0402] Meanwhile, the MR service device (900) and the DTaaS server (1200) may be connected via the interface (Interface APIs) of the MR AMS. Here, the connection between the MR service device (900) and the interface (Interface APIs) of the MR AMS may be established via a wireless network connection. In this case, the MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the MR service device (900).

[0403] In this way, when connected to the MR AMS server (1100), the MR service device (900) can provide at least some of the information received from the connected components (e.g., the vehicle's camera (310), the sensing unit (120), or the user input input through the user input unit (210)) to the MR AMS server (1100) via a network connection. Then, the MR AMS server (1100) can provide 3D map data for providing mixed reality to the MR service device (900) in response to the provided information.

[0404] For example, the MR service device (900) can transmit information about objects detected around the vehicle, as well as information about the vehicle's speed, direction, and current location. In addition, it can provide information about the driving path to the MR AMS server (1100). Then, the MR AMS server (1100) can provide 3D map data of an area according to the current location of the vehicle to the MR service device (900) based on the information provided by the MR service device (900).

[0405] In this case, the MR AMS server (1100) may determine POI information based on the current vehicle location, vehicle speed, and vehicle driving path, and may provide 3D map data that further includes the determined POI information to the 3D building map. In addition, the MR AMS server (1100) may provide 3D map data that further includes information on the situation around the vehicle to the MR service device (900) based on the information on objects around the vehicle provided.

[0406] Meanwhile, the MR service device (900) can render a mixed reality image based on the 3D map data provided from the MR AMS server (1100). For example, the MR service device (900) can control the MR renderer (920) to display a 3D map screen including models of buildings around the vehicle based on the provided 3D map data. In addition, the MR renderer (920) can display a graphic object corresponding to the vehicle on the 3D map screen, and display graphic objects corresponding to the provided POI data and situation information around the vehicle on the 3D map screen.

[0407] Accordingly, an image of a virtual environment (in other words, a mixed reality (MR) image, or an MR navigation screen, or an MR navigation interface) including a three-dimensional building model similar to the shape of the current vehicle and buildings around the vehicle and graphic objects corresponding to the vehicle can be output to a display (251) equipped in the vehicle, for example, a Center Information Display (CID), a Head Up Display (HUD), a Rear Sheet Information (RSI), or a Rear Sheet Entertainment (RSE).

[0408] In this case, information related to the vehicle's driving and the surroundings of the vehicle can be provided to the driver through the virtual environment. Through the 3D map data, i.e., the digital twin map (hereinafter referred to as the "DT map"), the MR service device (900) according to an embodiment of the present invention can provide the driver with a mixed reality service.

[0409] Meanwhile, the MR AMS server (1100) can determine 3D map data and POI information or situation information around each vehicle that can be provided together with the 3D map data based on information collected from not only the MR service device (900) equipped in one vehicle but also the MR service devices (900) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect from multiple vehicles in the form of a cloud server and generate 3D map data for mixed reality based on the collected information. In addition, it can be implemented to provide mixed reality service to at least one MR service device (900) equipped in different vehicles based on the generated 3D map data.

[0410] Therefore, for convenience of explanation, in the following, a cloud or server that includes the MR AMS server (1100) and the DTaaS server (1200), and provides information such as metadata (e.g., service metadata, 3D assets), 3D polygon maps, and digital twin maps for providing mixed reality services, i.e., a digital twin map (DT map), is referred to as an MR service cloud (1000).

[0411] As illustrated in FIG. 8b, the MR service device (900) (or MR navigation engine) may include an MR AMS client (910) and an MR renderer (920).

[0412] In addition, in order to implement the MR navigation function, which is one of the MR services, the MR service device (900) can transmit and receive data with the AR service device (800) (or AR engine) and the navigation system (770) equipped in the vehicle.

[0413] The MR AMS client (910) may include a context manager (911), a scene manager (913), and a UX scenario database (914).

[0414] Additionally, the MR renderer (920) may include a DTaaS client (921), an MR visualization unit (922), and a 3D HMI framework (923).

[0415] The MR AMS client (910) can collect vehicle location information, user input, user feedback information, payment information, etc. and transmit them to the MR AMS server (1100) located outside the vehicle.

[0416] The MR AMS server (1100) can transmit at least one of metadata, service metadata, and 3D assets required to provide MR service to the MR AMS client (910) based on information received from the MR AMS client.

[0417] The MR AMS client (910) can transmit data received from the MR AMS server (910) to the MR renderer (920).

[0418] The MR renderer (920) can create a digital twin map using a 3D polygon map received from a DTaaS server (1200) and an image received from an MR AMS client (910) or a camera (310) installed in a vehicle.

[0419] Additionally, the MR renderer (920) can render data received from the MR AMS client (920) into an MR object that can be overlapped on a digital twin map, and can create a mixed reality (MR) image by overlapping the rendered MR object on the digital twin map.

[0420] Thereafter, the MR renderer (920) can output the generated mixed reality image to a display (251) equipped in the vehicle.

[0421] All components described in this specification can be implemented as separate hardware modules, and can be understood as components implemented in software block units as needed.

[0422] Below, each component that constitutes the MR service platform will be described in more detail with reference to the attached drawings.

[0423] Figure 8c is a conceptual diagram for explaining the MR AMS client of the present invention.

[0424] The MR AMS client (910) is equipped in a vehicle and can provide a mixed reality automotive meta service (MR AMS).

[0425] The MR AMS client (910) may include a context manager (911) that requests a context corresponding to a user request (or user input) from an MR AMS server (1100) installed outside the vehicle, a scene manager (913) that manages MR scene information provided to a display (251) installed in the vehicle, and a UX scenario database (914) that provides a UX rule to at least one of the context manager (911) and the scene manager (913).

[0426] Additionally, the MR AMS client (910) may further include an interface API (912) that calls a function for communicating with the MR AMS server (1100) provided outside the vehicle.

[0427] The above interface API (912) may be configured with one or more functions configured to perform communication with the MR AMS server (1100), and may use these functions to convert a data format or message format to transmit data to the MR AMS server (1100), or to convert the format of data received from the MR AMS server (1100).

[0428] The interface API (921) can transmit a context request output from the context manager (911) to the MR AMS server (1100) and receive a 3D asset corresponding to the requested context from the MR AMS server (912).

[0429] Here, the context may refer to situational information and information corresponding to the situation the vehicle is in. Furthermore, the context may also include the meaning of content.

[0430] The above 3D asset may refer to 3D object data corresponding to the requested context. Furthermore, the 3D asset may refer to a 3D graphic object that overlaps or can be updated on a digital twin image (or digital twin map).

[0431] The MR AMS client (910) may be included in the MR service device (900).

[0432] The MR service device (900) may include a user interaction handler (901) that, when a user input is received through an input unit (210) provided in the vehicle, generates an action corresponding to the user input and transmits the action to the context manager.

[0433] The above user interaction handler (901) may be included in the MR service device (900) or may be included in the MR AMS client (910).

[0434] For example, when a user input of “Find nearby Starbucks” is received through the vehicle input unit (210), the user interaction handler (901) can generate an action corresponding to the user input (e.g., “Search POI”) and transmit it to the context manager (911) provided in the MR AMS client (910).

[0435] For example, the action may be determined by a motion matching a word included in the user input, and the action may be named a command or control command.

[0436] The above context manager (911) can generate a command to request a context corresponding to an action received from a user interaction handler (901) and transmit the command to the MR AMS server (1100) via an interface API (912).

[0437] The above command may be generated based on an action (e.g., “Search POI”) received from the user interaction handler (901), and may be formed to include, for example, the current vehicle location, the type of POI to be found, and radius information (e.g., GET “Starbucks” (type of POI) WITHIN “500m” (radius) FROM “37.7795, -122.4201” (current vehicle location (latitude, longitude)).

[0438] The context manager (911) can receive current scene information (current scene) currently being output from the vehicle from the scene manager (913) and receive UX rules from the UX scenario database (914).

[0439] Additionally, the context manager (911) can receive navigation information including the current route and current location from the navigation handler (902) that handles information of the navigation system (770).

[0440] The navigation handler (902) may be provided in the MR service device (900) or may be provided in the MR AMS client (910).

[0441] The context manager (911) can generate a command to request the context based on at least one of the current scene information, the UX rule, and the navigation information.

[0442] The current scene information may include screen information currently being output on the vehicle's display (251). For example, the current scene information may include information about a mixed reality image in which an MR object and an MR interface overlap on a digital twin map.

[0443] Additionally, at least one of the context manager (911) and the scene manager (913) of the present invention can receive sensor data processed through a sensor data adapter (903) that processes information sensed through the vehicle's sensing unit (120).

[0444] The above sensor data adapter (903) may be provided in the MR service device (900) or may be provided in the MR AMS client (910). The sensor data adapter (903) may transmit processed sensor data to an AR engine handler (904) that handles data transmitted to the AR engine (or AR service device) (800).

[0445] The interface API (912) can receive metadata of a context corresponding to the command and / or a 3D asset corresponding to the context from the MR AMS server (1100).

[0446] Thereafter, the interface API (912) can transmit the received metadata and / or 3D assets to the scene manager (913).

[0447] The scene manager (913) can generate UI data using the UX rules received from the UX scenario database (914) and the metadata and 3D assets received from the interface API (912).

[0448] Thereafter, the scene manager (913) can transmit the generated UI data to the MR renderer (920) that renders the generated UI data to be output as mixed reality (MR) or a mixed reality image on the display (251) provided in the vehicle.

[0449] Additionally, the scene manager (913) can further transmit the generated UI data to an AR engine handler (904) configured to handle an AR service device (800) equipped in the vehicle.

[0450] The UX rule stored in the UX scenario database (914) may refer to information about the rules, forms, formats or templates for creating a screen, UX or user interface provided by the MR service device, and these UX rules may be predefined for each type of data.

[0451] Additionally, UX rules can be updated or modified by users or administrators.

[0452] Figure 8d is a conceptual diagram for explaining the MR AMS server of the present invention.

[0453] Referring to FIG. 8d, an MR AMS server (1100) provided offboard in a vehicle and providing a Mixed Reality Automotive Meta Service (MR AMS) may include an interface API (1101) for calling a function for communicating with an MR AMS client provided in the vehicle, a service aggregation manager (1110) for requesting and receiving a context corresponding to a request received from the MR AMS client from a service provider, and a data integration manager (1120) for loading a 3D asset corresponding to the received context from a database (3D Assets for MR Navigation Database) (1130).

[0454] The above interface API (1101) may be named a server interface API (1101) to distinguish it from the interface API (912) of the MR AMS client (910) equipped in the vehicle.

[0455] Additionally, the interface API (912) of the MR AMS client (910) may be named a vehicle interface API or an MR AMS client interface API.

[0456] The interface API (1101) provided in the MR AMS server (1100) can transmit a user request (or context request) received from the MR AMS client to the service aggregation manager (1110).

[0457] The above interface API may include a first interface API (1101) that calls a function for performing communication with the MR AMS client (910) and a second interface API (1102a, 1102b, 1102c) that calls a function for performing communication with the service aggregation manager (1110) and the service provider (1300a, 1300b, 1300c).

[0458] The above second interface API (1102a, 1102b, 1102c) can receive service data and / or map data through the interface API provided to the service provider (1300a, 1300b, 1300c).

[0459] The interface APIs provided in the second interface API (1102a, 1102b, 1102c) and the service provider (1300a, 1300b, 1300c) may include functions configured to perform mutual data transmission and reception and to convert data formats or message formats, and these functions may be used to convert data formats or message formats to enable mutual data transmission and reception.

[0460] The service aggregation manager (1110) can request the requested context from different service providers based on the type of context requested from the MR AMS client (910) equipped in the vehicle.

[0461] Specifically, the service aggregation manager (1110) may, if the type of the requested context is a first type of context, request the first type of context from a first service provider (1300a) that provides the first type of context, and if the type of the requested context is a second type of context different from the first type, request the second type of context from a second service provider (1300b) that provides the second type of context.

[0462] For example, if the type of requested context is about a POI (e.g., “Starbucks”), the service aggregation manager (1110) can request and receive context (or POI data) about the POI from the first service provider (1300a) that provides information about the POI.

[0463] Additionally, if the type of the requested context is a view of a street, the service aggregation manager (1110) can request and receive context (or imagery data) for the view of a street from a second service provider (1300b) that provides information about the view of the street.

[0464] Additionally, if the type of the requested context is a service, the service aggregation manager (1110) may request and receive context related to the service (or data about the service (e.g., service ratings or prices)) from a third-party service provider (1300c) that provides information related to the service.

[0465] Additionally, the interface API (1101) can request a confirmed service API (Expand service API calls) to the service aggregation manager (1110) based on a service (or context request) requested from the MR AMS client (910).

[0466] The service aggregation manager (1110) can request and receive information corresponding to a confirmed service from a service provider (1300a, 1300b, 1300c) based on the extended service API request, and use the information to create a service API and output it to the data integration manager (1120).

[0467] The above data integration manager (1120) can perform data enhancement based on the service API received from the service aggregation manager (1110), generate a metadata package for the requested context, and transmit it to the vehicle's MR AMS client (910) through the interface API (1101).

[0468] The above metadata package may include the 3D asset and service metadata described above. Here, the service metadata may refer to metadata for providing a service corresponding to the requested context.

[0469] Meanwhile, the interface API (1101) can transmit a 3D asset loaded from the data integration manager (1120) to the MR AMS client (910).

[0470] Meanwhile, the MR AMS server (1100) of the present invention may further include the context manager (911) described above.

[0471] That is, the context manager (911) may be included in the MR AMS client (910) and installed in the vehicle unit, may be included in the MR AMS server (1100) and installed in the server (cloud) unit, or may be installed in both units.

[0472] The above context manager (911), when equipped in the MR AMS server (1100), can be configured to manage a context corresponding to a request received from the MR AMS client (910).

[0473] The above context manager (911) may include a context handler (911a) that handles and parses a context request, a context interpreter (911b) that manages a session for interpreting a context request and creates a context set using a data model, and a context graph database (Context Graph DB or MR Context DB) (911c) that stores the data model.

[0474] Here, the context handler (911a) can receive a user request input to the MR AMS client through the interface API (1101), parse the received user request, and transmit it to the context interpreter (911b).

[0475] The above context interpreter (911b) can create a query for a context request corresponding to the user request after creating a session, and request and receive a context data model corresponding to the query from the context graph database (911c).

[0476] The context interpreter (911b) requests a context corresponding to the context data model from the service aggregation manager (1110), and the service aggregation manager (1110) can request and receive context data corresponding to the context data model from the service provider (1300a, 1300b, 1300c).

[0477] The above service aggregation manager (1110) can request and receive a 3D asset (and / or service metadata) corresponding to the requested context from the data integration manager (1120), and transmit the context data received from the service provider and the 3D asset (and / or service metadata) received from the data integration manager to the context interpreter (911b).

[0478] The context interpreter (911b) can transmit the received context data and the 3D asset to the MR AMS client (910) equipped in the vehicle through the context handler (911a) and the interface API (1101).

[0479] Meanwhile, the context manager (911) may further include a context recorder (911d) that extracts a recommended context based on the above-mentioned generated context set, and a context controller (911e) (or context tracker) that manages contexts that must be periodically acquired.

[0480] The above context recorder (911d) may request the context interpreter (911b) to generate a query to recommend a service that can replace the specific service when the completed context data contains information that a specific service is unavailable.

[0481] Figure 9 is a conceptual diagram for explaining the DTaaS server of the present invention.

[0482] Referring to FIG. 9, the DTaaS (Digital Twin as a Service, or Digital Transformation as a Service) server (1200) of the present invention is installed outside a vehicle and can provide a mixed reality automotive meta service (MR AMS). Specifically, the DTaaS server (1200) can provide a digital twin map or data necessary for creating a digital twin map (e.g., all types of information regarding a 3D polygon map or an object overlapping on a digital twin).

[0483] The DTaaS server (1200) may include a DTaaS API (1210) that calls a function for communicating with an MR service device (900) equipped in a vehicle, a database (Digital Twins Maps DB) (1220) that stores a digital twin map and a renderable three-dimensional polygon map provided to the MR service device, and a processor (1280) that transmits a three-dimensional polygon map corresponding to the location information of the vehicle received from the MR service device to the MR service device through the DTaaS API.

[0484] In addition, the DTaaS server (1200) may further include a communication unit (TeleCommunicatino Unit, TCU) (1290) that performs communication with the MR AMS server (1100) that is installed outside the vehicle and provides MR AMS service.

[0485] In addition, the DTaaS server (1200) may further include a digital twin map generation unit (Digital Twin Representation and Update Unit) (1230) that creates a digital twin map by matching an actual captured image to a 3D polygon map stored in the database (1220).

[0486] In addition, the DTaaS server (1200) may further include a dynamic model database (Dynamics Modeling DB) (1240) that stores dynamic information about moving objects received from at least one of the MR service device (900) and the MR AMS server (1100) and a scenario database (Scenarios DB) (1250) that stores information related to scenarios that can be implemented in a digital twin.

[0487] In addition, the DTaaS server (1200) may further include a simulation unit (1260) that performs a simulation corresponding to a user request on the digital twin and a visualization unit (1270) that visualizes information to be implemented on the digital twin.

[0488] All of the components described above can be implemented as independent hardware (e.g., chips or modules), and can also be implemented as software-blocked components as needed.

[0489] The above DTaaS server (1200) can transmit and receive data through the DTaaS API (1210) not only with the vehicle (100), but also with a server (FMS Server) (1280) that provides a fleet management service (or vehicle fleet management service) and a server (1290) that provides a city planning service.

[0490] For example, the DTaaS server (1200) can collect log information collected from each server from at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290).

[0491] Afterwards, the DTaaS server (1200) can store the collected log information in a log database.

[0492] The DTaaS server (1200) can provide a digital twin map for visualization in at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290) based on the collected log information.

[0493] Additionally, the DTaaS server (1200) may transmit at least one of event notification information, simulation information, and visualization information to at least one of the vehicle (100), the FMS server (1280), and the urban planning service provision server (1290) based on the received log information.

[0494] FIG. 10 is a block diagram illustrating the structure of a 3D map view generation device (1300) connected to a cloud server (1350) according to an embodiment of the present invention.

[0495] Referring to FIG. 10, the MR AMS (Mixed Reality Automotive Meta Service) server (1100) described in FIG. 8d can be connected to various service providers (1351) that provide online map services, such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP. Furthermore, based on the results of aggregating map data provided from the connected service providers (1351), shape information of each building included in the map, for example, floor coordinate (footprint) information and height information of a building, can be aggregated and the aggregated information can be provided to DTaaS (1352). Here, DTaaS (1352) may refer to a DTaaS server (1200), i.e., a server or device that provides a service using a digital twin map.

[0496] Meanwhile, DTaaS (1352) may be connected to a POI database in which POI service data for each building or each area included in the map data is stored. In addition, it may be connected to a 3D model database in which data of a 2.5D polygon model for each building included in the map data is stored. Here, the 2.5D polygon model is a polygon model that can provide a building volume and may be a polygon model without a texture formed on the surface. The DTaaS (1352) may receive service data related to POI from the connected POI database, and may receive data of 2.5D polygon models of each building included in the map data of an area from the connected 3D model database.

[0497] The processor (1330) of the 3D map view generation device (1300) can receive various information related to the driving of the vehicle from the camera (310), the sensing unit (120), and the navigation system (770). For example, the processor (1330) can receive information on objects detected in front, behind, or to the side of the vehicle from the camera (310). In addition, the processor (1330) can receive information related to the speed, driving direction, and current location (GPS) of the vehicle from the sensing unit (120) including sensors connected to each component of the vehicle including the driving system (710). In addition, the processor (1330) can receive information related to the driving path of the vehicle from the navigation system (770).

[0498] Meanwhile, the 3D map view creation device (1300) and DTaaS (1352) may be connected through the interface (Interface APIs) of the MR AMS server (1100). In this case, the 3D map view creation device (1300) may be configured to correspond to the MR AMS client (910).

[0499] Here, the interface (Interface APIs) of the 3D map view creation device (1300) and the MR AMS server (1100) can be connected via a wireless network connection. In this case, the MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the 3D map view creation device (1300).

[0500] In this way, when connected to the MR AMS server (1100), the 3D map view generation device (1300) can provide at least some of the information received from the connected components to the MR AMS server (1100) via a network connection. Then, the MR AMS server (1100) can provide 3D map data for providing mixed reality to the 3D map view generation device (1300) in response to the provided information.

[0501] The 3D map data provided here is map data including a polygon model provided in the 3D model DB, and may include 2.5D map data providing a simple polygon model or 3D map data providing a three-dimensional virtual object.

[0502] For example, the 3D map view generation device (1300) can provide information on objects detected around the vehicle, information on the speed and direction of the vehicle, information on the current location of the vehicle, and information on the driving path of the vehicle to the MR AMS server (1100). Then, the MR AMS server (1100) can provide 2.5D map data of an area according to the current location of the vehicle to the 3D map view generation device (1300) based on the information provided from the 3D map view generation device (1300).

[0503] In this case, the MR AMS server (1100) may determine POI information based on the current vehicle location, direction, speed, and driving path, and may provide 3D map data that further includes the determined POI information to the 3D building map. In addition, the MR AMS server (1100) may provide 3D map data that further includes information on the situation around the vehicle to the 3D map view generation device (1300) based on the information on objects around the vehicle provided.

[0504] Meanwhile, the 3D map view generation device (1300) can render a mixed reality image based on the 3D map data provided from the MR AMS server (1100). For example, the 3D map view generation device (1300) can control an MR renderer to display a 3D map screen including models of buildings around the vehicle based on the provided 3D map data. In addition, the device can display a graphic object corresponding to the vehicle on the 3D map screen, and display graphic objects corresponding to the provided POI data and situation information around the vehicle on the 3D map screen.

[0505] Accordingly, an image of a virtual environment including a three-dimensional building model similar to the shape of the current vehicle and the buildings around the vehicle and graphic objects corresponding to the vehicle can be output to a display unit (251) such as a CID (Center Information Display), HUD (Head Up Display), RSI (Rear Sheet Information), or RSE (Rear Sheet Entertainment).

[0506] In this case, information related to the vehicle's driving and the surroundings of the vehicle can be provided to the driver through the virtual environment. Through the 3D map data, i.e., the digital twin map (hereinafter referred to as the "DT map"), the 3D map view generation device (1300) according to an embodiment of the present invention can provide the driver with a mixed reality service.

[0507] Meanwhile, the MR AMS server (1100) can determine 3D map data and POI information or situation information around each vehicle that can be provided together with the 3D map data based on information collected from not only a 3D map view generation device (1300) equipped in one vehicle but also 3D map view generation devices (1300) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect information from multiple vehicles in the form of a cloud server and generate 3D map data for mixed reality based on the collected information. In addition, it can be implemented to transmit MR information for providing a mixed reality service to at least one 3D map view generation device (1300) equipped in different vehicles based on the generated 3D map data.

[0508] For convenience of explanation, in the following, the MR AMS server (1100) that provides 3D map data, i.e., a digital twin map (DT map), for providing a mixed reality service by being connected to the DTaaS (1352) and the DTaaS (1352) are collectively referred to as a cloud server (1350).

[0509] Meanwhile, the 3D map view generation device (1300) may be a device that controls a display equipped in a vehicle through an interface unit. Alternatively, the 3D map view generation device (1300) may of course be a display device equipped in the vehicle. In this case, the 3D map view generation device (1300), that is, the 3D map view generation device (1300), may be equipped with a display unit, and may receive the updated DT map on which the real-world texturing has been performed from a cloud server, and directly display an MR view image including the received DT map on the display unit equipped in the 3D map view generation device (1300).

[0510] In the following description, for convenience of explanation, it is referred to as a 3D map view generation device (1300).

[0511] Meanwhile, a vehicle (100) related to the present invention may include a 3D map view generation device (1300).

[0512] The 3D map view generation device (1300) is capable of controlling at least one of the components described in FIG. 7. From this perspective, the 3D map view generation device (1300) may be a control unit (170).

[0513] Without being limited thereto, the 3D map view generation device (1300) may be a separate configuration independent from the control unit (170). When the 3D map view generation device (1300) is implemented as a component independent from the control unit (170), the 3D map view generation device (1300) may be provided in a part of the vehicle (100).

[0514] Hereinafter, for convenience of explanation, the 3D map view generation device (1300) will be described as a separate component independent from the control unit (170). The functions (operations) and control methods described for the 3D map view generation device (1300) in this specification can be performed by the control unit (170) of the vehicle. That is, all contents described in relation to the 3D map view generation device (1300) can be analogically applied to the control unit (170) in the same / similar manner.

[0515] In addition, the 3D map view generation device (1300) described in this specification may include some of the components described in FIG. 7 and various components provided in the vehicle. In this specification, for convenience of explanation, the components described in FIG. 7 and various components provided in the vehicle will be described by assigning separate names and drawing numbers.

[0516] Figure 11 is a conceptual diagram for explaining the above 3D map view generation device (1300).

[0517] A 3D map view generation device (1300) according to one embodiment of the present invention may include a communication unit (1310), an interface unit (1320), a memory (1340), and a processor (1330).

[0518] The above communication unit (1310) may be configured to perform wireless communication with at least one of the electrical components equipped in the vehicle (for example, the electrical components equipped in the vehicle as shown in FIG. 7).

[0519] In addition, the communication unit (1310) may be configured to perform communication with devices other than the vehicle, such as a mobile terminal, a server, another vehicle, infrastructure installed on the road, etc.

[0520] The above communication unit (1310) may be the communication device (400) described above, and may include at least one of the components included in the communication device (400).

[0521] The interface unit (1320) can communicate with at least one of the components provided in the vehicle.

[0522] Specifically, the interface unit (1320) can be configured to perform wired communication with at least one of the components provided in the vehicle as shown in FIG. 7.

[0523] The above interface unit (1320) receives sensing information from one or more sensors provided in the vehicle (100).

[0524] The above interface unit (1320) may be referred to as a sensor data collector.

[0525] The above interface unit (1320) collects (receives) information sensed through sensors provided in the vehicle (e.g., sensors for detecting vehicle operation (V.Sensors) (e.g., heading, throttle, break, wheel, etc.) and sensors for sensing vehicle surrounding information (S.Sensors) (e.g., Camera, Radar, LiDAR, Sonar, etc.)).

[0526] The above interface unit (1320) can transmit information sensed through a sensor equipped in the vehicle to the communication unit (1310) (or processor (1330)) so that it is reflected in a high-precision map.

[0527] The interface unit (1320) may, for example, serve as a passageway with electrical components installed in the vehicle through the vehicle interface unit (130).

[0528] The interface unit (1320) can exchange data with the vehicle interface unit (130).

[0529] The interface unit (1320) can be connected to a vehicle and serve as a passage for supplying electric energy.

[0530] For example, the route guidance device can be powered on by receiving electric energy from the vehicle's power supply unit (190) through the interface unit (1320).

[0531] Meanwhile, the present invention may include a memory (1340) that stores data supporting various functions of the 3D map view creation device (1300). For example, the memory (1340) may store a plurality of application programs (or applications) that can be executed by the processor (1330), data for the operation of the 3D map view creation device (1300), and commands.

[0532] As an example of such data, the memory (1340) may store building reference information, which is information on actual buildings corresponding to each of the building virtual objects included in the map data. The building reference information may store information on the height of the actual building, planar shape information (e.g., footprint information) which is the shape of the land occupied by the actual building, and information on the actual location of each building, i.e., the latitude and longitude of each building. In addition, the building reference information may include information on the shape of each side of each building or various three-dimensional shapes of the building. In addition, the building reference information may include information related to the use, type, and age of each building corresponding to each building virtual object.

[0533] The above memory (1340) can store building reference information corresponding to at least one of the building virtual objects included in the map data. Hereinafter, an area of ​​the memory (1340) where the building reference information is stored will be referred to as a building reference information storage unit (1343).

[0534] In addition, the memory (1340) may store building image setting values ​​including setting values ​​for angles and projection directions for generating building virtual objects, modeling setting values ​​for 2.5D building virtual objects formed of polygons, or modeling setting values ​​for 3D building virtual objects having a three-dimensional shape, as information for modeling the building virtual objects. In this way, based on the building reference information and building image setting values ​​stored in the memory (1340), the processor (1330) may calculate various building setting values ​​such as height, area, size, area of ​​each side, margin value, and offset value of the building virtual object to be generated, and may generate each building virtual object based on the calculated building setting values. Here, the building setting value may include a value related to at least one of the height, location, planar shape, three-dimensional shape, and shape of each side of the building included in the building reference information.

[0535] In addition, the memory (1340) can further store data for generating surface materials, i.e., surface textures, of each building virtual object. For example, the memory (1340) can store images of a portion of a building including shapes of various building elements. The images are images for forming the surface texture of the building (hereinafter, texture images) and can be formed as 2D images. In addition, the memory (1340) can store data of images (hereinafter, normal maps) corresponding to normal maps having different vector values ​​according to different materials so as to provide texture to the texture images. In addition, images (hereinafter, albedo maps) corresponding to albedo maps having material-specific color values ​​for each different material can be stored.

[0536] Meanwhile, the memory (1340) may store 2D images including shapes corresponding to building components. The 2D image may be an image including images corresponding to at least one building element for describing the texture of the building. In addition, each 2D image may include different building elements depending on the type, and the shapes of the included building elements may also be different. The 2D image may be a texture image that can form a surface texture (material) of the building virtual object in a form that is tiled on the surface of the building virtual object.

[0537] Meanwhile, the texture image may be synthesized with the outer surface of the building virtual object in a tiled form as described above. Here, the tiling may refer to a process of filling the surface area of ​​the building virtual object with tiles normalized to a certain size in a horizontal and vertical direction without overlapping each other. Accordingly, the texture image may refer to a tile, which is a unit image synthesized on the outer surface of the building virtual object through the tiling. Therefore, in the following description, a texture image texturing the outer surface of the building virtual object in a tiling manner may also be referred to as a tile. Here, the texturing may refer to an operation process of adding a texture representing a surface texture (material) to the surface area of ​​the building virtual object and synthesizing a texture on the surface area of ​​the building virtual object using a tile.

[0538] Additionally, the surface area of ​​the virtual object may be composed of a plurality of different sub-areas. In this case, each of the plurality of sub-areas may be matched with a different tile, and each of the plurality of areas may be tiled using different matching tiles.

[0539] Meanwhile, the above tiles may be grouped into different groups according to the characteristics of the building to which the tiles can be applied, such as the type, purpose, or size of the building. For example, the tiles may be grouped into residential buildings, commercial buildings, or officetel buildings. In this case, the tiles grouped into different groups, i.e., the tiles grouped into the residential building group, the tiles grouped into the commercial building group, and the tiles grouped into the officetel building group, may be tiles normalized to different sizes according to each group.

[0540] In addition, some of the tiles may be associated with at least one other tile. For example, the associated tiles may be tiles that are similar in shape and color. That is, the associated tiles may be tiles that differ only in at least a portion of their shape or in color. Alternatively, the associated tiles may be tiles that are similar to a certain degree in at least one of their shape and color. These associated tiles may reflect changes over time. For example, tiles that have the same shape but differ only in color may be used to reflect changes over time in a portion of a specific building.

[0541] Meanwhile, memory (1340) may contain association information regarding tiles that are interconnected in this manner. Furthermore, through association information, a specific tile can be associated with another tile. Therefore, by utilizing this association information, even tiles with different shapes or colors can be associated with each other.

[0542] The tiles stored in the above memory (1340) may form a database including each tile and tile information corresponding to each tile. In this case, the tile information stored in the database may include address information of each tile, information on a group to which each tile is included (group information), and, if there is another tile associated with the corresponding tile, association information including information on the other associated tile. Hereinafter, a database including tiles and tile information corresponding to each tile will be referred to as a tile database (DB) (hereinafter referred to as a tile DB) (1341).

[0543] Meanwhile, the memory (1340) may store map data including virtual objects. Here, the map data may be formed by connecting map data from multiple different regions in tile form. In this case, each map tile may represent a logical rectangular section corresponding to a certain area, and may be used to subdivide a wide map area. In other words, the map data may be formed by data from map tiles each corresponding to a different region.

[0544] Meanwhile, the above map data may be map data including data of a so-called 2.5D polygon model. In addition, the map data may include high-definition (HD) map data having high accuracy. The map data may be linked to a navigation system (770) and a driving system (700) provided in a vehicle (100), and the memory (1340) may provide the map data to various linked systems and components under the control of the processor (1330). Such map data may be stored in a map data storage unit (1342).

[0545] Meanwhile, a 3D map view generation device (1300) according to an embodiment of the present invention may include a processor (1330) that generates a 3D map as a digital twin using at least one of an image captured by a camera (310) provided in a vehicle (100), 2D map data (e.g., HD map data), and 3D map data (e.g., map data including data of a 2.5D polygon model). The processor (1330) controls each connected component and may control the overall operation of the 3D map view generation device (1300).

[0546] Additionally, the processor (1330) can overlap (or superimpose, output) graphic objects related to route guidance on a digital twinned 3D map.

[0547] Here, the graphic object related to route guidance refers to an object output as mixed reality (MR), and may include various types of objects (e.g., POI objects, carpet-shaped objects, 3D objects, etc.) required to perform route guidance. Here, the graphic object related to the route guidance may also be referred to as an MR object.

[0548] When the destination of the vehicle (100) is set, the processor (1330) can determine a route for the vehicle (100) to drive through the navigation system (770). And, when the driving route of the vehicle (100) is determined, the processor (1330) can determine at least one virtual object (e.g., an object output as the mixed reality (MR)) to perform texturing on a digital twin 3D map for route guidance.

[0549] Meanwhile, in order to determine virtual objects to be displayed on the digital twinned 3D map, the processor (1330) may first determine a map area in which to generate the virtual objects, i.e., to model. To this end, the processor (1330) may first detect, from map data stored in the memory (1340), at least one map tile around a map tile corresponding to an area in which the vehicle (100) is located, according to the current location of the vehicle (100) and the driving direction of the vehicle (100). For example, the processor (1330) may determine an area corresponding to 5 horizontal and 5 vertical maps centered on the map tile in which the vehicle (100) is located, i.e., a total of 25 map tile data, as a map area, according to a zoom level of a specified map, and may detect map tile data corresponding to the determined map area from the map data.

[0550] And the processor (1330) can detect buildings included in the detected map area. And it can detect building reference information corresponding to the detected buildings from the building reference information storage unit (1343). And the setting values ​​of each building image related to the detected building reference information can be determined. Here, the building image setting values ​​are information for modeling each building virtual object, and may include a setting value of a projection direction according to an angle or distance for generating the building virtual objects according to the driving direction of the vehicle (100) and / or the angle at which the MR view is displayed, a setting value of modeling according to a 2.5D virtual object or a setting value of modeling according to a 3D virtual object, for example, a vector value such as a coordinate value of a vertex or a normal line.

[0551] Once the building image setting values ​​are determined, the processor (1330) can calculate building setting values ​​corresponding to each building included in the map area based on the determined building image setting values ​​and building reference information of each building. Here, the building setting values ​​can include the height of the building virtual object to be created, the side area of ​​each side, the plane area, etc. Then, once the calculation of the building setting values ​​is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included in the map area based on the building setting values.

[0552] And when the modeling of virtual objects included in the above map area is completed, the processor (1330) can determine virtual objects to be displayed on the display unit (251) through rendering among the modeled virtual objects. To this end, the processor (1330) can determine a visible area of ​​a frustum shape formed according to the driving direction of the vehicle (100) based on the current position of the vehicle (100), determine building virtual objects included in the visible area of ​​the frustum shape as objects to be rendered, and perform texturing on the determined building virtual objects.

[0553] Meanwhile, the processor (1330) may pre-model building virtual objects corresponding to each building included in the map area, and render and display only at least some of the modeled building virtual objects on the display unit according to the visible area of ​​the frustum shape determined according to the position and driving direction of the vehicle (100). In this case, the texturing may be performed during the rendering process. Therefore, by pre-modeling objects to be rendered and selectively rendering only some of the modeled objects as needed, texturing of virtual objects that are not displayed on the display unit can be prevented. Therefore, the amount of computation required for the texturing can be reduced, and the time for virtual objects to be displayed on the display unit can be significantly reduced.

[0554] Moreover, the processor (1330) may be implemented as a plurality of processors. For example, the processor (1330) may be implemented as a processor that models the building virtual objects and a processor that renders some of the modeled virtual objects as different processors. For example, the processor that models the virtual objects may be a central processing unit (Central Processing Unit) that interprets various program commands of the 3D map view generation device (1300) according to an embodiment of the present invention and performs data calculations and processing. On the other hand, the processor that renders some of the pre-modeled virtual objects may be a GPU (Graphics Processing Unit) that processes graphic calculations in the 3D map view generation device (1300) according to an embodiment of the present invention and outputs result values. In this case, since the modeling of the virtual object and the rendering of the modeled virtual object can be processed in parallel on different processors, the computational load on the processor can be significantly reduced compared to the case where all processes are performed on a single processor, and by displaying the virtual object by performing rendering from objects that have been modeled in advance, the time required until the virtual object is displayed on the display unit can also be significantly shortened. Furthermore, the 3D map view generation device (1300) according to the embodiment of the present invention described above generates modeling of building virtual objects corresponding to each building included in the map area in advance, and renders only some virtual objects according to some areas (e.g., the visible area of ​​the frustum shape) among the generated building virtual objects, thereby further shortening the time for displaying the building virtual objects, and thus real-time provision of an MR view according to the driving of the vehicle (100) can be made possible.

[0555] In this way, since the present invention has a configuration that pre-generates modeling of building virtual objects corresponding to each building included in the map area, when the map area needs to be updated according to the location and driving direction of the vehicle (100), the processor (1330) can additionally detect at least one map tile data from the map data through the update. In addition, the virtual objects of the buildings included in the area of ​​the additionally detected map tile can be further modeled. On the other hand, the processor (1330) can delete virtual objects of the buildings included in the area of ​​the map tile that has deviated from the map area according to the update of the map area for efficient memory utilization.

[0556] Meanwhile, when the destination of the vehicle (100) or the route to the destination is determined, the processor (1330) may additionally detect at least one map tile data from the map data based on the determined destination or the route to the destination. In this case, the additionally detected at least one map tile data may be map tile data that are connected to each other. In addition, when the additionally detected at least one map tile data is additionally detected, the processor (1330) may model virtual objects corresponding to each of the buildings included in the additionally detected map tile data.

[0557] In this case, virtual objects modeled from the additionally detected map tile data may be rendered and displayed on the display unit (251) regardless of the driving information of the vehicle (100) when a preset condition is met. For example, the preset condition may be met when a user requests a search for a route to a destination or a check of the surroundings of the destination. In this case, regardless of the current driving status of the vehicle (100), building virtual objects modeled from the additionally detected map tile data may be rendered and displayed along the shortest path to the set destination or the searched driving path to the destination. In this case, texturing of the building virtual objects may be performed during the rendering process.

[0558] Meanwhile, texturing of the above-mentioned modeled virtual objects can be performed in a manner in which texture images are tiled, and can be performed by sequentially combining multiple sub-regions constituting the virtual object.

[0559] For example, the virtual object corresponding to the target building may have a surface area divided into multiple layers according to the exterior appearance of the target building, and a partial area may be formed for each of the divided surface areas. For example, the processor (1330) may divide the target building into a bottom layer, a top layer, and at least one middle layer, and different tiles may be matched to each partial area.

[0560] In this case, the top floor is distinguished when it has a shape or form that is distinct from the middle floor, such as a roof, an antenna, or a spire. If the target building does not have a structure placed on the roof of the building, such as a roof, an antenna, or a spire, the processor (1330) may distinguish the target building only by the bottom floor and at least one middle floor.

[0561] Meanwhile, in the case of an intermediate layer, the processor (1330) can be divided into at least one layer according to a preset floor height or window arrangement, etc. For example, if the intermediate layers have different colors or shapes (e.g., different window arrangements), the layers with the different colors or shapes can be distinguished as different intermediate layers. That is, in the case of floors designated as commercial spaces and floors designated as residential spaces, such as in a mixed-use building, the floors designated as commercial spaces and the floors designated as residential spaces can each be distinguished as different intermediate layers.

[0562] In this case, different layers, for example, the ground floor, at least one intermediate layer, and the top layer, can be matched with different tiles. And based on the different matching tiles, tiling can be performed for each layer, i.e., each sub-region (e.g., the ground floor, at least one intermediate layer, and the top layer). Then, each sub-region for which tiling has been performed can be sequentially combined in a preset order based on combination information specified in the building reference information corresponding to the target building.

[0563] Meanwhile, the tiling may be sequentially performed based on a tiling order specified in the building reference information. For example, each of the sub-regions may be sequentially combined starting from a sub-region corresponding to a floor close to the ground. In this case, tiling may be performed first for a sub-region corresponding to an area in contact with the ground, i.e., the bottom floor, and then tiling may be performed for a sub-region corresponding to the next order (e.g., the first middle floor) on top of the sub-region already tiled. Finally, tiling may be performed for a sub-region corresponding to the uppermost floor (e.g., the top floor). In this way, by sequentially performing tiling for each sub-region in a preset order for the virtual object, texturing may be performed for the entire surface area of ​​the virtual object.

[0564] In this way, texturing of a virtual object according to an embodiment of the present invention can be performed based on the sequential combination of each tiled sub-region. Therefore, texturing of a virtual object according to an embodiment of the present invention can also be referred to as procedural texturing.

[0565] Meanwhile, the tile DB (1341) may be provided from a preset server that is communicatively connected to the 3D map view generation device (1300). For example, the preset server may be a cloud server. In this case, the cloud server may be the same server as the cloud server that provides map data (e.g., map data including 2.5D polygon objects), but of course, they may be different servers. In this case, that is, if they are different servers, the cloud server that provides the map data will be described as a first cloud server, and the cloud server that provides the tile DB (1341) and building reference information will be described as a second cloud server.

[0566] When the above tile DB (1341) is provided, the processor (1330) can determine tiles to be tiled on the virtual object surface area of ​​each building included in an area defined based on driving information of the vehicle (100) based on information collected around the vehicle (100).

[0567] For example, the processor (1330) can segment an image of a building around a vehicle (100) acquired through a camera (310) into a plurality of partial regions and detect a tile matching each of the segmented images (i.e., partial region images of the target building). To this end, the processor (1330) can calculate a similarity for each tile included in the tile DB (1341) for each of the partial region images, and determine a tile matching each of the partial region images based on the calculated similarity.

[0568] For example, the processor (1330) can detect feature information for a certain partial region image. And, among the tiles included in the tile DB (1341), at least one tile having a feature similar to a feature detected from the certain partial region image can be detected. And, for each of the at least one detected tiles, a similarity to the certain partial region image can be calculated, and a certain tile having the highest calculated similarity can be matched as a tile corresponding to the certain partial region image.

[0569] For this tile matching, the processor (1330) may utilize a pre-trained artificial intelligence (AI) model. In this case, the AI ​​model may recognize a partial region image from features detected in the partial region image, and detect a tile having features most similar to the feature points of the recognized partial region image.

[0570] Here, the feature points of the partial region image can be detected in various ways. For example, the shape, form, or color pattern of the image can be detected as the feature point. In this case, the more distinct the shape, form, or color pattern of the partial region image, the more clearly the AI ​​model can detect the pattern from the partial region image, resulting in a higher recognition rate.

[0571] On the other hand, if it is difficult to detect a pattern of shape, form or color from the partial region image due to low image quality, brightness difference, shaking, etc., the artificial intelligence model may not be able to detect the feature points of the partial region image, in which case a low recognition rate may be produced. Then, based on the recognition rate, the artificial intelligence model may not detect a tile corresponding to the partial region image. In other words, the artificial intelligence model may determine whether to detect a tile corresponding to any one of the partial region images based on the produced recognition rate.

[0572] Here, the artificial intelligence model may be included in the 3D map view generation device (1300) according to an embodiment of the present invention as a separate component from the processor (1330) (not shown). Alternatively, the artificial intelligence model may be part of the processor (1330), or the processor (1330) may perform the function of the artificial intelligence model. In this case, the artificial intelligence model may be understood as the same component as the processor (1330).

[0573] Meanwhile, when the tiles corresponding to each partial region image are determined, the processor (1330) can generate combination information related to the combination of each partial region image based on the acquired building image. For example, the combination information may be information about the location of each partial region image or the order in which tiling is performed for the acquired building image.

[0574] And the processor (1330) can model virtual objects of buildings located in an area defined according to the location of the vehicle (100) from map data. And, among the area, building reference information of buildings located in at least a part of the area (hereinafter referred to as a map area) defined according to the driving information of the vehicle (100) can be detected. And, the processor (1330) can perform texturing on virtual objects corresponding to the detected building reference information among the modeled virtual objects.

[0575] In more detail, the processor (1330) can sequentially tile surface areas of virtual objects corresponding to the detected building reference information using tiles pre-designated for each sub-area.

[0576] For example, if the target building is formed of three layers, including a ground floor, a first intermediate floor, and a second intermediate floor, the processor (1330) may sequentially tile tiles matching each sub-area in a preset order onto the surface area of ​​a virtual object corresponding to the target building. Here, the preset order may be an order from the ground.

[0577] Accordingly, tiles matching the bottom area, i.e., the partial area (bottom layer) included in the building reference information, can be tiled on the surface of a virtual object in an area in contact with the ground in the virtual object in the map data corresponding to the target building. Then, when tiling for the bottom layer is completed, the processor (1330) can tile tiles matching the next partial area (first intermediate layer) in a preset order on the partial area (bottom layer) for which tiling has been completed. Then, when tiling for the first intermediate layer is completed, the processor (1330) can tile tiles matching the next partial area (second intermediate layer) in a preset order on the partial area (first intermediate layer) for which tiling has been completed. In this way, by sequentially tiling the tiles matching each partial area on the surface area of ​​the target building, texturing can be performed on the virtual object in the map data corresponding to the target building.

[0578] Here, the height between each floor of the target building may correspond to an integer multiple of the tile. For example, if the height between each floor of the target building is equal to the height of the tile, the processor (1330) may perform tiling for each sub-region by arranging tiles matching each sub-region in a non-overlapping manner.

[0579] To this end, the processor (1330) may determine an area in which the tiling is to be performed based on the left and right margin values ​​preset from the corners of each side area of ​​each virtual object (building virtual object) and the horizontal and vertical lengths of each side area of ​​the building according to the building reference information. In this case, each building surface may be divided into a plurality of sub-areas based on a floor divider that divides the top and bottom, the top layer, and at least one middle layer. In addition, the processor (1330) may configure the left and right margin values ​​and the top and bottom margin values ​​or the width of the floor divider as margins for each of the plurality of sub-areas, and determine the remaining area excluding the margins as an area in which the tiling is to be performed. In addition, in this case, the inter-floor height or inter-floor width of the bottom or top layer may be formed differently from the inter-floor height or inter-floor width of the middle layer. Additionally, among the tiles stored in the tile DB (1341), the tiles corresponding to the middle layer may be tiles that differ in at least one of width or height from the tiles corresponding to the bottom layer and the top layer.

[0580] And the processor (1330) can determine the number of tiles to be tiled horizontally or the number of tiles to be tiled vertically (tiling offset) in each sub-area where the tiling is to be performed according to the width or height of the determined area. And the processor (1330) can distinguish tiles that are tiled continuously in the surface area of ​​the building virtual object, i.e., each building element included in the texture image.

[0581] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different areas of the texture image by each color value, and accordingly, different areas of the texture image, i.e., different building elements, can be distinguished by color channel.

[0582] And the processor (1330) can generate mask maps for masking building elements for each distinguished building element in each surface area of ​​the tiled building virtual object. The mask map is a map formed so that building elements other than a certain building element are hidden in the surface area of ​​the tiled building virtual object, and may mean an image used to specify only one building element among each building element included in the texture image. In other words, the mask map corresponding to a specific building element may be an image in which other areas are hidden except for an area corresponding only to the specific building element, and may be an image for applying a specific effect only to an area corresponding to the specific building element.

[0583] Meanwhile, when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps with shading effects so that each mask map can display a different material texture.

[0584] Normally, when there is a curve in the surface of an object, the direction of the surface changes due to the curve, and light is reflected differently depending on the changed surface direction, and the curve of the surface of the object can be felt by the reflected light that is reflected differently. The normal map is used to express the curved surface using the reflected light of the curve rather than the actual curve, and may mean an image having a 3D vector value that represents the 3D direction of the surface of the object required for calculating the degree of shading corresponding to the reflected light reflected by the surface curve of the specific material when light incident on a specific material is incident on the surface of the specific material. When such a normal map is used, even if it is a 2D image that does not have a three-dimensional shape, the reflected light corresponding to the curve according to the specific material can be expressed for the light incident on the building, so the texture of the specific material can be formed.

[0585] Accordingly, the processor (1330) can blend normal maps corresponding to different materials with respect to the mask maps corresponding to each building element. Then, a 3D vector value according to the normal map can be applied to the building element corresponding to each mask map, so that a material texture according to the normal map can be expressed.

[0586] Here, in the case of an area covered by the mask map, the 3D vector value may not be applied even when the normal map is blended. Therefore, the normal map to which the mask map is blended may be textured with a specific material only in an area not covered by the blended mask map. In this case, the area not covered by the mask map may be an area corresponding to a specific building element specified by the mask map. Therefore, the normal map to which the mask map is blended may be an image to which a 3D vector value is applied so that a surface texture of a specific material is expressed only in a specific building element that is not masked by the mask map.

[0587] Meanwhile, the mask map may be performed for each distinguished building element in each surface area of ​​the building virtual object in which texture images are tiled. Accordingly, multiple mask maps may be generated for each building element in the surface area of ​​the building virtual object. Each mask map may be blended with a normal map of a different material. The processor (1330) may perform blending between normal maps to which the mask maps are blended.

[0588] Accordingly, the processor (1330) can generate a single normal map (hereinafter referred to as a multi-blended normal map) in which 3D vector values ​​corresponding to surface textures of different materials are applied to areas corresponding to different building elements, respectively, for the surface area of ​​the building virtual object through blending between normal maps to which the mask map is blended. Then, by synthesizing the generated multi-blended normal map to the surface area of ​​the building virtual object, the areas corresponding to each building element of the building virtual object can be textured to have different surface textures.

[0589] Here, the processor (1330) can generate a normal map in which 3D vectors corresponding to different surface textures are applied to areas corresponding to each building element included in the texture image for each texture image tiled on the surface area of ​​the building virtual object. Then, the generated normal map, i.e., the multi-blended normal map, can be synthesized for each texture image arranged on the surface area of ​​the building virtual object. In this case, since the texture image is synthesized in a tiling manner on the surface area of ​​the building virtual object, the multi-blended normal map synthesized for each texture image can also be synthesized in a tiling manner on the surface area of ​​the building virtual object.

[0590] Meanwhile, a normal map having 3D vector values ​​corresponding to the surface texture of the specific material applied may be blended with an albedo map for representing the color of the specific material. In this case, the albedo map may be an image having a color value unique to the specific material. Accordingly, a normal map having 3D vector values ​​corresponding to the texture of the specific surface material may form a pair with an albedo map having a color value unique to the specific surface material, and may be blended together.

[0591] Therefore, when an albedo map is blended with a normal map, the normal map may have a 3D vector value corresponding to the surface texture of a specific material as well as a color value unique to the specific material. Therefore, when a mask map corresponding to a specific building element and a normal map are blended, and the albedo map is further blended to the blended normal map, the specific building element corresponding to the mask map may have a color value unique to the specific material as well as a surface texture of the specific material. In this case, a multi-blended normal map corresponding to the entire surface area of ​​a building virtual object or a texture image tiled on the building virtual object may be an image in which 3D vector values ​​having different material textures are applied to different areas corresponding to a plurality of building elements, and each area is expressed with a color of a different material.

[0592] Meanwhile, when texturing of a surface area of ​​a building virtual object is completed through the multi-blending normal map, the processor (1330) can control the MR service device (900) to MR render the building virtual object for which texturing has been completed and display it on the display unit (251). In this case, the processor (1330) can control the MR service device (900) and the display unit (251) of the vehicle (100) through the interface unit (1320).

[0593] Meanwhile, the processor (1330) may determine at least one of the colors of the normal map corresponding to each building element based on an image captured from the camera of the vehicle (100). For example, the processor (1330) may obtain an image corresponding to a visible area of ​​a frustum shape according to the current position of the vehicle (100) and the driving direction of the vehicle (100) from the camera of the vehicle (100). Then, the objects included in the received image may be classified by type and a representative color may be determined for each type of the classified object. In this case, the representative color may be the RGB average value of the color of each object classified as a specific type, or the color having the highest ratio among the colors of each object classified as the specific type.

[0594] For example, the processor (1330) can detect colors corresponding to building objects from the acquired image, and determine a specific color having the highest RGB average value or ratio of the detected colors as a representative color of the building objects included in the image.

[0595] And the processor (1330) can determine at least one color among the colors of the normal map corresponding to each building element as the determined representative color. Accordingly, at least one color among the colors of the normal map corresponding to each building element of the building virtual object can be determined from an image acquired from the camera of the vehicle (100).

[0596] Meanwhile, the above description describes modeling building virtual objects corresponding to buildings included in the map area and texturing and rendering some of the modeled building virtual objects. However, it is also possible to model and render road virtual objects corresponding to road shapes included in the map area.

[0597] To model the above road virtual object, the processor (1330) may generate a plurality of nodes along a road shape included in the map area from map data. Then, the plurality of generated nodes may be defined as segments, and polygon objects having a specific width may be generated for each segment, and the generated polygon objects may be combined to model the road virtual object. In this case, the polygon objects may be round objects having a specific width.

[0598] Meanwhile, the processor (1330) may utilize a morphology operation to more smoothly express the edge of the road object. The morphology operation is one of the techniques for analyzing and transforming the shape and structure of an object within an image, and may include erosion, dilation, opening, and closing operations. Here, the processor (1330) may more smoothly express the road virtual object composed of the polygon objects through a closing operation used when filling a groove or cutting off a protruding part among the morphology operation techniques.

[0599] Meanwhile, FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view creation device (1300) and an MR service device (900) according to an embodiment of the present invention.

[0600] First, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can collect driving information of the vehicle (100). Here, the driving information of the vehicle (100) can include information on the speed, location, and driving direction of the vehicle (100) detected through the sensing unit (120). In addition, a map area for modeling virtual objects can be determined based on the collected driving information of the vehicle (100).

[0601] Meanwhile, when a map area is determined, the processor (1330) can detect building reference information of buildings included in the determined map area from the object reference information storage unit (1343). Then, when the building reference information is detected, building image setting values, which are information for modeling building virtual objects corresponding to each building, are determined, and building virtual objects corresponding to each building can be modeled based on the determined building image setting values ​​and the building reference information of each building.

[0602] And the processor (1330) can determine any texture image as a tile from the tile DB (1341) for each building or for each partial region (e.g., floor region, at least one middle region, top region) of each building. And the texture image determined as a tile can be tiled for each building or for each partial region of each building. And mask maps corresponding to different components of the building included in the tiled texture image are blended with normal maps corresponding to different materials, and each normal map blended with the mask map is blended with an albedo map having a color corresponding to each material, so that the different components of the building can be textured to have different materials and different surface textures. And by blending the textured normal maps, a multi-blended normal map corresponding to a specific surface region or partial region of the building virtual object or a specific texture image can be generated. And, by synthesizing the generated multi-blended normal map to a specific surface area or partial area of ​​the building virtual object, or by tiling a specific surface area or partial area of ​​the building virtual object along a tiled texture image, a surface texture for a surface area of ​​the building virtual object can be formed. That is, a textured virtual object can be generated.

[0603] Meanwhile, the MR service device (900) can model the location of the self-vehicle (100), pedestrians and other vehicles around the vehicle (100), POI (Position Of Interest) information, route information, and road information and terrain information around the vehicle (100) based on information provided from the vehicle system, i.e., GPS, ADAS, and navigation system of the vehicle (100). In addition, the MR service device (900) can configure a UX scene including the textured virtual objects using the modeled information and information about the modeled virtual objects provided from the 3D map view generation device (1300). In addition, the MR renderer can be controlled to generate a map image including a camera view, 3D effects, and GUI (graphics rendering), and the image can be displayed through the display unit (251).

[0604] Hereinafter, a method of determining a map area, modeling virtual objects included in the determined map area, and rendering and displaying the modeled virtual objects by a 3D map view generation device (1300) according to an embodiment of the present invention will be described in more detail with reference to the attached drawings.

[0605] FIG. 13 is a flowchart illustrating an operation process of a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention for determining a map area, modeling virtual objects included in the determined map area, blending the modeled virtual objects into a normal map having 3D vector values ​​of different materials to texture them, and rendering and displaying the textured virtual objects. FIG. 14 is an exemplary diagram illustrating an example of a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention for updating a map area according to vehicle driving information. FIG. 15 is a conceptual diagram illustrating map tile background modeling described in FIG. 14 in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0606] First, referring to FIG. 13, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can obtain driving information about the vehicle's location, speed, driving direction, destination, and path to the destination through the navigation system (770) and the vehicle's sensing unit (120) (S1300).

[0607] When driving information of a vehicle (100) is acquired, the processor (1330) can determine a map area based on a map tile corresponding to the location of the vehicle (100). The map area can be determined differently depending on the preset zoom level of the map.

[0608] For example, the map area may be an area formed by 25 map tiles centered on the map tile according to the location of the current vehicle (100) when the zoom level of the currently set map is 17. In this case, when the zoom level of the map is lowered, an area formed by a greater number of map tiles centered on the map tile according to the location of the current vehicle (100) may be determined as the map area, and when the zoom level of the map is higher, a map area formed by a smaller number of map tiles (e.g., 9) centered on the map tile according to the location of the current vehicle (100) may be determined. When the map area is determined, the processor (1330) may detect data of map tiles included in the determined map area from map data stored in the memory (1340).

[0609] Meanwhile, the above map area may be updated according to driving information of the vehicle (100). For example, when the vehicle (100) leaves the map tile where it is currently located due to driving of the vehicle (100), the processor (1330) may update the map area so that the map area includes map tiles within a range according to the zoom level of the currently set map based on the map tile into which the vehicle (100) newly enters.

[0610] FIG. 14 is an exemplary diagram showing an example in which a processor (1330) of a 3D map view generation device (1300) according to an embodiment of the present invention updates a map area according to driving information of a vehicle (100).

[0611] First, referring to (a) of FIG. 14, (a) of FIG. 14 is a drawing showing an example of a map area (1402) formed with 25 map tiles centered on a map tile according to the location of the current vehicle (100) according to the currently set zoom level '17' of the map. In this case, the map area (1402) may be formed centered on the first map tile (1400) in which the current vehicle (100) is located.

[0612] In this state, if the vehicle (100) moves from the first map tile (1400) to the second map tile (1410) according to the movement of the vehicle (100), the processor (1330) can update the map area (1402) including a number of map tiles (25) determined according to the preset zoom level of the map centered on the first map tile (1400), as shown in FIG. 14b, to a map area (1412) including a number of map tiles (25) determined according to the preset zoom level of the map centered on the second map tile (1410) according to the position of the moved vehicle (100). Accordingly, the processor (1330) can detect data of map tiles corresponding to the updated map area (1412) from map data stored in the memory (1340). In this case, the processor (1330) can detect at least one map tile (1420) that is connected to each other from the map data of the memory (1340), as shown in FIG. 14b.

[0613] Meanwhile, when a map area is determined from data of detected map tiles, the processor (1330) can model virtual objects corresponding to buildings and roads included in the determined map area (S1304).

[0614] In the above step S1304, the processor (1330) can detect buildings included in the detected map area. And, building reference information corresponding to the detected buildings can be detected from the building reference information storage unit (1343). And, the setting values ​​of each building image related to the detected building reference information can be determined. Here, the building image setting values ​​are information for modeling each building virtual object, and may include a setting value of a projection direction according to an angle or distance for generating the building virtual objects according to the driving direction of the vehicle (100) and / or an angle at which the MR view is displayed, a setting value of modeling according to a 2.5D virtual object or a setting value of modeling according to a 3D virtual object, for example, a vector value such as a coordinate value of a vertex or a normal line.

[0615] Once the building image setting values ​​are determined, the processor (1330) can calculate building setting values ​​corresponding to each building included in the map area based on the determined building image setting values ​​and building reference information of each building. Here, the building setting values ​​can include the height of the building virtual object to be created, the side area of ​​each side, the plane area, etc. Then, once the calculation of the building setting values ​​is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included in the map area based on the building setting values.

[0616] Meanwhile, as mentioned in step S1302, the processor (1330) may update the map area according to the driving information of the vehicle (100). In this case, if new map tile data is additionally detected as shown in FIG. 14b by updating the map area, the processor (1330) may additionally model virtual objects corresponding to buildings and roads included in the added map tiles. On the other hand, if the map area is updated as shown in FIG. 14b, map tiles (1430) may be excluded from the map area. In this case, the processor (1330) may delete virtual objects generated from map tiles (1430) excluded from the map area, thereby increasing the efficiency of the memory (1340).

[0617] Here, the processor (1330) may maintain at least some of the virtual objects generated from the excluded map tiles, depending on the building type, user-specified status, POI status, etc. In this case, the maintained virtual objects may be stored in the memory (1340) and may be reused when the corresponding map tile is detected again in the future. In other words, in the case of a building virtual object corresponding to a building frequently visited by the user or a building specified by the user, the virtual object may be stored and reused regardless of whether the map tile containing the building leaves the map area.

[0618] Meanwhile, the step S1304 may include a process of modeling a road virtual object corresponding to the road shape of each map tile included in the map area.

[0619] In this case, the processor (1330) can generate a plurality of nodes along the road shape included in the map area in step S1304. Then, the generated plurality of nodes can be defined as each segment, and the road virtual object can be modeled by generating polygon objects having a specific width for each segment and combining the generated polygon objects. In this case, the polygon objects can be round objects having a specific width.

[0620] Meanwhile, the processor (1330) may utilize morphological operations to more smoothly express the edges of the road virtual object. In particular, the processor (1330) may temporarily increase the radius of the round polygon objects and then restore it to its original state in order to perform a closing operation among the morphological operations. In this case, the degree of smoothness of the edge portion of the road virtual object may vary depending on the radius of the polygon objects and the number of times the closing operation is performed.

[0621] Additionally, the processor (1330) can raise the generated road virtual object above the ground or form it lower than the ground level. In this case, if the height of the road virtual object is lower than the ground level, the ground around the road can exhibit a curb effect, forming a curb depending on the height difference between the ground level and the road virtual object. In this case, by varying the height difference between the road virtual object and the ground level depending on the width or length of the road or the characteristics of the road, the road virtual object can be expressed differently depending on the characteristics of the actual road.

[0622] In addition, the processor (1330) may also form an object corresponding to a river or stream similar to the method of generating the road virtual object. In this case, the height of the virtual object corresponding to the river or stream may be formed so as to be lower than the height of the ground by a preset depth. In this case, similar to the road virtual object, by making the height difference between the virtual object corresponding to the river or stream and the ground different depending on the width or length of the river or stream, the virtual object corresponding to the river or stream may be expressed differently depending on the characteristics of the actual river or stream.

[0623] Meanwhile, in step S1304, when virtual objects such as buildings and roads of each map tile included in the map area are modeled, the processor (1330) can determine an area according to driving information obtained from the vehicle (100) (S1306).

[0624] The area determined in the above step S1306 is an area determined based on the location of the vehicle (100) or the driving direction of the vehicle (100), and may be a map area determined based on the current driving information of the vehicle (100), that is, an area corresponding to at least a portion of at least one map tile detected from the map data storage unit (1342) of the memory (1340). For example, the area may be an area corresponding to the surroundings of the current location of the vehicle (100) within a first map tile according to the current location of the vehicle (100). Alternatively, the area may be an area corresponding to the front of the vehicle (100) in a first map tile including the current location of the vehicle (100) according to the driving direction of the vehicle (100). Alternatively, the area may be an area including a portion of the first map tile and an area corresponding to a portion of at least one other map tile.

[0625] Meanwhile, the area determined in the above step S1306 may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), i.e., a user field of view area according to the driving direction of the vehicle (100). In this case, the user field of view area has a form that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to perspective projection that can form a sense of perspective, and thus may have the shape of a frustum with the location of the vehicle (100) as a vanishing point.

[0626] Here, an area determined according to the driving information of the vehicle (100) in the step S1306 may be an area where virtual objects displayed as a digital twin map can be displayed on the display unit (251). That is, it may be an area displayed as a digital twin map to the user of the vehicle (100) through the MR view, and may be a map area shown to the user of the vehicle (100), i.e., a visible area. Hereinafter, among the map areas determined according to the driving information of the current vehicle (100), an area determined in the step S1306 that is displayed on the display unit according to the driving information of the vehicle (100) will be referred to as a visible area.

[0627] When the visible area is determined in step S1306, the processor (1330) may determine some of the virtual objects to be rendered among the virtual objects modeled in step S1304 according to the determined visible area (S1308). The objects determined in step S1308 may be only some of the virtual objects included in the visible area among the virtual objects modeled in step S1304, that is, the map tiles detected as the current map area. In this case, the modeled objects determined as the rendering target in step S1308, that is, the virtual objects, may include a plurality of building virtual objects and a plurality of road virtual objects or parts of road virtual objects.

[0628] Meanwhile, when the virtual objects to be rendered are determined in step S1308, the processor (1330) can tile a preset texture image, i.e., a tile, for each of the building virtual objects among the determined virtual objects (S1310). Here, the tiling can be performed for each side of each building virtual object determined as the rendering target, and can be performed for the side areas of each building virtual object excluding a preset offset area (or margin area) from the corner of each side.

[0629] Here, at least one of the above-described building virtual objects may have its side region divided into multiple sub-regions. For example, the side region may be divided into a bottom region, a middle region, and a top region, and at least one of the bottom region, the middle region, and the top region may be further divided into multiple regions. In this case, the processor (1330) may tile different texture images for each of the multiple divided regions.

[0630] In addition, the processor (1330) may tile an arbitrary texture image for each building virtual object or for each partial region of each building virtual object. In this case, texture images corresponding to each building virtual object or each partial region of each building virtual object may be determined in advance. In this case, the building reference information, which is information on an actual building corresponding to each building virtual object and stored in the memory (1340), may include information on a texture image corresponding to each building virtual object or each partial region of each building virtual object, and the processor (1330) may determine a texture image corresponding to each building virtual object or each partial region of each building virtual object when determining a building image setting value based on the driving information of the vehicle (100).

[0631] And, based on the height, area, side area of ​​the building included in the reference information of each building virtual object, and the offset area (or margin area), at least one of the horizontal length and the vertical length of the area in which the texture image is to be tiled in each building virtual object can be calculated, and the number of texture images to be tiled in the horizontal or vertical direction can be calculated according to the calculated length. That is, the processor (1330) can calculate information for modeling building virtual objects based on the reference information of the building including the building image setting value and the height of the actual building, and at the same time calculate a building setting value including information on the area of ​​each side area in which the texture images are to be tiled in each side area for each building virtual object and the number of texture images to be tiled in each side area.

[0632] And the processor (1330) can tile texture images in the side areas of each rendering target building virtual object based on information about texture images corresponding to each building virtual object or a partial area of ​​each building virtual object, and information about the number of tilings of the side areas of each virtual object and the texture images included in the calculated building setting value.

[0633] Meanwhile, when tiling of texture images for side areas of each building virtual object is completed in step S1310, the processor (1330) can generate a normal map including 3D vector values ​​that form different textures for each building element included in the texture image, which is a tiled 2D image (S1312).

[0634] In this case, the normal map may be a normal map (hereinafter referred to as a multi-blended normal map) obtained by blending multiple normal maps generated for each building element included in the texture image. In order to generate the multi-blended normal map, the processor (1330) may generate mask maps for masking building elements for each distinguished building element in each surface area of ​​the building virtual object in which the texture images are tiled.

[0635] In this case, the mask map is a map formed so that building elements other than one building element are hidden in the surface area of ​​the building virtual object on which the texture images are tiled, and may mean an image used to specify only one building element among each building element included in the texture image.

[0636] And when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps to which shading effects are applied so that the textures of different materials can be represented in each mask map. In this case, as described above, since the mask map is an image that covers the remaining area except for the corresponding specific building element, when the normal map is blended, a 3D vector value forming a texture corresponding to the specific material corresponding to the normal map can be blended only in an area not covered by the mask map and an area corresponding to the specific building element corresponding to the mask map.

[0637] Meanwhile, the processor (1330) may generate a mask map for each distinguished building element in each surface area of ​​the building virtual object in which texture images are tiled. Accordingly, multiple mask maps may be generated for each building element, and each mask map may be blended with a normal map of a different material. In addition, the processor (1330) may generate a multi-blended normal map in which 3D vector values ​​corresponding to surface textures of different materials are applied to areas corresponding to different building elements, respectively, with respect to the surface area of ​​the building virtual object, by blending normal maps in which the mask maps are blended with each other. In addition, by synthesizing the generated multi-blended normal map to the surface area of ​​the building virtual object, the processor (1330) may texture the surface areas of each building virtual object so that the areas corresponding to each building element of the building virtual object each have a different surface texture.

[0638] Here, when generating a mask map for each building element, the processor (1330) may further blend at least some of the plurality of mask maps generated for each building element with an albedo map, which is a map image having a color corresponding to a specific material. In this case, as at least some of the plurality of mask maps generated for each building element further have a color according to the albedo map, color mapping can be performed with a color corresponding to a specific material.

[0639] In this case, the processor (1330) may determine a color corresponding to at least some of the plurality of mask maps generated for each building element based on the results of analyzing an image captured from the camera of the vehicle (100). For example, the processor (1330) may detect objects corresponding to buildings among image objects included in an image captured from the camera of the vehicle (100). Then, colors may be detected from the detected building image objects, and based on the detected colors, a dominant color or an average color of actual buildings around the vehicle (100) may be determined as a representative color. And, by blending an albedo map corresponding to a color determined as a representative color with at least some of a plurality of mask maps generated for each building element, blending mask maps to which the albedo map is blended to generate the multi-blended normal map, and synthesizing the generated multi-blended normal map to each building virtual object, at least some of the building elements of the building virtual objects currently determined as rendering targets in step S1308 can be textured to have colors similar to those of actual buildings around the vehicle (100).

[0640] Meanwhile, when texturing is completed in step S1312, the processor (1330) may render the building virtual objects for which texturing has been completed to generate images corresponding to the building virtual objects. In addition, the processor (1330) may render other virtual objects, including at least one road virtual object included in the visible area, to generate images corresponding to each virtual object (S1314).

[0641] And the processor (1330) can control the vehicle interface unit (1320) to display virtual objects including each building virtual object and road virtual object generated through the above rendering on the display unit (251) (S1316).

[0642] As described above, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention has a configuration that detects map tiles within a certain area (map area) set centered on the vehicle (100), models virtual objects within the detected map tiles in advance, and renders and images only some of the objects in the driving direction of the vehicle (100) among the pre-modeled objects. In addition, the processor (1330) synthesizes a normal map so that a 3D vector value corresponding to a specific material for each building element is applied only to some of the objects to be rendered, so that a texture according to a specific material for each building element can be formed by the applied 3D vector value during the rendering. In this way, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can shorten the time required for modeling by rendering pre-modeled objects when displaying virtual objects as an MR view using a digital twin map. In addition, the number of virtual objects to be rendered is also limited to the visible area to minimize the number, and the surface area of ​​each building virtual object can be textured to have the three-dimensional effect of various materials using a texture image, which is a 2D image, and a normal map having 3D vector values ​​corresponding to different materials. In this way, by performing texturing using only 2D images, the amount of computation required for texturing can be greatly reduced, and as a result, the time required for rendering can be greatly shortened.

[0643] In this way, since the amount of computation and time required to display virtual objects in an MR view using the digital twin map can be significantly reduced, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can perform rendering of virtual objects to be displayed through the MR view during the time when the app for displaying the MR view using the digital twin map is running, i.e., during the run time. In other words, real-time run-time 3D map modeling can be possible.

[0644] Meanwhile, as described above, the present invention has a configuration for pre-modeling virtual objects included in a plurality of map tile data according to the location of the vehicle (100) and rendering some of the modeled virtual objects, including texturing. In this case, the map tile data can be updated according to the location of the vehicle (100), and the detection of such map tile data can be performed through background processing. In this case, since the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention pre-models virtual objects included in the map tile data when the map tile data is detected according to the determined map area, the modeling of the virtual objects can also be performed through the background processing. The modeling of virtual objects performed through background processing in this way will be referred to as background modeling.

[0645] Meanwhile, the processor (1330) according to an embodiment of the present invention has a configuration that models virtual objects in a map area in advance through background modeling, and renders a portion of virtual objects that have already been modeled based on a visible area determined according to driving information of the vehicle (100). Here, the modeling of the virtual objects is performed through background processing, and the rendering of the modeled virtual objects is performed through foreground processing according to the execution of the user's MR view app, so the 3D map view generation device (1300) according to an embodiment of the present invention has a configuration in which the modeling and rendering of virtual objects are performed separately. Therefore, when the 3D map view generation device (1300) according to an embodiment of the present invention has a plurality of processors, the modeling of the virtual objects and the rendering of the modeled virtual objects may be performed by different processors, respectively.

[0646] FIG. 15 is a block diagram illustrating a configuration in which modeling and rendering of virtual objects are performed by different processes in a 3D map view generation device (1300) according to an embodiment of the present invention having multiple processors.

[0647] Referring to FIG. 15, a 3D map view generation device (1300) according to an embodiment of the present invention, i.e., an embedded system, may include a first processor that loads a plurality of map tiles through an API based on the current location (GPS Position) of a vehicle (100) in conjunction with a navigation map, and a second processor that renders an image of a virtual object displayed on a display unit (251). Here, the first processor may be a central processing unit (CPU) of the 3D map view generation device (1300) according to the embodiment of the present invention, and the second processor may be a GPU (Graphics Processing Unit) that processes graphic operations on the display device and outputs a result value.

[0648] In this case, the first processor may load a plurality of map tiles according to the location of the vehicle (100) and perform modeling on virtual objects included in the loaded plurality of map tiles. In this case, the first processor may perform modeling of the virtual objects in a background modeling manner. In addition, the modeled virtual objects may be stored in a buffer. In this case, the modeled objects may be simple polygonal (2.5D) virtual objects without surface texturing.

[0649] In this case, at least one of the plurality of map tiles may be added when the map area is updated according to the location of the vehicle (100). Then, the first processor may additionally model virtual objects included in at least one additional map tile according to the updated map area. Then, the buffer may be updated so that the modeled virtual objects are added. Accordingly, virtual objects corresponding to each object, including buildings and roads included in the entire updated map area, may be stored in the buffer.

[0650] Meanwhile, when an app for displaying an MR view is executed by a user, the second processor can render the modeled virtual objects stored in the buffer. During the rendering process, the second processor can perform texturing by tiling a texture image to a side area of ​​each modeled building virtual object, and synthesizing a normal map to which a 3D vector corresponding to a different material is applied to each area of ​​each building element included in the tiled texture image to the side area of ​​each building virtual object. In this case, the 3D vector is for providing a shading effect that forms a texture of a specific material through a shading effect, and the second processor, as a custom shader, can perform rendering to generate an image of each building virtual object, including texturing that includes shading according to the 3D vector applied to each building element.

[0651] Therefore, the 3D map view generation device (1300) according to an embodiment of the present invention can process modeling of virtual objects and rendering of modeled virtual objects in parallel on different processors. Therefore, the computational load applied to the processor can be significantly reduced compared to a case where all processes are performed on a single processor, and by displaying virtual objects by performing rendering from pre-modeled objects, the time required until the virtual objects are displayed on the display unit can also be significantly shortened. Accordingly, when an app related to the MR view is executed, rendering of 3D virtual objects included in the digital twin map can be performed during the runtime time, which is the time when the app is executed, and real-time provision of a digital twin map including virtual objects with high-quality textures can be possible.

[0652] However, for convenience of explanation, the following description assumes that modeling and rendering are performed by a single processor (1330). However, it should be understood that the present invention is not limited thereto.

[0653] Meanwhile, in step S1316 of FIG. 13, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can simultaneously display a screen showing information collected around the vehicle according to the ADAS system and an MR view screen according to the digital twin map. FIG. 16 is an exemplary diagram illustrating such an example.

[0654] Referring to FIG. 16, the processor (1330) of the 3D map view generation device (1300) can display information on surrounding vehicles and obstacles based on information collected through the vehicle interface unit (1320) and an ADAS (Advanced Driver Assistance Systems) view image based on driving information of the vehicle (100) in one area (1600) of the display unit (251). ADAS refers to an active safety device that detects a dangerous situation with a sensor or a camera, warns the driver of the risk of an accident, and helps the driver make a judgment and respond. The ADAS view image can include information on objects corresponding to the surrounding situation of the vehicle, for example, a vehicle object and other vehicles and obstacles detected around the vehicle, and can include route information such as the current driving speed or driving time of the vehicle (100) or the expected time required to reach the destination.

[0655] Meanwhile, the processor (1330) can display an MR view image in another area (1610) on the display unit (251). The MR view image is a view image including a building virtual object or a road virtual object created as a 3D object, and can display a digital twin map formed by the virtual objects.

[0656] Meanwhile, according to the above description, it has been mentioned that the processor (1330) of the 3D map view generation device (1300) according to the embodiment of the present invention can determine an area to display virtual objects by rendering them according to the driving information of the vehicle (100), i.e., a visible area, in step S1306 of FIG. 13. Here, the visible area may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), i.e., a user viewing area according to the driving direction of the vehicle (100). In this case, the user viewing area has a form that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to perspective projection that can form a sense of perspective, and thus may have the shape of a frustum with the location of the vehicle (100) as a vanishing point. Therefore, the above-mentioned visible area can be formed in a frustum shape with the position of the vehicle (100) as the vanishing point.

[0657] FIG. 17a is an exemplary diagram illustrating a frustum-shaped visible area formed across a plurality of map tiles or a single map tile, depending on the position of the vehicle and the driving direction of the vehicle, in a 3D map view generation device (1300) according to an embodiment of the present invention. FIG. 17b illustrates an example of a visible area formed to include a plurality of map tiles.

[0658] First, referring to (a) of FIG. 17A, the processor (1330) can form a frustum-shaped visible area (1700) formed along the driving direction of the vehicle (100) with the current location (1720) of the vehicle (100) as a reference point. In this case, the processor (1330) can perform rendering only on virtual objects located within the formed visible area (1700). That is, even within one map tile, virtual objects located in areas other than the visible area (1700) may not be rendered.

[0659] Here, rendering may refer to a process or calculation process of generating an image using a computer program. Therefore, even if it is a modeled virtual object, that is, even if a virtual object is created, if the rendering is not performed, an image corresponding to the virtual object may not be created. In addition, if an image is not created, it may not be displayed on the display unit (251). That is, the processor (1330) may selectively determine objects to be displayed on the display unit (251) among the modeled virtual objects based on a visible area formed according to the driving direction of the vehicle (100).

[0660] Accordingly, as shown in (a) of FIG. 17A, images corresponding to each virtual object can be generated through a rendering process for virtual objects within an area included in the visible area (1700). However, images of virtual objects within an area included in the visible area (1700) may not be generated. Therefore, the processor (1330) can reduce the amount of computation required for rendering and shorten the computation time by rendering only some of the virtual objects modeled based on the visible area.

[0661] Meanwhile, the visible area (1700) may be formed across a plurality of map tiles, as shown in (b) of FIG. 17, depending on the zoom level of the set map. In addition, the plurality of map tiles may be a part of a map area set around the current location (1720) of the vehicle (100). In this case, since the processor (1330) models virtual objects based on the map area, images of virtual objects in an area included in the visible area (1700) among the plurality of map tiles may be generated through a rendering process according to the visible area (1700) formed across the plurality of map tiles, and may be displayed on the display unit (251).

[0662] Meanwhile, Fig. 17b is an example diagram showing an example in which a visible area formed in a frustum shape includes multiple map tiles.

[0663] As shown in Fig. 17b, when a plurality of map tiles are included in the visible area (1700), the processor (1330) can perform modeling on virtual objects included in each of the plurality of map tiles included in the visible area (1700). In addition, the virtual objects for which modeling has been performed can be rendered in a preset order or by selecting only some of the virtual objects.

[0664] For example, the processor (1330) may perform modeling and rendering in order of proximity to the vehicle based on the current location (1720) of the vehicle. Accordingly, objects included in the first map tiles (1751) that are close to the current location (1720) of the vehicle may be modeled and rendered before objects included in the second map tiles (1752) that are further away from the current location (1720) of the vehicle.

[0665] Alternatively, the processor (1330) may model and render only a portion of the objects included in each of the map tiles included within the visible area (1700) based on a distance from the current location (1720) of the vehicle.

[0666] For example, the processor (1330) may perform modeling and rendering for all objects included in each map tile of the first map tiles (1751), but may perform modeling and rendering for only some of the objects included in each map tile of the second map tiles (1752). In this case, the processor (1330) may model and render only objects corresponding to roads and terrain among the objects included in the second map tiles (1752), and may not perform modeling and rendering for general building objects.

[0667] In this way, the modeling and rendering priorities for objects included in each map tile can be determined based on their type. These object type-specific priorities can be included in the data for the objects for the map tile (hereinafter referred to as "map tile data"), along with geometry data containing the shape information of each object, as attribute data corresponding to each object.

[0668] In this way, the order in which objects included in a map tile are modeled and rendered according to their type is determined by attribute information, and the processor (1330) can sequentially model (procedural modeling) and render (procedural rendering) objects included in a map tile according to the order, i.e., priority.

[0669] Meanwhile, by using the above priority, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention may model and render a specific object with priority over other objects. For example, the processor (1330) may give a higher priority to objects such as landmarks than to general building objects. In this case, the building object corresponding to the landmark may be modeled and rendered with priority over or earlier than other general building objects. For example, if the second map tiles (1752) include a building object corresponding to a landmark, the processor (1330) may perform modeling and rendering for the landmark building object together with the road objects and terrain objects included in the second map tiles (1752).

[0670] Meanwhile, FIG. 18a is an exemplary diagram showing an example of texture images being tiled on modeled building virtual objects in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0671] When the building virtual objects determined as rendering targets (hereinafter referred to as rendering target virtual objects) in step S1308 of the above-described FIG. 13 are determined, the processor (1330) may tile a preset texture image in each side area for each of the building virtual objects among the determined virtual objects. In this case, the processor (1330) may detect a side area of ​​the building virtual object excluding a preset offset area (or margin area) inward from each side edge of the building virtual object, and may tile the texture image in the detected side area.

[0672] First, referring to (a) of FIG. 18a, the processor (1330) can determine the left and right margins (1811, 1812) of the side area of ​​the building virtual object determined as the rendering target according to the preset left and right margin values. According to the left and right margin values, the coordinate range of the area (1810) where the texture image is to be tiled in the horizontal direction can be determined.

[0673] For example, if the left and right margin values ​​are 0.1, the lower left point of the side area of ​​the building virtual object (2520) can be defined as (0, 0) and the upper right point can be defined as (1, 1). Then, as shown in Fig. 18a, the width of the tiling area (1810) excluding the left and right margins can be calculated as 0.8. In addition, if the building width of the actual building corresponding to the building virtual object (2520) is 14 m, the actual horizontal width of the tiling area (1810) can be calculated as 11.2 m.

[0674] Meanwhile, the texture image (1800) may be a normalized tile image and may have a preset width. In this case, if the horizontal width of the texture image (1800) is 4 m, dividing the actual horizontal width of the tiling area (1810), 11.2 m, by the horizontal width of the texture image (1800), 4 m, may yield a value of 2.8. In this case, since the number of texture images (1800) is an integer, rounding may yield a value of 3, and the yielded value 3 may be the number of texture images (1800) to be tiled in the tiling area (1810) of the side area of ​​the building virtual object (2520) set according to the preset left and right margin values. And the processor (1330) inputs (0.1, 0.9), which are the left and right X-axis coordinates of the tiling area (1810), as input values ​​of the remap function and inputs (0, 3) as a new range according to the remap function, so that up to three tiles (texture images (1800)) can be successively attached to the tiling area (1810) in the horizontal direction in the tiling area (1810).

[0675] In addition, the processor (1330) can set multiple partial regions based on the bottom layer, which is the first floor, according to preset upper and lower margin values. For example, the processor (1330) can set a floor divider corresponding to the space between the bottom layer and the upper layer of the partial region. In this case, the space between the set floor dividers based on the bottom layer can be determined as the partial region where texture images are to be tiled.

[0676] Here, the side region of the building virtual object may include multiple subregions. For example, the side region of the building virtual object may be divided into a bottom region (1852), a middle region (1850), and a top region (1851), and at least one of the bottom region (1852), the middle region (1850), and the top region (1851) may be further divided into multiple regions. In this case, the processor (1330) may tile different texture images for each of the multiple divided regions.

[0677] In this case, the processor (1330) can obtain a mask map corresponding to each sub-region based on the UV coordinate system corresponding to each sub-region on each building side. For example, the processor (1330) can define the lower left point of the side region of the building virtual object as (0, 0) and the upper right point as (1, 1). Then, by defining the Y-axis coordinate in the UV coordinate system according to the margin value of the sub-region in which the layer division is formed, the vertical width corresponding to the sub-region can be defined, and the mask map corresponding to the sub-region can be obtained according to the defined vertical width. In this case, the mask map corresponding to the sub-region can be a mask map in which a height lower than the Y-axis coordinate according to the margin value of the sub-region has a value of 0, and a height higher than the Y-axis coordinate has a value of 1. In this case, when blending with a normal map is performed, a part having a value of 0 maintains a value of 0, but a part having a value of 1 can have a 3D vector value of the normal map.

[0678] As an example of such a partial region, if the margin value of the partial region where the layer division is formed is 0.8, the processor (1330) can determine the coordinates of the upper right point of the partial region as (1, 0.8). Then, the processor (1330) can obtain a mask map in which an area corresponding to a height less than 80% of the total height of the building virtual object from the floor surface of the building virtual object has a value of 0, and an area corresponding to a height greater than that has a value of 1. That is, an area corresponding to the upper edge of the building virtual object from a height of 80% or more of the total height of the building virtual object can be formed as a partial region (e.g., a tower region).

[0679] Meanwhile, the processor (1330) can tile different texture images for each sub-region (1850, 1851, 1852) of the building virtual object (2560) in the same manner as the method described in (a) of FIG. 18A. In this case, as shown in (b) of FIG. 18A, the top region (1851) can be tiled with the first texture image (1801), the bottom region (1852) can be tiled with the second texture image (1802), and the middle region (1850) can be tiled with the third texture image (1800). Accordingly, the building virtual object (2650) can be tiled with different texture images for each different sub-region.

[0680] In this way, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can diversify the exterior shape (facade design) of the building virtual object by dividing each building virtual object into a plurality of different regions and tiling different texture images in each of the divided regions.

[0681] FIG. 18b illustrates examples of facade designs of a building virtual object expressed in various ways by using different partial regions and tiled texture images in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0682] First, Figure 18b (a) illustrates an example of a building virtual object formed by three sub-regions: a ground floor, a middle floor, and a top floor. Figures 18b (b) and (c) illustrate examples of a building virtual object formed by only the ground and middle floors. In this way, by varying the number of sub-regions, building virtual objects can be expressed with different exterior shapes.

[0683] In addition, the processor (1330) can vary the texture images tiled in each sub-region for each building virtual object. In this case, since not only the number of sub-regions but also the tiled texture images vary, the number of possible combinations can increase. Therefore, a more diverse exterior representation of the building virtual object can be achieved.

[0684] Meanwhile, the processor (1330) can blend normal maps having 3D vector values ​​corresponding to different materials for each building element in step S1312 of FIG. 13 so that the images can be displayed with different materials for each building element included in the tiled texture image of the building virtual object during rendering.

[0685] FIG. 19 is a flowchart illustrating in more detail the texturing process of step S1312 of FIG. 13, which forms different surface textures for each component of a building in a 3D map view generation device (1300) according to an embodiment of the present invention. FIG. 20 is an exemplary diagram illustrating an example of normal maps blended to mask maps corresponding to different building elements and a multi-blended normal map generated by blending normal maps together, according to the process described in FIG. 19.

[0686] First, referring to FIG. 19, the processor (1330) can identify and distinguish building elements included in each texture image for each texture image tiled in the tiling area of ​​the building virtual object (S1900).

[0687] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different areas of the texture image by each color value, and accordingly, different areas of the texture image, i.e., different building elements, can be distinguished by color channels. For example, if the texture image includes at least one window element, the window element may have different color values ​​from the rest of the building background. In addition, if the window element includes a glass element corresponding to a glass area, a window frame element corresponding to a window frame area, and a blind element corresponding to a blind area, the glass element, the window frame element, and the blind element may each have different color values. Therefore, the processor (1330) can distinguish and identify areas corresponding to a plurality of different building elements included in the texture image based on the color values. In this case, the texture image may be referred to as a labeling map, which is a map image that labels a plurality of different building elements.

[0688] Meanwhile, when each building element included in the texture image is distinguished and identified, the processor (1330) can generate mask maps corresponding to each building element (S1902).

[0689] In this case, the mask map corresponding to a specific building element may be an image having a value of 1 in an area corresponding to a specific building element identified by a specific color value in the texture image, and a value of 0 in the remaining areas.

[0690] For example, if the texture image includes at least one window element, the mask map corresponding to the window element may be a mask map that has a value of 1 only in the area corresponding to the window element and a value of 0 in the remaining area. On the other hand, the mask map corresponding to the building wall may be a mask map that has a value of 1 only in the building wall portion, i.e., in the remaining area of ​​the texture image excluding the window element, and a value of 0 in the remaining area.

[0691] In addition, if the window element includes the glass element, the window frame element, and the blind element, the processor (1330) can generate a glass mask map in which only the area corresponding to the glass element in each window element has a value of 1 and the rest has a value of 0, a window frame mask map in which only the area corresponding to the window frame element in each window element has a value of 1 and the rest has a value of 0, and a blind mask map in which only the area corresponding to the blind element in each window element has a value of 1 and the rest has a value of 0.

[0692] When mask maps corresponding to each building element are generated for each building element included in the tiled texture image in this way, the processor (1330) can blend each generated mask map with a normal map having a 3D vector value corresponding to a different material (S1904). In this case, the normal map means an image having a 3D vector value indicating a 3D direction of the surface of an object required for calculating the degree of shading corresponding to the reflected light reflected by the surface curvature of the specific material when light incident on the specific material is incident on the specific material, in a graphic processing device, and may be a 2D image capable of expressing the texture of the specific material through the degree of shading, i.e., the shading effect.

[0693] This blending of the normal map and the mask map may be a process similar to a multiplication process. Therefore, when the normal map and the mask map are blended, the 3D vector value applied to the normal map may be applied only to the area that is not masked with a value of 0 in the mask map, i.e., the area corresponding to a specific building element. Therefore, only the area corresponding to the specific building element corresponding to the mask map may express a texture according to a specific material. In this case, the processor (1330) may blend the mask map corresponding to each building element with a normal map having a 3D vector value of a different material, thereby allowing each building element to have a texture of a different material.

[0694] In the above step S1904, when each mask map is blended with each normal map, the processor (1330) may further blend an albedo map having a different color for each of the mask maps and the blended normal maps. In this case, the albedo map may be an image having a color value unique to a specific material. In addition, a pair may be formed with a normal map having a 3D vector value corresponding to the texture of a specific surface material.

[0695] Then, the processor (1330) can further blend the albedo map forming a pair with the normal map for each normal map blended with the mask map (S1906). In this case, blending with the albedo map can be applied similarly to the multiplication process as with blending with the normal map. Therefore, a color value can be applied only to a part having a value of 1 in the mask map, and accordingly, a normal map capable of expressing the texture of a specific material and the unique color of the material can be generated for a specific building element.

[0696] Meanwhile, in step S1906, when the mask maps and blended normal maps corresponding to each component of the building are blended with the albedo maps, the processor (1330) can blend the blended normal maps of the building elements with each other (S1908). In this case, the blending of the normal maps may be a union concept, which may mean a process of combining the areas to which 3D vector values ​​(and color values ​​according to the albedo map) are applied between the normal maps into a single map image, a multi-blended normal map. Therefore, when the blending process of step S1906 is completed, a normal map (multi-blended normal map) to which 3D vector values ​​(and albedo color values) of different normal maps are applied for each building element included in a single texture image may be generated.

[0697] And the processor (1330) can perform texturing on the side area of ​​the building virtual object by synthesizing the multi-blending normal map into the tiling area among the side areas of the building virtual object (S1910).

[0698] In this case, the multi-blending normal map may be generated for the entire tiled area in which texture images are tiled in the side area or partial area of ​​the building virtual object. In this case, the processor (1330) may synthesize the multi-blending normal map generated in a size corresponding to the entire side area or partial area of ​​the building virtual object onto the side area or partial area of ​​the building virtual object.

[0699] Alternatively, the multi-blending normal map may be generated for any one texture image tiled in a side area or partial area of ​​the building virtual object. In this case, the processor (1330) may synthesize the multi-blending normal map for the side area or partial area of ​​the building virtual object by tiling the multi-blending normal map for each of the texture images tiled in the side area or partial area of ​​the building virtual object.

[0700] Meanwhile, FIG. 20 illustrates an example of generating a multi-blending normal map including areas to which different 3D vector values ​​corresponding to each of a plurality of building elements included in a texture image are applied according to the operation process described in FIG. 19.

[0701] Referring to FIG. 20, the processor (1330) can distinguish and identify each building element from a texture image (1800) including a window element. In this case, if the window element includes a glass element and a window frame element (2002), the processor (1330) can identify the wall element (2001), the glass element, and the window frame element (2002) from the texture image (1800).

[0702] And the processor (1330) can generate a mask map corresponding to each identified building element. In this case, as shown in FIG. 20, a mask map (2011) corresponding to the wall element can be generated in which only the area corresponding to the wall element (2001) has a value of 1 and the remaining areas, for example, the areas corresponding to the glass element and the window frame element (2002), have a value of 0. In addition, a mask map (2012) corresponding to the window frame element can be generated in which only the area corresponding to the window frame element (2002) has a value of 1 and the remaining areas, for example, the areas corresponding to the glass element and the wall element (2001), have a value of 0.

[0703] And the processor (1330) can blend the first normal map (2021) and the second normal map (2022), which are map images to which 3D vector values ​​corresponding to different materials are applied, for each generated mask map (2011, 2012). In this case, the first normal map (2021) may be an image to which a 3D vector value capable of expressing the first material with a shading effect is applied, and the second normal map (2022) may be an image to which a 3D vector value capable of expressing the second material with a shading effect is applied.

[0704] Then, the 3D vector value of the blended normal map can be synthesized only in the area having the value 1, excluding the area having the value 0 in each mask map. Therefore, when the mask map (2011) corresponding to the wall element and the first normal map (2021) are blended, a normal map (2031) synthesized with a 3D vector value that can express the first material only in the area corresponding to the wall element in the texture image (1800) can be generated, as shown in FIG. 20. In addition, when the mask map (2012) corresponding to the window frame element and the second normal map (2022) are blended, a normal map (2032) synthesized with a 3D vector value that can express the second material only in the area corresponding to the window frame element in the texture image (1800) can be generated, as shown in FIG. 20.

[0705] Then, the processor (1330) can blend a normal map (2031) in which 3D vector values ​​that can express a first material only in an area corresponding to the wall element are synthesized, and a normal map (2032) in which 3D vector values ​​that can express a second material only in an area corresponding to the window frame element are synthesized. In this case, blending is performed between normal maps, and each normal map can be synthesized into a single multi-blended normal map. Therefore, as shown in FIG. 20, a multi-blended normal map (2050) can be generated in which 3D vector values ​​that can express a first material in an area corresponding to the wall element (2001) are synthesized, and 3D vector values ​​that can express a second material in an area corresponding to the window frame element (2002) are synthesized. That is, a normal map can be generated in which 3D vector values ​​that can express different materials in each area corresponding to each building element included in the texture image are synthesized.

[0706] Meanwhile, in the description of Fig. 19 described above, the albedo map having a material-specific color is blended into the normal map to which the mask map is blended, thereby mapping the material-specific color to each building element. However, in contrast to this, the colors of virtual objects within the visible area can also be determined as colors similar to the colors of buildings within the actual visible area based on images captured from the camera of the vehicle (100).

[0707] FIG. 21 is a flowchart illustrating an operation process of mapping the colors of virtual building objects based on the colors of actual buildings within a visible area in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0708] Referring to FIG. 21, when step S1904 of blending the mask maps generated for each building element in FIG. 19 into a normal map having 3D vector values ​​corresponding to different materials is performed, the processor (1330) can acquire an image of the surroundings of the vehicle (100) through the camera of the vehicle (100) (S2100). In this case, the camera may be a camera that captures an image of the front area of ​​the vehicle (100), and the acquired image may be an image of the front of the vehicle (100) acquired according to the driving direction of the vehicle (100), i.e., an image of the visible area.

[0709] And the processor (1330) can identify each object included in the acquired image and partition the acquired image into multiple regions according to the type of each identified object. That is, the processor (1330) can partition the regions where each object included in the acquired image is displayed on the image into different regions for each object (S2102).

[0710] For example, if the acquired image is an image including trees, buildings, the sky, and a road, the processor (1330) may, in step S2102, segment the image into an area where each tree is displayed, an area where each building is displayed, an area where the sky is displayed, and an area where the road is displayed. Then, a dominant color may be detected in each segmented area, and each area may be displayed with the detected dominant color. In this case, an image may be acquired in which the area where each object included in the image is displayed is segmented with the dominant color in each area.

[0711] To this end, the processor (1330) may utilize an artificial intelligence module (not shown) trained through preset learning data. In this case, the training may be training to identify each region on the image as a different object based on shape and color. Accordingly, the artificial intelligence module, upon completion of the training, may identify a region of the input image as belonging to one of the preset object types based on the pattern formed by the shape and color displayed in the region.

[0712] And the processor (1330) can classify the areas on the image divided by each object by type of the object. And a representative color can be determined from the areas on the image divided by type of each object (S2104).

[0713] In this case, one or more areas on the segmented image can be classified as having the same object type. For example, an area where each building is displayed on the image can be classified into areas corresponding to each building, and a dominant color can be detected for each segmented area. In this state, the processor (1330) can classify each area on the image classified by each object according to the type of each object. Accordingly, at least one area identified as a 'building' can be classified as an area corresponding to the object type 'building'. In addition, the processor (1330) can determine a representative color corresponding to the object type 'building' based on the color of each of the plurality of areas classified as areas corresponding to the object type 'building'.

[0714] For example, the processor (1330) may determine the color with the largest proportion among the colors of each of the areas corresponding to the object type 'building' as the representative color corresponding to the object type 'building'. Alternatively, the processor (1330) may calculate the average color of the colors of each of the areas corresponding to the object type 'building' and determine the calculated average color as the representative color corresponding to the object type 'building'.

[0715] When a representative color corresponding to the object type 'building' is determined, the processor (1330) can map at least some of the building elements of the building virtual object within the visible area to the determined representative color (S2106). For example, the processor (1330) can blend at least some of the normal maps blended to the mask maps corresponding to each of the different building elements with an albedo map having a representative color corresponding to the object type 'building'.

[0716] FIG. 22 is an exemplary diagram showing an example of determining colors mapped to texture images through images captured from a vehicle in a 3D map view generation device (1300) according to an embodiment of the present invention.

[0717] First, (a) of FIG. 22 illustrates an example of an image captured from a camera of a vehicle (100). In this case, the image may be an image captured from the front of the vehicle (100) along the driving direction of the vehicle (100).

[0718] Then, the processor (1330) can identify objects included in the acquired image. And, as shown in (b) of Fig. 22, the areas identified as each object can generate a color map image in which each area is displayed with a dominant color.

[0719] As shown in (b) of FIG. 22, when a color map image is generated, the processor (1330) can distinguish the area of ​​each object identified in the generated color map image by the type of each object. That is, as shown in (b) of FIG. 22, in the color map image, a first area (2201) corresponding to a first building object, a second area (2202) corresponding to a second building object, a third area (2203) corresponding to a third building object, a fourth area (2204) corresponding to a fourth building object, and a fifth area (2205) corresponding to a fifth building object can be distinguished as areas corresponding to the object type 'building'.

[0720] Then, the processor (1330) can calculate the color proportion of each area corresponding to the object type "building." In this case, areas with the same color can be considered as a single area. Accordingly, the first area (2201) and the fifth area (2205) with the same color can be considered as a single area.

[0721] In this case, as shown in (c) of Fig. 22, the color proportions determined from each area identified as a building object in the image can be calculated. In this case, similar to the building object, the color proportions of road objects or sky objects can also be calculated.

[0722] In this case, as shown in (c) of FIG. 22, if the first color, which is the dominant color of the first region (2201) and the fifth region (2205), has the largest proportion, the processor (1330) can determine the representative color of the building virtual object with the first color. Then, the processor (1330) can generate an albedo map having the first color, and blend at least some of the normal maps blended with the albedo map having the first color and the mask map corresponding to each different building element. Accordingly, a multi-blended normal map can be generated by blending at least one normal map to which the albedo map of the first color is blended. Therefore, texturing of the building virtual object can be performed so that at least some of the building elements of the building virtual object are displayed with the first color.

[0723] Meanwhile, according to the above description, the processor (1330) of the 3D map view generation device (1300) according to the embodiment of the present invention has been described to enable different textures to be formed for each building element by using a mask map and a normal map corresponding to each building element of the building virtual object during the rendering process. In this case, the processor (1330) groups the mask maps corresponding to specific building elements into a plurality of groups, and controls the specific building elements to be displayed differently for each group, thereby further enhancing the diversity of each building virtual object.

[0724] FIG. 23 illustrates an example of a 3D map view generation device (1300) according to an embodiment of the present invention, in which window elements are assumed as examples of specific building elements to be grouped, window elements formed in a building virtual object are grouped into one or more groups, and the building virtual object is displayed in various ways using the grouped window elements.

[0725] For example, when the processor (1330) of the 3D map view generation device (1300) generates a multi-blended normal map in units of texture images that are tiled on a building virtual object, a plurality of mask maps (hereinafter referred to as glass mask maps) corresponding to a specific building element formed for each of the tiled texture images, for example, a glass element corresponding to a glass area of ​​a window element, may be formed. In this case, the processor (1330) groups the plurality of mask maps corresponding to each glass element into a plurality of groups, and maps different colors to each group (blending albedo maps of different colors) or blends normal maps, thereby allowing the glass elements corresponding to each group to have different colors or different materials.

[0726] In this case, the processor (1330) can arbitrarily group the plurality of mask maps regardless of the tiling order of the texture image corresponding to each mask map. Alternatively, if the side area of ​​the building virtual object is formed of a plurality of partial areas, the processor (1330) can group building elements of different multiple areas into the same group. Accordingly, any building element can be controlled as the same group regardless of the location or partial area of ​​the building virtual object, and different groups can be controlled to have different colors or materials.

[0727] In this case, the processor (1330) can group a plurality of glass mask maps formed for the building virtual object (3300) into two groups. By mapping different colors to each group of the glass mask maps, as shown in (a) of FIG. 23, an effect can be produced in which some (2301) and the remaining some (2302) of the windows of the building virtual object (3000) are turned on with different colors.

[0728] Meanwhile, similarly, the processor (1330) groups a plurality of glass mask maps formed for the building virtual object (2300) into two groups and assigns different colors to each group, thereby allowing some (2312) of the windows of the building virtual object (2300) to be turned off, and the remaining some (2311) to be turned on, as shown in (b) of FIG. 23.

[0729] In this case, the processor (1330) can change the mask maps grouped into each group into different groups when a preset condition is met. In this case, some randomly determined mask maps grouped into the first group can be changed to the second group. In addition, some randomly determined mask maps grouped into the second group can be changed to the first group. In addition, the processor (1330) can map different colors or materials to the mask maps for each group so that specific building elements having different states (e.g., lighting color or lighting turn-on state) can be randomly changed. In this case, the preset condition may include a case where a preset time elapses.

[0730] Meanwhile, the window element may further include a blind element that covers at least a portion of the glass area of ​​the window area formed by each texture image. The blind element may be formed by mask maps generated for the glass area of ​​each window area, separately from the mask maps formed from each texture image. For example, the processor (1330) may determine a blind area that covers at least a portion of the glass area from a window area identified from texture images tiled in a side area of ​​a building virtual object. Then, a mask map may be generated that has a value of 1 for the glass area covered by the blind area, i.e., the glass area, and a value of 0 for the glass area not covered by the blind area. In this case, the generated mask map may be a blind mask map for representing the blind element that covers at least a portion of the glass area with a different material or color from the glass area.

[0731] Here, the processor (1330) may generate blind areas with different sizes for each blind mask map so that the glass areas covered by the blinds are different from each other. In addition, the generated blind mask map may be arbitrarily further blended with at least some of the glass areas formed in the building virtual object. In this case, as the blind mask map and the glass area are synthesized through blending, at least some of the glass area may be covered by the blind area of ​​the blind mask map.

[0732] Accordingly, as shown in (c) of FIG. 23, an effect may be produced in which some of the glass areas of the window area formed in the building virtual object (2300), i.e., the glass areas, are covered by blinds. In addition, the areas covered by the blinds (2330) for each glass area (2320) may produce different effects. In addition, the processor (1330) may change the blind mask map blended to each glass area when a preset condition is met, thereby changing the size of the blind area covering each glass area. In this case, the preset condition may include a case in which a preset time elapses.

[0733] Meanwhile, by grouping at least one mask map constituting the window element of each building virtual object into multiple groups and mapping the grouped mask maps to different colors or materials, it is possible to produce various changes in the appearance of building virtual objects according to specific conditions or environments. For example, the processor (1330) can display each building virtual object differently according to the driving time of the vehicle (100) by blending each group of window elements of each building virtual object to have a specific color or material according to the driving time of the vehicle (100).

[0734] FIG. 24 is an exemplary diagram showing an example of a 3D map view generation device (1300) according to an embodiment of the present invention displaying building virtual objects differently depending on time zone.

[0735] First, referring to (a) of Fig. 24, (a) of Fig. 24 illustrates a case where the time zone in which the vehicle (100) is driving is daytime. In this case, the processor (1330) can map at least some of the window elements of each building virtual object (2400) around the vehicle object (1720) to a color having a darker illuminance than the illuminance of the sky area, thereby indicating that at least some of the windows of each building virtual object are turned off or relatively dark.

[0736] On the other hand, if the time zone during which the vehicle (100) is driven is night or evening, the processor (1330) can indicate that at least some of the window elements of each building virtual object (2400) around the vehicle object (1720) are turned on by mapping them to a color having an illuminance higher than a preset illuminance, as shown in (b) of FIG. 24.

[0737] Meanwhile, in the above description, it was explained as an example that a specific building element identified from texture images tiled on each building virtual object is changed by changing a group depending on whether a preset condition is met. However, it is also possible to dynamically change the characteristics of the specific building element by linking it to specific content.

[0738] FIG. 25 is a flowchart illustrating an operation process of a 3D map view generation device (1300) according to an embodiment of the present invention, in which at least one of the characteristics defining a building element formed in a building virtual object is changed in conjunction with multimedia content.

[0739] During the operation process of the above-described FIG. 13, when step S1316 of controlling the vehicle interface unit (1320) to display virtual objects including each building virtual object and road virtual object generated through rendering on the display unit (251) is performed, the processor (1330) can detect multimedia content played in the vehicle (100). In addition, if there is multimedia content such as sound sources and videos played in the vehicle, the characteristics of the multimedia content played can be analyzed (S2500).

[0740] In the above step S2500, the processor (1330) can analyze various features from the multimedia content being played. For example, if the multimedia content being played is audio content, the processor (1330) can analyze the beat speed of the audio content being played in the vehicle, the amplitude of a specific audio signal included in the audio content, the sound field of the specific audio signal, etc. In addition, if the multimedia content being played is video content, in addition to the features analyzed for the audio content, the processor (1330) can additionally analyze the amount of change in the image over a certain period of time, etc.

[0741] And the processor (1330) can map at least one of the parameters of the building element included in the texture image tiled for each building virtual object, i.e., the values ​​that determine the size, position, and color of the building element, to at least one of the analyzed features (S2502). And the processor (1330) can change at least one of the parameters of the building element mapped to each feature according to the features of the multimedia content that change according to the multimedia content being played (S2504).

[0742] For example, the processor (1330) may map the size of a glass element among the building elements to the amplitude of a specific frequency band of a reproduced audio signal. In this case, the size of the glass element formed in the building virtual object may vary depending on changes in the amplitude of the mapped frequency band.

[0743] In this case, the processor (1330) can map the amplitude of different frequency bands for each building virtual object to the size of the glass element. Accordingly, the glass elements formed in each building virtual object can have different sizes depending on the amplitude change of different frequency bands for each building virtual object.

[0744] Additionally, the processor (1330) may map the color of the glass element to other features analyzed from the audio signal, independently of the size of the glass element. For example, the processor (1330) may map the color of the glass element to the pitch or duration of the audio signal, and may cause the color of the glass element to change depending on the pitch or duration of the audio signal being played. In this case, not only the size but also the color of the glass element may change depending on the audio signal being played.

[0745] Here, the processor (1330) can map parameters related to various building elements, as well as the glass element, to the characteristics of the analyzed multimedia content. For example, the processor (1330) can map the size or color of the glass element, as well as the blind element synthesized with the glass element through blending, to the characteristics of the analyzed multimedia content.

[0746] Alternatively, the processor (1330) may map the size of the tiling area, i.e., the side area of ​​the building virtual object where the texture tiles are tiled, as well as the building elements, to the characteristics of the analyzed multimedia content. For example, the processor (1330) may change the size or location of the tiling area of ​​the building virtual object according to the characteristics of the analyzed multimedia content by changing the margin value used to determine the tiling area in the building virtual object according to the characteristics of the analyzed multimedia content.

[0747] Meanwhile, the processor (1330) can change the building elements to which the features of the multimedia content are mapped based on whether a preset condition is met (S2506). For example, the processor (1330) can change the features mapped to the size of the glass element to be mapped to the color of the glass element. Alternatively, the processor (1330) can change the features of the multimedia content mapped to each building virtual object between building virtual objects.

[0748] That is, if a parameter of at least one building element of the first building virtual object is mapped to an amplitude of the first frequency band and a parameter of at least one building element of the second building virtual object is mapped to an amplitude of the second frequency band, the processor (1330) may map the parameter of at least one building element of the first building virtual object to an amplitude of the second frequency band and may map the parameter of at least one building element of the second building virtual object to an amplitude of the first frequency band depending on whether the above-described condition is satisfied. Accordingly, the diversity of each building virtual object may be further enhanced by allowing each building virtual object to have a different shape and appearance.

[0749] Fig. 26 is an exemplary diagram showing an example in which window elements formed in each building virtual object are changed in conjunction with the characteristics of multimedia content according to the operation process of Fig. 25.

[0750] As described in the operation process of the above-described FIG. 25, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention can map at least one of the features of a building element included in each building virtual object to any one of the features analyzed from multimedia content played in the vehicle (100). In addition, as the multimedia content is played, the feature of the building element linked to any one of the features of the multimedia content is changed, thereby producing diversity as if each building virtual object is changed in conjunction with the multimedia content played in the vehicle (100).

[0751] Accordingly, as shown in (a) of FIG. 26, when the window elements of each building virtual object are linked to any one of the features of the multimedia content played in the vehicle (100), at least one (2601) of the window elements of each building virtual object may be lit according to the multimedia content played. In this case, different window elements may be mapped to features of different multimedia contents, or different window elements may be mapped to features of different multimedia contents for each building virtual object.

[0752] Accordingly, when the mapped feature changes as the multimedia content is played, for example, when the tempo becomes faster, the number of window elements (2601) displayed as turned on in each building virtual object, that is, displayed in a color corresponding to the turned-on state, or window elements (2602) displayed in a color corresponding to the turned-off state, may increase, or the speed at which the window elements (2601) are lit in each building virtual object may become faster. On the other hand, when the tempo becomes slower, the number of window elements (2601) displayed as turned on in each building virtual object may decrease, or the speed at which the window elements (2601) are lit in each building virtual object may become slower.

[0753] In addition, the window elements that are turned on or off (indicated by a color corresponding to the turned-off state) in each building virtual object may differ from each other. Accordingly, as shown in (b) of Fig. 26, the states of the window elements displayed in each building virtual object may differ from each other as multimedia content is played.

[0754] Meanwhile, in Fig. 26, the case where the status of window elements displayed in each building virtual object changes as multimedia content is played is described, but it is of course also possible that the area of ​​each building virtual object in which the building elements are displayed may change. In this case, the building elements in the side area of ​​each building virtual object may be formed by a texture image that is tiled to each building virtual object. In addition, the texture image may be tiled in the tiling area excluding the upper, lower, and left and right margin values ​​from the side area of ​​each building virtual object. Accordingly, the processor (1330) may change the area of ​​each building virtual object in which the building elements are displayed as the multimedia content is played by mapping the upper, lower, and left and right margin values ​​to features analyzed from the multimedia content being played.

[0755] Referring to (a) of Fig. 27, (a) of Fig. 27 illustrates a tiling area (2750) in which texture images are tiled in a side area of ​​a building virtual object (2700). In this case, the tiling area (2750) may be an area excluding preset left and right margins (2701, 2702) and top and bottom margins (2711, 2712) at the left and right and top and bottom edges of the side area of ​​the building virtual object (2700). In addition, preset or arbitrary texture images may be continuously synthesized through tiling in the tiling area (2750), and building elements of the building virtual object (2700) may be formed according to the shape included in the tiled texture image.

[0756] In this case, the processor (1330) can map the left and right margins (2701, 2702) and the top and bottom margins (2711, 2712) to features analyzed from the multimedia content being played. In this case, the processor (1330) can change the left and right margins (2701, 2702) and the top and bottom margins (2711, 2712) within a range from a preset minimum value to a preset maximum value according to any one of the features analyzed from the multimedia content being played.

[0757] Accordingly, when the characteristics of the multimedia content being played mapped to the left and right margins (2701, 2702) and the top and bottom margins (2711, 2712) decrease (e.g., decrease in amplitude of a specific frequency), the left and right margins (2701, 2702) and the top and bottom margins (2711, 2712) may decrease. Then, as the left and right margins (2701, 2702) and the top and bottom margins (2711, 2712) decrease, the tiling area (2750) may expand, and as the tiling area (2750) expands, the size of each building element included in the tiling area (2750) may increase, as shown in (b) of FIG. 27.

[0758] Conversely, when the characteristics of the multimedia content being played mapped to the left and right margins (2701, 2702) and the top and bottom margins (2711, 2712) increase (e.g., increase in amplitude of a specific frequency), the left and right margins (2701, 2702) and the top and bottom margins (2711, 2712) may increase. Then, as the left and right margins (2701, 2702) and the top and bottom margins (2711, 2712) increase, the tiling area (2750) may decrease, and as the tiling area (2750) decreases, the size of each building element included in the tiling area (2750) may decrease.

[0759] Meanwhile, the processor (1330) may map only one of the margin values ​​of the left and right margins (2701, 2702) and the top and bottom margins (2711, 2712) to a specific feature of the multimedia content being played. For example, the processor (1330) may map the bottom margin (2712) to a first feature of the multimedia content being played. Then, the value of the bottom margin (2712) may change according to the playback of the multimedia content, and when a value related to the mapped feature increases, the value may increase. Then, as the value of the bottom margin (2712) increases in the side area of ​​the building virtual object (2700), the entire tiling area (2750) may be compressed horizontally (the vertical width may decrease) and moved upward, as shown in (c) of FIG. 27. Accordingly, the size of the area of ​​the building virtual object in which each building element is displayed may change or its position may vary depending on the playback of the multimedia content.

[0760] Meanwhile, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention, in addition to displaying each building object displayed on the 3D map view in conjunction with the multimedia content being played, may also display the building objects displayed on the 3D map view differently for each building object based on the characteristics of the multimedia content being played. Accordingly, it is possible to prevent the building objects displayed on the 3D map view from changing in the same manner, and to prevent the changes in the building objects from becoming monotonous.

[0761] In order to display each building object differently in this way, the processor (1330) may display the building objects displayed on the 3D map view in colors with different brightness based on the characteristics of the multimedia content played in the vehicle. To this end, the processor (1330) may link the analysis result of the multimedia content played to the 3D coordinates (hereinafter defined as world coordinates) at which each building object is displayed on the 3D map view, so that the brightness of the window elements of each building object may be different for each building object.

[0762] FIG. 28 is a conceptual diagram for explaining a process in which a 3D map view generation device according to an embodiment of the present invention determines the brightness of window elements for each building object according to the characteristics of analyzed multimedia content (hereinafter, media characteristics).

[0763] Referring to FIG. 28, the processor (1330) can divide the components for determining the brightness of the window element applied to each building object into a common component (2810) commonly applied to each building object and an individual component (2820) individually applied to each building object. In this case, the characteristics of the multimedia content played in the vehicle are linked to all building objects, and may be characteristics commonly applied to all building objects. Accordingly, the characteristics of the multimedia content can be classified as a common component (2810).

[0764] In this case, the processor (1330) can first determine common components. To this end, the processor (1330) can perform an analysis of the multimedia content currently being played in the vehicle through the media characteristic analyzer (2811). For example, the media characteristic analyzer (2811) can analyze the multimedia content being played differently depending on the type of multimedia content.

[0765] For example, if the multimedia content being played is audio content, the media characteristic analyzer (2811) can separate the audio content being played into frequency bands (2812-1, 2812-2, …, 2812-n) as shown in FIG. 28. In this case, each separated frequency band can become a detailed characteristic of the audio content being played. That is, the number of frequency bands separated from the audio content can determine the number of characteristics of the audio content, and the amplitude values ​​detected in each separated frequency band can be determined as the detailed characteristics (2812-1, 2812-2, …, 2812-n) of the audio content.

[0766] In contrast, the media characteristic analyzer (2811) may separate representative colors from the video content playback screen when the multimedia content being played is video content. In this case, the separated representative colors may become detailed characteristics of the video content, and the number of detailed characteristics may be determined based on the number of representative colors.

[0767] Meanwhile, when the common component (2810) is determined, the processor (1330) can determine individual components (2820) to be applied to each building object. In this case, the processor (1330) can use a random number generator (2821) that receives world coordinates unique to each building object as seed values ​​and generates random values ​​so that different brightnesses can be determined for each building object.

[0768] In this case, the processor (1330) can determine the output range of the random numbers output from the random number generator (2821) according to the number of characteristics of the multimedia content analyzed by the media characteristic analyzer (2811). That is, if the number of detailed characteristics of the multimedia content analyzed by the media characteristic analyzer (2811) is 8 (i.e., n=8), the processor (1330) can control the random number generator (2821) so that a value of a minimum of 1 and a maximum of 8 is generated based on the input seed value.

[0769] Then, the random number generator (2821) can receive the world coordinates of a building object that will determine the brightness of the window elements as a seed value, and generate a random integer within the range of the determined minimum and maximum values. In this case, since the building objects each have their own unique world coordinates in the 3D map view, if the coordinates of each building object are input as a seed value, the random number values ​​generated for each building object can be generated with minimal duplication. In this case, if the random number generated by the random number generator (2821) is not an integer (non-integer), the processor (1330) can perform a floor operation or the like to convert the non-integer random number into an integer.

[0770] And the processor (1330) can input an integerized random number to the feature filter (2823) and determine one feature index corresponding to the integerized random number. Here, the feature filter (2823) may be a filter for determining one of the detailed features of the multimedia content analyzed through the common component (2810) analysis. Accordingly, the feature filter (2823) may be provided with index nodes corresponding to each of the detailed features of the multimedia content analyzed through the common component (2810) analysis, and may determine one index node corresponding to the input random number. For example, when the input random number is '2', the feature filter (2823) may determine an index node corresponding to the second feature among the features of the detailed multimedia content.

[0771] Meanwhile, each index node may correspond to a specific characteristic of the multimedia content. Accordingly, if the multimedia content is audio content, each index node may correspond to an amplitude value detected from each frequency band separated from the audio content. On the other hand, if the multimedia content is video content, each index node may correspond to a representative color detected from the screen on which the video content is played.

[0772] In this case, as shown in Fig. 28, assuming that the analyzed multimedia content is audio content, the characteristic filter (2823) can determine an amplitude value detected from one of the frequency bands separated from the audio content as an amplitude value corresponding to the input building object, based on an index node selected according to an input random number.

[0773] And the processor (1330) can input the amplitude value determined through the characteristic filter (2823) into the brightness correction unit (2824). And, for each building object, a brightness value corresponding to the determined amplitude value can be added to the basic window brightness set as the default to calculate a correction brightness value. And, according to the calculated correction brightness value, the brightness of window elements included in the building object can be determined. In this case, the processor (1330) can correct the color brightness of the albedo map blended to the window elements according to the calculated correction brightness value. Accordingly, the brightness of the window elements of the building object can be changed. That is, the brightness of the window elements of the building object displayed on the 3D map view can be changed in conjunction with the characteristics of the multimedia content being played.

[0774] Meanwhile, for a frequency band where an amplitude value is not detected, the basic window brightness value can be calculated as a correction brightness value, but for a frequency band where an amplitude value is detected, a brightness value different from the basic window brightness value can be calculated as a correction brightness value.

[0775] Additionally, the basic window brightness may vary depending on the vehicle's driving time, i.e., whether it is daytime or nighttime. Accordingly, as the vehicle's driving time changes, the corrected brightness values ​​calculated for each building object may vary.

[0776] Additionally, the amplitude values ​​of each frequency band of the audio content may change depending on the playback of the audio content. Furthermore, if the amplitude values ​​of each frequency band change, the correction brightness values ​​determined based on the amplitude values ​​of each frequency band may change.

[0777] Meanwhile, since the world coordinates of building objects are different for each building object, the random numbers generated by the random number generator may not overlap. In addition, since the characteristic values ​​of multimedia are selected based on different, non-overlapping random numbers, different correction brightness values ​​can be produced for each building object. Accordingly, the 3D map view generation device (1300) according to an embodiment of the present invention can display window elements with different brightness for each building object even if the building objects are close to each other.

[0778] Meanwhile, the window elements to which the corrected brightness value is applied in the above-described Fig. 28 may be limited to window elements that indicate a turned-on state in each building object. Accordingly, if the turned-on window elements are changed, the window elements to which the corrected brightness is applied may also be changed accordingly. In addition, if the shape or size of the window elements is changed due to a change in the shape of the building object or the shape of the window elements, the shape or size of the area (window area) to which the corrected brightness is applied may also change.

[0779] Meanwhile, the above description describes an example in which the brightness of window elements of each building object is differently corrected based on the amplitude values ​​of separated frequency bands when the multimedia content being played is audio content.

[0780] However, even if the multimedia content being played is audio content, it is of course possible to correct the color of the window elements of each building object based on an image related to the audio content. In this case, the image related to the audio content may be an image reminiscent of the audio content, such as an album art image of the audio content.

[0781] An album art image, also called an album cover image, may refer to a picture or photograph printed on the front cover of an album included in an LP (Long Play record) or CD (Compact Disk) containing audio content. The album art image is a representative image corresponding to the audio content and may be provided together with the audio content by the audio content production company. In addition, the processor (1330) may search for the album art image as an image related to the audio content, and determine the color of at least one of the building elements constituting the building objects on the 3D map view based on the result of analyzing the album art image.

[0782] FIG. 29 is a conceptual diagram illustrating a process in which a 3D map view generation device (1300) according to an embodiment of the present invention determines the color of a building object based on an album art image of media being played.

[0783] Referring to FIG. 29, the processor (1330) of the 3D map view generation device (1300) can search for an album art image (2900) corresponding to the audio content being played. Then, a set number of representative colors (2912-1, 2912-2, ... 2912-n) can be detected from the searched album art image (2900) (2911). In this case, the representative colors (2912-1, 2912-2, ... 2912-n) detected from the album art image (2900) can be determined as detailed characteristics of the audio content corresponding to the album art image (2900). In this case, the determined detailed characteristics can be used as a common component (2810).

[0784] Meanwhile, the processor (1330) may generate an integer random number through a random number generator (not shown) that receives the world coordinates of each building object as a seed value, similar to that described in the above-described FIG. 28. Then, the integer random number may be input to a feature filter (2920) that includes index nodes corresponding to each of the detailed features, i.e., representative colors (2912-1, 2912-2, ... 2912-n) detected from the album art image (2900). Then, based on the input random number, the detailed feature to be applied to each building object, i.e., the representative color, may be determined.

[0785] In this case, if the component of the building object to which the representative color is to be applied is a window element, the processor (1330) may determine a correction color of the window elements included in each building object by blending (2930) the currently set basic window color and the representative color determined for each building object. In this case, the processor (1330) may correct the color of the albedo map blended to the window elements according to the determined correction color. Accordingly, the color of the window elements of the building object may be changed. That is, the color of the window elements of the building object displayed on the 3D map view may be changed in conjunction with the characteristics of the multimedia content being played.

[0786] Therefore, when a specific audio content is being played, each building object on the 3D map view displayed on the display unit of the vehicle may have window elements displayed in a tone similar to the representative color of the album art image of the audio content being played, and accordingly, each building object on the 3D map view may have improved color harmony with the album art image of the audio content being played.

[0787] In this case, as described above, since the coordinates (world coordinates) of each building object are unique, a random number can be generated for each building object with minimal duplication. Therefore, even for adjacent building objects, different representative colors can be determined, and accordingly, window elements for adjacent building objects can be displayed with different compensation colors.

[0788] Meanwhile, in the above description, a window element was used as an example of a component of a building object to which a correction color is applied, but this is merely an example to help understand the present invention, and the present invention is not limited thereto. In other words, it goes without saying that the correction color may be applied to the color of a glass element, a window frame element, or a wall element of a building object, rather than a window element, through an operation process identical or similar to that of FIG. 29.

[0789] Meanwhile, in the above description, it was mentioned that the image related to the audio content is an album art image, for example, but in the case of video content, the color of the window element may be changed according to the description of Fig. 29 based on the representative color analysis result of the related image. In this case, the image related to the video content may be the screen on which the video content is played.

[0790] In addition, the processor (1330) may apply different correction colors to different building elements for each building object. For example, the correction color may be applied to a window element of one building object, while the correction color may be applied to a wall element of another building object. In this case, the processor (1330) may change the colors of different building components for each building object according to detailed features analyzed from multimedia content (e.g., audio content). That is, if the building element to which the correction color is applied is a window element, the color of the window element of the building object may be changed according to features analyzed from the audio content. On the other hand, if the building element to which the correction color is applied is a wall element, the color of the wall element of the building object may be changed according to features analyzed from the audio content.

[0791] Meanwhile, in the above description, the results of the analysis of the characteristics of the multimedia content being played were used to correct the brightness or color of a specific building element. However, it is also possible to correct the material values ​​of specific building elements for each building object by blending a normal map or albedo map that has been corrected by synthesizing various characteristic results analyzed from multimedia content.

[0792] FIG. 30 is a conceptual diagram illustrating a process in which a 3D map view generation device (1300) according to an embodiment of the present invention determines different material values ​​for each building object based on a plurality of characteristics detected from multimedia being played.

[0793] Referring to FIG. 30, the processor (1330) of the 3D map view generation device (1300) can analyze various characteristics of the multimedia content being played.

[0794] For example, if the multimedia content being played is audio content, the audio content being played can be separated by frequency band, and the amplitude values ​​detected for each separated frequency band can be detected as first characteristics of the audio content. In addition, the processor (1330) can detect representative colors from an album art image searched for the audio content being played, and the detected representative colors can be detected as second characteristics of the audio content. In addition, the processor (1330) can determine one of the first characteristics and one of the second characteristics based on a random number generated from a random number generator (not shown) that receives the world coordinates of the building object as a seed and produces a random number. In addition, the material of a specific building element of the building object can be determined by blending an albedo map or a normal map corrected according to the determined one of the first characteristics, the determined one of the second characteristics, and a basic material set as a default into a mask map.

[0795] For example, the first characteristic may determine the brightness of a specific building element of the building object. And the second characteristic may determine the color of a specific building element of the building object. And the processor (1330) may blend the albedo map corrected according to the determined brightness and color into a normal map according to a preset basic material. Accordingly, the processor (1330) may correct the preset basic material for the specific building element with a material having a brightness corresponding to any one of the first characteristics and a color corresponding to any one of the second characteristics.

[0796] Meanwhile, even when the multimedia content being played is video content, the processor (1330) can similarly detect multiple detailed characteristics from the video content being played. For example, the processor (1330) can separate audio content included in the video content being played by frequency band and detect amplitude values ​​detected for each separated frequency band as first characteristics of the video content. In addition, the processor (1330) can detect representative colors from the screen of the video content being played and detect the detected representative colors as second characteristics of the video content. In addition, the processor (1330) can determine one of the first characteristics and one of the second characteristics based on the world coordinates of the building object, and blend the albedo map corrected according to the determined first characteristic and second characteristic with the normal map corresponding to the basic material of the building object. Accordingly, the processor (1330) can reflect the detailed characteristics analyzed from the video content being played and correct the material expressing the specific building element so that the brightness or color of the specific building element varies for each building object displayed on the 3D map view.

[0797] Meanwhile, as described above, the 3D map view generation device (1300) according to an embodiment of the present invention can express the material of a building element constituting a building object using a normal map blended into a mask map, and express the color or brightness of the building element using an albedo map blended into the mask map. Therefore, for convenience of explanation, the following description will describe the color or brightness of at least one building element of the building object instead of the process of blending the albedo map or normal map into the mask map. That is, in the following description, the process of changing or displaying the color, brightness, or material of at least one building element constituting one side of a building may be a process including a process of blending the albedo map or normal map into the mask map.

[0798] Meanwhile, in the above description, an example of correcting the brightness of a specific building element (e.g., a window element) according to the amplitude of each band of a frequency band separated from audio content, and correcting the color of a specific building element (e.g., a window element) from an image related to audio content, i.e., an album image, has been described. However, this is merely an example of the present invention assumed for the convenience of explanation, and it is to be understood that the present invention is not limited thereto. That is, the color or material of a specific building element may be changed instead of the brightness according to the amplitude of each band of a frequency band (e.g., corrected with an RGB value corresponding to the amplitude value), and the brightness or material of a specific building element may be changed instead of the color by receiving a luminance or illuminance value corresponding to the RGB value of a representative color detected from an album image.

[0799] Meanwhile, a 3D map view generation device (1300) according to an embodiment of the present invention may determine at least a portion of one side of at least one building object as an area where video content is displayed, and play the video content in the determined area, in order to depict a building object on which a video advertisement is played. Accordingly, a 3D map view including building objects on which a video advertisement is played can be provided, thereby providing a 3D map view more similar to the real world.

[0800] Meanwhile, when video content is displayed on a part of a building object in this way, the processor (1330) of the 3D map view generation device (1300) can provide a user UX that provides a bezel-less feel by displaying the color outside the area where the video content is played in a color similar to the color of the video content being played, thereby expanding the color tone of the video content being played. In this case, the boundary of the area where the video content is displayed is smoothly expanded, thereby providing a sense of visual unity between the area outside the video content display area and the video content display area, thereby maximizing the advertising effect of the video content.

[0801] FIG. 31 illustrates an example in which, for the playback of such bezel-less video content, a 3D map view generation device (1300) according to an embodiment of the present invention determines the color of a surrounding area based on the color of video content displayed on a side area of ​​a building object. In this case, the surrounding area, i.e., the component of the building object whose color is determined based on the color of the video content, is assumed to be a window element. However, this is only an assumption for the convenience of explanation, and it is obvious that the present invention is not limited thereto. That is, it is obvious that the surrounding area may be a wall element of the building object.

[0802] First, referring to (a) of FIG. 31, the processor (1330) can divide the display area (3100) where the video content is to be displayed into the number of horizontal windows of the building, as shown in (a) of FIG. 31. Then, the processor (1330) can determine an area (ambient light area) (3110, 3120) where color is propagated based on a color value detected from the video screen being played, based on the display area (3100).

[0803] In this case, the ambient light area may be determined as an area including at least one building component in a direction orthogonal to the direction in which the display area (3100) is divided. Accordingly, as shown in (a) of FIG. 31, the processor (1330) may determine areas including one window element in the upper or lower direction of each of the divided display areas as ambient light areas (3110, 3120), as shown in (a) of FIG. 31.

[0804] In this case, since the area where the video content is displayed is horizontally divided by the number of building windows, as shown in (a) of Fig. 31, if the horizontal length of the display area (3100) where the video content is displayed has a length corresponding to 8 window elements, it can be divided into 8 areas. In addition, each of the divided display areas can have different window elements of each ambient light area adjacent to it, and different divided display areas can correspond to adjacent window elements.

[0805] Accordingly, the first region, the second region, the third region, the seventh region, and the eighth region of the first ambient light region (3110) may correspond to the first region, the second region, the third region, the seventh region, and the eighth region of the divided display region (3100), respectively. And the a region, the b region, the c region, the g region, and the h region of the second ambient light region (3120) may correspond to the a region, the b region, the c region, the g region, and the h region of the divided display region (3100), respectively. [0806...

Claims

In the 3D map view generation device, An interface unit that receives sensing information collected from at least one sensor equipped in a vehicle; Memory for storing map data for multiple map tiles; and A processor is included that controls the interface unit to receive driving information of the vehicle, models building objects included in map data of a plurality of map tiles determined based on the driving information, generates a plurality of texture images including shapes of different building elements, and generates a 3D map view image including rendered building objects by tiling the texture images on each side of the modeled building object models. The above processor, A 3D map view generation device characterized in that it analyzes a plurality of detailed features from multimedia content played in the vehicle, links at least one of the analyzed detailed features to at least one of the brightness, color, and material of a specific building element, and displays each building object so that the specific building element changes for each building object according to the playback of the multimedia content. In the first paragraph, the processor, A 3D map view generation device characterized in that any one of the analyzed detailed features to be linked to any one of the brightness, color, and material of the specific building element is determined based on a random number generated based on the unique coordinates of each building object. In the first paragraph, the processor, A 3D map view generation device characterized in that it separates the frequency band of the multimedia content being played into a plurality of bands and links the amplitude of each separated frequency band to one of the brightness, color, and material of the specific building element as the detailed feature. In the third paragraph, the processor, Searching for images related to multimedia content being played, and linking a plurality of representative colors detected from the searched images to one of the brightness, color, and material of the specific building element as the detailed features, Images related to the above multimedia content are: A 3D map view generation device characterized in that, if the multimedia content is audio content, it is an album art image, and if the multimedia content is video content, it is a screen on which the video content is played. In the first paragraph, the processor, A 3D map view generation device characterized in that it changes at least one of the brightness or color of the specific building element by adding a brightness value according to at least one of the analyzed detailed features to the basic brightness set for the specific building element, or by blending a color according to at least one of the analyzed detailed features to the basic color set for the specific building element. In the first paragraph, the processor, A 3D map view generation device characterized in that, for each building object, at least one of brightness, color, and material is differently determined for the specific building element to be linked to the detailed features. In the first paragraph, the processor, A 3D map view generation device characterized in that it displays video content on a portion of one side of a specific building object among the above building object models, and corrects the color of an area surrounding a video display area where the video content is displayed among the one side of the specific building object to a representative color analyzed from the video content being played. In the seventh paragraph, the processor, A 3D map view generation device characterized in that the video display area is divided into a plurality of areas based on specific building elements constituting one side of the specific building object, representative colors are detected for each of the divided areas, and the colors of the specific building elements around each of the divided areas are corrected to the detected representative colors. In the first paragraph, the processor, Modeling a display area object in which tiling of the texture image is restricted and inserting the modeled display area object into a specific building object model, and inserting a 3D animation object in which at least one of movement, color, and shape changes according to detailed features analyzed from multimedia content played in the vehicle or the driving state of the vehicle inside the inserted display area object, At least one side of the above display area object, A 3D map view generation device characterized in that it is exposed to one side of the above specific building object model. In the 3D map view generation device, An interface unit that receives sensing information collected from at least one sensor equipped in a vehicle; Memory for storing map data for multiple map tiles; and A processor is included that controls the interface unit to receive driving information of the vehicle, models building objects included in map data of a plurality of map tiles determined based on the driving information, generates a plurality of texture images including shapes of different building elements, and generates a 3D map view image including rendered building objects by tiling the texture images to each side area of ​​the modeled building object models. The above processor, A 3D map view generation device characterized in that it changes the brightness or color of at least some of the specific building elements constituting at least one side of the building objects based on the collected sensing information or the driving information of the vehicle. In the 10th paragraph, the processor, A 3D map view generation device characterized in that it detects an event occurring in the real world based on the collected sensing information, determines a graphic object related to the detected event, detects at least some of specific building elements constituting one side of one of the building objects according to the shape of the determined graphic object, and displays the detected building elements in a color or brightness that is distinct from other building elements, thereby displaying the graphic object related to the detected event on one side of one of the building objects. In the 11th paragraph, the processor, A 3D map view generation device characterized in that it determines a media façade area where the graphic object can be displayed by excluding a building margin value from one side of the above-mentioned building object, and substitutes a mask image corresponding to the graphic object into the determined media façade area to produce a shape of the graphic object with an aspect ratio changed according to the aspect ratio of the determined media façade area. In the 12th paragraph, the processor, A 3D map view generation device characterized in that it detects at least some of specific building elements constituting one side of one of the building objects according to the shape of a graphic object included in a mask image substituted into the media facade area, and changes the color or brightness of at least some of the detected building elements to form a visual pattern corresponding to the shape of the graphic object. In the 12th paragraph, the processor, Checking whether the size of the media facade area determined from one side of the above one building object is greater than or equal to the size according to the minimum scale value of the graphic object, and displaying a graphic object related to the detected event on one side of the above one building object according to the check result, The minimum scale value of the above graphic object is: A 3D map view generation device characterized in that the type of the graphic object is determined according to any one of the distance between the 3D map view camera providing the 3D map view and the one of the building objects or the display magnification of the 3D map view. In the 14th paragraph, the processor, A 3D map view generation device characterized in that, based on the above check result, at least one other building object adjacent to the one building object is further detected, a mask image corresponding to the graphic object is substituted into a display area formed by one side of the one building object and at least one side of the at least one other building object, and a color or brightness of specific building elements detected from the side surfaces of a plurality of building objects is changed based on the substituted mask image to form a visual pattern corresponding to the shape of the graphic object. In the 10th paragraph, the processor, Detecting environmental information around an actual building corresponding to a specific building object, and displaying the brightness or color of at least one side of the specific building object differently according to the detected environmental information, The above specific building object is, A 3D map view generation device characterized by having a preset landmark building object. In the 10th paragraph, the processor, Detecting the status information of the above vehicle, and changing the number or ratio of building elements in which at least one of brightness or color is displayed differently among specific building elements constituting one side of at least one building object based on the status information of the detected vehicle, At least one building object above, A 3D map view generation device characterized in that the building object is around a vehicle object corresponding to the vehicle on the 3D map view. In the 10th paragraph, the processor, Detecting route guidance information of the vehicle, and further displaying additional information related to the detected route guidance information on at least one side of building objects surrounding the vehicle object corresponding to the vehicle on the 3D map view, The above additional information is: A 3D map view generation device characterized in that some of the building elements forming one side of the above building object are displayed in a pattern formed by different colors or brightnesses. In claim 18, the processor, A 3D map view generation device characterized in that, when the 3D map view is changed to a top view in which building objects are displayed in the shape of a footprint area, the additional information is displayed in a footprint area corresponding to at least one of the building objects around the vehicle object. In a control method of a 3D map view generation device, A step of receiving driving information of a vehicle; A step of determining at least one map tile included in a field of view based on the driving information and detecting map data of the determined map tile; A step of modeling building objects included in the above detected map data; A step of generating multiple texture images containing shapes of different building elements; A step of generating a 3D map view image including rendered building objects by tiling the generated texture image on each side of the modeled building object models; A step of analyzing multiple detailed features from multimedia content played in the vehicle; A step of linking at least one of the analyzed detailed features to at least one of the brightness, color, and material of a specific building element based on the unique coordinates of each building object; and, A control method for a 3D map view generation device, characterized in that it includes a step of displaying at least one of the brightness, color, and material of the specific building element linked to the analyzed detailed features differently for each building object according to the playback of the multimedia content.

Images