Mixed reality-based 3D map view generation device and control method of the device
The 3D map view generating device addresses the lack of accurate textures and obscuration in current 3D maps by distorting terrain and limiting building heights, ensuring clear and intuitive route guidance in mixed reality applications.
Patent Information
- Application Number
- KR1020257036706
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2024-08-02
- Filing Date
- 2025-07-09
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2045-07-09
AI Technical Summary
Current 3D maps generated using digital twin technology lack accurate building textures and are obscured by building objects, making it difficult to provide intuitive and accurate route guidance in mixed reality applications.
A 3D map view generating device that distorts terrain and road objects based on vehicle position and viewing conditions, limits building object heights, and applies high-quality textures to virtual objects in real-time, ensuring clear guidance positions.
Provides high-quality, real-time virtual objects with accurate textures and prevents guidance obscuration, enabling intuitive route guidance even at distances from the camera.
Smart Images

Figure 112025121950997-PCT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a device for generating a 3D map view based on mixed reality. Background Technology
[0002] Recently, following Augmented Reality (AR), which outputs graphic objects through a vehicle's windshield or HUD (Head Up Display) or additionally outputs graphic objects to the real world using images captured by a camera, there has been active development of technology related to Mixed Reality (MR), which can provide various simulation information related to vehicles by applying Digital Twin (DT) technology.
[0003] As part of the development of technologies related to such mixed reality, methods to provide information related to route guidance to drivers using mixed reality are being actively researched. In the case of route guidance using mixed reality, there is an advantage in that it can provide various information that the driver located in the cockpit cannot see, such as displaying a graphic object corresponding to the vehicle on a digitized 3D map through the digital twin technology, providing information about a driving route that the driver has not yet driven through the map and the graphic object, or providing a field of view such as a bird's-eye view.
[0004] Meanwhile, in order for a driver to obtain accurate information through route guidance using mixed reality, the degree of correspondence between the map digitized through the aforementioned digital twin technology—that is, the digital twin map—and the real world must be high. To achieve this, it is necessary to provide a 3D map rather than a 2D map. Accordingly, service providers currently offering digital twin maps provide 3D maps created by mapping 2D maps into 3D.
[0005] However, current service providers use building modeling based on satellite or aerial photographs to provide 3D mapping. When using satellite or aerial photographs in this way, the shooting angle is high, making it difficult to obtain images of various sides of the building, and consequently, there is a problem in that it is difficult to provide accurate textures of the building. Accordingly, conventional 3D maps are provided in the form of polygon data containing the building's footprint coordinates and height.
[0006] On the other hand, there is a problem in that 3D maps containing modeling provided in the form of polygon data do not include the accurate textures of the buildings. Such 3D maps composed of polygon modeling only represent the location and 3D volume of buildings and differ from the actual shape of the buildings; this difference causes a discrepancy between the digital twin map and the real world, making it difficult to provide accurate information to drivers.
[0007] In addition, in the case of a 3D map, multiple building objects may be displayed between the camera providing the 3D map view and the guidance location according to the currently set path. Therefore, the 3D map has a problem in that the guidance location may be partially obscured by the building objects, or the path guidance may be provided in a form that is difficult for the user to identify due to the distance between the 3D map view camera and the guidance location. The problem to be solved
[0008] The present invention aims to solve the aforementioned problems and other problems.
[0009] One objective of the present invention is to provide a 3D map view generating device and a control method for the device, which can display virtual objects having high-quality texture materials similar to real-world buildings in real time without delay according to changes in the driving state of a vehicle.
[0010] In addition, another objective of the present invention is to provide a 3D map view generating device and a method for controlling the device that can provide a more realistic mixed reality-based path guidance service through a digital twin map comprising virtual objects equipped with more realistic textures corresponding to each side shape of a building.
[0011] In addition, another objective of the present invention is to provide a 3D map view generating device and a method for controlling the device, wherein the guidance position is not obscured by building objects located between the guidance position and a 3D map view camera that provides a field of view of the 3D map view.
[0012] In addition, another objective of the present invention is to provide a 3D map view generating device and a control method for the device that provide a 3D map that enables a user to identify the guidance location more intuitively and easily, even when the distance between the guidance location and the 3D map view camera is greater than a certain distance. means of solving the problem
[0013] A 3D map view generating device according to an embodiment of the present invention comprises an interface unit for receiving driving information including the position of a vehicle, a memory for storing a map object including at least one of a terrain object and a road object included in map data, and a processor for generating a 3D map view including a guidance position for guiding the driving path of the vehicle based on the driving information. The processor determines a distance at which visibility can be secured for the guidance position based on at least one of the distance between the position of the vehicle and the guidance position, the viewing conditions of the 3D map view, and the driving state of the vehicle; maintains the vertex position of the map object in a first area corresponding to within the distance at which visibility can be secured based on the position of the vehicle, and changes the position of at least some vertices constituting the map object based on a defined coordinate transformation function in a second area exceeding the distance at which visibility can be secured based on the position of the vehicle, generates a modified 3D map view such that the map object of the second area is continuous with the map object of the first area based on the at least some vertices whose positions have been changed, and controls the guidance position to be displayed on the map object of the second area in the modified 3D map view. A 3D map view generating device according to an embodiment of the present invention comprises an interface unit that receives driving information of a vehicle and receives object information regarding dynamic objects around the vehicle, a memory that stores terrain objects and static objects included in map data, and a processor that generates a 3D map view including guidance positions for guiding the driving path of the vehicle, wherein the processor determines a visibility-unsecured area where the guidance positions are not displayed in a visible state in the 3D map view as a second area, and causes the terrain objects of the second area to be deformed by changing the positions of at least some vertices constituting terrain objects included in the second area based on a defined coordinate transformation function.The method is characterized by detecting a static object corresponding to the second area from the map data, detecting a dynamic object located in the second area or displayed corresponding to the second area based on the object information, determining the display position of the static object and the display position of the dynamic object based on a transformation coordinate system corresponding to the transformed terrain object of the second area, and rendering so that the static object and the dynamic object are displayed together on the transformed terrain object to generate a 3D map view including the guidance position, the static object, and the dynamic object.
[0014] In one embodiment, the processor determines the first distance based on at least one of the height of the 3D map view camera providing the field of view of the 3D map view, the gaze direction vector of the 3D map view camera, and the driving state of the vehicle.
[0015] In one embodiment, the processor distorts at least some of the terrain and road objects by modifying the positions of at least some of the vertices constituting the terrain and road objects detected from the map data based on a vertex coordinate transformation function defined by a preset distortion model and distortion parameters, wherein the distortion model is defined in the form of an exponential function and the distortion parameters include the slope value of the exponential function of the distortion model determined according to a scenario in which the 3D map view is provided.
[0016] In one embodiment, the processor varies the curvature of the terrain and road objects depending on the distance between the guidance location for guiding the driving path of the vehicle and the vehicle or the driving speed of the vehicle, and the guidance location is characterized as being at least one of a destination, a waypoint for route guidance, or a specific location set by a user.
[0017] In one embodiment, the processor distorts the terrain and road objects with a distortion rate greater than or equal to a preset minimum distortion rate, wherein the minimum distortion rate is a distortion rate that distorts the terrain objects such that the guidance position exists within a preset distance from the intersection point of the 3D map view camera providing the field of view of the 3D map view and the distorted terrain object.
[0018] In one embodiment, the processor distorts a terrain object corresponding to a map area within a preset distance centered on the current location of the vehicle, and controls the interface unit so that virtual objects included in the distorted map area are displayed on the display unit of the vehicle centered on the location of the vehicle, wherein the distorted map area is a distorted area that is bent into a hemispherical shape centered on the location of the vehicle such that the ground becomes higher as it moves away from the vehicle object corresponding to the vehicle.
[0019] In one embodiment, the processor, when there are multiple guidance locations for guiding the driving path of the vehicle, distinguishes a map area according to each of the multiple guidance locations based on the distance from the vehicle, distorts the terrain and road objects so as to be bent according to different curvatures for each distinguished area, and the different curvatures include curvatures having different polarities (positive (+), negative (-)).
[0020] In one embodiment, the processor models building objects detected from the map data, places the modeled building object models below a footprint area, which is the shape of a ground area occupied by each building object and included in a map area curved according to the distortion of the terrain object, protrudes at least some of the placed building object models from the footprint area according to the driving information of the vehicle, and renders at least some of the building object models protruding from the footprint area to display a 3D map view including building objects included in the curved map area.
[0021] In one embodiment, the processor is characterized by controlling the height of the curved map area so that building object models included in the area exceeding a preset height do not protrude from the footprint area, thereby limiting the rendering of building objects included in the area exceeding the preset height.
[0022] In one embodiment, the processor is characterized by restricting the maximum height protruding from the footprint area corresponding to each building object differently according to the distance exceeding the first distance for building object models included in the area outside the area corresponding to the first distance among the curved map areas.
[0023] In one embodiment, the processor controls the building object model so that it does not protrude from the footprint area corresponding to the building object in the case where the protrusion of the building object model from the footprint area is maximally restricted among the curved map areas, and the area where the protrusion of the building object model is maximally restricted is characterized as a map area where the height of the curved map area reaches a preset height.
[0024] In one embodiment, the preset height is characterized as being the height of a point corresponding to a guidance position for guiding the driving path of the vehicle in the curved map area.
[0025] In one embodiment, the processor is characterized in that building objects included in an area greater than the preset height in the curved map area do not limit the maximum height protruding from the footprint area.
[0026] In one embodiment, the processor is characterized by protruding each building object placed below the footprint area from the footprint area in a direction perpendicular to a horizontal plane parallel to the vehicle's driving plane or to a ground shape inclined with respect to the horizontal plane along the curved map area.
[0027] In one embodiment, the processor is characterized by expanding the size of modeled building objects included in a map area outside the area within the first distance in proportion to the distance from the vehicle.
[0028] In one embodiment, the processor is characterized by determining the size magnification ratio of modeled building objects included in a map area outside the area within the first distance based on the straight-line distance between the center of the 3D map view camera providing the field of view of the 3D map view and each of the modeled building objects.
[0029] In one embodiment, the building objects whose size is expanded are characterized as being building objects around a guidance location for guiding the driving path of the vehicle, building objects corresponding to a pre-set type of building, or building objects according to a pre-set POI (Point Of Interest).
[0030] In one embodiment, the processor renders other modeled objects on the distorted terrain and road objects after the terrain and road objects are modeled and rendered to be bent at a preset curvature, wherein the other modeled objects include dynamic objects including building objects and surrounding vehicles recognized based on information obtained from an ADAS (Advanced Driver Assistance System).
[0031] In one embodiment, the processor is characterized by, when terrain objects and road objects included in the map data are modeled and rendered, modeling building objects included in the map data, and generating the map view image including rendered building objects by tiling a plurality of texture images including the shapes of different building elements on each side of the modeled building objects.
[0032] In addition, a control method for a 3D map view generation device according to an embodiment of the present invention comprises: receiving driving information of a vehicle; determining at least one map tile included in a field of view based on the driving information and detecting map data of the determined map tile; detecting vertices constituting terrain and road objects from the map data; modifying the coordinates of at least some of the vertices constituting the terrain and road objects based on a pre-set distortion model and a vertex coordinate transformation function defined by distortion parameters; modeling and rendering a distorted terrain and road object such that at least some of the vertices with modified coordinates are bent at a pre-set curvature; modeling a building object below the terrain and road object in which at least some of the coordinates are distorted; rendering a building object on at least some of the terrain and road object in which at least some of the coordinates are distorted in response to a user request to display a map view image; and displaying a map view image in which building objects are rendered on the terrain and road object in which at least some of the coordinates are distorted on a display unit of the vehicle. Effects of the invention
[0033] The effects of the 3D map view generating device and the control method of the device according to an embodiment of the present invention are described as follows.
[0034] First, the present invention can pre-load map data of a region corresponding to the area surrounding a vehicle based on the vehicle's location according to the vehicle's driving information from map data of a wide area, and pre-model virtual objects corresponding to buildings and roads included in said loaded region. Furthermore, by rendering only at least some of the modeled virtual objects according to the vehicle's driving information, the invention has the advantage of providing a digital twin map containing high-quality virtual objects similar to real-world buildings and roads in real time according to the vehicle's driving information.
[0035] Second, for at least some of the modeled building objects, at least one mask map corresponding to different parts of a texture image representing each part of the building, and a normal map which is an image having 3D vector values that are superimposed on the mask map and form different textures, are superimposed on the texture image so that each component of the building represented by the texture image has a different material, and the texture image in which the mask map and the normal map are superimposed to form a surface material can be tiled on each face of the at least some of the building objects. Furthermore, by rendering the at least some of the building objects on which the texture image is tiled and displaying them on a display unit, the texturing of the building objects can be performed with a small amount of computation while including building objects having high-quality texture materials similar to the actual building, and there is an advantage of being able to provide a digital twin map including a building model with a high-quality texture similar to the actual building.
[0036] In addition, the present invention has the effect of enabling the user to identify the guidance location more intuitively and easily, even when the distance between the guidance location in the 3D map view and the 3D map view camera is greater than a certain distance, by distorting the terrain object so that it bends according to the distance from the vehicle based on the position of the vehicle on the 3D map.
[0037] In addition, the present invention has the effect of preventing the guidance position from being obscured by the building objects by limiting the height of building objects located within a certain distance from the guidance position among the building objects located between the 3D map view, the guidance position, and the 3D map view camera. Brief explanation of the drawing
[0038] FIG. 1 is a drawing illustrating the exterior of a vehicle according to an embodiment of the present invention. FIG. 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles. FIGS. 3 and 4 are drawings illustrating the interior of a vehicle according to an embodiment of the present invention. FIGS. 5 and 6 are drawings referenced to describe an object according to an embodiment of the present invention. FIG. 7 is a block diagram referenced to describe a vehicle according to an embodiment of the present invention. FIG. 8a is a conceptual diagram illustrating the AR service platform of the present invention. FIG. 8b is a conceptual diagram illustrating an MR service platform for providing MR services of the present invention. FIG. 8c is a conceptual diagram illustrating the MR AMS client of the present invention. FIG. 8d is a conceptual diagram illustrating the MR AMS server of the present invention. FIG. 9 is a conceptual diagram illustrating the DTaaS server of the present invention. FIG. 10 is a block diagram illustrating the structure of a 3D map view generation device according to an embodiment of the present invention. FIG. 11 is a block diagram illustrating the structure of a 3D map view generation device according to an embodiment of the present invention in more detail. FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view generation device and an MR service device according to an embodiment of the present invention. FIG. 13 is a flowchart illustrating the operation process of a processor of a 3D map view generation device according to an embodiment of the present invention, which models virtual objects included in a map area and renders and displays the modeled virtual objects. FIG. 14 is an exemplary diagram illustrating an example in which a processor of a 3D map view generating device according to an embodiment of the present invention updates a map area according to driving information of a vehicle. FIG. 15 is a block diagram illustrating a configuration in which the modeling and rendering of virtual objects are performed by different processes in a 3D map view generation device according to an embodiment of the present invention having a plurality of processors. FIG. 16 is an exemplary diagram illustrating an example in which a screen showing information collected from around a vehicle according to an ADAS system and an MR view screen according to a digital twin map are simultaneously displayed in a 3D map view generating device according to an embodiment of the present invention. FIG. 17a is an exemplary diagram illustrating a frustum-shaped visible area formed across a plurality of map tiles or a single map tile according to the position of the vehicle and the driving direction of the vehicle in a 3D map view generating device according to an embodiment of the present invention. FIG. 17b is an example illustrating an example in which a frustum-shaped visible area shown in FIG. 17a includes a plurality of map tiles. FIG. 18a is an exemplary diagram illustrating an example in which texture images are tiled in a margin width and at least one preset texturing area in a 3D map view generating device according to an embodiment of the present invention. FIG. 18b is an exemplary diagram illustrating examples of facade designs of virtual building objects expressed in various ways by varying partial regions and tiled texture images in a 3D map view generating device according to an embodiment of the present invention. FIG. 19 is a flowchart illustrating a texturing process for forming different surface textures on each component of a building in a 3D map view generating device according to an embodiment of the present invention. FIG. 20 is an exemplary diagram illustrating an example in which normal maps are blended into a mask map corresponding to different building elements and a multi-blending normal map is generated by blending the normal maps together in a 3D map view generation device according to an embodiment of the present invention. FIG. 21 is a flowchart illustrating the operation process of mapping the colors of virtual building objects based on the colors of actual buildings within a visible area in a 3D map view generating device according to an embodiment of the present invention. FIG. 22 is an exemplary diagram illustrating an example in which a color mapped to texture images is determined through an image captured from a vehicle in a 3D map view generating device according to an embodiment of the present invention. FIG. 23 is a conceptual diagram illustrating the concept of GPU shader processing that distorts terrain objects in a 3D map view generating device according to an embodiment of the present invention. FIG. 24 is an illustrative diagram showing an example in which a terrain object is distorted according to the distance from the vehicle in a 3D map view generating device according to an embodiment of the present invention. FIG. 25 is an example diagram illustrating an example in which terrain objects are distorted differently depending on the driving speed of a vehicle in a 3D map view generating device according to an embodiment of the present invention. FIG. 26 is an exemplary diagram illustrating an example in which path guidance is provided according to a distorted terrain object in a 3D map view generating device according to an embodiment of the present invention. FIG. 27 is an exemplary diagram illustrating an example in which terrain objects are distorted so that the terrain between a plurality of guidance positions differs from one another according to the distance from the vehicle in a 3D map view generating device according to an embodiment of the present invention. FIG. 28 illustrates an example of a top-down view provided in a 3D map view generating device according to an embodiment of the present invention, in which terrain objects are distorted according to distance from the vehicle. FIG. 29 is an exemplary diagram illustrating an example in which a building object is displayed on a distorted terrain object according to the height of the distorted terrain object in a 3D map view generating device according to an embodiment of the present invention. FIG. 30 illustrates an example where the guidance location is obscured by building objects when the terrain object is not distorted. FIG. 31 is an illustrative diagram showing different examples of displaying building objects displayed on distorted terrain objects in a 3D map view generating device according to an embodiment of the present invention. FIG. 32 is an exemplary diagram illustrating examples in which the height of building objects is limited according to the display method of building objects displayed on a distorted terrain object in a 3D map view generating device according to an embodiment of the present invention. FIG. 33 is an illustrative diagram showing an example in which building objects displayed on a distorted terrain object are displayed differently depending on the distance in a 3D map view generating device according to an embodiment of the present invention. FIG. 34 is an exemplary diagram illustrating an example in which dynamic objects are displayed according to distorted terrain in a 3D map view generating device according to an embodiment of the present invention. Specific details for implementing the invention
[0039] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.
[0040] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.
[0041] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.
[0042] Singular expressions include plural expressions unless the context clearly indicates otherwise.
[0043] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0044] The concept of a vehicle as described in this specification may include automobiles and motorcycles. Hereinafter, vehicles will be described primarily in the context of automobiles.
[0045] The vehicle described in this specification may be a concept that includes all of the following: an internal combustion engine vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, etc.
[0046] In the following description, the left side of the vehicle refers to the left side of the vehicle's direction of travel, and the right side of the vehicle refers to the right side of the vehicle's direction of travel.
[0047] FIG. 1 is a drawing illustrating the exterior of a vehicle according to an embodiment of the present invention.
[0048] FIG. 2 is a drawing of a vehicle according to an embodiment of the present invention viewed from various external angles.
[0049] FIGS. 3 and 4 are drawings illustrating the interior of a vehicle according to an embodiment of the present invention.
[0050] FIGS. 5 and 6 are drawings referenced to describe an object according to an embodiment of the present invention.
[0051] FIG. 7 is a block diagram referenced to describe a vehicle according to an embodiment of the present invention.
[0052] Referring to FIGS. 1 to 7, the vehicle (100) may include a wheel that rotates by a power source and a steering input device (510) for controlling the direction of travel of the vehicle (100).
[0053] The vehicle (100) may be an autonomous vehicle.
[0054] The vehicle (100) can be switched to autonomous driving mode or manual mode based on user input.
[0055] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on user input received through the user interface device (200).
[0056] The vehicle (100) can be switched to an autonomous driving mode or a manual mode based on driving situation information. The driving situation information can be generated based on object information provided by an object detection device (300).
[0057] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information generated by the object detection device (300).
[0058] For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information received through the communication device (400).
[0059] The vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on information, data, and signals provided by an external device.
[0060] When the vehicle (100) is operated in autonomous driving mode, the autonomous vehicle (100) can be operated based on the driving system (700).
[0061] For example, an autonomous vehicle (100) can be operated based on information, data, or signals generated from a driving system (710), a vehicle exit system (740), and a parking system (750).
[0062] When the vehicle (100) is operated in manual mode, the autonomous vehicle (100) can receive user input for driving through the driving control device (500). Based on the user input received through the driving control device (500), the vehicle (100) can be operated.
[0063] The overall length refers to the length from the front to the rear of the vehicle (100), the overall width refers to the width of the vehicle (100), and the overall height refers to the length from the bottom of the wheels to the roof. In the following description, the overall length direction (L) may refer to the direction that serves as the reference for measuring the overall length of the vehicle (100), the overall width direction (W) may refer to the direction that serves as the reference for measuring the overall width of the vehicle (100), and the overall height direction (H) may refer to the direction that serves as the reference for measuring the overall height of the vehicle (100).
[0064] As illustrated in FIG. 7, the vehicle (100) may include a user interface device (200), an object detection device (300), a communication device (400), a driving operation device (500), a vehicle driving device (600), a driving system (700), a navigation system (770), a sensing unit (120), a vehicle interface unit (130), a memory (140), a control unit (170), and a power supply unit (190).
[0065] According to an embodiment, the vehicle (100) may include additional components other than those described herein, or may not include some of the described components.
[0066] The user interface device (200) is a device for communication between the vehicle (100) and the user. The user interface device (200) can receive user input and provide information generated by the vehicle (100) to the user. The vehicle (100) can implement a UI (User Interfaces) or UX (User Experience) through the user interface device (200).
[0067] The user interface device (200) may include an input unit (210), an internal camera (220), a bio-sensing unit (230), an output unit (250), and a processor (270).
[0068] According to an embodiment, the user interface device (200) may include additional components other than the described components, or may not include some of the described components.
[0069] The input unit (200) is for receiving information from a user, and the data collected from the input unit (120) can be analyzed by the processor (270) and processed into a control command of the user.
[0070] The input section (200) may be positioned inside the vehicle. For example, the input section (200) may be positioned in an area of the steering wheel, an area of the instrument panel, an area of the seat, an area of each pillar, an area of the door, an area of the center console, an area of the head lining, an area of the sun visor, an area of the windshield, or an area of the window.
[0071] The input unit (200) may include a voice input unit (211), a gesture input unit (212), a touch input unit (213), and a mechanical input unit (214).
[0072] The voice input unit (211) can convert the user's voice input into an electrical signal. The converted electrical signal can be provided to the processor (270) or the control unit (170).
[0073] The voice input unit (211) may include one or more microphones.
[0074] The gesture input unit (212) can convert the user's gesture input into an electrical signal. The converted electrical signal can be provided to the processor (270) or the control unit (170).
[0075] The gesture input unit (212) may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input.
[0076] According to an embodiment, the gesture input unit (212) can detect a user's three-dimensional gesture input. To this end, the gesture input unit (212) may include a light output unit that outputs a plurality of infrared lights or a plurality of image sensors.
[0077] The gesture input unit (212) can detect the user's three-dimensional gesture input through the Time of Flight (TOF) method, the structured light method, or the disparity method.
[0078] The touch input unit (213) can convert the user's touch input into an electrical signal. The converted electrical signal can be provided to the processor (270) or the control unit (170).
[0079] The touch input unit (213) may include a touch sensor for detecting a user's touch input.
[0080] According to an embodiment, the touch input unit (213) can be formed integrally with the display unit (251) to implement a touch screen. Such a touch screen can provide both an input interface and an output interface between the vehicle (100) and the user.
[0081] The mechanical input unit (214) may include at least one of a button, a dome switch, a jog wheel, and a jog switch. An electrical signal generated by the mechanical input unit (214) may be provided to a processor (270) or a control unit (170).
[0082] The mechanical input section (214) can be placed in the steering wheel, center fascia, center console, cockpit module, door, etc.
[0083] The internal camera (220) can acquire an image of the vehicle interior. The processor (270) can detect the state of the user based on the image of the vehicle interior. The processor (270) can acquire the user's gaze information from the image of the vehicle interior. The processor (270) can detect the user's gestures from the image of the vehicle interior.
[0084] The biometric detection unit (230) can acquire the user's biometric information. The biometric detection unit (230) includes a sensor capable of acquiring the user's biometric information, and can acquire the user's fingerprint information, heart rate information, etc. by using the sensor. The biometric information can be used for user authentication.
[0085] The output unit (250) is intended to generate output related to sight, hearing, or touch.
[0086] The output unit (250) may include at least one of a display unit (251), an acoustic output unit (252), and a haptic output unit (253).
[0087] The display unit (251) can display graphic objects corresponding to various information.
[0088] The display unit (251) may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.
[0089] The display unit (251) can be formed as a layered structure with the touch input unit (213) or as an integral unit to implement a touch screen.
[0090] The display unit (251) can be implemented as a HUD (Head Up Display). When the display unit (251) is implemented as a HUD, the display unit (251) may be equipped with a projection module to output information through an image projected onto a windshield or window.
[0091] The display unit (251) may include a transparent display. The transparent display may be attached to a windshield or a window.
[0092] A transparent display can display a predetermined screen while having a predetermined transparency. To have transparency, the transparent display may include at least one of a transparent TFEL (Thin Film Electroluminescent), a transparent OLED (Organic Light-Emitting Diode), a transparent LCD (Liquid Crystal Display), a transparent transparent display, and a transparent LED (Light Emitting Diode) display. The transparency of the transparent display can be adjusted.
[0093] Meanwhile, the user interface device (200) may include a plurality of display units (251a to 251g).
[0094] The display unit (251) may be placed in a part of the steering wheel, a part of the instrument panel (521a, 251b, 251e), a part of the seat (251d), a part of each pillar (251f), a part of the door (251g), a part of the center console, a part of the head lining, a part of the sun visor, or implemented in a part of the windshield (251c) or a part of the window (251h).
[0095] The sound output unit (252) converts an electrical signal provided from the processor (270) or the control unit (170) into an audio signal and outputs it. To this end, the sound output unit (252) may include one or more speakers.
[0096] The haptic output unit (253) generates a tactile output. For example, the haptic output unit (253) can operate by vibrating the steering wheel, seat belt, and seat (110FL, 110FR, 110RL, 110RR) so that the user can perceive the output.
[0097] The processor (270) can control the overall operation of each unit of the user interface device (200).
[0098] According to an embodiment, the user interface device (200) may include a plurality of processors (270) or may not include a processor (270).
[0099] If the user interface device (200) does not include a processor (270), the user interface device (200) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).
[0100] Meanwhile, the user interface device (200) can be named as a vehicle display device.
[0101] The user interface device (200) can be operated under the control of the control unit (170).
[0102] The object detection device (300) is a device for detecting an object located outside the vehicle (100).
[0103] The objects may be various objects related to the operation of the vehicle (100).
[0104] Referring to FIGS. 5 and 6, the object may include a lane (OB10), other vehicles (OB11), pedestrians (OB12), motorcycles (OB13), traffic signals (OB14, OB15), light, roads, structures, speed bumps, terrain features, animals, etc.
[0105] A lane (OB10) may be a driving lane, a lane adjacent to a driving lane, or a lane on which an oncoming vehicle travels. A lane (OB10) may be a concept that includes the left and right lines forming the lane.
[0106] Other vehicles (OB11) may be vehicles driving in the vicinity of the vehicle (100). Other vehicles may be vehicles located within a predetermined distance from the vehicle (100). For example, other vehicles (OB11) may be vehicles that are ahead of or behind the vehicle (100).
[0107] A pedestrian (OB12) may be a person located near a vehicle (100). A pedestrian (OB12) may be a person located within a predetermined distance from the vehicle (100). For example, a pedestrian (OB12) may be a person located on a sidewalk or a roadway.
[0108] A two-wheeled vehicle (OB12) may refer to a vehicle that moves using two wheels and is located around a vehicle (100). A two-wheeled vehicle (OB12) may be a vehicle having two wheels located within a predetermined distance from the vehicle (100). For example, a two-wheeled vehicle (OB13) may be a motorcycle or bicycle located on a sidewalk or roadway.
[0109] Traffic signals may include traffic lights (OB15), traffic signs (OB14), patterns or text drawn on the road surface.
[0110] The light may be light generated from a lamp equipped in another vehicle. The light may be light generated from a street light. The light may be sunlight.
[0111] Roads may include road surfaces, curves, slopes such as uphill and downhill, etc.
[0112] A structure may be an object located near a road and fixed to the ground. For example, a structure may include streetlights, street trees, buildings, utility poles, traffic lights, and bridges.
[0113] Topographical features may include mountains, hills, etc.
[0114] Meanwhile, objects can be classified into moving objects and stationary objects. For example, moving objects may include other vehicles and pedestrians. For example, stationary objects may include traffic signals, roads, and structures.
[0115] The object detection device (300) may include a camera (310), radar (320), lidar (330), ultrasonic sensor (340), infrared sensor (350), and processor (370).
[0116] According to an embodiment, the object detection device (300) may include additional components other than the described components, or may not include some of the described components.
[0117] The camera (310) may be positioned at an appropriate location outside the vehicle to acquire images of the vehicle's exterior. The camera (310) may be a mono camera, a stereo camera (310a), an AVM (Around View Monitoring) camera (310b), or a 360-degree camera.
[0118] For example, the camera (310) may be positioned in the interior of the vehicle, close to the front windshield, to acquire an image of the front of the vehicle. Alternatively, the camera (310) may be positioned around the front bumper or radiator grille.
[0119] For example, the camera (310) may be positioned in the interior of the vehicle, close to the rear glass, to obtain an image of the rear of the vehicle. Alternatively, the camera (310) may be positioned around the rear bumper, trunk, or tailgate.
[0120] For example, the camera (310) may be positioned in the interior of the vehicle in close proximity to at least one of the side windows to obtain an image of the side of the vehicle. Alternatively, the camera (310) may be positioned around the side mirror, fender, or door.
[0121] The camera (310) can provide the acquired image to the processor (370).
[0122] The radar (320) may include an electromagnetic wave transmitter and a receiver. The radar (320) may be implemented as a pulse radar or a continuous wave radar based on the principle of radio wave emission. Among the continuous wave radar methods, the radar (320) may be implemented as a Frequency Modulated Continuous Wave (FMCW) or Frequency Shift Keying (FSK) method depending on the signal waveform.
[0123] The radar (320) can detect an object based on the Time of Flight (TOF) method or the phase-shift method using electromagnetic waves, and can detect the location of the detected object, the distance from the detected object, and the relative speed.
[0124] The radar (320) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.
[0125] The lidar (330) may include a laser transmitter and a receiver. The lidar (330) may be implemented using a Time of Flight (TOF) method or a phase-shift method.
[0126] The lidar (330) can be implemented as a driven or non-driven type.
[0127] When implemented as a driven type, the lidar (330) is rotated by a motor and can detect objects around the vehicle (100).
[0128] When implemented as a non-driven type, the lidar (330) can detect an object located within a predetermined range relative to the vehicle (100) by optical steering. The vehicle (100) may include a plurality of non-driven lidars (330).
[0129] Lidar (330) can detect an object based on the Time of Flight (TOF) method or the phase-shift method using laser light medium, and can detect the position of the detected object, the distance from the detected object, and the relative velocity.
[0130] The lidar (330) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.
[0131] The ultrasonic sensor (340) may include an ultrasonic transmitter and a receiver. The ultrasonic sensor (340) can detect an object based on ultrasound and detect the position of the detected object, the distance from the detected object, and the relative speed.
[0132] The ultrasonic sensor (340) can be placed at an appropriate location outside the vehicle to detect an object located in front, behind, or to the side of the vehicle.
[0133] The infrared sensor (350) may include an infrared transmitter and a receiver. The infrared sensor (340) can detect an object based on infrared light and detect the position of the detected object, the distance from the detected object, and the relative speed.
[0134] An infrared sensor (350) can be placed at an appropriate location outside the vehicle to detect an object located in front, behind, or to the side of the vehicle.
[0135] The processor (370) can control the overall operation of each unit of the object detection device (300).
[0136] The processor (370) can detect and track objects based on the acquired image. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object through an image processing algorithm.
[0137] The processor (370) can detect and track an object based on the reflected electromagnetic waves that are reflected back from the object after the transmitted electromagnetic waves are reflected. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object based on the electromagnetic waves.
[0138] The processor (370) can detect and track an object based on the reflected laser light that is reflected back from the object by the transmitted laser. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object based on the laser light.
[0139] The processor (370) can detect and track an object based on the reflected ultrasound that is reflected back from the object after the transmitted ultrasound is reflected. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object based on the ultrasound.
[0140] The processor (370) can detect and track an object based on the reflected infrared light that is reflected back from the object after the transmitted infrared light is reflected. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed to the object based on the infrared light.
[0141] According to an embodiment, the object detection device (300) may include a plurality of processors (370) or may not include a processor (370). For example, each of the camera (310), radar (320), lidar (330), ultrasonic sensor (340), and infrared sensor (350) may individually include a processor.
[0142] If the object detection device (300) does not include a processor (370), the object detection device (300) may be operated under the control of the processor or control unit (170) of the device in the vehicle (100).
[0143] The object detection device (400) can be operated under the control of the control unit (170).
[0144] The communication device (400) is a device for performing communication with an external device. Here, the external device may be another vehicle, a mobile terminal, or a server.
[0145] A communication device (400) may include at least one of a transmitting antenna, a receiving antenna, an RF (Radio Frequency) circuit capable of implementing various communication protocols, and an RF element to perform communication.
[0146] The communication device (400) may include a short-range communication unit (410), a location information unit (420), a V2X communication unit (430), an optical communication unit (440), a broadcast transmission and reception unit (450), and a processor (470).
[0147] According to an embodiment, the communication device (400) may include additional components other than the described components, or may not include some of the described components.
[0148] The short-range communication unit (410) is a unit for short-range communication. The short-range communication unit (410) can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.
[0149] The short-range communication unit (410) can form a short-range wireless communication network (Wireless Area Networks) to perform short-range communication between the vehicle (100) and at least one external device.
[0150] The location information unit (420) is a unit for obtaining location information of the vehicle (100). For example, the location information unit (420) may include a Global Positioning System (GPS) module or a Differential Global Positioning System (DGPS) module.
[0151] The V2X communication unit (430) is a unit for performing wireless communication with a server (V2I: Vehicle to Infra), other vehicles (V2V: Vehicle to Vehicle), or pedestrians (V2P: Vehicle to Pedestrian). The V2X communication unit (430) may include an RF circuit capable of implementing communication protocols with infrastructure (V2I), communication between vehicles (V2V), and communication with pedestrians (V2P).
[0152] The optical communication unit (440) is a unit for communicating with an external device via light. The optical communication unit (440) may include an optical transmitter that converts an electrical signal into an optical signal and transmits it externally, and an optical receiver that converts a received optical signal into an electrical signal.
[0153] According to an embodiment, the light emitter may be formed to be integrated with a lamp included in the vehicle (100).
[0154] The broadcast transmission / reception unit (450) is a unit for receiving a broadcast signal from an external broadcast management server or transmitting a broadcast signal to a broadcast management server via a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, and a data broadcast signal.
[0155] The processor (470) can control the overall operation of each unit of the communication device (400).
[0156] According to an embodiment, the communication device (400) may include a plurality of processors (470) or may not include a processor (470).
[0157] If the communication device (400) does not include a processor (470), the communication device (400) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).
[0158] Meanwhile, the communication device (400) can implement a vehicle display device together with the user interface device (200). In this case, the vehicle display device may be named a telematics device or an AVN (Audio Video Navigation) device.
[0159] The communication device (400) can be operated under the control of the control unit (170).
[0160] The driving control device (500) is a device that receives user input for driving.
[0161] In manual mode, the vehicle (100) can be operated based on a signal provided by the driving control device (500).
[0162] The driving control device (500) may include a steering input device (510), an acceleration input device (530), and a brake input device (570).
[0163] The steering input device (510) can receive input regarding the direction of travel of the vehicle (100) from a user. The steering input device (510) is preferably formed in the shape of a wheel so that steering input can be made by rotation. According to an embodiment, the steering input device may be formed in the shape of a touch screen, a touch pad, or a button.
[0164] The acceleration input device (530) can receive input from a user for accelerating the vehicle (100). The brake input device (570) can receive input from a user for decelerating the vehicle (100). The acceleration input device (530) and the brake input device (570) are preferably formed in the form of pedals. According to an embodiment, the acceleration input device or the brake input device may be formed in the form of a touch screen, a touch pad, or a button.
[0165] The driving operation device (500) can be operated according to the control of the control unit (170).
[0166] The vehicle drive unit (600) is a device that electrically controls the operation of various devices within the vehicle (100).
[0167] The vehicle drive unit (600) may include a power train drive unit (610), a chassis drive unit (620), a door / window drive unit (630), a safety device drive unit (640), a lamp drive unit (650), and an air conditioning drive unit (660).
[0168] According to an embodiment, the vehicle drive unit (600) may include additional components other than the described components, or may not include some of the described components.
[0169] Meanwhile, the vehicle drive unit (600) may include a processor. Each unit of the vehicle drive unit (600) may individually include a processor.
[0170] The power train drive unit (610) can control the operation of the power train device.
[0171] The power train drive unit (610) may include a power source drive unit (611) and a transmission drive unit (612).
[0172] The power source drive unit (611) can control the power source of the vehicle (100).
[0173] For example, when a fossil fuel-based engine is the power source, the power source drive unit (610) can perform electronic control of the engine. By doing so, the output torque of the engine can be controlled. The power source drive unit (611) can adjust the engine output torque according to the control of the control unit (170).
[0174] For example, if the power source is an electric energy-based motor, the power source drive unit (610) can perform control over the motor. The power source drive unit (610) can adjust the rotational speed, torque, etc. of the motor according to the control of the control unit (170).
[0175] The transmission drive unit (612) can perform control over the transmission.
[0176] The transmission drive unit (612) can adjust the state of the transmission. The transmission drive unit (612) can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or parking (P).
[0177] Meanwhile, when the engine is the power source, the transmission drive unit (612) can adjust the gear engagement state in the forward (D) state.
[0178] The chassis drive unit (620) can control the operation of the chassis device.
[0179] The chassis drive unit (620) may include a steering drive unit (621), a brake drive unit (622), and a suspension drive unit (623).
[0180] The steering drive unit (621) can perform electronic control of the steering apparatus within the vehicle (100). The steering drive unit (621) can change the direction of travel of the vehicle.
[0181] The brake drive unit (622) can perform electronic control of the brake apparatus within the vehicle (100). For example, it can reduce the speed of the vehicle (100) by controlling the operation of the brakes placed on the wheels.
[0182] Meanwhile, the brake drive unit (622) can control each of the plurality of brakes individually. The brake drive unit (622) can control the braking force applied to the plurality of wheels differently.
[0183] The suspension drive unit (623) can perform electronic control of the suspension apparatus within the vehicle (100). For example, the suspension drive unit (623) can control the suspension apparatus when there is a curvature in the road surface to reduce the vibration of the vehicle (100).
[0184] Meanwhile, the suspension drive unit (623) can control each of the multiple suspensions individually.
[0185] The door / window drive unit (630) can perform electronic control of the door apparatus or window apparatus within the vehicle (100).
[0186] The door / window drive unit (630) may include a door drive unit (631) and a window drive unit (632).
[0187] The door drive unit (631) can perform control over the door device. The door drive unit (631) can control the opening and closing of a plurality of doors included in the vehicle (100). The door drive unit (631) can control the opening or closing of the trunk or tail gate. The door drive unit (631) can control the opening or closing of the sunroof.
[0188] The window drive unit (632) can perform electronic control of the window apparatus. It can control the opening or closing of a plurality of windows included in the vehicle (100).
[0189] The safety device drive unit (640) can perform electronic control of various safety apparatuses within the vehicle (100).
[0190] The safety device drive unit (640) may include an airbag drive unit (641), a seatbelt drive unit (642), and a pedestrian protection device drive unit (643).
[0191] The airbag drive unit (641) can perform electronic control of the airbag apparatus within the vehicle (100). For example, the airbag drive unit (641) can control the airbag to deploy when danger is detected.
[0192] The seatbelt drive unit (642) can perform electronic control of the seatbelt apparatus in the vehicle (100). For example, the seatbelt drive unit (642) can control the seatbelt to secure the passenger to the seat (110FL, 110FR, 110RL, 110RR) using the seatbelt when danger is detected.
[0193] The pedestrian protection device drive unit (643) can perform electronic control of the hood lift and pedestrian airbag. For example, the pedestrian protection device drive unit (643) can control the hood lift up and the pedestrian airbag to deploy when a collision with a pedestrian is detected.
[0194] The lamp driving unit (650) can perform electronic control of various lamp apparatuses within the vehicle (100).
[0195] The air conditioning drive unit (660) can perform electronic control of the air conditioner in the vehicle (100). For example, the air conditioning drive unit (660) can control the air conditioner to operate and supply cold air into the vehicle when the temperature inside the vehicle is high.
[0196] The vehicle drive unit (600) may include a processor. Each unit of the vehicle drive unit (600) may individually include a processor.
[0197] The vehicle drive unit (600) can be operated under the control of the control unit (170).
[0198] The driving system (700) is a system that controls various operations of the vehicle (100). The driving system (700) can be operated in an autonomous driving mode.
[0199] The operation system (700) may include a driving system (710), an exit system (740), and a parking system (750).
[0200] According to an embodiment, the operating system (700) may include additional components other than the components described, or may not include some of the components described.
[0201] Meanwhile, the operating system (700) may include a processor. Each unit of the operating system (700) may individually include a processor.
[0202] Meanwhile, according to an embodiment, if the operating system (700) is implemented in software, it may be a sub-concept of the control unit (170).
[0203] Meanwhile, according to an embodiment, the driving system (700) may be a concept that includes at least one of a user interface device (200), an object detection device (300), a communication device (400), a vehicle driving device (600), and a control unit (170).
[0204] The driving system (710) can drive the vehicle (100).
[0205] The driving system (710) receives navigation information from the navigation system (770) and provides a control signal to the vehicle driving device (600) to drive the vehicle (100).
[0206] The driving system (710) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100).
[0207] The driving system (710) can receive a signal from an external device through a communication device (400) and provide a control signal to a vehicle driving device (600) to drive the vehicle (100).
[0208] The vehicle dispatch system (740) can perform the dispatch of the vehicle (100).
[0209] The vehicle exit system (740) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform the exit of the vehicle (100).
[0210] The vehicle output system (740) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to perform the output of the vehicle (100).
[0211] The vehicle output system (740) can receive a signal from an external device through a communication device (400) and provide a control signal to a vehicle driving device (600) to perform the output of the vehicle (100).
[0212] The parking system (750) can perform parking of the vehicle (100).
[0213] The parking system (750) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).
[0214] The parking system (750) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).
[0215] The parking system (750) can receive a signal from an external device through a communication device (400) and provide a control signal to a vehicle driving device (600) to perform parking of the vehicle (100).
[0216] The navigation system (770) can provide navigation information. The navigation information may include at least one of map information, set destination information, path information according to the set destination, information about various objects on the path, lane information, and current location information of the vehicle.
[0217] The navigation system (770) may include memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system (770).
[0218] According to an embodiment, the navigation system (770) can receive information from an external device through a communication device (400) and update previously stored information.
[0219] According to an embodiment, the navigation system (770) may be classified as a sub-component of the user interface device (200).
[0220] The sensing unit (120) can sense the state of the vehicle. The sensing unit (120) may include an attitude sensor (e.g., yaw sensor, roll sensor, pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight sensing sensor, a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.
[0221] The sensing unit (120) can acquire sensing signals regarding vehicle attitude information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / reverse information, battery information, fuel information, tire information, vehicle lamp information, vehicle interior temperature information, vehicle interior humidity information, steering wheel rotation angle, vehicle exterior illumination, pressure applied to the accelerator pedal, pressure applied to the brake pedal, etc.
[0222] The sensing unit (120) may further include, in addition, an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.
[0223] The vehicle interface unit (130) can serve as a channel for various types of external devices connected to the vehicle (100). For example, the vehicle interface unit (130) may be equipped with a port that can be connected to a mobile terminal, and may be connected to a mobile terminal through said port. In this case, the vehicle interface unit (130) can exchange data with the mobile terminal.
[0224] Meanwhile, the vehicle interface unit (130) can serve as a channel for supplying electrical energy to a connected mobile terminal. When a mobile terminal is electrically connected to the vehicle interface unit (130), the vehicle interface unit (130) can provide electrical energy supplied from the power supply unit (190) to the mobile terminal under the control of the control unit (170).
[0225] The memory (140) is electrically connected to the control unit (170). The memory (140) can store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory (140) can be a hardware storage device such as ROM, RAM, EPROM, flash drive, hard drive, etc. The memory (140) can store various data for the overall operation of the vehicle (100), such as programs for processing or control by the control unit (170).
[0226] According to an embodiment, the memory (140) may be formed integrally with the control unit (170) or implemented as a sub-component of the control unit (170).
[0227] The control unit (170) can control the overall operation of each unit within the vehicle (100). The control unit (170) may be named an Electronic Control Unit (ECU).
[0228] The power supply unit (190) can supply power necessary for the operation of each component according to the control of the control unit (170). In particular, the power supply unit (190) can receive power from a battery inside the vehicle, etc.
[0229] One or more processors and control units (170) included in the vehicle (100) may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.
[0230] FIG. 8a is a conceptual diagram illustrating the AR service platform of the present invention.
[0231] The AR service platform providing the AR service of the present invention may be named an AR service system.
[0232] The above AR service platform may include a server (850) that is provided outside the vehicle and collects and processes information necessary for AR services and transmits it to the vehicle, and an AR service device (800) that is provided in the vehicle and provides AR services using the information transmitted from the server.
[0233] The statement that the server (850) collects and processes information necessary for the AR service and transmits it to the vehicle may include the meaning that the server (850) collects and processes information necessary for the AR service and transmits it to the AR service device (800) equipped in the vehicle.
[0234] The above AR service device (800) can vary the information provided as an AR service based on the vehicle's condition.
[0235] That is, the AR service device (800) of the present invention can dynamically adjust (variably) the information and amount of information to be displayed as AR according to the vehicle's situation and select the information to be emphasized.
[0236] In addition, the AR service platform of the present invention can control the AR service provided in the vehicle to vary according to specific conditions, such as vehicle conditions and advertising exposure conditions.
[0237] Conventional AR navigation has limitations in that it is difficult to reflect the latest information when displaying destinations or major Points of Interest (POIs) because it utilizes data stored in map data, and it cannot provide POIs that include real-time attributes, such as refueling or parking.
[0238] On the other hand, the AR service platform of the present invention can fuse vehicle location information, map data, multiple sensor data, real-time POI information, advertising / event information, etc., and display them on AR navigation.
[0239] For example, for displaying AR information, the AR service device (800) of the present invention receives AR service information from a server based on the vehicle's current location and navigation route / guidance information, and can process it into a form for display on an AR navigation screen.
[0240] For example, the AR service device (800) of the present invention can reconfigure real-time AR display information. The AR service device (800) can reconfigure service data received from a server to be displayed on an AR navigation screen by determining the display format, size, location, exposure method, etc. of the AR content in consideration of driving conditions (e.g., varying the POI exposure location and size according to driving speed, changing the service information exposure location according to traffic conditions, adjusting the AR Wall display location and exposure time, etc.).
[0241] In addition, the AR service device (800) of the present invention can analyze the frequency of exposure of AR display information through user feedback.
[0242] The server (850) collects user input information (input information such as touch, order, etc.) for AR service content, performs an analysis of the frequency of content exposure, and can adjust the service content exposure policy based on the information.
[0243] Through this configuration, the present invention can fuse various external service contents to be displayed in AR navigation and provide various services through POI information including real-time attributes.
[0244] In addition, the present invention enables the display of various forms of AR content, such as advertisements, events, and major landmark information, as well as POI information.
[0245] In addition, a new user experience of AR navigation can be presented through the UX scenario-based embodiment proposed in the present invention.
[0246] The present invention may provide a service platform structure and an AR information display method (UX) that dynamically adjusts the amount of information (POI data, advertisement) to be displayed in AR according to vehicle conditions and advertisement exposure conditions, a module that collects POI information and commerce service information for AR expression and processes them into a form that is easy to render in an AR engine, a module that processes specific POI information to be emphasized according to the situation inside / outside the vehicle, a module that collects vehicle situation information and applies UX policies according to the situation, and an AR engine module that renders AR objects (Group Poi, Mini Poi, 3D Object, Event wall, etc.) according to UX policies.
[0247] In addition, the present invention may provide a client module for transmitting and receiving interaction and data between front and rear seat displays of a vehicle, a Service App module for exposing commerce service information linked to POIs, a client module for collecting user actions regarding advertisements such as AR ad object exposure results and clicks, and a cloud module for collecting and analyzing user actions regarding advertisements such as AR ad object exposure results and clicks.
[0248] Referring to FIG. 8a, the AR service platform of the present invention may include a server (850) which is an off-board configuration located outside the vehicle and an AR service device (800) which is an on-board configuration provided in the vehicle.
[0249] First, the server (850) may include a POI Data Aggregator (851), an Ads manager (852), an Ads Monitoring (853), a Service & Ads Manager (854), a Commerce Manager (855), a DB Connector (856), and a Dashboard (857).
[0250] The POI Data Aggregator (851) can receive information required for AR services from multiple external servers and convert / integrate it into the message format of the AR service platform.
[0251] The Ads manager (852) can manage ad information / content and ad campaigns (ad exposure conditions).
[0252] The Ads Monitoring unit (853) can collect and store ad impressions and click results.
[0253] The Service & Ads Manager (854) can insert advertising information that matches the exposure conditions into the service information and provide it to the client.
[0254] The Commerce Manager (855) can collect commerce service integration / payment information.
[0255] The database connector (DB Connector) (856) can store / query advertising content, advertising impression result information, and commerce payment information.
[0256] The dashboard (857) can display the real-time AR service status visualizing the results of ad impressions and payment history.
[0257] Additionally, the server (850) may further include an AR service cloud API (or a data conversion unit) for converting information transmitted from the AR service device (800) of the vehicle into a data format available to the server, and for converting information processed / generated by the server into a data format available to the AR service device (800).
[0258] Meanwhile, the AR service device (800) may include a client (810) including a cloud interface, a commerce app, a CID-RSE interaction manager, a policy manager, an ad monitoring, a driving context, a personalized recommendation, etc., and an AR engine (820) including a POI renderer, a display manager, a touch manager, etc.
[0259] The client (810) can receive POI information, advertisements, etc. from the server.
[0260] Additionally, the client (810) can send and receive order / payment information with the server (850) and send the ad display results to the server (850).
[0261] The AR engine (820) can transmit data such as the number of times an AR object output in AR is touched and the number of times it is exposed to the client (810).
[0262] Additionally, the AR engine (820) can transmit and receive front / rear seat (CID, RSE) linkage data with the client (810) and can output an AR object according to the AR display policy received from the client (810).
[0263] Additionally, the AR engine (820) can determine the type of AR object, output location, POI type, output size, etc., provided through the AR service based on data collected from a gyroscope sensor, camera, communication unit, navigation, ADAS (Advanced Driver Assistance Systems), GPS, etc., equipped in the vehicle.
[0264] The AR service device (800) equipped in the vehicle (on-board) can render service content in AR to display data received from a cloud server as AR on the front camera image.
[0265] Additionally, the AR service device (800) can mediate data transmission between the server and the AR engine, such as collecting ad posting result data from the AR engine and transmitting it to the server.
[0266] In addition, the AR service device (800) can link data generated via AR between CID-RSE (i.e., front / rear seats).
[0267] Additionally, the AR service device (800) can perform data management for the AR display policy, and specifically, can provide AR display policy data according to the driving situation to the AR engine.
[0268] Additionally, the AR service device (800) can provide situation awareness and personalized services, and specifically, can provide AR objects to the AR engine according to driving conditions (speed, TBT (Turn-by-Turn), etc.) using in-vehicle data.
[0269] In this specification, an example of providing an AR service is described by overlaying AR information (or AR objects, AR content, POI information, etc.) onto an image captured (received, processed) through a camera equipped in a vehicle.
[0270] However, not limited thereto, the AR service described in this specification can be applied in the same or similar manner to various methods of implementing augmented reality, such as when AR information is displayed directly on the windshield of a vehicle as an overlay on real-world space based on the driver or passenger, or when AR information is displayed through a HUD (Head-up Display).
[0271] The input data (input information) used to provide AR services and the output data (output information) provided through the AR service platform are as follows.
[0272] First, the types of input data may include map data (navigation information), service content information (POI, advertisements, etc.), dynamic information, vehicle sensor information, historical information, and driving-related information.
[0273] Map data (navigation information) may include route information to a destination (navigation route), guidance information (Turn-by-Turn), road shape ahead (Road / Lane), multiple map attribute information (road type / attribute, width of road and lane, curvature, gradient, speed limit, etc.), Localization Object information (Road marking, Traffic sign, etc.).
[0274] Service content information (POI, advertisements, etc.) may include POI information received from multiple service providers, advertisement information available at the current location, and real-time information for reservation / payment services such as gas stations, charging stations, and parking lots.
[0275] Dynamic information may include traffic information (road-level traffic, lane-level traffic), event information (accidents, hazard warnings, etc.), weather information, V2X (V2V, V2I) (Vehicle to Everything, Vehicle to Vehicle, Vehicle to Infrastructure), etc.
[0276] Vehicle sensor information may include current location information (GPS / DR), camera input information (ADAS information, object recognition information), and V2X (real-time surrounding situation information collectible through V2V and V2I).
[0277] Historical information may include information on past driving routes, traffic history (e.g., traffic volume by time of day), and communication speeds according to zone and time.
[0278] Driving-related information may include the driving mode (manual, autonomous driving, semi-autonomous driving, ADAS function activation status, etc.), whether the vehicle has entered a destination or waypoint, and whether it has entered a parking lot.
[0279] Output information that can be provided through the AR service platform may include current location / path-based AR service display data.
[0280] Current location / route-based AR service display data may include points where AR advertisements can be displayed along the route (AR Wall, POI building highlights), selectable AR building information (selectable major building information such as landmarks), general POI information (POI summary information such as icons or speech bubbles), distant POI information (distance / direction display of important POI information that is not on the route but is helpful for driving), display information output when multiple POIs exist in the same building, destination building and real-time parking lot status information, real-time status information of gas stations / charging stations, and location-based advertisement / event information.
[0281] The AR service platform of the present invention can filter AR service information based on real-time information and determine a display method.
[0282] Specifically, the AR service platform can determine the number of real-time POI exposures based on driving speed, POI overlap removal, size adjustment, exposure time, etc.
[0283] In addition, the AR service platform can determine the method of displaying POIs based on the recognition of risk information, and specifically, can dynamically change the method of displaying POIs according to situations such as accidents, construction, or the recognition of multiple moving objects.
[0284] In addition, the AR service platform can dynamically change the POI display location when AR display visibility is degraded due to traffic.
[0285] In addition, the AR service platform can reconfigure AR display data for the front and rear seats; for example, by considering driving speed, hazard information, weather information, etc., it can minimize AR service information on the front seat display and reconfigure the rear seat display to show all displayable information.
[0286] The operation / function / control method of such an AR service platform may be implemented by a server or AR service device included in the AR service platform, or by the organic interaction between the server and the AR service device.
[0287] Referring to FIG. 8a, the configuration of the server (850) of the AR service platform is described in more detail as follows.
[0288] The service and advertising manager (854) can perform client request functions, POI information and advertising information integration (data processing & aggregation) functions, and client response functions.
[0289] Specifically, the client request function may include requesting / receiving POI information (location, category) via the Unified API, or requesting / receiving destination entrance location information via the Unified API (selecting one of destination coordinates / address / id).
[0290] Here, Unified API refers to an API defined in the AR service cloud that is independent of a specific data provider (to minimize client variation).
[0291] The POI information and advertising information integration (data processing & aggregation) function may include integrating POI information and advertising information within a radius of 000m of a location requested by a client (from data manager, ads manager) or integrating the entrance location of a destination requested by a client and POI advertising information (from data manager, ads manager).
[0292] Specifically, the POI information and advertising information integration function may include matching advertising information, including building wall and event wall information, with POI information, or, in cases where there are multiple POIs in the same building, a function to set priorities and filter on the server (e.g., keeping only partner companies and excluding other POI information).
[0293] Here, the filtering criteria may include assigning a priority score to each POI and comparing them.
[0294] The client response function may include transmitting POI information and advertising information via the Unified API, or transmitting destination entry locations and advertising information via the Unified API.
[0295] The data manager (not shown) included in the server (850) may include a POI information collection / transmission function, a building shape (polygon) information collection / transmission function, and a destination entrance information collection / transmission function.
[0296] The POI information collection / transmission function can perform the function of requesting POI information from a 3rd party API or transmitting POI information received from a 3rd party API (converted into a Unified API response format) to the Service & Ads Aggregator.
[0297] The building shape (polygon) information collection / transmission function can request building outer shape information from a 3rd party API / data set, or transmit POI information received from a 3rd party API (converted into Unified API response format) to the Service & Ads Aggregator.
[0298] The destination entry information collection / transmission function can perform the function of requesting destination entry information from a 3rd party API or transmitting destination entry information received from a 3rd party API (converted into a Unified API response format) to the Service & Ads Aggregator.
[0299] The Ads Manager (852) can provide a partner (advertising) company management interface, an ad campaign management interface, and an ad content management interface.
[0300] The Ads Monitoring unit (853) can perform the functions of receiving feedback on measuring the effectiveness of advertisements and delivering advertisement information.
[0301] The partner (advertising) vendor management interface allows for POI advertiser management (adding / modifying / deleting advertiser information) and general advertiser management (adding / deleting advertiser information).
[0302] POI-supported ad formats may include Brand POI pins, Building walls, 3D renderings, Event walls, etc., and the supported ad format for brand advertisements unrelated to actual POIs / locations (e.g., Coca-Cola advertisements, etc.) (general supported ad formats) may be Event walls.
[0303] The ad campaign management interface allows you to add, modify, or delete ad campaigns (ad location, type, time, etc.).
[0304] The ad content management interface can add / modify / view / delete content (POI brand icon images, building wall images, event wall images / videos, 3D rendering images) by ad format.
[0305] The function for receiving advertising effectiveness measurement feedback may include the function of receiving ad impression feedback sent by the client and transmitting it to the DB Manager (CPC / CMP / CPT&P).
[0306] The ad information delivery function may include a function to retrieve and deliver ad campaign information to be displayed within a radius of 000m of the location requested by the Service & Ads Aggregator (in the case of CPT&P, only ads that meet the time conditions are delivered).
[0307] The Commerce Manager (855) can perform client integration, external commerce service integration, and payment information management functions.
[0308] The client integration function can include receiving requests by integrating with the Unified API, converting the request content received via the Unified API into the external commerce API specification, converting data received from the external API into the Unified API's message format, and delivering data to the client.
[0309] The Commerce Manager can convert the request content received via the Unified API into an external commerce API specification and perform external service integration functions using the converted content.
[0310] Converting data received from an external API into the Unified API message format may refer to the task of converting data received from external service integration into the Unified API.
[0311] The external commerce service integration function may include requesting a list of stores adjacent to the current location and receiving meta-information results, requesting detailed information about a specific store from the list and receiving results, requesting reservation / orders and receiving results, requesting service usage status and receiving results, and integrating member information of the commerce service and receiving results.
[0312] Here, the service usage status request & result reception can be used for sequence management and AR message pop-ups based on the service usage status (reservation completed / parking lot entry / parked / parking lot exit / reservation canceled).
[0313] The service's member information linkage & result reception can be used for linking information between commerce service members ↔ AR service members (OEM Connectis service members).
[0314] The payment information management function may include the collection of payment details (details, amounts) from external commerce services and the function of charging fees to external commerce companies based on the payment details.
[0315] The database connector (DB Connector) (856) can perform advertising effect measurement data management functions, commerce data management functions, advertiser data management functions, advertising content data management functions and advertising location data management functions.
[0316] The advertising effectiveness measurement data management function allows you to save / delete log data related to CPC / CPM / CPT&P, and view data (by POI, brand, time, and ad type).
[0317] The commerce data management function allows you to save / delete payment history (content, amount) from external commerce services and retrieve data (by POI, brand, time, and ad type).
[0318] The advertiser data management function allows you to save, modify, delete, and view advertiser information and ad campaign settings for each advertiser.
[0319] The advertising content data management function allows you to save, modify, delete, and view advertising content by linking with advertiser information.
[0320] The ad location data management function allows for the management of Event wall section coordinates and Building wall coordinates (by brand) for displaying AR ads, and can be categorized into coordinates directly registered by users and specific coordinates obtained through integration with company APIs.
[0321] The Service Dashboard (857) can perform advertising effectiveness measurement data visualization and commerce service data visualization functions.
[0322] The data visualization function for measuring advertising effectiveness can provide CPC: charts on total ad clicks by company / brand (viewable by period), CPC: aggregate charts on total ad clicks (viewable by period), CPM: charts on total ad impressions by company / brand (viewable by period), CPM: aggregate charts on total ad impressions (viewable by period), CPT&P: charts on ad clicks by company / brand (viewable by period), and CPT&P: charts on ad impressions by company / brand (viewable by period).
[0323] These charts can be provided in various forms, such as bar graphs, line graphs, pie charts, word graphs, and geospatial graphs.
[0324] Although CPT&P is billed per hour rather than per click or impression, it can be used as data to measure the effectiveness of exposure.
[0325] The commerce service data visualization function can provide charts for cumulative payment amounts by company (viewable by period) and charts for total cumulative payment amounts (viewable by period).
[0326] FIG. 8b is a conceptual diagram illustrating an MR service platform for providing the MR service of the present invention.
[0327] The present invention can provide an MR service platform capable of providing a Mixed Reality Automotive Meta Service (MR AMS) (hereinafter referred to as MR service).
[0328] The MR service platform may be named an MR service system, MR navigation system, MR platform, MR system, etc.
[0329] An MR service platform refers to a platform capable of providing services based on mixed reality and may include various independent components.
[0330] For example, the MR service platform may include an MR service device (900) (or referred to as an MR navigation engine) equipped on the vehicle side (onboard), an MR AMS (hereinafter referred to as an MR AMS server) (1100) equipped on the server side (or cloud side) (offboard) outside the vehicle, and a DTaaS (Digital Twin as a Service) server (1200).
[0331] The above MR service device (900) may include an MR AMS client (910) and an MR renderer (920).
[0332] The MR service described in this specification can be understood as a mixed reality navigation service for vehicles. That is, the MR service platform of the present invention can provide a vehicle interface implemented in mixed reality to a user riding in a vehicle.
[0333] The MR service provided by the MR service platform of the present invention can provide an experience of the digital world through a display inside the vehicle while driving in the real world.
[0334] Specifically, MR services can provide drivers with interactive user experiences such as navigation, safe driving, POI (Point of Interest), and entertainment in a virtual 3D space that fuses real-world information with the digital world.
[0335] Through this, the MR service platform of the present invention can provide various UX (User Experience) free from spatiotemporal constraints compared to existing camera-based (or HUD (Head-up Display) based) AR (Augmented Reality).
[0336] Here, the digital world refers to a digital twin or a digitally twinned smart city, and real-world information may include infrastructure data such as V2X (Vehicle to Everything) and C-ITS (Cooperative-Intelligent Transport Systems) and / or surrounding perception data sensed by sensors equipped in autonomous vehicles.
[0337] In addition, the fusion described above may include the concept of fusing vehicle and infrastructure sensor data as well as the MR service cloud (1000) (or MR AMS cloud) and MR service device (900) for implementing the MR service platform.
[0338] In addition, interactive should be understood as a concept that includes not only mixed reality navigation but also high-quality 3D image rendering and user interaction.
[0339] Meanwhile, the Mixed Reality (MR) described in this specification may refer to an environment in which virtual reality is integrated with the real world so that physical objects in reality and virtual objects can interact.
[0340] Mixed Reality (MR) can encompass the meanings of Augmented Reality (AR), which adds virtual information based on reality, and Augmented Virtuality (AV), which adds real-world information to a virtual environment.
[0341] In other words, by providing a smart environment where reality and the virtual are seamlessly connected, users can enjoy rich experiences. For instance, users can interact with a virtual pet placed in their palm or play games in a virtual environment built inside a real-world room. They can also virtually rearrange household furniture or create an environment where people located remotely can gather and work together.
[0342] A mixed reality vehicle meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention can help the user prepare for road shapes and actions in advance by expressing a future driving route as a preview.
[0343] In addition, a mixed reality automotive meta service (MR AMS) using mixed reality (MR) according to one embodiment of the present invention may improve advertising effects or induce an increase in service usage rates by focusing on a specific Point of Interest (POI).
[0344] In addition, the Mixed Reality Automotive Meta Service (MR AMS) using Mixed Reality (MR) according to one embodiment of the present invention is not dependent on a specific map provider and can fuse data from various map providers.
[0345] The present invention can provide an MR navigation function as one of the mixed reality vehicle meta services.
[0346] In the MR navigation function, since augmented reality objects are implemented in a virtual world rather than overlapping them on the real world, difficulties such as forward obscuration issues and achieving matching quality that occur in AR navigation can be resolved.
[0347] Accordingly, the present invention can improve user UX by effectively expressing various contexts that were difficult to express in existing navigation through the MR navigation function.
[0348] To this end, the MR service platform of the present invention may provide a method for managing MR context, a method and system for acquiring metadata and 3D assets for providing MR AMS.
[0349] The MR service platform of the present invention can render related service metadata and 3D assets into MR navigation by defining the MR Context required in the digital world and modeling service metadata for acquiring it.
[0350] Accordingly, the present invention provides an experience of the digital world through a display inside the vehicle while driving in the real world, and can recommend and provide various additional HMI (Human Machine Interface) services to the user by utilizing the enhanced visibility and attention of MR navigation.
[0351] Hereinafter, an MR service platform according to an embodiment of the present invention for providing the previously described MR service will be described.
[0352] Referring to FIG. 8b, the MR service platform (or MR service system) of the present invention may include an MR service cloud (1000) provided outside the vehicle and an MR service device (900) provided in the vehicle.
[0353] The MR service cloud (1100) provided offboard of the vehicle may include at least one of an MR AMS (Mixed Reality Automotive Meta Service) server (1100) and a DTaaS (Digital Twin as a Service) server (1200).
[0354] The MR service device (900) onboard the vehicle may include an MR AMS client (910) and an MR renderer (920).
[0355] The above MR service device (900) can provide a navigation function (or navigation application) by interacting with the AR service device (800) and the navigation system (770).
[0356] Information required for the above navigation function can be received through user input (or user request) input through a camera (310), a sensing unit (120), and a user input unit (210) equipped in the vehicle.
[0357] In addition, information required for the above navigation function can be received through an MR service cloud (1000) provided offboard of the vehicle, and information sensed or processed in the vehicle can be transmitted to the MR service cloud (1000) to receive information required in each vehicle.
[0358] The MR AMS (Mixed Reality Automotive Meta Service) server (1100) can be connected to various service providers (1300a, 1300b, 1300c) that provide online map services, such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP, as illustrated in FIG. 8d. Based on the result of aggregating map data provided by the connected service providers (1300a, 1300b, 1300c), shape information of each building included in the map (e.g., footprint coordinate information and height information of the building) can be aggregated, and the aggregated information can be provided to the DTaaS server (1200). Here, the DTaaS server (1200) may refer to a server or device that provides DTaaS, that is, a service using a digital twin map.
[0359] The above DTaaS may mean Digital Twin as a Service or Digital Transformation as a Service.
[0360] Meanwhile, the DTaaS server (1200) can be connected to a POI database in which POI service data for each building or each area included in the map data is stored. It can also be connected to a 3D model database in which data of a 3D polygon model (or 3D polygon map) for each building included in the map data is stored. Here, the 3D polygon model is a polygon model capable of providing a building volume, and may be a polygon model in which no texture is formed on the surface. The DTaaS server (1200) can receive service data related to POI from the connected POI database and can receive data of 3D polygon models of each building included in the map data of one area from the connected 3D model database.
[0361] The processor (not shown) of the MR service device (900) can receive various information related to the driving of the vehicle from the object detection device (300), the sensing unit (120), and the navigation system (770). For example, the processor can receive information about an object detected in front, behind, or to the side of the vehicle from the camera (310) of the object detection device (300).
[0362] Additionally, the processor may receive information related to the vehicle's speed, driving direction, current vehicle location (GPS), etc. from a sensing unit (120) which includes sensors connected to each component of the vehicle, including the driving system (710). It may also receive information related to the vehicle's driving path from a navigation system (770).
[0363] Meanwhile, the above-mentioned MR service device (900) and DTaaS server (1200) can be connected through the interface APIs of the MR AMS. Here, the connection between the above-mentioned MR service device (900) and the interface APIs of the MR AMS can be established through a wireless network connection. In this case, the above-mentioned MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the above-mentioned MR service device (900).
[0364] In this way, when connected to the MR AMS server (1100), the MR service device (900) can provide at least some of the information received from the connected components (e.g., the vehicle's camera (310), the sensing unit (120), or user input received through the user input unit (210)) to the MR AMS server (1100) via a network connection. Then, the MR AMS server (1100) can provide three-dimensional map data for providing mixed reality to the MR service device (900) in response to the provided information.
[0365] For example, the MR service device (900) can transmit information about objects detected around the vehicle, as well as information about the vehicle's speed, direction, and current vehicle location. It can also provide information about the driving path to the MR AMS server (1100). Then, based on the information received from the MR service device (900), the MR AMS server (1100) can provide 3D map data of a specific area according to the current vehicle location to the MR service device (900).
[0366] In this case, the MR AMS server (1100) may determine POI information based on the current location of the vehicle, the speed of the vehicle, and the driving path of the vehicle, and may provide 3D map data that further includes the determined POI information to the 3D building map. Additionally, the MR AMS server (1100) may provide 3D map data that further includes information about the situation around the vehicle based on the information about objects around the vehicle provided to the MR service device (900).
[0367] Meanwhile, the MR service device (900) can render a mixed reality image based on 3D map data provided by the MR AMS server (1100). For example, the MR service device (900) can control an MR renderer (920) to display a 3D map screen including models of buildings around a vehicle based on the provided 3D map data. Additionally, the MR renderer (920) can display a graphic object corresponding to the vehicle on the 3D map screen, and can display graphic objects corresponding to the provided POI data and situational information around the vehicle on the 3D map screen.
[0368] Accordingly, a virtual environment image (in other words, a mixed reality (MR) image, or an MR navigation screen, or an MR navigation interface) containing a 3D building model similar to the current vehicle and the building shape around the vehicle and a graphic object corresponding to the vehicle can be output to a display (251) equipped in the vehicle, for example, a Center Information Display (CID), a Head Up Display (HUD), a Rear Sheet Information (RSI), or a Rear Sheet Entertainment (RSE).
[0369] In this case, information related to the driving of the vehicle and the situation around the vehicle can be provided to the driver through the virtual environment. Through the three-dimensional map data, that is, the Digital Twin Map (hereinafter DT Map), the MR service device (900) according to an embodiment of the present invention can provide a mixed reality service to the driver.
[0370] Meanwhile, the MR AMS server (1100) can determine three-dimensional map data and POI information or situational information around each vehicle that can be provided together with the three-dimensional map data, based on information collected from not only the MR service device (900) equipped in one vehicle but also from the MR service device (900) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect from multiple vehicles in the form of a cloud server and generate three-dimensional map data for mixed reality based on the collected information. It can also be implemented to provide mixed reality services to at least one MR service device (900) equipped in different vehicles based on the generated three-dimensional map data.
[0371] Therefore, for the convenience of explanation, the cloud or server that includes the above MR AMS server (1100) and the above DTaaS server (1200) and provides information such as metadata (e.g., service metadata, 3D assets), 3D polygon maps, and digital twin maps for providing mixed reality services, and provides 3D map data, i.e., digital twin maps (DT maps), will be referred to as the MR service cloud (1000).
[0372] As illustrated in FIG. 8b, the MR service device (900) (or MR navigation engine) may include an MR AMS client (910) and an MR renderer (920).
[0373] In addition, to implement an MR navigation function, which is one of the MR services, the MR service device (900) can transmit and receive data with an AR service device (800) (or AR engine) and a navigation system (770) equipped in a vehicle.
[0374] The MR AMS client (910) may include a context manager (911), a scene manager (913), and a UX scenario database (914).
[0375] Additionally, the MR renderer (920) may include a DTaaS client (921), an MR visualization unit (922), and a 3D HMI framework (923).
[0376] The MR AMS client (910) can collect vehicle location information, user input, user feedback information, payment information, etc., and transmit them to an MR AMS server (1100) located outside the vehicle.
[0377] The MR AMS server (1100) can transmit at least one of metadata, service metadata, and 3D assets required to provide MR services to the MR AMS client (910) based on information received from the MR AMS client.
[0378] The MR AMS client (910) can transmit data received from the MR AMS server (910) to the MR renderer (920).
[0379] The MR renderer (920) can generate a digital twin map using a 3D polygon map received from the DTaaS server (1200) and an image received from the MR AMS client (910) or the camera (310) equipped in the vehicle.
[0380] Additionally, the MR renderer (920) can render data received from the MR AMS client (920) into an MR object that can be overlapped on a digital twin map, and generate a mixed reality (MR) image by overlapping the rendered MR object onto the digital twin map.
[0381] Afterwards, the MR renderer (920) can output the generated mixed reality image to a display (251) equipped in the vehicle.
[0382] All components described in this specification may be implemented as separate hardware modules and, if necessary, understood as components implemented in block units in software.
[0383] Below, each component constituting the MR service platform will be described in more detail with reference to the attached drawings.
[0384] FIG. 8c is a conceptual diagram illustrating the MR AMS client of the present invention.
[0385] The MR AMS client (910) is equipped in a vehicle and can provide a Mixed Reality Automotive Meta Service (MR AMS).
[0386] The MR AMS client (910) may include a context manager (911) that requests a context corresponding to a user request (or user input) from an MR AMS server (1100) located outside the vehicle, a scene manager (913) that manages MR scene information provided to a display (251) located in the vehicle, and a UX scenario database (914) that provides a UX rule to at least one of the context manager (911) and the scene manager (913).
[0387] Additionally, the MR AMS client (910) may further include an interface API (912) that calls a function to communicate with the MR AMS server (1100) located outside the vehicle.
[0388] The above interface API (912) may be composed of one or more functions configured to communicate with the MR AMS server (1100), and using these functions, data or message formats may be converted to transmit data to the MR AMS server (1100), or the format of data received from the MR AMS server (1100) may be converted.
[0389] The interface API (921) can transmit a context request output from the context manager (911) to the MR AMS server (1100) and receive a 3D asset corresponding to the requested context from the MR AMS server (912).
[0390] Here, the above context may refer to situational information and may refer to information corresponding to the situation faced by the vehicle. Additionally, the above context may include the meaning of content.
[0391] The above 3D asset may refer to 3D object data corresponding to the requested context. Additionally, the above 3D asset may refer to a 3D graphic object that can be overlapped or newly updated on a digital twin image (or digital twin map).
[0392] The MR AMS client (910) may be included in the MR service device (900).
[0393] The MR service device (900) may include a User Interaction Handler (901) that, when a user input is received through an input unit (210) provided in the vehicle, generates an action corresponding to the user input and transmits it to the context manager.
[0394] The above user interaction handler (901) may be included in the MR service device (900) or in the MR AMS client (910).
[0395] For example, when a user interaction handler (901) receives a user input "Find nearby Starbucks" through the vehicle's input unit (210), it can generate an action corresponding to the user input (e.g., "Search POI") and transmit it to a context manager (911) equipped in an MR AMS client (910).
[0396] For example, the above action may be determined by an action that matches a word included in the user input, and the above action may be named a command or a control command.
[0397] The above context manager (911) can generate a command to request a context corresponding to an action received from a user interaction handler (901) and transmit the command to an MR AMS server (1100) through an interface API (912).
[0398] The above command may be generated based on an action (e.g., "Search POI") received from the user interaction handler (901), and may be formed to include, for example, the current vehicle location, the type of POI to be found, and radius information (e.g., GET "Starbucks" (type of POI) WITHIN "500m" (radius) FROM "37.7795, -122.4201" (current vehicle location (latitude, longitude)).
[0399] The context manager (911) can receive current scene information currently being output from the vehicle from the scene manager (913) and receive UX rules from the UX scenario database (914).
[0400] Additionally, the context manager (911) can receive navigation information including the current path and current location from a navigation handler (902) that handles information of the navigation system (770).
[0401] The navigation handler (902) may be provided in the MR service device (900) or in the MR AMS client (910).
[0402] The context manager (911) can generate a command to request the context based on at least one of the current scene information, the UX rule, and the navigation information.
[0403] The current scene information may include screen information currently being displayed on the vehicle's display (251). For example, the current scene information may include information regarding a mixed reality image in which an MR object and an MR interface are overlaid on a digital twin map.
[0404] In addition, at least one of the context manager (911) and scene manager (913) of the present invention can receive sensor data processed through a sensor data adapter (903) that processes information sensed through the sensing unit (120) of the vehicle.
[0405] The sensor data adapter (903) may be provided in an MR service device (900) or in an MR AMS client (910). The sensor data adapter (903) may also transmit the processed sensor data to an AR engine handler (904) that handles the data transmitted to the AR engine (or AR service device) (800).
[0406] The interface API (912) can receive metadata of the context corresponding to the command and / or a three-dimensional asset corresponding to the context from the MR AMS server (1100).
[0407] Afterward, the interface API (912) can transmit the received metadata and / or 3D assets to the scene manager (913).
[0408] The scene manager (913) can generate UI data using UX rules received from the UX scenario database (914), metadata received from the interface API (912), and 3D assets.
[0409] Afterwards, the scene manager (913) can transmit the generated UI data to an MR renderer (920) that renders the data to be output as mixed reality (MR) or mixed reality video on a display (251) equipped in the vehicle.
[0410] Additionally, the scene manager (913) can further transmit the generated UI data to an AR engine handler (904) formed to handle an AR service device (800) equipped in a vehicle.
[0411] The UX rules stored in the UX scenario database (914) may refer to information regarding rules, forms, formats, or templates for creating screens, UX, or user interfaces provided by the MR service device, and these UX rules may be predefined for each type of data.
[0412] In addition, UX rules can be updated or modified by users or administrators.
[0413] FIG. 8d is a conceptual diagram illustrating the MR AMS server of the present invention.
[0414] Referring to FIG. 8d, an MR AMS server (1100) that is provided offboard of a vehicle and provides a Mixed Reality Automotive Meta Service (MR AMS) may include an interface API (1101) that calls a function to communicate with an MR AMS client provided in the vehicle, a Service Aggregation Manager (1110) that requests and receives a context corresponding to a request received from the MR AMS client from a service provider, and a data Integration Manager (1120) that loads a 3D asset corresponding to the received context from a database (3D Assets for MR Navigation Database) (1130).
[0415] The above interface API (1101) may be named as a server interface API (1101) to distinguish it from the interface API (912) of the MR AMS client (910) equipped in the vehicle.
[0416] Additionally, the interface API (912) of the MR AMS client (910) may be named the vehicle interface API or the MR AMS client interface API.
[0417] The interface API (1101) provided in the MR AMS server (1100) can transmit a user request (or context request) received from the MR AMS client to the service aggregation manager (1110).
[0418] The above interface API may include a first interface API (1101) that calls a function to perform communication with the MR AMS client (910) and a second interface API (1102a, 1102b, 1102c) that the service aggregation manager (1110) calls a function to perform communication with the service provider (1300a, 1300b, 1300c).
[0419] The above second interface API (1102a, 1102b, 1102c) can receive service data and / or map data through an interface API provided in a service provider (1300a, 1300b, 1300c).
[0420] The interface API provided in the second interface API (1102a, 1102b, 1102c) and the service provider (1300a, 1300b, 1300c) may include a function formed to perform mutual data transmission and reception and to convert data format or message format, and may transmit and receive mutual data by converting data format or message format using such function.
[0421] The service aggregation manager (1110) can request the requested context from different service providers based on the type of context requested by the MR AMS client (910) equipped in the vehicle.
[0422] Specifically, the service aggregation manager (1110) may request the first type of context from a first service provider (1300a) that provides the first type of context if the requested context is of the first type of context, and may request the second type of context from a second service provider (1300b) that provides the second type of context if the requested context is of the second type different from the first type.
[0423] For example, if the type of requested context is related to a POI (e.g., "Starbucks"), the service aggregation manager (1110) may request and receive context (or POI data) related to the POI from a first service provider (1300a) that provides information related to the POI.
[0424] Additionally, if the type of requested context is a view of a certain street, the service aggregation manager (1110) may request and receive context (or imagery data) for said street view from a second service provider (1300b) that provides information about the street view.
[0425] Additionally, if the type of requested context is a service, the service aggregation manager (1110) may request and receive the context related to the service (or data about the service (e.g., service ratings or prices)) from a third service provider (1300c) that provides information related to the service.
[0426] Additionally, the interface API (1101) can request expanded service APIs (Expand service API calls) from the service aggregation manager (1110) based on the service (or context request) requested by the MR AMS client (910).
[0427] The service aggregation manager (1110) can request and receive information corresponding to the confirmed service from the service provider (1300a, 1300b, 1300c) based on the extended service API request, and use this to create a service API and output it to the data integration manager (1120).
[0428] The data integration manager (1120) can perform data enhancement based on the service API received from the service aggregation manager (1110), generate a metadata package for the requested context, and transmit it to the vehicle's MR AMS client (910) via the interface API (1101).
[0429] The above metadata package may include the previously described 3D asset and service metadata. Here, service metadata may refer to metadata for providing a service corresponding to the requested context.
[0430] Meanwhile, the interface API (1101) can transmit the 3D asset loaded from the data integration manager (1120) to the MR AMS client (910).
[0431] Meanwhile, the MR AMS server (1100) of the present invention may further include the context manager (911) described above.
[0432] That is, the context manager (911) may be included in the MR AMS client (910) and provided at the vehicle end, or included in the MR AMS server (1100) and provided at the server (cloud) end, or provided at both ends.
[0433] The above context manager (911), when provided in the MR AMS server (1100), may be configured to manage a context corresponding to a request received from an MR AMS client (910).
[0434] The above context manager (911) may include a context handler (911a) that handles and parses context requests, a context interpreter (911b) that manages sessions for interpreting context requests and creates a set of contexts using a data model, and a context graph database (Context Graph DB or MR Context DB) (911c) that stores the data model.
[0435] Here, the context handler (911a) can receive a user request input to the MR AMS client through the interface API (1101), parse the received user request, and pass it to the context interpreter (911b).
[0436] The above context interpreter (911b) can generate a query for a context request corresponding to the user request after session creation, and request and receive a context data model corresponding to the query from the context graph database (911c).
[0437] The context interpreter (911b) requests a context corresponding to the context data model from the service aggregation manager (1110), and the service aggregation manager (1110) can request and receive context data corresponding to the context data model from the service providers (1300a, 1300b, 1300c).
[0438] The service aggregation manager (1110) may request and receive a three-dimensional asset (and / or service metadata) corresponding to the requested context from the data integration manager (1120), and transmit the context data received from the service provider and the three-dimensional asset (and / or service metadata) received from the data integration manager to the context interpreter (911b).
[0439] The context interpreter (911b) can transmit the received context data and the 3D asset to the MR AMS client (910) equipped in the vehicle through the context handler (911a) and the interface API (1101).
[0440] Meanwhile, the context manager (911) may further include a context recommender (911d) that extracts a recommended context based on the generated set of contexts, and a context controller (911e) (or Context tracker) that manages the contexts to be acquired periodically.
[0441] The above context recommender (911d) may request the context interpreter (911b) to generate a query to recommend a service that can replace the specific service if the completed context data contains information that a specific service cannot be used.
[0442] FIG. 9 is a conceptual diagram illustrating the DTaaS server of the present invention.
[0443] Referring to FIG. 9, the DTaaS (Digital Twin as a Service, or Digital Transformation as a Service) server (1200) of the present invention is provided outside the vehicle and can provide a Mixed Reality Automotive Meta Service (MR AMS). Specifically, the DTaaS server (1200) can provide a digital twin map or provide data necessary to generate a digital twin map (e.g., a 3D polygon map or all kinds of information regarding objects overlapping on the digital twin).
[0444] The DTaaS server (1200) may include a DTaaS API (1210) that calls a function to communicate with an MR service device (900) equipped in a vehicle, a database (Digital Twins Maps DB) (1220) that stores a digital twin map and a renderable 3D polygon map provided to the MR service device, and a processor (1280) that transmits a 3D polygon map corresponding to the location information to the MR service device through the DTaaS API based on the location information of the vehicle received from the MR service device.
[0445] Additionally, the DTaaS server (1200) may further include a communication unit (TeleCommunication Unit, TCU) (1290) that communicates with an MR AMS server (1100) that is provided outside the vehicle and provides MR AMS services.
[0446] Additionally, the DTaaS server (1200) may further include a Digital Twin Representation and Update Unit (1230) that generates a digital twin map by matching a 3D polygon map stored in the database (1220) with an actual captured image.
[0447] Additionally, the DTaaS server (1200) may further include a Dynamics Modeling DB (1240) that stores dynamic information regarding moving objects received from at least one of the MR service device (900) and the MR AMS server (1100), and a Scenarios DB (1250) that stores information related to scenarios that can be implemented in the digital twin.
[0448] Additionally, the DTaaS server (1200) may further include a simulation unit (1260) that performs a simulation corresponding to a user request on the digital twin and a visualization unit (1270) that visualizes information to be implemented on the digital twin.
[0449] All the components described above can be implemented as independent hardware (e.g., chips or modules), and may also be implemented as software-blocked components as needed.
[0450] The above DTaaS server (1200) can transmit and receive data through the DTaaS API (1210) not only to the vehicle (100) but also to a server (FMS Server) (1280) that provides fleet management services (or vehicle group management services) and a server (1290) that provides city planning services.
[0451] For example, the DTaaS server (1200) can collect log information collected from each server from at least one of the vehicle (100), the FMS server (1280), and the city planning service provider server (1290).
[0452] Afterwards, the DTaaS server (1200) can store the collected log information in a log database.
[0453] The DTaaS server (1200) can provide a digital twin map for visualization in at least one of the vehicle (100), FMS server (1280), and urban planning service provider server (1290) based on collected log information.
[0454] Additionally, the DTaaS server (1200) can transmit at least one of event notification information, simulation information, and visualization information to at least one of a vehicle (100), an FMS server (1280), and an urban planning service provider server (1290) based on the received log information.
[0455] FIG. 10 is a block diagram illustrating the structure of a 3D map view generating device (1300) connected to a cloud server (1350) according to an embodiment of the present invention.
[0456] Referring to FIG. 10, the MR AMS (Mixed Reality Automotive Meta Service) server (1100) described in FIG. 8d can be connected to various service providers (1351) that provide online map services, such as OSM (Open Street Map), Mapbox, HERE, WRLD, and BingMAP. Based on the result of aggregating map data provided by the connected service providers (1351), shape information of each building included in the map—for example, footprint coordinate information and height information of the building—can be aggregated and the aggregated information can be provided to DTaaS (1352). Here, DTaaS (1352) may refer to a DTaaS server (1200), that is, a server or device that provides services using a digital twin map.
[0457] Meanwhile, DTaaS (1352) can be connected to a POI database in which POI service data for each building or each area included in the map data is stored. It can also be connected to a 3D model database in which 2.5D polygon model data for each building included in the map data is stored. Here, the 2.5D polygon model is a polygon model capable of providing building volume, and may be a polygon model in which no texture is formed on the surface. The DTaaS (1352) can receive service data related to POI from the connected POI database and can receive data of 2.5D polygon models of each building included in the map data of one area from the connected 3D model database.
[0458] The processor (1330) of the 3D map view generating device (1300) can receive various information related to the driving of the vehicle from the camera (310), the sensing unit (120), and the navigation system (770). For example, the processor (1330) can receive information about an object detected in front, behind, or to the side of the vehicle from the camera (310). In addition, the processor (1330) can receive information related to the vehicle's speed, driving direction, and current vehicle location (GPS) from the sensing unit (120), which includes sensors connected to each component of the vehicle, including the driving system (710). In addition, it can receive information related to the vehicle's driving path from the navigation system (770).
[0459] Meanwhile, the 3D map view generating device (1300) and DTaaS (1352) can be connected through the interface APIs of the MR AMS server (1100). In this case, the 3D map view generating device (1300) may be configured to correspond to the MR AMS client (910).
[0460] Here, the interface APIs between the 3D map view generating device (1300) and the MR AMS server (1100) can be connected via a wireless network connection. In this case, the MR AMS server (1100) may be a network server or a cloud server that is wirelessly connected to the 3D map view generating device (1300).
[0461] In this way, when connected to the MR AMS server (1100), the 3D map view generating device (1300) can provide at least some of the information received from the connected components to the MR AMS server (1100) via a network connection. Then, in response to the provided information, the MR AMS server (1100) can provide 3D map data for providing mixed reality to the 3D map view generating device (1300).
[0462] The 3D map data provided here is map data including polygon models provided in the 3D model DB, and may include 2.5D map data providing simple polygon models or 3D map data providing three-dimensional virtual objects.
[0463] For example, the 3D map view generating device (1300) can provide the MR AMS server (1100) with information about objects detected around the vehicle, the vehicle's speed, direction, current vehicle location, and the vehicle's driving path. Then, based on the information received from the 3D map view generating device (1300), the MR AMS server (1100) can provide the 3D map view generating device (1300) with 2.5D map data of a specific area corresponding to the current vehicle location.
[0464] In this case, the MR AMS server (1100) may determine POI information based on the current vehicle's location, direction, speed, and driving path, and may provide 3D map data that further includes the determined POI information to the 3D building map. Additionally, the MR AMS server (1100) may provide 3D map data that further includes information about the situation around the vehicle based on the information about objects around the vehicle provided to the 3D map view generating device (1300).
[0465] Meanwhile, the 3D map view generating device (1300) can render a mixed reality image based on 3D map data provided by the MR AMS server (1100). For example, the 3D map view generating device (1300) can control an MR renderer to display a 3D map screen including models of buildings around a vehicle based on the provided 3D map data. Additionally, it can display a graphic object corresponding to the vehicle on the 3D map screen, and display graphic objects corresponding to the provided POI data and situational information around the vehicle on the 3D map screen.
[0466] Accordingly, an image of a virtual environment containing a 3D building model similar to the current vehicle and the shape of the building surrounding the vehicle, and a graphic object corresponding to the vehicle, can be output to a display unit (251) such as a CID (Center Information Display), HUD (Head Up Display), RSI (Rear Sheet Information), or RSE (Rear Sheet Entertainment).
[0467] In this case, information related to the driving of the vehicle and the situation around the vehicle can be provided to the driver through the virtual environment. Through the three-dimensional map data, that is, the Digital Twin Map (hereinafter DT Map), the 3D map view generating device (1300) according to an embodiment of the present invention can provide a mixed reality service to the driver.
[0468] Meanwhile, the MR AMS server (1100) can determine three-dimensional map data and POI information or situational information around each vehicle that can be provided together with the three-dimensional map data, based on information collected from a 3D map view generating device (1300) equipped in one vehicle as well as from a 3D map view generating device (1300) equipped in multiple vehicles. In this case, the MR AMS server (1100) can collect information from multiple vehicles in the form of a cloud server and generate three-dimensional map data for mixed reality based on the collected information. Then, based on the generated three-dimensional map data, it can be implemented to transmit MR information for providing mixed reality services to at least one 3D map view generating device (1300) equipped in different vehicles.
[0469] For convenience of explanation, below, the MR AMS server (1100) that is connected to the above DTaaS (1352) and provides 3D map data, i.e., a digital twin map (DT map), for providing mixed reality services, and the above DTaaS (1352) will be collectively referred to as the cloud server (1350).
[0470] Meanwhile, the 3D map view generating device (1300) may be a device that controls a display provided in a vehicle through an interface unit. Alternatively, the 3D map view generating device (1300) may be a display device provided in a vehicle. In this case, the 3D map view generating device (1300), that is, the 3D map view generating device (1300), may be equipped with a display unit, and may receive an updated DT map with the real-image texturing from a cloud server and directly display an MR view image including the received DT map on the display unit provided in the 3D map view generating device (1300).
[0471] For convenience of explanation, the following description will be referred to as a 3D map view generating device (1300).
[0472] Meanwhile, the vehicle (100) related to the present invention may include a 3D map view generating device (1300).
[0473] The 3D map view generating device (1300) is capable of controlling at least one of the components described in FIG. 7. In this regard, the 3D map view generating device (1300) may be a control unit (170).
[0474] Not limited thereto, the 3D map view generating device (1300) may be a separate configuration independent of the control unit (170). When the 3D map view generating device (1300) is implemented as a component independent of the control unit (170), the 3D map view generating device (1300) may be provided in a part of the vehicle (100).
[0475] Hereinafter, for the convenience of explanation, the 3D map view generating device (1300) will be described as a separate component independent of the control unit (170). The functions (operations) and control methods described for the 3D map view generating device (1300) in this specification can be performed by the control unit (170) of the vehicle. That is, all contents described in relation to the 3D map view generating device (1300) can be applied by analogy to the control unit (170) in the same or similar way.
[0476] In addition, the 3D map view generating device (1300) described in this specification may include a part of the component described in FIG. 7 and various components provided in the vehicle. For convenience of explanation, the component described in FIG. 7 and various components provided in the vehicle will be described by assigning separate names and reference numerals.
[0477] FIG. 11 is a conceptual diagram for explaining the 3D map view generating device (1300).
[0478] A 3D map view generating device (1300) according to one embodiment of the present invention may include a communication unit (1310), an interface unit (1320), a memory (1340), and a processor (1330).
[0479] The above communication unit (1310) may be formed to perform wireless communication with at least one of the electronic components equipped in the vehicle (e.g., the electronic components equipped in the vehicle as seen in FIG. 7).
[0480] In addition, the communication unit (1310) may be configured to communicate with devices other than a vehicle, such as a mobile terminal, a server, another vehicle, or infrastructure provided on a road.
[0481] The above communication unit (1310) may be the communication device (400) described above and may include at least one of the components included in the communication device (400).
[0482] The interface unit (1320) can communicate with at least one of the components provided in the vehicle.
[0483] Specifically, the interface section (1320) may be formed to perform wired communication with at least one of the components provided in the vehicle shown in FIG. 7.
[0484] The interface unit (1320) receives sensing information from one or more sensors provided in the vehicle (100).
[0485] The above interface unit (1320) may be referred to as a sensor data collector.
[0486] The above interface unit (1320) collects (receives) information sensed through sensors provided in the vehicle (e.g., sensors that detect vehicle operation (V.Sensors) (e.g., heading, throttle, brake, wheel, etc.) and sensors that sense surrounding information of the vehicle (S.Sensors) (e.g., Camera, Radar, LiDAR, Sonar, etc.)).
[0487] The above interface unit (1320) can transmit information sensed through a sensor equipped in the vehicle to a communication unit (1310) (or processor (1330)) so that it is reflected in a high-precision map.
[0488] The interface section (1320) can serve as a passage to the electronic components equipped in the vehicle through, for example, the vehicle interface section (130).
[0489] The interface unit (1320) can exchange data with the vehicle interface unit (130).
[0490] The interface section (1320) can serve as a channel connected to the vehicle to receive electrical energy.
[0491] For example, the route guidance device can turn on the power by receiving electrical energy from the vehicle's power supply unit (190) through the interface unit (1320).
[0492] Meanwhile, the present invention may include a memory (1340) that stores data supporting various functions of the 3D map view generating device (1300). For example, the memory (1340) may store a plurality of application programs (or applications) that can be executed by the processor (1330), data for the operation of the 3D map view generating device (1300), and instructions.
[0493] As an example of such data, the memory (1340) may store building reference information, which is information about actual buildings corresponding to each of the building virtual objects included in the map data. The building reference information may store the height of the actual building, planar shape information (e.g., footprint information) which is the shape of the land occupied by the actual building, and information about the actual location of each building, that is, the latitude and longitude of each building. In addition, the building reference information may include information about the shape of each side of each building or various three-dimensional shapes of the building. In addition, the building reference information may include information related to the use, type, age, etc. of each building corresponding to each building virtual object.
[0494] The memory (1340) can store building reference information corresponding to at least one of the building virtual objects included in the map data. Hereinafter, a region on the memory (1340) where the building reference information is stored will be referred to as the building reference information storage unit (1343).
[0495] Additionally, the memory (1340) may store building image setting values as information for modeling the building virtual objects, including setting values for the angle and projection direction for creating the building virtual objects, modeling setting values for a 2.5D building virtual object formed by polygons, or modeling setting values for a 3D building virtual object having a three-dimensional shape. Based on the building reference information and building image setting values stored in the memory (1340), the processor (1330) can calculate various building setting values, such as the height, area, size, area of each side, margin value, and offset value of the building virtual object to be created, and create each building virtual object based on the calculated building setting values. Here, the building setting values may include values related to at least one of the building height, location, planar shape, three-dimensional shape, and shape of each side included in the building reference information.
[0496] Additionally, the memory (1340) can store additional data for creating the surface material, i.e., the surface texture, of each building virtual object. For example, the memory (1340) can store an image of a part of a building that includes the shape of various building elements. The image is an image (hereinafter referred to as a texture image) for forming the surface texture of the building and can be formed as a 2D image. Additionally, the memory (1340) can store data of images (hereinafter referred to as normal maps) corresponding to normal maps that have different vector values according to different materials so as to provide texture to the texture image. Additionally, it can store images (hereinafter referred to as albedo maps) corresponding to albedo maps that have color values unique to the material for each different material.
[0497] Meanwhile, the memory (1340) may store 2D images that include shapes corresponding to the components of a building. The 2D images may be images that include at least one image corresponding to a building element for describing the texture of the building. Additionally, each 2D image may include different building elements depending on the type, and the shapes of the included building elements may also differ. The 2D images may be texture images that can form a surface texture (material) of the building virtual object in a form that is tiled on the surface of the building virtual object.
[0498] Meanwhile, the above texture image can be composited with the outer surface of the building virtual object in a tiling form as described above. Here, tiling may refer to a process of filling the surface area of the building virtual object with tiles normalized to a certain size in the horizontal and vertical directions continuously without overlapping each other. Accordingly, the above texture image may refer to a tile, which is a unit image composited with the outer surface of the building virtual object through tiling. Therefore, in the following description, the texture image that is textured on the outer surface of the building virtual object in a tiling manner may be referred to as a tile. Here, texturing refers to adding a texture representing a surface material to the surface area of the building virtual object, and may refer to the operation process of compositing a texture using tiles on the surface area of the building virtual object.
[0499] Additionally, the surface area of the virtual object may be composed of multiple different sub-regions. In this case, a different tile may be matched to each of the multiple sub-regions, and tiling may be performed on each of the multiple regions using the matching different tiles.
[0500] Meanwhile, the above tiles may be grouped into different groups according to the characteristics of the building to which they can be applied, such as the type, use, or size of the building. For example, the above tiles may be grouped into residential buildings, commercial buildings, or officetel buildings. In this case, the tiles grouped into different groups—namely, the tiles grouped into residential buildings, the tiles grouped into commercial buildings, and the tiles grouped into officetel buildings—may be tiles normalized to different sizes according to each group.
[0501] Furthermore, some of the aforementioned tiles may be associated with at least one other tile. For example, the associated tiles may be tiles that are similar in shape and color. That is, the associated tiles may differ in at least a part of their shape or only in color. Alternatively, the associated tiles may be tiles that are similar in at least one of their shape and color to a certain degree or more. Such associated tiles can reflect changes over time. For example, in the case of tiles that have the same shape but differ only in color, they can be used to reflect changes over time in a specific part of a building.
[0502] Meanwhile, the memory (1340) may contain association information for tiles that are associated with each other in this way. And through this association information, a specific tile can be associated with another tile. Therefore, by using this association information, even if the tiles have different shapes or colors, they can be associated with each other.
[0503] The tiles stored in the memory (1340) may form a database containing each tile and tile information corresponding to each tile. In this case, the tile information stored in the database may include address information of each tile, information of the group to which each tile is included (group information), and association information including information of other tiles associated with the corresponding tile if there are other tiles associated with the tile. Hereinafter, the database containing tiles and tile information corresponding to each tile will be referred to as a tile database (Database, DB) (hereinafter referred to as Tile DB) (1341).
[0504] Meanwhile, the memory (1340) can store map data containing virtual objects. Here, the map data may be formed by connecting map data of multiple different regions in a tile form. In this case, each map tile may represent a logical rectangular section corresponding to a certain area, and may be intended to subdivide an extensive map area. That is, the map data may be formed from data of map tiles corresponding to different regions.
[0505] Meanwhile, the above map data may be map data including data of a so-called 2.5D polygon model. Additionally, the above map data may include HD (High-definition) map data having high accuracy. The above map data may be linked to a navigation system (770) and a driving system (700) equipped in a vehicle (100), and the memory (1340) may provide the above map data to various linked systems and components under the control of the processor (1330). Such map data may be stored in a map data storage unit (1342).
[0506] Meanwhile, a 3D map view generating device (1300) according to an embodiment of the present invention may include a processor (1330) that generates a digital twin 3D map using at least one of an image captured from a camera (310) provided in a vehicle (100), 2D map data (e.g., HD map data), and 3D map data (e.g., map data including data of a 2.5D polygon model). The processor (1330) controls each connected component and can control the overall operation of the 3D map view generating device (1300).
[0507] In addition, the processor (1330) can overlap (or superimpose, output) graphic objects related to path guidance on a digital twin 3D map.
[0508] Here, the graphic object related to route guidance refers to an object output as mixed reality (MR) and may include various types of objects necessary for performing route guidance (e.g., POI objects, carpet-shaped objects, 3D objects, etc.). Here, the graphic object related to route guidance may also be named as an MR object.
[0509] When the destination of the vehicle (100) is set, the processor (1330) can determine the route that the vehicle (100) will travel through the navigation system (770). And when the driving route of the vehicle (100) is determined, the processor (1330) can determine at least one virtual object (e.g., an object output to the mixed reality (MR)) to perform texturing on a digital twin 3D map for route guidance.
[0510] Meanwhile, to determine virtual objects to be displayed on the digital twin 3D map, the processor (1330) may first determine a map area to create, that is, to model, the virtual objects. To do this, the processor (1330) may first detect at least one map tile around a map tile corresponding to the area where the vehicle (100) is located, according to the current location of the vehicle (100) and the driving direction of the vehicle (100), from map data stored in memory (1340). For example, the processor (1330) may determine a map area corresponding to a total of 25 map tile data, arranged in a row and column of 5 around the map tile where the vehicle (100) is located, according to the zoom level of a designated map, and detect map tile data corresponding to the determined map area from the map data.
[0511] And the processor (1330) can detect buildings included in the detected map area. And it can detect building reference information corresponding to the detected buildings from the building reference information storage unit (1343). And it can determine the setting values of each building image associated with the detected building reference information. Here, the building image setting values are information for modeling each building virtual object, and may include setting values for the projection direction according to the angle or distance for creating the building virtual objects according to the driving direction of the vehicle (100) and / or the angle at which the MR view is displayed, setting values for modeling according to 2.5D virtual objects or setting values for modeling according to 3D virtual objects, for example, vector values such as the coordinate values of vertices or normals.
[0512] When the building image setting value is determined, the processor (1330) can calculate building setting values corresponding to each building included within the map area based on the determined building image setting value and the building reference information of each building. Here, the building setting value may include the height of the building virtual object to be created, the side area of each side, the plan area, etc. And when the calculation of the building setting values is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included within the map area based on the building setting value.
[0513] And when the modeling of virtual objects included in the map area is completed, the processor (1330) can determine which virtual objects among the modeled virtual objects will be displayed on the display unit (251) through rendering. To this end, the processor (1330) can determine a frustum-shaped visible area formed according to the driving direction of the vehicle (100) based on the current position of the vehicle (100), determine the building virtual objects included in the frustum-shaped visible area as objects to be rendered, and perform texturing on the determined building virtual objects.
[0514] Meanwhile, the processor (1330) may pre-model building virtual objects corresponding to each of the buildings included in the map area, and render only at least some of the modeled building virtual objects according to the visible area of the frustum shape determined by the position and driving direction of the vehicle (100) and display them on the display unit. In this case, the texturing may be performed during the rendering process. Therefore, by pre-modeling the objects to be rendered and selectively rendering only some of the modeled objects as needed, texturing of virtual objects that are not displayed on the display unit can be prevented. Thus, the amount of computation associated with texturing can be reduced, and the time for virtual objects to be displayed on the display unit can be significantly reduced.
[0515] Furthermore, the processor (1330) may be implemented as a plurality of processors. For example, the processor (1330) may be implemented as a processor that models the building virtual objects and a processor that renders some of the modeled virtual objects, each being a different processor. As an example, the processor that models the virtual objects may be a Central Processing Unit that interprets various program instructions of the 3D map view generating device (1300) according to an embodiment of the present invention and performs data calculations and processing. On the other hand, the processor that renders some of the pre-modeled virtual objects may be a Graphic Processing Unit (GPU) that processes graphic calculations in the 3D map view generating device (1300) according to an embodiment of the present invention and outputs a result value. In this case, since the modeling of virtual objects and the rendering of the modeled virtual objects can be processed in parallel on different processors, the computational load applied to the processor can be significantly reduced compared to the case where all processes are performed on a single processor, and by performing rendering from pre-modeled objects to display virtual objects, the time required until the virtual objects are displayed on the display unit can also be significantly shortened. Furthermore, the 3D map view generation device (1300) according to the embodiment of the present invention described above can pre-generate models of building virtual objects corresponding to each building included in the map area, and by rendering only some of the virtual objects according to a part of the generated building virtual objects (e.g., the visible area of the frustum shape), the time required to display the building virtual objects can be further shortened, and accordingly, real-time provision of an MR view according to the driving of the vehicle (100) can be made possible.
[0516] As such, the present invention has a configuration for pre-generating modeling of building virtual objects corresponding to each building included in the map area. Therefore, when the map area needs to be updated according to the position and driving direction of the vehicle (100), the processor (1330) can detect at least one map tile data by adding it from the map data through the update. Additionally, virtual objects of buildings included in the area of the additionally detected map tile can be further modeled. On the other hand, for efficient memory utilization, the processor (1330) can delete virtual objects of buildings included in the area of the map tile that has deviated from the map area according to the update of the map area.
[0517] Meanwhile, when the destination or path to the destination of the vehicle (100) is determined, the processor (1330) may additionally detect at least one map tile data from the map data based on the determined destination or path to the destination. In this case, the at least one additionally detected map tile data may be map tile data connected to each other. And when the processor (1330) additionally detects at least one additional map tile data, it may model virtual objects corresponding to each of the buildings included in the additionally detected map tile data.
[0518] In this case, virtual objects modeled from the additionally detected map tile data can be rendered and displayed on the display unit (251) regardless of the driving information of the vehicle (100) when a preset condition is satisfied. For example, the preset condition may be satisfied when a user requests a route search to a destination or requests a check of the area around the destination. In this case, regardless of the current driving state of the vehicle (100), virtual building objects modeled from the additionally detected map tile data can be rendered and displayed along the shortest route to the set destination or the driving route searched to the destination. In this case, texturing of the virtual building objects may be involved in the rendering process.
[0519] Meanwhile, texturing for the above-mentioned modeled virtual objects can be performed by tiling texture images, or by sequentially combining multiple partial regions constituting the virtual objects.
[0520] For example, a virtual object corresponding to the above target building may have its surface area divided into multiple layers according to the exterior of the above target building, and a partial area may be formed for each divided surface area. For example, the processor (1330) may divide the above target building into a bottom layer, a top layer, and at least one middle layer, and different tiles may be matched for each partial area.
[0521] In this case, the top floor is distinguished when it has a shape or form that is distinct from the intermediate floor, such as a roof, antenna, or spire, and if the target building does not have a structure placed on the roof of the building such as a roof, antenna, or spire, the processor (1330) may distinguish the target building with only a bottom floor and at least one intermediate floor.
[0522] Meanwhile, in the case of intermediate floors, the processor (1330) may be divided into at least one floor according to a pre-set floor height or arrangement of windows, etc. For example, if intermediate floors have different colors or shapes (e.g., different arrangement of windows), the floors having different colors or different shapes may be divided into different intermediate floors. That is, in the case of floors designated as commercial spaces and floors designated as residential spaces, such as in a mixed-use building, the floors designated as commercial spaces and the floors designated as residential spaces may each be divided into different intermediate floors.
[0523] In this case, different floors, for example, the bottom floor, at least one intermediate floor, and the top floor, may each be matched with different tiles. Then, based on the matched different tiles, tiling can be performed for each floor, that is, each partial area (e.g., bottom floor, at least one intermediate floor, top floor). Then, each partial area where tiling has been performed can be sequentially combined according to a pre-set order based on combination information specified in the building reference information corresponding to the target building.
[0524] Meanwhile, the above tiling may be performed sequentially based on the tiling order specified in the building reference information. For example, each of the above partial regions may be combined sequentially starting from the partial region corresponding to the floor closest to the ground. In this case, tiling for the partial region corresponding to the area in contact with the ground, i.e., the floor layer, may be performed first, and then tiling for the partial region corresponding to the next order (e.g., the first intermediate layer) may be performed on top of the tiled partial region. Finally, tiling for the partial region corresponding to the top floor (e.g., the top floor) may be performed. By ensuring that tiling for each partial region is performed sequentially in a pre-set order for the virtual object in this manner, texturing of the entire surface area of the virtual object can be performed.
[0525] As such, texturing of a virtual object according to an embodiment of the present invention can be performed according to a sequential combination of each tiled partial region. Accordingly, texturing of a virtual object according to an embodiment of the present invention may also be referred to as procedural texturing.
[0526] Meanwhile, the tile DB (1341) may be provided from a pre-configured server that is connected to the 3D map view generation device (1300). For example, the pre-configured server may be a cloud server. In this case, the cloud server may be the same server as the cloud server that provides map data (e.g., map data including 2.5D polygon objects), but it is also possible that they are different servers. In this case, that is, if they are different servers, the cloud server that provides the map data will be described as the first cloud server, and the cloud server that provides the tile DB (1341) and building reference information will be described as the second cloud server.
[0527] When the above tile DB (1341) is provided, the processor (1330) can determine tiles to be tiled on the virtual object surface area of each building included in a defined area based on the driving information of the vehicle (100) based on information collected around the vehicle (100).
[0528] For example, the processor (1330) can fragment an image of a building surrounding a vehicle (100) obtained through a camera (310) according to multiple partial regions and detect a tile that matches each of the fragmented images (i.e., images of partial regions of the target building). To do this, the processor (1330) can calculate a similarity for each of the partial region images to each tile included in the tile DB (1341), and can determine a tile that matches each of the partial region images based on the calculated similarity.
[0529] For example, the processor (1330) can detect feature information for any one partial region image. And among the tiles included in the tile DB (1341), it can detect at least one tile having a feature similar to the feature detected from any one partial region image. Then, for each of the at least one detected tile, it can calculate the similarity to any one partial region image and match any one tile with the highest calculated similarity to the tile corresponding to any one partial region image.
[0530] For such tile matching, the processor (1330) may use a pre-trained Artificial Intelligence (AI) model. In this case, the AI model may recognize the partial region image from features detected in any one partial region image and detect any one tile having features most similar to the feature points of the recognized partial region image.
[0531] Here, feature points of the partial region image can be detected in various ways. For example, the shape, form, or color pattern of the image can be detected as the feature points. In this case, the more distinct the shape, form, or color pattern of the partial region image, the clearer the pattern the artificial intelligence model can detect from the partial region image, and a high recognition rate can be calculated.
[0532] On the other hand, if it is difficult to detect patterns of shape, form, or color from the partial region image due to low image quality, differences in light intensity, shaking, etc., the AI model may fail to detect feature points of the partial region image, and in this case, a low recognition rate may be calculated. Then, based on the recognition rate, the AI model may not detect the tile corresponding to the partial region image. That is, the AI model may decide whether to detect the tile corresponding to any one of the partial region images based on the calculated recognition rate.
[0533] Here, the artificial intelligence model may be included in the 3D map view generating device (1300) according to an embodiment of the present invention as a separate component from the processor (1330) (not shown). Alternatively, the artificial intelligence model may be part of the processor (1330) or the processor (1330) may perform the function of the artificial intelligence model. In this case, the artificial intelligence model may be understood as the same component as the processor (1330).
[0534] Meanwhile, when the above processor (1330) determines a tile corresponding to each partial area image, it can generate combination information related to the combination of each partial area image based on the acquired building image. For example, the combination information may be location information of each partial area image or information regarding the order in which tiling is performed with respect to the acquired building image.
[0535] And the processor (1330) can model virtual objects of buildings located in a region defined according to the position of the vehicle (100) from map data. And among the said region, it can detect building reference information of buildings located in at least a portion of the said region (hereinafter map region) defined according to the driving information of the vehicle (100). And the processor (1330) can perform texturing on virtual objects corresponding to the detected building reference information among the modeled virtual objects.
[0536] More specifically, the processor (1330) can sequentially tile the surface regions of virtual objects corresponding to the detected building reference information using pre-specified tiles for each partial region.
[0537] For example, if the above target building is formed with three layers in appearance, such as a floor layer, a first intermediate layer, and a second intermediate layer, the processor (1330) can tile tiles that match each partial area sequentially to the surface area of a virtual object corresponding to the above target building according to a preset order. Here, the preset order may be in order of proximity to the ground.
[0538] Accordingly, tiles matching the bottom area, that is, the partial area (bottom layer) included in the building reference information, can be tiled onto the surface of the virtual object in the area in contact with the ground in the virtual object within the map data corresponding to the target building. And when the tiling for the bottom layer is completed, the processor (1330) can tile tiles matching the next partial area (first intermediate layer) according to a preset order onto the tiled partial area (bottom layer). And when the tiling for the first intermediate layer is completed, the processor (1330) can tile tiles matching the next partial area (second intermediate layer) according to a preset order onto the tiled partial area (first intermediate layer). By sequentially tiling the tiles matching each partial area onto the surface area of the target building in this way, texturing for the virtual object within the map data corresponding to the target building can be performed.
[0539] Here, the height between each floor of the target building may correspond to an integer multiple of the tile. For example, if the height between each floor of the target building is the same as the height of the tile, the processor (1330) can perform tiling for each part area by arranging tiles that match each part area in a continuous manner without overlapping each other.
[0540] To this end, the processor (1330) can determine the area where the tiling is to be performed based on the left and right margin values set from the corners of each side area of each virtual object (building virtual object) and the lateral and longitudinal lengths of each side area of the building according to the building reference information. In this case, each building face may be divided into multiple partial areas based on floor dividers that distinguish the top, bottom, left, right, bottom layer, top layer, and at least one middle layer. The processor (1330) may configure the left and right margin values and the top and bottom margin values or the width of the floor dividers as margins for each of the multiple partial areas, and determine the remaining area excluding the margins as the area where the tiling is to be performed. In this case, the floor height or floor width of the bottom layer or top layer may be formed differently from the floor height or floor width of the middle layer. Additionally, among the tiles stored in the tile DB (1341), the tiles corresponding to the middle layer may be tiles that differ from the tiles corresponding to the bottom layer and the top layer in at least one of their width or height.
[0541] And the processor (1330) can determine the number of tiles to be tiled in the horizontal direction or the number of tiles to be tiled in the vertical direction (tiling offset) for each partial area where the tiling is to be done, according to the width or height of the determined area. And the processor (1330) can distinguish each building element included in the tile, i.e., the texture image, for the tiles that are tiled continuously on the surface area of the building virtual object.
[0542] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different areas of the texture image by each color value, and accordingly, can distinguish different areas of the texture image, i.e., different building elements, by color channel.
[0543] And the processor (1330) can generate mask maps for masking building elements for each of the separated building elements in each surface area of the building virtual object in which texture images are tiled. The mask map is a map formed such that other building elements are obscured except for one building element in the surface area of the building virtual object in which texture images are tiled, and may mean an image used to specify only one building element among each building element included in the texture image. That is, the mask map corresponding to a specific building element is an image in which other areas are obscured except for the area corresponding to the specific building element, and may be an image for applying a specific effect only to the area corresponding to the specific building element.
[0544] Meanwhile, when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps with shading effects applied so that different material textures can be displayed on each mask map.
[0545] Typically, when there is curvature on the surface of an object, the direction of the surface changes due to the curvature, and light is reflected differently depending on the changed surface direction; the curvature of the object's surface can be perceived through the reflected light that is reflected differently. The normal map is designed to represent the surface with curvature by utilizing the reflected light of the curvature rather than the actual curvature, and may refer to an image having 3D vector values representing the three-dimensional direction of the object's surface that is necessary for a graphics processing device to calculate the degree of shading corresponding to the reflected light reflected by the surface curvature of the specific material when light incident on the specific material is used. When using such a normal map, even if it is a 2D image that does not have a three-dimensional shape, the reflected light corresponding to the curvature according to the specific material can be expressed with respect to the lighting incident on the building, thereby allowing the texture of the specific material to be formed.
[0546] Accordingly, the processor (1330) can blend normal maps corresponding to different materials with respect to mask maps corresponding to each building element. Then, 3D vector values according to the normal maps are applied to the building elements corresponding to each mask map, so that the texture according to the normal maps can be expressed.
[0547] Here, in the case of areas obscured by the mask map, the 3D vector values may not be applied even when the normal map is blended. Therefore, the normal map blended with the mask map may be textured with a specific material only in the areas not obscured by the blended mask map. In this case, the areas not obscured by the mask map may be areas corresponding to specific building elements specified by the mask map. Therefore, the normal map blended with the mask map may be an image to which 3D vector values are applied so that the surface texture of a specific material is expressed only in the specific building elements not masked by the mask map.
[0548] Meanwhile, the above mask map can be performed for each of the separated building elements in each surface area of the building virtual object where texture images are tiled. Therefore, for the surface area of the building virtual object, a plurality of mask maps can be generated for each building element. And each mask map can be blended with a normal map of a different material. And the processor (1330) can perform blending between the normal maps to which the mask maps are blended.
[0549] Accordingly, the processor (1330) can generate a single normal map (hereinafter referred to as a multi-blending normal map) in which 3D vector values corresponding to different material surface textures are applied to regions corresponding to different building elements on the surface area of a virtual building object through blending between the normal maps that are blended with the mask map. Then, by compositing the generated multi-blending normal map onto the surface area of the virtual building object, the regions corresponding to each building element of the virtual building object can be textured so that they each have different surface textures.
[0550] Here, the processor (1330) can generate a normal map for each texture image tiled to the surface area of the building virtual object, in which 3D vectors corresponding to different surface textures are applied to the areas corresponding to each building element included in the texture image. Then, the generated normal map, i.e., a multi-blending normal map, can be composited for each texture image placed on the surface area of the building virtual object. In this case, since the texture image is composited in a tiling manner on the surface area of the building virtual object, the multi-blending normal map composited for each texture image can also be composited in a tiling manner on the surface area of the building virtual object.
[0551] Meanwhile, a normal map to which 3D vector values corresponding to the surface texture of the specific material are applied can be blended with an albedo map to represent the color of the specific material. In this case, the albedo map may be an image having the unique color value of the specific material. Therefore, a normal map having 3D vector values corresponding to the texture of the specific surface material can form a pair with an albedo map having the unique color value of the specific surface material and can be blended together.
[0552] Therefore, when an albedo map is blended with a normal map, the normal map may have not only 3D vector values corresponding to the surface texture of a specific material but also color values unique to the specific material. Thus, when a mask map and a normal map corresponding to a specific building element are blended, and the albedo map is further blended with the blended normal map, the specific building element corresponding to the mask map may have not only the surface texture of the specific material but also color values unique to the specific material. In this case, the multi-blending normal map corresponding to the entire surface area of the building virtual object or the texture image tiled on the building virtual object may be an image in which 3D vector values having different material textures are applied to different areas corresponding to multiple building elements, and each is expressed in a different material color.
[0553] Meanwhile, when texturing of the surface area of the building virtual object is completed through the multi-blending normal map, the processor (1330) can control the MR service device (900) to render the textured building virtual object into MR and display it on the display unit (251). In this case, the processor (1330) can control the MR service device (900) and the display unit (251) of the vehicle (100) through the interface unit (1320).
[0554] Meanwhile, the processor (1330) may determine at least one of the colors of the normal map corresponding to each building element based on an image captured from the camera of the vehicle (100). For example, the processor (1330) may obtain an image from the camera of the vehicle (100) that corresponds to a frustum-shaped visible area according to the current position of the vehicle (100) and the driving direction of the vehicle (100). Then, the objects included in the received image may be classified by type, and a representative color may be determined for each type of classified object. In this case, the representative color may be the RGB average value of the colors of each object classified into a specific type, or the color with the highest proportion among the colors of each object classified into the specific type.
[0555] For example, a processor (1330) can detect colors corresponding to building objects from an acquired image and determine a specific color with the highest RGB average value or ratio of the detected colors as the representative color of the building objects included in the image.
[0556] And the processor (1330) can determine at least one color among the colors of the normal map corresponding to each building element as the determined representative color. Thus, at least one of the colors of the normal map corresponding to each building element of the building virtual object can be determined from an image obtained from the camera of the vehicle (100).
[0557] Meanwhile, although the above description explained modeling building virtual objects corresponding to buildings included in the map area and texturing and rendering some of the modeled building virtual objects, it is also possible to model and render road virtual objects corresponding to road shapes included in the map area.
[0558] To model the above-mentioned road virtual object, the processor (1330) can generate a plurality of nodes along the road shape included in the map area from the map data. Then, the generated plurality of nodes can be defined as respective segments, and the road virtual object can be modeled by generating polygon objects having a specific width for each segment and combining the generated polygon objects. In this case, the polygon objects may be round objects having a specific width.
[0559] Meanwhile, the processor (1330) may use morphology operations to express the edges of the road objects more smoothly. The morphology operations are one of the techniques for analyzing and transforming the shape and structure of objects within an image and may include erosion, dilation, opening, and closing operations. Here, the processor (1330) can express the virtual road objects composed of polygon objects more smoothly through the closing operation, which is used to fill in grooves or cut away protruding parts among the morphology operation techniques.
[0560] Meanwhile, FIG. 12 is a conceptual diagram illustrating the operation flow of a 3D map view generating device (1300) and an MR service device (900) according to an embodiment of the present invention.
[0561] First, the processor (1330) of the 3D map view generating device (1300) according to an embodiment of the present invention can collect driving information of a vehicle (100). Here, the driving information of the vehicle (100) may include information on the speed, position, and driving direction of the vehicle (100) detected through the sensing unit (120). And based on the collected driving information of the vehicle (100), a map area to model virtual objects can be determined.
[0562] Meanwhile, when a map area is determined, the processor (1330) can detect building reference information of buildings included in the determined map area from the building reference information storage unit (1343). When the building reference information is detected, building image setting values, which are information for modeling building virtual objects corresponding to each building, are determined, and building virtual objects corresponding to each building can be modeled based on the determined building image setting values and the building reference information of each building.
[0563] And the processor (1330) can determine any texture image as a tile from the tile DB (1341) for each building or for each partial area of each building (e.g., floor area, at least one intermediate area, top area). And the texture image determined as a tile can be tiled for each building or for each partial area of each building. And the mask map corresponding to different components of the building included in the tiled texture image can be blended with a normal map corresponding to a different material, and each normal map blended with the mask map can be blended with an albedo map having a color corresponding to a material, so that the different components of the building have different materials and different surface textures. And by blending the textured normal maps together, a multi-blending normal map corresponding to a specific surface area or entire partial area of the building virtual object or a specific texture image can be generated. And, a surface texture for a surface area of a building virtual object can be formed by compositing the generated multi-blending normal map over a specific surface area or a partial area of the building virtual object, or by tiling the specific surface area or partial area of the building virtual object along a tiled texture image. That is, a textured virtual object can be created.
[0564] Meanwhile, the MR service device (900) can model the location of the vehicle (100), pedestrians and other vehicles around the vehicle (100), POI (Position Of Interest) information, route information, and road and terrain information around the vehicle (100) based on map data, based on information provided from the vehicle system, namely the GPS, ADAS, and navigation system of the vehicle (100). Then, using the modeled information and the information about the modeled virtual objects provided from the 3D map view generation device (1300), a UX scene including the textured virtual objects can be constructed. Then, by controlling the MR renderer, a map image including a camera view, 3D effects, and GUI can be generated (graphic rendering) and displayed through the display unit (251).
[0565] Hereinafter, we will examine in more detail, with reference to the attached drawings, how a 3D map view generating device (1300) according to an embodiment of the present invention determines a map area, models virtual objects included in the determined map area, and renders and displays the modeled virtual objects.
[0566] FIG. 13 is a flowchart illustrating the operation process of a processor (1330) of a 3D map view generating device (1300) according to an embodiment of the present invention, which determines a map area, models virtual objects included in the determined map area, textures the modeled virtual objects by blending them into a normal map having 3D vector values of different materials, and renders and displays the textured virtual objects. FIG. 14 is an example diagram illustrating an example in which the processor (1330) of a 3D map view generating device (1300) according to an embodiment of the present invention updates a map area according to vehicle driving information. FIG. 15 is a conceptual diagram for explaining the map tile background modeling described in FIG. 14 in a 3D map view generating device (1300) according to an embodiment of the present invention.
[0567] First, referring to FIG. 13, the processor (1330) of the 3D map view generating device (1300) according to an embodiment of the present invention can obtain driving information regarding the vehicle's location, speed, driving direction, destination, route to destination, etc. through the navigation system (770) and the vehicle's sensing unit (120) (S1300).
[0568] When driving information of the vehicle (100) is obtained, the processor (1330) can determine a map area based on a map tile corresponding to the location of the vehicle (100). The map area may be determined differently depending on the pre-set zoom level of the map.
[0569] For example, the map area may be an area formed by 25 map tiles centered around the map tile corresponding to the current vehicle (100) position when the zoom level of the currently set map is 17. In this case, if the zoom level of the map is lowered, the map area may be determined to be an area formed by a larger number of map tiles centered around the map tile corresponding to the current vehicle (100) position, and if the zoom level of the map is higher, the map area may be determined to be a map area formed by a smaller number of map tiles (e.g., 9) centered around the map tile corresponding to the current vehicle (100) position. And when the map area is determined, the processor (1330) can detect data of the map tiles included in the determined map area from the map data stored in the memory (1340).
[0570] Meanwhile, the above map area can be updated according to the driving information of the vehicle (100). For example, when the vehicle (100) moves away from the map tile where it is currently located due to the driving of the vehicle (100), the processor (1330) can update the map area so that the map area includes map tiles within a range according to the zoom level of the currently set map based on the map tile where the vehicle (100) newly enters.
[0571] FIG. 14 is an exemplary diagram illustrating an example in which a processor (1330) of a 3D map view generating device (1300) according to an embodiment of the present invention updates a map area according to driving information of a vehicle (100).
[0572] First, referring to FIG. 14(a), FIG. 14(a) is a drawing showing an example of a map area (1402) formed by 25 map tiles centered on the map tile corresponding to the current vehicle (100)'s location according to the currently set map zoom level '17'. In this case, the map area (1402) can be formed centered on the first map tile (1400) where the current vehicle (100) is located.
[0573] In this state, if the vehicle (100) moves from the first map tile (1400) to the second map tile (1410) according to the movement of the vehicle (100), the processor (1330) can update the map area from the map area (1402), which includes a number (25) of map tiles determined by the zoom level of the map set around the first map tile (1400) as shown in FIG. 14 (b), to the map area (1412), which includes a number (25) of map tiles determined by the zoom level of the map set around the second map tile (1410) according to the position of the moved vehicle (100). Accordingly, the processor (1330) can detect the data of the map tiles corresponding to the updated map area (1412) from the map data stored in the memory (1340). In this case, the processor (1330) can detect at least one map tile (1420) connected to each other from the map data of the memory (1340), as shown in (b) of FIG. 14.
[0574] Meanwhile, when a map area is determined from the data of the detected map tiles, the processor (1330) can model virtual objects corresponding to buildings and roads included in the determined map area (S1304).
[0575] In the above step S1304, the processor (1330) can detect buildings included in the detected map area. And it can detect building reference information corresponding to the detected buildings from the building reference information storage unit (1343). And it can determine the setting values of each building image associated with the detected building reference information. Here, the building image setting values are information for modeling each building virtual object, and may include setting values for the projection direction according to the angle or distance for creating the building virtual objects according to the driving direction of the vehicle (100) and / or the angle at which the MR view is displayed, setting values for modeling according to 2.5D virtual objects or setting values for modeling according to 3D virtual objects, for example, vector values such as the coordinate values of vertices or normals.
[0576] When the building image setting value is determined, the processor (1330) can calculate building setting values corresponding to each building included within the map area based on the determined building image setting value and the building reference information of each building. Here, the building setting value may include the height of the building virtual object to be created, the side area of each side, the plan area, etc. And when the calculation of the building setting values is completed, the processor (1330) can create, i.e., model, virtual objects corresponding to each building included within the map area based on the building setting value.
[0577] Meanwhile, as mentioned in step S1302 above, the processor (1330) can update the map area according to the driving information of the vehicle (100). In this case, if data of new map tiles is additionally detected as shown in (b) of FIG. 14 due to the update of the map area, the processor (1330) can additionally model virtual objects corresponding to buildings and roads included in the added map tiles. On the other hand, if the map area is updated as shown in (b) of FIG. 14, map tiles (1430) that are excluded from the map area may occur. In this case, the processor (1330) can increase the efficiency of memory (1340) by deleting virtual objects generated from the map tiles (1430) that are excluded from the map area.
[0578] Here, the processor (1330) may retain at least some of the virtual objects generated from the excluded map tiles depending on the building type, user-defined status, or POI status. In this case, the retained virtual objects may be stored in memory (1340) and may be reused when the corresponding map tile is detected again in the future. That is, in the case of a building virtual object corresponding to a building frequently visited by the user or a building designated by the user, it may be stored and reused regardless of whether the map tile containing the building is outside the map area.
[0579] Meanwhile, the above S1304 step may include a process of modeling a road virtual object corresponding to the road shape of each map tile included in the map area.
[0580] In this case, the processor (1330) can generate a plurality of nodes along the road shape included in the map area in step S1304. Then, the generated plurality of nodes can be defined as respective segments, and the road virtual object can be modeled by generating polygon objects having a specific width for each segment and combining the generated polygon objects. In this case, the polygon objects may be round objects having a specific width.
[0581] Meanwhile, the processor (1330) may use morphological operations to express the edges of the road virtual object more smoothly. In particular, the processor (1330) may temporarily increase the radius of the round polygon objects and then restore it to its original state to perform a closing operation during the morphological operations. In this case, the degree of smoothness of the edge portion of the road virtual object may vary depending on the radius of the polygon objects and the number of times the closing operation is performed.
[0582] Additionally, the processor (1330) can form the generated road virtual object higher than the ground or lower than the ground. In this case, if the height of the road virtual object is lower than the ground, the ground around the road may exhibit a curb effect forming a curb depending on the height difference between the ground and the road virtual object. In this case, the height difference between the road virtual object and the ground may be different depending on the width or length of the road or the characteristics of the road, so that the road virtual object may be expressed differently according to the characteristics of the actual road.
[0583] In addition, the processor (1330) may form an object corresponding to a river or stream in a manner similar to the method of creating the road virtual object. In this case, the height of the virtual object corresponding to the river or stream may be formed so that it is lower than the height of the ground by a preset depth. In this case, similar to the road virtual object, the height difference between the virtual object corresponding to the river or stream and the ground may be different depending on the width or length of the river or stream, so that the virtual object corresponding to the river or stream may be represented differently according to the characteristics of the actual river or stream.
[0584] Meanwhile, in step S1304 above, when virtual objects such as buildings and roads of each map tile included in the map area are modeled, the processor (1330) can determine a one area according to driving information obtained from the vehicle (100) (S1306).
[0585] The area determined in step S1306 above is an area determined based on the location of the vehicle (100) or the driving direction of the vehicle (100), and may be a map area determined according to the driving information of the current vehicle (100), that is, an area corresponding to at least a part of at least one map tile detected from the map data storage unit (1342) of the memory (1340) as a map area. For example, the said area may be an area corresponding to the area around the current location of the vehicle (100) within a first map tile according to the current location of the vehicle (100). Or, the said area may be an area corresponding to the front of the vehicle (100) in a first map tile containing the current location of the vehicle (100) according to the driving direction of the vehicle (100). Or, the said area may be an area including a part of the first map tile and an area corresponding to a part of at least one other map tile.
[0586] Meanwhile, the area determined in step S1306 above may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), that is, a user field of view area according to the driving direction of the vehicle (100). In this case, the user field of view area may have a shape that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to a perspective projection that can form a sense of perspective, and thus may have the shape of a frustum with the position of the vehicle (100) as the vanishing point.
[0587] Here, the area determined according to the driving information of the vehicle (100) in step S1306 may be an area where virtual objects displayed as a digital twin map can be displayed on the display unit (251). That is, it may be a map area shown to the user of the vehicle (100) as a digital twin map through an MR view, i.e., a visible area. Hereinafter, among the map areas determined according to the driving information of the current vehicle (100), the area determined in step S1306 that is displayed on the display unit according to the driving information of the vehicle (100) will be referred to as the visible area.
[0588] When a visible area is determined in step S1306, the processor (1330) can determine some of the virtual objects to be rendered among the virtual objects modeled in step S1304 according to the determined visible area (S1308). Among the objects modeled in step S1304, that is, virtual objects included in the map tiles detected as the current map area, only some of the virtual objects included in the visible area can be determined in step S1308. In this case, the modeled objects determined as rendering targets in step S1308, i.e., the virtual objects, may include a plurality of building virtual objects and a plurality of road virtual objects or a portion of road virtual objects.
[0589] Meanwhile, when the virtual objects to be rendered are determined in step S1308, the processor (1330) can tile a pre-set texture image, i.e., a tile, for each of the building virtual objects among the determined virtual objects (S1310). Here, the tiling can be performed on each side of each building virtual object determined as a rendering target, and can be performed on the side areas of each building virtual object excluding a pre-set offset area (or margin area) from the corner of each side.
[0590] Here, at least one of the above-mentioned building virtual objects may have a side area that can be divided into multiple sub-areas. For example, the side area may be divided into a bottom area, a middle area, and a top area, and at least one of the bottom area, middle area, and top area may be further divided into multiple areas. In this case, the processor (1330) may tile different texture images for each of the divided multiple areas.
[0591] Additionally, the processor (1330) may tile arbitrary texture images for each building virtual object or for each partial area of each building virtual object. In this case, texture images corresponding to each building virtual object or each partial area of each building virtual object may be predetermined. In this case, building reference information, which is information about the actual building corresponding to each building virtual object stored in memory (1340), may include information about texture images corresponding to each building virtual object or each partial area of each building virtual object, and the processor (1330) may determine the texture images corresponding to each building virtual object or each partial area of each building virtual object when determining building image setting values based on the driving information of the vehicle (100).
[0592] And based on the height, area, side area of the building included in the reference information of each building virtual object, and the offset area (or margin area), at least one of the horizontal length and vertical length of the area where texture images are to be tiled in each building virtual object can be calculated, and the number of texture images to be tiled in the horizontal or vertical direction can be calculated according to the calculated length. That is, the processor (1330) can calculate information for modeling building virtual objects based on the building reference information including the building image setting value and the actual building height, and at the same time, calculate a building setting value that includes information on the width of each side area where texture images are to be tiled in each building virtual object and the number of texture images to be tiled in each side area.
[0593] And the processor (1330) can tile texture images to the side regions of each building virtual object to be rendered based on information about texture images corresponding to each building virtual object or a partial region of each building virtual object, and information about the side regions of each virtual object included in the calculated building setting value and the number of tilings of the texture images.
[0594] Meanwhile, when the tiling of texture images for the side regions of each building virtual object is completed in step S1310, the processor (1330) can generate a normal map containing 3D vector values that form different textures for each building element included in the texture image, which is a tiled 2D image (S1312).
[0595] In this case, the normal map may be a normal map (hereinafter referred to as a multi-blending normal map) that is blended with a plurality of normal maps generated for each building element included in the texture image. To generate the multi-blending normal map, the processor (1330) may generate mask maps for masking building elements for each separated building element in each surface area of the building virtual object where the texture images are tiled.
[0596] In this case, the mask map is a map formed such that other building elements are obscured except for one building element in the surface area of a building virtual object where texture images are tiled, and it may refer to an image used to specify only one building element among each building element included in the texture image.
[0597] And when mask maps corresponding to each building element are generated, the processor (1330) can blend different normal maps with shading effects applied so that different material textures can be displayed on each mask map. In this case, as described above, since the mask map is an image that covers the remaining area excluding only the corresponding specific building element, when the normal map is blended, the 3D vector value forming the texture corresponding to the specific material corresponding to the normal map can be blended only in the area not covered by the mask map and the area corresponding to the specific building element corresponding to the mask map.
[0598] Meanwhile, the processor (1330) can generate a mask map for each of the separated building elements in each surface area of the building virtual object where texture images are tiled. Thus, multiple mask maps can be generated for each building element, and each mask map can be blended with a normal map of a different material. Then, the processor (1330) can generate the multi-blending normal map by blending the normal maps blended with the mask maps with each other, so that 3D vector values corresponding to different material surface textures are applied to the areas corresponding to different building elements for the surface area of the building virtual object. Then, by compositing the generated multi-blending normal map onto the surface area of the building virtual object, the processor (1330) can texture the surface areas of each building virtual object so that the areas corresponding to each building element of the building virtual object each have different surface textures.
[0599] Here, when the processor (1330) generates a mask map for each building element, it may further blend at least some of the plurality of mask maps generated for each building element with an albedo map, which is a map image having a color corresponding to a specific material. In this case, at least some of the plurality of mask maps generated for each building element may have a color according to the albedo map, and thus color mapping can be performed with a color corresponding to a specific material.
[0600] In this case, the processor (1330) can determine a color corresponding to at least some of the plurality of mask maps generated for each building element based on the result of analyzing an image captured by the camera of the vehicle (100). For example, the processor (1330) can detect objects corresponding to buildings among the image objects included in the image captured by the camera of the vehicle (100). Then, it can detect colors from the detected building image objects and determine the dominant color or average color of the actual buildings around the vehicle (100) as the representative color based on the detected colors. Then, by blending an albedo map corresponding to the color determined as the representative color with at least some of the multiple mask maps generated for each building element, and by blending the mask maps blended with the albedo map to generate the multi-blending normal map, and by compositing the generated multi-blending normal map onto each building virtual object, at least some of the building elements of the building virtual objects currently determined as rendering targets in step S1308 can be textured to have colors similar to actual buildings around the vehicle (100).
[0601] Meanwhile, when texturing is completed in step S1312, the processor (1330) can render the textured building virtual objects to generate an image corresponding to the building virtual objects. Additionally, the processor (1330) can render other virtual objects including at least one road virtual object included in the visible area to generate an image corresponding to each virtual object (S1314).
[0602] And the processor (1330) can control the vehicle interface unit (1320) so that virtual objects including each building virtual object and road virtual object generated through the rendering are displayed on the display unit (251) (S1316).
[0603] As described above, the processor (1330) of the 3D map view generating device (1300) according to an embodiment of the present invention has a configuration in which map tiles within a certain area (map area) set around the vehicle (100) are detected, virtual objects within the detected map tiles are pre-modeled, and only some of the objects according to the driving direction of the vehicle (100) among the pre-modeled objects are rendered and converted into images. Furthermore, the processor (1330) can synthesize a normal map so that 3D vector values corresponding to a specific material are applied to each building element only to the rendered objects, thereby allowing a texture according to a specific material to be formed for each building element by the applied 3D vector values during rendering. In this way, the processor (1330) of the 3D map view generating device (1300) according to an embodiment of the present invention can reduce the time required for modeling by rendering the pre-modeled objects when displaying virtual objects as an MR view using a digital twin map. In addition, the number of virtual objects to be rendered is also limited to the visible area to minimize their number, and the surface area of each building virtual object can be textured to have a three-dimensional quality of various materials by using a texture image, which is a 2D image, and a normal map having 3D vector values corresponding to different materials. By performing texturing using only 2D images in this way, the amount of computation required for texturing can be significantly reduced, and consequently, the time required for rendering can be significantly shortened.
[0604] In this way, since the amount of computation and the time required to display virtual objects in an MR view using the digital twin map can be significantly reduced, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention may be able to perform rendering of virtual objects to be displayed through the MR view during the time when the app for displaying the MR view using the digital twin map is running, that is, during the run time. In other words, real-time run-time 3D map modeling may be possible.
[0605] Meanwhile, as described above, the present invention has a configuration in which virtual objects included in a plurality of map tile data are pre-modeled according to the position of the vehicle (100), and some of the modeled virtual objects are rendered including texturing. In this case, the map tile data can be updated according to the position of the vehicle (100), and the detection of such map tile data can be performed through background processing. In this case, the processor (1330) of the 3D map view generating device (1300) according to an embodiment of the present invention pre-models the virtual objects included in the map tile data when the map tile data is detected according to a determined map area, so the modeling of the virtual objects can also be performed through the background processing. The modeling of virtual objects through background processing in this manner is referred to as background modeling.
[0606] Meanwhile, the processor (1330) according to the embodiment of the present invention has a configuration that pre-models virtual objects in a map area through background modeling and renders a portion of the virtual objects that have already been modeled based on a visible area determined by the driving information of the vehicle (100). Here, the modeling of the virtual objects is performed through background processing, and the rendering of the modeled virtual objects is performed through foreground processing that occurs according to the execution of the user's MR view app. Therefore, the 3D map view generating device (1300) according to the embodiment of the present invention has a configuration in which the modeling and rendering of virtual objects are performed separately from each other. Accordingly, if the 3D map view generating device (1300) according to the embodiment of the present invention is equipped with a plurality of processors, the modeling of the virtual objects and the rendering of the modeled virtual objects may be performed by different processors, respectively.
[0607] FIG. 15 is a block diagram illustrating a configuration in which the modeling and rendering of virtual objects are performed by different processes in a 3D map view generating device (1300) according to an embodiment of the present invention having multiple processors.
[0608] Referring to FIG. 15, the 3D map view generating device (1300), i.e., the embedded system according to an embodiment of the present invention, may include a first processor that loads a plurality of map tiles via an API based on the current location (GPS Position) of the vehicle (100) linked with a navigation map, and a second processor that renders an image of a virtual object displayed on a display unit (251). Here, the first processor may be a Central Processing Unit (CPU) of the 3D map view generating device (1300) according to an embodiment of the present invention, and the second processor may be a Graphic Processing Unit (GPU) that processes graphic operations on the display device and outputs a result value.
[0609] In this case, the first processor can perform modeling of virtual objects included in the loaded multiple map tiles while loading multiple map tiles according to the position of the vehicle (100). In this case, the first processor can perform modeling of the virtual objects using a background modeling method. The modeled virtual objects can be stored in a buffer. In this case, the objects being modeled may be virtual objects in the form of simple polygons (2.5D) without surface texturing.
[0610] In this case, at least one of the plurality of map tiles may be added when the map area is updated according to the position of the vehicle (100). Then, the first processor may additionally model virtual objects included in the at least one added map tile according to the updated map area. And the buffer may be updated so that the modeled virtual objects are added. Thus, virtual objects corresponding to each object including buildings and roads included in the entire updated map area may be stored in the buffer.
[0611] Meanwhile, when an app for displaying an MR view is executed by a user, the second processor can render the modeled virtual objects stored in the buffer. During the rendering process, the second processor can perform texturing by tiling a texture image onto the side area of each modeled building virtual object and compositing a normal map onto the side of each building virtual object, wherein a 3D vector corresponding to a different material is applied to each area of each building element included in the tiled texture image. In this case, the 3D vector is intended to provide a shading effect that forms the texture of a specific material through shading effects, and the second processor can perform rendering that generates an image of each building virtual object, including texturing that includes shading based on the 3D vector applied to each building element, as a custom shader.
[0612] Accordingly, the 3D map view generation device (1300) according to an embodiment of the present invention can perform the modeling of virtual objects and the rendering of the modeled virtual objects in parallel on different processors. Therefore, compared to the case where all processes are performed on a single processor, the computational load applied to the processor can be significantly reduced, and by performing rendering from pre-modeled objects to display virtual objects, the time required until the virtual objects are displayed on the display unit can also be significantly shortened. Accordingly, when an app related to the MR view is executed, rendering of three-dimensional virtual objects included in the digital twin map can be performed during the runtime, which is the time the app is executed, and real-time provision of a digital twin map containing virtual objects with high-quality textures can be made possible.
[0613] However, for the convenience of explanation, the following description assumes that modeling and rendering are performed by a single processor (1330). However, it goes without saying that the present invention is not limited thereto.
[0614] Meanwhile, in step S1316 of FIG. 13, the processor (1330) of the 3D map view generating device (1300) according to an embodiment of the present invention can simultaneously display a screen showing information collected from around a vehicle according to an ADAS system and an MR view screen according to a digital twin map. FIG. 16 is an illustrative diagram showing such an example.
[0615] Referring to FIG. 16, the processor (1330) of the 3D map view generating device (1300) can display an ADAS (Advanced Driver Assistance Systems) view image in a part area (1600) of the display unit (251) based on information about surrounding vehicles and obstacles collected through the vehicle interface unit (1320) and driving information of the vehicle (100). ADAS refers to an active safety device that detects dangerous situations using sensors or cameras, warns the driver of the risk of an accident, and helps the driver make a judgment and take action. The ADAS view image may include information about objects corresponding to the surrounding situation of the vehicle, such as vehicle objects and other vehicles and obstacles detected around the vehicle, and may include route information such as the current driving speed of the vehicle (100), driving time, or estimated time to the destination.
[0616] Meanwhile, the processor (1330) can display an MR view image on another area (1610) on the display unit (251). The MR view image is a view image that includes a building virtual object or a road virtual object created as a 3D object, and can display a digital twin map formed by the virtual objects.
[0617] Meanwhile, according to the above description, the processor (1330) of the 3D map view generating device (1300) according to an embodiment of the present invention may determine, in step S1306 of FIG. 13, an area to render and display virtual objects according to the driving information of the vehicle (100), i.e., a visible area. Here, the visible area may be formed in the form of an area that can be captured as an image by the camera of the vehicle (100) according to the driving direction of the vehicle (100), i.e., a user field of view area according to the driving direction of the vehicle (100). In this case, the user field of view area may have a shape that becomes narrower as it approaches the vehicle (100) and wider as it moves away from the vehicle (100) according to a perspective projection in which a sense of perspective can be formed, and thus may have the shape of a frustum with the position of the vehicle (100) as the vanishing point. Therefore, the visible area may be formed in the shape of a frustum with the position of the vehicle (100) as the vanishing point.
[0618] FIG. 17a is an exemplary illustration showing a frustum-shaped visible area formed across a plurality of map tiles or a single map tile according to the position of the vehicle and the driving direction of the vehicle in a 3D map view generating device (1300) according to an embodiment of the present invention. FIG. 17b is an example showing a visible area formed to include a plurality of map tiles.
[0619] First, referring to (a) of FIG. 17a, the processor (1330) can form a frustum-shaped visible area (1700) formed along the driving direction of the vehicle (100) with the current position (1720) of the vehicle (100) as a reference point. In this case, the processor (1330) can perform rendering only on virtual objects located within the formed visible area (1700). That is, even within a single map tile, virtual objects located in areas other than the visible area (1700) may not be rendered.
[0620] Here, rendering may refer to a process or computational process of generating an image using a computer program. Therefore, even if a virtual object is modeled, that is, even if a virtual object is created, if the rendering is not performed, an image corresponding to the virtual object may not be generated. And if an image is not generated, it may not be displayed on the display unit (251). That is, the processor (1330) can selectively determine which objects among the modeled virtual objects to display on the display unit (251) based on the visible area formed according to the driving direction of the vehicle (100).
[0621] Accordingly, as shown in (a) of FIG. 17a, images corresponding to each virtual object can be generated through a rendering process for virtual objects within the area included in the visible area (1700). However, images of virtual objects may not be generated for virtual objects within the area included in the visible area (1700). Therefore, the processor (1330) can reduce the amount of computation required for rendering and shorten the computation time by rendering only some of the virtual objects that have been modeled based on the visible area.
[0622] Meanwhile, the above-mentioned visible area (1700) may be formed across a plurality of map tiles, as shown in FIG. 17 (b), depending on the zoom level of the set map. The plurality of map tiles may be part of a map area set around the current location (1720) of the vehicle (100). In this case, since the processor (1330) models virtual objects based on the map area, images of virtual objects in the area included in the visible area (1700) among the plurality of map tiles are generated through a rendering process according to the visible area (1700) formed across the plurality of map tiles and can be displayed on the display unit (251).
[0623] Meanwhile, FIG. 17b is an example illustrating an example in which a visible area formed in such a frustum shape includes a plurality of map tiles.
[0624] As shown in FIG. 17b, when a plurality of map tiles are included within a visible area (1700), the processor (1330) can perform modeling for virtual objects included in each of the plurality of map tiles included within the visible area (1700). Then, the modeled virtual objects can be rendered according to a preset order or by selecting only some of the virtual objects.
[0625] For example, the processor (1330) can perform modeling and rendering in order of proximity to the vehicle based on the vehicle's current location (1720). Accordingly, objects included in the first map tiles (1751) that are close to the vehicle's current location (1720) can be modeled and rendered before the second map tiles (1752) that are further away from the vehicle's current location (1720).
[0626] Alternatively, the processor (1330) may model and render only a portion of the objects included in each of the map tiles included within the visible area (1700) according to the distance from the reference point of the vehicle's current location (1720).
[0627] For example, the processor (1330) may perform modeling and rendering for all objects included in each map tile in the case of the first map tiles (1751), but may perform modeling and rendering for only some of the objects included in each map tile in the case of the second map tiles (1752). In this case, the processor (1330) may model and render only the objects corresponding to roads and terrain among the objects included in the second map tiles (1752), and may not perform modeling and rendering for general building objects.
[0628] In this way, the priority for modeling and rendering of objects included in each map tile can be determined according to their type. This priority based on the type of object can be included in the object data for the map tile (hereinafter referred to as map tile data), along with geometry data containing shape information of each object, as attribute data corresponding to each object.
[0629] In this way, the order in which objects included in the map tile are modeled and rendered according to their types is determined by the attribute information, and the processor (1330) can model (procedural modeling) and render (procedural rendering) the objects included in the map tile sequentially according to the above order, that is, priority.
[0630] Meanwhile, using the above priority, the processor (1330) of the 3D map view generation device (1300) according to an embodiment of the present invention may cause a specific object to be modeled and rendered with priority over other objects. For example, the processor (1330) may assign a higher priority to objects such as landmarks than to general building objects. In this case, the building object corresponding to the landmark may be modeled and rendered with priority or first over other general building objects. As an example, if the second map tiles (1752) contain a building object corresponding to a landmark, the processor (1330) may perform modeling and rendering of the landmark building object together with the road objects and terrain objects included in the second map tiles (1752).
[0631] Meanwhile, FIG. 18a is an example diagram illustrating an example in which texture images are tiled onto modeled virtual building objects in a 3D map view generating device (1300) according to an embodiment of the present invention.
[0632] When the building virtual objects (hereinafter referred to as rendering target virtual objects) determined as rendering targets in step S1308 of FIG. 13 are determined, the processor (1330) can tile a preset texture image to each side area for each of the building virtual objects among the determined virtual objects. In this case, the processor (1330) can detect the side area of the building virtual object excluding the preset offset area (or margin area) inward from each side corner of the building virtual object, and tile the texture image to the detected side area.
[0633] First, referring to (a) of FIG. 18a, the processor (1330) can determine the left and right margins (1811, 1812) of the side area of the building virtual object determined as the rendering target according to the left and right margin values that are pre-set for the side area of the building virtual object. According to the left and right margin values, the coordinate range of the area (1810) to tile the texture image in the horizontal direction can be determined.
[0634] For example, if the left and right margin values are 0.1, the bottom-left point of the side area of the building virtual object (2520) can be defined as (0, 0) and the top-right point as (1, 1). Then, as shown in FIG. 18a, the width of the tiling area (1810) excluding the left and right margins can be calculated as 0.8. And if the building width of the actual building corresponding to the building virtual object (2520) is 14m, the actual width of the tiling area (1810) can be calculated as 11.2m.
[0635] Meanwhile, the texture image (1800) is a normalized tile image and may have a pre-set width. In this case, if the width of the texture image (1800) is 4m, a value of 2.8 can be calculated by dividing the actual width of the tiling area (1810), which is 11.2m, by the width of the texture image (1800), which is 4m. In this case, since the number of texture images (1800) is an integer, a value of 3 can be calculated by rounding, and the calculated value of 3 may be the number of texture images (1800) to be tiled in the tiling area (1810) of the side area of the building virtual object (2520) set according to the pre-set left and right margin values. And the processor (1330) inputs the left and right X-axis coordinates (0.1, 0.9) of the tiling area (1810) as input values for the remap function and inputs a new range according to the remap function as (0, 3), so that up to 3 tiles (texture images (1800)) are attached to the tiling area (1810) in the horizontal direction in a continuous manner.
[0636] In addition, the processor (1330) can set multiple partial regions based on the floor layer, which is the first layer, according to the pre-set upper and lower margin values. For example, the processor (1330) can set a floor divider corresponding to the space between the floor layer and the upper part of the partial region. In this case, the space between the floor dividers set based on the floor layer can be determined as the partial region where texture images will be tiled.
[0637] Here, the side area of the building virtual object may include multiple sub-regions. For example, the side area of the building virtual object may be divided into a bottom area (1852), a middle area (1850), and a top area (1851), and at least one of the bottom area (1852), the middle area (1850), and the top area (1851) may be further subdivided into multiple areas. In this case, the processor (1330) may tile different texture images for each of the subdivided multiple areas.
[0638] In this case, the processor (1330) can obtain a mask map corresponding to each partial area based on a UV coordinate system corresponding to each partial area on each side of the building. For example, the processor (1330) can define the bottom-left point of the side area of the building virtual object as (0, 0) and the top-right point as (1, 1). Then, by defining the Y-axis coordinate in the UV coordinate system according to the margin value of the partial area where the floor division is formed, the vertical width corresponding to the partial area is defined, and a mask map corresponding to the partial area can be obtained according to the defined vertical width. In this case, the mask map corresponding to the partial area may be a mask map in which the height below the Y-axis coordinate according to the margin value of the partial area has a value of 0, and the height above the Y-axis coordinate has a value of 1. In this case, when blending with the normal map is performed, the part having a value of 0 maintains the value of 0, but the part having a value of 1 may have the 3D vector value of the normal map.
[0639] As an example of such a partial area, if the margin value of the partial area where the layer division is formed is 0.8, the processor (1330) can determine the upper right corner coordinates of the partial area as (1, 0.8). Then, the processor (1330) can obtain a mask map in which the area corresponding to a height less than 80% of the total height of the building virtual object from the floor surface of the building virtual object has a value of 0, and the area corresponding to a height greater than that has a value of 1. That is, the area corresponding to the upper corner of the building virtual object from a height greater than 80% of the total height of the building virtual object can be formed as a partial area (e.g., a tower area).
[0640] Meanwhile, the processor (1330) can tile different texture images for each partial area (1850, 1851, 1852) of the building virtual object (2560) in the same manner as described in (a) of FIG. 18a. In this case, as shown in (b) of FIG. 18a, the top area (1851) can be tiled with a first texture image (1801), and the floor area (1852) can be tiled with a second texture image (1802). And the middle area (1850) can be tiled with a third texture image (1800). Thus, the building virtual object (2650) can have different texture images tiled for each different partial area.
[0641] In this way, the processor (1330) of the 3D map view generating device (1300) according to an embodiment of the present invention can diversify the facade design of the building virtual object by dividing each building virtual object into different multiple regions and tiling different texture images into each divided region.
[0642] FIG. 18b illustrates examples of facade designs of virtual building objects that are expressed in various ways by varying partial regions and tiled texture images in a 3D map view generating device (1300) according to an embodiment of the present invention.
[0643] First, FIG. 18b (a) illustrates an example of a virtual building object formed into three partial regions: a bottom layer, a middle layer, and a top layer. FIG. 18b (b) and (c) illustrate examples of virtual building objects formed into partial regions only with a bottom layer and a middle layer. In this way, virtual building objects can be represented to have different external shapes by varying the number of each partial region.
[0644] In addition, the processor (1330) can make the texture images tiled in each partial area different for each building virtual object. In this case, since not only the number of partial areas but also the tiled texture images are different, the number of possible combinations can be increased. Therefore, it is possible to represent the appearance of a more diverse building virtual object.
[0645] Meanwhile, the processor (1330) can blend a normal map having a 3D vector value corresponding to a different material for each building element in step S1312 of FIG. 13 so that for each building element included in the texture image tiled on the building virtual object, the image can be displayed with a different material during rendering.
[0646] FIG. 19 is a flowchart illustrating in more detail the texturing process of step S1312 of FIG. 13, which forms different surface textures on each component of a building in a 3D map view generating device (1300) according to an embodiment of the present invention. FIG. 20 is an example diagram illustrating an example in which normal maps are blended into a mask map corresponding to different building elements according to the process described in FIG. 19, and a multi-blending normal map is generated by blending the normal maps together.
[0647] First, referring to FIG. 19, the processor (1330) can identify and distinguish building elements included in each texture image for each texture image tiled in the tiling area of a building virtual object (S1900).
[0648] For example, each building element included in the texture image may have a different color. In this case, the processor (1330) can distinguish different regions of the texture image by each color value, and accordingly, distinguish different regions of the texture image, i.e., different building elements, by color channel. For example, if the texture image includes at least one window element, the window element may have a different color value from the other building background. Also, if the window element includes a glass element corresponding to a glass area, a window frame element corresponding to a window frame area, and a blind element corresponding to a blind area, the glass element, window frame element, and blind element may each have different color values. Therefore, the processor (1330) can distinguish and identify regions corresponding to multiple different building elements included in the texture image based on color values. In this case, the texture image may be a map image that labels multiple different building elements and may be named a labeling map.
[0649] Meanwhile, when each building element included in the texture image is distinguished and identified, the processor (1330) can generate mask maps corresponding to each building element (S1902).
[0650] In this case, the mask map corresponding to a specific building element may be an image having a value of 1 in the area corresponding to the specific building element identified by a specific color value in the texture image, and a value of 0 in the remaining area.
[0651] For example, if the texture image includes at least one window element, the mask map corresponding to the window element may be a mask map in which only the area corresponding to the window element has a value of 1 and the remaining area has a value of 0. On the other hand, the mask map corresponding to the building wall may be a mask map in which only the building wall portion, that is, the remaining part of the texture image excluding the window element, has a value of 1 and the remaining part has a value of 0.
[0652] Additionally, if the window element includes the glass element, the window frame element, and the blind element, the processor (1330) can generate a glass mask map in which only the area corresponding to the glass element in each window element has a value of 1 and the rest has a value of 0, a window frame mask map in which only the area corresponding to the window frame element in each window element has a value of 1 and the rest has a value of 0, and a blind mask map in which only the area corresponding to the blind element in each window element has a value of 1 and the rest has a value of 0.
[0653] When mask maps corresponding to each building element are generated for each building element included in the tiled texture image, the processor (1330) can blend each generated mask map with a normal map having a 3D vector value corresponding to a different material (S1904). In this case, the normal map refers to an image having a 3D vector value representing the three-dimensional direction of an object surface that is required to calculate the degree of shading corresponding to the reflected light that is reflected by the surface curvature of the specific material when light incident on the specific material is reflected by the graphic processing device, and may be a 2D image capable of expressing the texture of the specific material through the degree of shading, that is, the shading effect.
[0654] The blending of such normal maps and mask maps can be a process similar to multiplication. Therefore, when the normal maps and mask maps are blended, the 3D vector values applied to the normal map can be applied only to the areas that are not masked with a value of 0 in the mask map, that is, the areas corresponding to specific building elements. Thus, the texture according to a specific material can be expressed only to the areas corresponding to specific building elements corresponding to the mask map. In this case, the processor (1330) may make each building element have a texture of a different material by blending the mask map corresponding to each building element with a normal map having 3D vector values of a different material.
[0655] When each mask map is blended with each normal map in step S1904 above, the processor (1330) may further blend an albedo map having a different color for each normal map blended with the mask map. In this case, the albedo map may be an image having a color value unique to a specific material. And it may form a pair with a normal map having a 3D vector value corresponding to the texture of a specific surface material.
[0656] Then, the processor (1330) can further blend an albedo map that forms a pair with the normal map for each normal map blended with the mask map (S1906). In this case, the blending with the albedo map can be applied similarly to a multiplication process, just like the blending with the normal map. Thus, color values can be applied only to the parts of the mask map that have a value of 1, and accordingly, a normal map can be generated that can express the texture of a specific material and the unique color of that material for a specific building element.
[0657] Meanwhile, in step S1906 above, when normal maps blended with mask maps corresponding to each component of the building are blended with albedo maps respectively, the processor (1330) can blend the building elements blended normal maps with each other (S1908). In this case, the blending of the normal maps is a union concept and may mean a process of combining the areas where 3D vector values (and color values according to the albedo map) are applied to each normal map into a single map image, which is a multi-blending normal map. Therefore, when the blending process of step S1906 is completed, a normal map (multi-blending normal map) in which 3D vector values (and albedo color values) of different normal maps are applied to each building element included in a single texture image can be generated.
[0658] And the processor (1330) can perform texturing on the side area of the building virtual object by compositing the multi-blending normal map into the tiling area of the side area of the building virtual object (S1910).
[0659] In this case, the multi-blending normal map may be generated for the entire tiled area where texture images are tiled in a side area or partial area of the building virtual object. In this case, the processor (1330) may composite the multi-blending normal map, generated with a size corresponding to the entire side area or partial area of the building virtual object, onto the side area or partial area of the building virtual object.
[0660] Alternatively, the multi-blending normal map may be generated for any one texture image tiled to a side area or partial area of a building virtual object. In this case, the processor (1330) may composite the multi-blending normal map to a side area or partial area of a building virtual object by tiling the multi-blending normal map to each of the texture images tiled to a side area or partial area of the building virtual object.
[0661] Meanwhile, FIG. 20 illustrates an example in which a multi-blending normal map is generated, comprising regions to which different 3D vector values corresponding to each of the multiple building elements included in the texture image are applied, according to the operation process described in FIG. 19.
[0662] Referring to FIG. 20, the processor (1330) can distinguish and identify each building element from a texture image (1800) containing a window element. In this case, if the window element includes a glass element and a window frame element (2002), the processor (1330) can identify the wall element (2001), the glass element, and the window frame element (2002) respectively from the texture image (1800).
[0663] And the processor (1330) can generate a mask map corresponding to each identified building element. In this case, as shown in FIG. 20, a mask map (2011) corresponding to a wall element can be generated in which only the area corresponding to the wall element (2001) has a value of 1, and the remaining area, for example, the area corresponding to the glass element and the window frame element (2002), has a value of 0. Additionally, a mask map (2012) corresponding to a window frame element can be generated in which only the area corresponding to the window frame element (2002) has a value of 1, and the remaining area, for example, the area corresponding to the glass element and the wall element (2001), has a value of 0.
[0664] And the processor (1330) can blend a first normal map (2021) and a second normal map (2022), which are map images with 3D vector values corresponding to different materials, for each generated mask map (2011, 2012). In this case, the first normal map (2021) may be an image with 3D vector values applied that can express the first material with a shading effect, and the second normal map (2022) may be an image with 3D vector values applied that can express the second material with a shading effect.
[0665] Then, the 3D vector values of the blended normal map can be synthesized only in the areas having a value of 1, excluding the areas having a value of 0 in each mask map. Thus, when the mask map (2011) corresponding to the wall element and the first normal map (2021) are blended, as shown in FIG. 20, a normal map (2031) can be generated in which 3D vector values capable of expressing the first material only in the areas corresponding to the wall element in the texture image (1800) are synthesized. Additionally, when the mask map (2012) corresponding to the window frame element and the second normal map (2022) are blended, as shown in FIG. 20, a normal map (2032) can be generated in which 3D vector values capable of expressing the second material only in the areas corresponding to the window frame element in the texture image (1800) are synthesized.
[0666] Then, the processor (1330) can blend a normal map (2031) in which 3D vector values capable of expressing a first material only in the area corresponding to the wall element are synthesized, and a normal map (2032) in which 3D vector values capable of expressing a second material only in the area corresponding to the window frame element are synthesized. In this case, blending is performed between the normal maps, and each normal map can be synthesized into a single multi-blending normal map. Thus, as shown in FIG. 20, a multi-blending normal map (2050) can be generated in which 3D vector values capable of expressing a first material are synthesized in the area corresponding to the wall element (2001) and 3D vector values capable of expressing a second material are synthesized in the area corresponding to the window frame element (2002). That is, a normal map in which 3D vector values capable of expressing different materials can be synthesized for each area corresponding to each building element included in the texture image can be generated.
[0667] Meanwhile, the description of FIG. 19 above describes blending an albedo map having the material's inherent color with a normal map blended with a mask map to map the material's inherent color to each building element. However, unlike this, the color of virtual objects within the visible area may be determined as a color similar to the color of buildings included in the actual visible area based on an image captured from the camera of the vehicle (100).
[0668] FIG. 21 is a flowchart illustrating the operation process of mapping the colors of virtual building objects based on the colors of actual buildings within a visible area in a 3D map view generating device (1300) according to an embodiment of the present invention.
[0669] Referring to FIG. 21, when step S1904 is performed, which involves blending mask maps generated for each building element in FIG. 19 with normal maps having 3D vector values corresponding to different materials, the processor (1330) can acquire an image of the surroundings of the vehicle (100) through the camera of the vehicle (100) (S2100). In this case, the camera may be a camera that captures an image of the front area of the vehicle (100), and the acquired image may be an image of the front of the vehicle (100) acquired according to the driving direction of the vehicle (100), that is, an image of the visible area.
[0670] And the processor (1330) can identify each object included in the acquired image and divide the acquired image into multiple regions according to the type of each identified object. That is, the processor (1330) can divide the regions where each object included in the acquired image is displayed on the image into different regions for each object (S2102).
[0671] For example, if the acquired image is an image containing trees, buildings, the sky, and roads, the processor (1330) may, in step S2102, divide the image into areas where each tree is displayed, areas where each building is displayed, areas where the sky is displayed, and areas where roads are displayed. Then, the dominant color in each divided area may be detected, and each area may be displayed with the detected dominant color. In this case, an image may be acquired in which the areas where each object included in the image is displayed are divided by the dominant color in each area.
[0672] To this end, the processor (1330) may use an artificial intelligence module (not shown) trained through pre-set training data. In this case, the training may be for identifying each region on an image as a different object according to its shape and color. Accordingly, the artificial intelligence module that has completed the training may identify a region of an image as belonging to one of the pre-set object types based on the pattern formed by the shape and color displayed in a region of the input image.
[0673] And the processor (1330) can divide the regions on the image, which are partitioned by each object, according to the type of object. And it can determine a representative color from the regions on the image divided according to the type of each object (S2104).
[0674] In this case, one or more regions on the partitioned image may be classified as the same type of object. For example, the area where each building is displayed on the image may be partitioned into regions corresponding to each building, and a dominant color may be detected for each partitioned region. In this state, the processor (1330) can distinguish each region on the image partitioned for each object according to the type of each object. Thus, at least one region identified as 'building' may be classified as a region corresponding to the object type 'building'. And the processor (1330) can determine a representative color corresponding to the object type 'building' based on the color of each of the multiple regions classified as regions corresponding to the object type 'building'.
[0675] For example, the processor (1330) may determine the color with the largest proportion among the colors of each area corresponding to the object type 'building' as the representative color corresponding to the object type 'building'. Alternatively, the processor (1330) may calculate the average color of each color of the areas corresponding to the object type 'building' and determine the calculated average color as the representative color corresponding to the object type 'building'.
[0676] When a representative color corresponding to the object type 'building' is determined, the processor (1330) can map at least some of the building elements of the virtual building object within the visible area to the determined representative color (S2106). For example, the processor (1330) can blend at least some of the normal maps blended to the mask maps corresponding to each different building element with an albedo map having a representative color corresponding to the object type 'building'.
[0677] FIG. 22 is an example diagram illustrating an example in which a color mapped to texture images is determined through an image captured from a vehicle in a 3D map view generating device (1300) according to an embodiment of the present invention.
[0678] First, FIG. 22 (a) illustrates an example of an image captured by a camera of a vehicle (100). In this case, the image may be an image captured of the front of the vehicle (100) along the driving direction of the vehicle (100).
[0679] Then, the processor (1330) can identify objects included in the acquired image. And as shown in FIG. 22 (b), the regions identified as each object can generate a color map image in which each region is displayed with a dominant color.
[0680] When a color map image is generated as shown in FIG. 22 (b), the processor (1330) can separate the area of each object identified in the generated color map image according to the type of each object. That is, as shown in FIG. 22 (b), the first area (2201) corresponding to the first building object, the second area (2202) corresponding to the second building object, the third area (2203) corresponding to the third building object, the fourth area (2204) corresponding to the fourth building object, and the fifth area (2205) corresponding to the fifth building object in the color map image can be separated as areas corresponding to the object type 'building'.
[0681] Then, the processor (1330) can calculate the color weight of each of the areas corresponding to the object type 'building'. In this case, areas having the same color can be considered as one area. Therefore, the first area (2201) and the fifth area (2205) having the same color can be considered as one area.
[0682] In this case, as shown in (c) of FIG. 22, the proportions of colors determined from each area identified as a building object in the image can be calculated. In this case, the proportions of colors of road objects or sky objects can also be calculated, just like the building objects.
[0683] In this case, as shown in (c) of FIG. 22, if the first color, which is the dominant color of the first region (2201) and the fifth region (2205), occupies the largest proportion, the processor (1330) can determine the representative color of the building virtual object with the first color. Then, the processor (1330) can generate an albedo map having the first color and blend at least some of the normal maps that are blended with the mask maps corresponding to each of the different building elements with the albedo map having the first color. Thus, at least one normal map blended with the albedo map of the first color can be blended with each other to generate a multi-blending normal map. Therefore, texturing of the building virtual object can be performed so that at least some of the building elements of the building virtual object are displayed in the first color.
[0684] Meanwhile, in the case where a plurality of map tiles are included within a visible area (1700) as shown in FIG. 17b, the 3D map view generating device (1300) according to an embodiment of the present invention can perform modeling of virtual objects included in each of the plurality of map tiles included within the visible area (1700). The processor (1330) can generate an original mesh, which is a set of vertices having polygonal shapes of the objects, through the modeling. By modifying the coordinates of the vertices constituting the original mesh, a distorted 3D map view of the shape of at least one object included in the plurality of map tiles can be provided.
[0685] In this case, the distorted object may be a terrain object included in the plurality of map tiles. Then, the 3D map view generating device (1300) according to an embodiment of the present invention can provide a three-dimensional map view in which a plurality of objects including building objects are displayed on a distorted ground surface according to the distorted terrain object.
[0686] FIG. 23 is a conceptual diagram for explaining the concept of GPU shader processing that distorts an object in a 3D map view generating device according to an embodiment of the present invention.
[0687] Referring to FIG. 23, map data corresponding to each of a plurality of map tiles included within a visible area (1700) may include information on vertices for forming polygonal shapes of objects included in each map tile. Then, the processor (1330) may generate meshes (original mesh - mesh before distortion), which are sets of vertices, from the map data. In this case, the original mesh may be classified into a terrain mesh representing roads, curbs, and land within the map tile area, and a building mesh representing a plurality of buildings and 3D landmarks within the map tile area.
[0688] Then, the processor (1330) can receive the vertex coordinates of each original mesh. And based on a pre-set distortion model (2302) and a vertex coordinate transformation function defined by distortion parameters, it can perform vertex distortion processing (2303) to modify the coordinates of the vertices included in the original mesh, i.e., the original vertices.
[0689] Here, the distortion model (2302) may define a function capable of expressing the distorted shape of a vertex. For example, if a terrain object in the form of a horizontal plane is distorted into a hemispherical shape, the distortion model (2302) may be defined in the form of an exponential function with the current location of the vehicle as the origin. In this case, the distortion parameter may be a coefficient and a constant such as the slope of the function, and may be the slope value of the exponential function of the distortion model (2302), and may be a value controlled according to the UX scenario situation. And the vertex coordinate transformation function may be an exponential function defined by the slope value of the exponential function. Here, the UX scenario is a scenario in which a 3D map view is provided, and may include the method of providing the 3D map view (e.g., front view, top down view, etc.).
[0690] Meanwhile, when performing the above-mentioned vertex distortion processing (2303), if the distance between the vertices constituting the original mesh is greater than a certain distance, smooth distortion processing may not be possible. For example, in the case of a road object with the shape of a straight road with a distance of 1 km, the original mesh may be represented with only two vertices at the start and end. In this case, since there is no vertex that can specify the middle area of the road object, distortion of the road object may not be possible.
[0691] Accordingly, the processor (1330) may add a tessellation process (2301) to the original mesh detected from the map data if necessary. In this case, the tessellation process may mean a process of adding vertices between vertices that have a distance greater than the predetermined distance so that the distance between vertices is less than the predetermined distance.
[0692] The process of FIG. 23 described above may be performed in a processor (1330). Alternatively, it may be a process performed in a Graphic Processing Unit (GPU) that performs graphic processing in a 3D map view generating device (1300) according to an embodiment of the present invention under the control of the processor (1330). As an example, the process of FIG. 23 may represent shader processing, which is a set of software instructions used to calculate the rendering effect of the graphics hardware in the GPU. Therefore, in the following description, the processor (1330) may be a GPU.
[0693] FIG. 24 illustrates a cross-section of a three-dimensional map in which a plurality of building objects (2411) are rendered on a distorted terrain object (2500) when the terrain object (2500) is distorted according to the distance from the vehicle (2410) in a 3D map view generating device (1300) according to an embodiment of the present invention.
[0694] First, referring to FIG. 24 (a), the terrain object (2500) may not be distorted up to the first area (2401) from the vehicle (2310) to the first distance that is set. In this case, the first distance may be a distance at which visibility of navigation information and terrain shape can be secured. Therefore, from the location of the vehicle (2410) to the first area (2401) corresponding to the first distance, the terrain object may not be distorted, and the positions of the vertices of the original mesh may be maintained as they are.
[0695] And in the case of a second area (2402) which is an area where the distance from the vehicle (2410) exceeds the first distance, the processor (1330) can change the vertex positions of the original mesh (hereinafter referred to as terrain object mesh) constituting the terrain object (2500) based on a predefined coordinate transformation function.
[0696] In this case, the coordinate transformation function can modify the vertex positions of the terrain object mesh with the starting point of the second area (2402) as the initial value, such that the displacement increases as they move further away from the initial value. Accordingly, the processor (1330) can modify the position of the vertex of the terrain object mesh with a smaller displacement value the closer the vertex is to the point corresponding to the first distance from the starting point of the second area (2402), i.e., the distance from the self-velocity (2410), and modify the position of the vertex with a larger displacement value the further the vertex is from the point corresponding to the first distance from the self-velocity (2410).
[0697] Accordingly, as shown in FIG. 24 (a), distortion begins at a point where the distance from the self-property (2410) exceeds a first distance, and the terrain object (2500) can be distorted to have a greater curvature as the distance from the self-property (2410) increases. That is, the distortion of the terrain object (2500) can begin from an area (second area (2402)) that exceeds the first distance from the self-property (2410).
[0698] Meanwhile, the first distance may refer to a distance at which visibility of navigation information and terrain shape can be secured as described above. The visibility of the terrain shape may be determined by the height of the 3D map view camera (2400) or the angle of the 3D map view camera (2400). Here, the 3D map view camera (2400) may provide a field of view (2550) of the 3D map view including a navigation guidance position (2451). Additionally, the first distance may be determined according to the driving state of the vehicle (2410), for example, driving speed, etc. Therefore, the first distance may be determined based on driving information including the height of the 3D map view camera (2400), the directional angle of the gaze point vector of the 3D map view camera (2400), or the driving state of the vehicle (2410).
[0699] Meanwhile, the processor (1330) can determine in real time, according to the driving conditions of the vehicle (2410), a distortion model (exponential function) and a distortion parameter (slope) that define a coordinate transformation function that distorts the terrain object (2500).
[0700] For example, the processor (1330) can determine the first distance according to the driving path of the vehicle (2410). And it can determine the distortion parameter. As an example, the processor (1330) can set the first distance to 300m. And it can determine the degree of distortion of the terrain object (2500) by setting the distortion parameter.
[0701] In this case, if a right turn situation occurs 700m ahead, the ground shape corresponding to the right turn section (700m ahead of the vehicle (2410)) exceeding 300m (first distance) from the vehicle (2410) may be distorted by the slope according to the distortion parameter, as shown in FIG. 24 (a). Accordingly, the road and terrain shape of the area extending from the point exceeding 300m from the vehicle (2410) to the right turn section 700m ahead may be distorted, and a 3D map view screen including the distorted road and terrain shape may be displayed on the display unit.
[0702] Alternatively, the processor (1330) may determine the distortion rate of the terrain object (2500) based on the distance between the vehicle (2410) and the navigation guidance location (2451). Here, the navigation guidance location (2451) may be a destination or a waypoint for route guidance. Or it may be a specific location that is pre-set or arbitrarily set by the user.
[0703] In this case, the processor (1330) can increase the distortion rate of the terrain object as the distance between the navigation guidance location (2451) and the vehicle (2410) becomes closer. Therefore, as the distance to the vehicle (2410) becomes closer, the terrain object (2500) is distorted with a higher curvature, so the straight-line distance between the 3D map view camera (2400) and the navigation guidance location (2451) can become closer. Thus, the user can identify and observe the navigation guidance location (2451) from a closer distance.
[0704] On the other hand, the processor (1330) can reduce the distortion rate of the terrain object as the distance between the navigation guidance location (2451) and the vehicle (2410) increases. Therefore, as the distance from the vehicle (2410) increases, the terrain object (2500) is distorted with a lower curvature, so the straight-line distance between the 3D map view camera (2400) and the navigation guidance location (2451) can be increased.
[0705] Meanwhile, the processor (1330) determines a minimum distortion rate based on the distance between the vehicle (2410) and the navigation guidance location (2451), and can distort a terrain object (2500) (an area outside the first distance from the vehicle (2410) - a second area (2402)) with a distortion rate greater than or equal to the minimum distortion rate.
[0706] In this case, the processor (1330) can calculate a distortion rate that distorts the terrain object (2500) so that the navigation guidance position (2451) exists within a preset distance from the intersection point (2460) of the distortion surface of the terrain object (2500) corresponding to the second area (2402) and the upper vector (2405) of the field of view of the 3D map view camera (2400) as shown in FIG. 24 (b). The calculated distortion rate can be determined as the minimum distortion rate. The terrain object (2500) can be distorted so that the shape of the ground in the area (second area (2402)) where the distance from the vehicle (2410) exceeds the first distance is distorted by a distortion rate greater than the determined minimum distortion rate.
[0707] As described above, it has been mentioned that the curvature of the shape of the road and ground (hereinafter referred to as the ground shape) that is distorted at a first distance or after the first distance from the vehicle (2410) may vary depending on the driving situation of the vehicle (2410), for example, the driving speed. FIG. 25 is an example illustration showing an example in which a terrain object is distorted differently depending on the speed of the vehicle during the driving situation of the vehicle (2410).
[0708] First, referring to FIG. 25, FIG. 25 (a) illustrates an example of a ground shape according to a distorted terrain object (2500) in an area after the first distance from the vehicle (2410) when the driving speed of the vehicle (2410) is greater than or equal to the first speed. FIG. 25 (b) illustrates an example of a ground shape according to a distorted terrain object (2500) in an area after the first distance from the vehicle (2410) when the driving speed of the vehicle (2410) is less than or equal to the second speed. In this case, the first speed may be faster than the second speed.
[0709] In this case, when the speed of the vehicle (2410) is slow (second speed or lower), the shape of the ground after the first distance from the vehicle (2410) can be bent with a low curvature, as shown in FIG. 25 (b). On the other hand, when the speed of the vehicle (2410) is fast (first speed or higher), the shape of the ground after the first distance from the vehicle (2410) can be bent with a high curvature, as shown in FIG. 25 (a). That is, the higher the driving speed of the vehicle (2410), the higher the distortion rate of the terrain object (2500).
[0710] In this case, if the distortion rate of the terrain object (2500) increases, the distance between objects located near the vehicle (2410) (e.g., building objects) and the 3D map view camera (2400) may be shortened due to the bending of the ground shape caused by the distortion. Therefore, visibility of the area near the vehicle (2410) may be improved. On the other hand, if the distortion rate of the terrain object (2500) decreases, a wider field of view is secured due to the ground shape bent with low curvature, so a wider map area can be displayed on the 3D map view screen. Therefore, the user can more easily observe an area far from the vehicle (2410).
[0711] FIG. 26 is an illustrative diagram showing an example in which path guidance according to the distorted terrain object is provided in a 3D map view generating device according to an embodiment of the present invention.
[0712] First, referring to FIG. 26 (a), FIG. 26 (a) illustrates an example of distorting a terrain object detected from map data of map tiles included in a viewing area into a pre-set three-dimensional shape.
[0713] In this case, the processor (1330) may distort the map area visible within the field of view by bending the terrain object into a hemispherical shape so that the ground becomes higher as it moves away from the vehicle object (3000), centering on the location of the vehicle object (3000) corresponding to the location of the vehicle (100), as shown in (a) of FIG. 26. In this case, the vehicle object (3000) may be located at the center of the hemispherical distorted map area and may be the location with the lowest height among the hemispherical distorted map areas.
[0714] Therefore, as the distance from the vehicle object (3000) increases, the height of the ground displayed on the display unit (251) may increase, and in this case, the obscuration between the virtual building objects due to the step difference may be reduced. Also, as the height of the ground increases as the distance from the vehicle object (3000) increases, there may be no horizontal line, and thus the map area can be displayed without limitations due to the vanishing point.
[0715] Referring to FIG. 26(b), FIG. 26(b) shows an example in which an MR view screen based on a digital twin map is displayed without distortion of the terrain object depending on the position of the vehicle (100). In this case, since the terrain object has the shape of a map horizontal plane, the building objects displayed on the terrain object may become smaller according to the principle of perspective projection as the distance from the vehicle object (3000) increases. Also, since the building objects become smaller as they get closer to the horizon, virtual objects may no longer be displayed starting from the horizon. Furthermore, as the guidance position displayed on the display unit (251) is also obscured by other building objects, the objects in front of the vehicle that the user can identify in the 3D map view may be limited to within a certain distance from the vehicle object (3000).
[0716] Meanwhile, referring to (c) of FIG. 26, where the terrain object is distorted into a hemispherical shape centered on the position of the vehicle (100), the terrain object can be distorted so that the ground becomes higher as it moves away from the vehicle object (3000). In this case, the terrain object can be distorted into a hemispherical shape as shown in (a) of FIG. 26. And when the terrain object is distorted into a hemispherical shape, a horizontal line may not be formed due to the shape of the ground area according to the distorted terrain object.
[0717] Therefore, since the limitation by the horizontal line is removed, the field of view in front of the vehicle that can be displayed within the field of view in front of the vehicle can be extended. In addition, because a step is formed as it moves further away from the vehicle object (3000) due to the distortion, even if the zoom level of the map is the same, 3D objects located at a greater distance can be displayed on the screen of the display unit (251) without being obscured.
[0718] Meanwhile, although the above description used the case where there is only one navigation guidance location as an example, it is obvious that there may be multiple navigation guidance locations. In this case, the 3D map view generating device (1300) according to the embodiment of the present invention may distort the ground shape between each of the multiple navigation guidance locations with different curvatures.
[0719] FIG. 27 illustrates an example of distorting a terrain object (2500) so that the terrain between a plurality of guidance positions differs from one another according to the distance from the vehicle in a 3D map view generating device (1300) according to an embodiment of the present invention.
[0720] Referring to FIG. 27, if there are multiple navigation guidance locations (first guidance location (2451), second guidance location (2452)) after the first distance, the processor (1330) can divide the area into a first guidance area (2701) corresponding to the distance (first guidance distance) from after the first distance to the first guidance location (2451) and a second guidance area (2702) corresponding to the distance (second guidance distance) from after the first guidance location (2451) to the second guidance location (2452). For each divided area, at least a part of the terrain object (2500) can be distorted so that the ground shape is distorted with different curvatures.
[0721] In this case, the processor (1330) may not distort the ground shape of the area within the first distance (first area (2401)). And, among the distances from the vehicle (2410) to the first guidance position (2451), the area corresponding to the distance after the first distance, i.e., the first guidance area (2701), may be distorted by a part of the terrain object (2500) so that the ground shape corresponding to the first guidance area (2701) is distorted by a first curvature. And, among the distances from the vehicle (2410) to the second guidance position (2452), the area after the first guidance distance, i.e., the second guidance area (2702) corresponding to the second guidance distance, may be distorted by a part of the terrain object (2500) so that the ground shape corresponding to the second guidance distance is distorted by a second curvature.
[0722] For example, the first curvature may have a smaller value than the second curvature. Or the first curvature may have a larger value than the second curvature. Or the first curvature and the second curvature may be curvatures that distort the shape of the ground in different directions.
[0723] For example, as shown in FIG. 27 above, the first curvature may be a curvature that distorts the terrain object (2500) so that the ground shape is distorted in the direction of the sky (e.g., a curvature having a positive value). Conversely, the second curvature may be a curvature that distorts the terrain object (2500) so that the ground shape is distorted in the direction of the direction of the ground (e.g., a curvature having a negative value).
[0724] In this case, as shown in FIG. 27, due to the curvature having a negative value, the ground shape of the area between the first guide position (2451) and the second guide position (2452) (first guide area (2701)) can be bent in the opposite direction to the direction in which the ground shape bends in the second guide area (2702). Thus, as shown in FIG. 27, the distance between the first guide position (2701) and the second guide position (2702) can be shortened along the viewing direction of the 3D map view camera (2400). Therefore, the second guide position (2452), which is spaced apart from the vehicle (2410) by the first distance, the first guide distance, and the second guide distance, can be included within the field of view (2405) of the 3D map view camera (2400) together with the first guide position (2451). Therefore, multiple guidance positions (first guidance position (2451), second guidance position (2452)) can be displayed on a single screen.
[0725] In this way, the processor (1330) can determine the curvature to distort the ground shape between each of the plurality of guide positions so that a plurality of guide positions with different distances from the vehicle (2410) are all displayed on a single screen. That is, the processor (1330) can determine at least one of the size of the area where distortion occurs and the direction in which distortion occurs by setting different curvatures for the ground areas between the plurality of guide positions with different distances from the vehicle (2410).
[0726] In addition, the above description used the example that the first curvature and the second curvature have different polarities (positive or negative), but it goes without saying that the first curvature and the second curvature may have the same polarity. For example, if both the first curvature and the second curvature have positive values, the shape of the ground between the first guide position (2701) and the second guide position (2702) in FIG. 27 may be distorted downwards.
[0727] Meanwhile, the above terrain distortion can be applied to various types of 3D map views.
[0728] For example, in the above description, an example was described in which the shape of the ground in an area (second area) exceeding the area (first area) from the vehicle is distorted when a front view image corresponding to the forward direction of the vehicle is provided from a 3D map view camera positioned at a certain height from the vehicle. However, unlike such a front view, it is obvious that distortion of the terrain object can also be applied when a top-down view in the direction of the ground is provided from above the vehicle (2410).
[0729] FIG. 28 illustrates an example in which the shape of the ground is distorted when a top-down view is provided in a 3D map view generating device (1300) according to an embodiment of the present invention.
[0730] First, the upper drawing of FIG. 28 illustrates a cross-section of a case where a top-down view is provided centered on the position of the vehicle (2810). For example, when a top-down view is selected, such as at the request of a user, the processor (1330) can align the line of sight of the 3D map view camera (2800) perpendicular to the ground and move it to a preset height from the vehicle (2810). Accordingly, as shown in the lower drawing of FIG. 28, the 3D map view camera (2800) can provide a 3D map view in the form of looking down at the ground from above, centered on the position of the vehicle (2810).
[0731] In this case, the area shown by the 3D map view, i.e., the field of view, can be determined according to the field of view angle (2830) of the 3D map view camera (2800) and the height of the 3D map view camera (2800). In this case, if the field of view angle (2830) of the 3D map view camera (2800) is fixed, the area shown by the 3D map view can be determined according to the height of the 3D map view camera (2800). And the processor (1330) can determine at least one map tile as a field of view based on the determined field of view of the 3D map view camera (2800), and detect meshes for configuring each object, i.e., original meshes, from the information of objects included in the map data of each of the determined at least one map tile.
[0732] Meanwhile, as described above, the 3D map view generating device (1300) according to an embodiment of the present invention can ensure that no distortion occurs with respect to the ground shape of a first area (2801) within a first distance from the vehicle (2810). However, regarding the ground shape of a second area (2802) corresponding to a distance exceeding the first distance from the vehicle (2810) and reaching the field of view limit of the 3D map view camera (2800), i.e., a second distance, the terrain object (2500) can be distorted to bend according to a preset curvature.
[0733] Accordingly, objects (e.g., building objects) included in the map area corresponding to the first area (2801), which has a diameter corresponding to the first distance centered on the vehicle (2810), can be displayed in the form of a conventional top-down map view, as shown in the lower drawing of FIG. 28. On the other hand, objects included in the map area between the field of view limit of the 3D map view and the first area (2801) can be displayed distorted according to the shape of the ground that bends according to a preset curvature.
[0734] Meanwhile, as described above, the field of view of the 3D map view can be determined according to the height at which the 3D map view camera (2800) is positioned from the ground. Accordingly, a range in which the shape of the ground is not distorted, i.e., a first distance, can be determined. In other words, in the case of a top-down view, the first distance may vary depending on the height at which the 3D map view camera (2800) is positioned, and accordingly, the location and range of the area where the shape of the ground is distorted may vary.
[0735] Meanwhile, as described above, the 3D map view generating device (1300) according to an embodiment of the present invention can provide a 3D map view in which the ground shape is distorted. And since the distorted ground shape is bent according to a constant curvature depending on the distance from the vehicle (2410), the ground shape can be more bent as the distance from the vehicle (2410) increases.
[0736] In this case, if the ground shape is bent from the ground toward the sky due to distortion of the terrain object (2500), building objects located further away from the vehicle (2410) may be displayed at a higher position than the height of the vehicle (2410). In this case, the processor (1330) may display building objects rendered in the distorted ground shape differently depending on their distance from the vehicle (2410).
[0737] FIG. 29 is an exemplary diagram illustrating an example in which building objects are displayed on a distorted terrain object according to the height of the distorted terrain object in a 3D map view generating device according to an embodiment of the present invention.
[0738] Referring to FIG. 29, the processor (1330) can distort the terrain object (2500) so that the ground shape is bent according to a constant curvature for a second area (2402) that is outside a first area (2401) within a first distance from the vehicle (2410). Accordingly, as shown in FIG. 29, building objects displayed on the terrain object (2500) that is distorted as the distance from the vehicle (2410) increases can be displayed at a position higher than the height of the vehicle (2410).
[0739] In this case, the processor (1330) can render building objects only up to an area corresponding to a preset height (2900). That is, the processor (1330) can normally render modeled building objects for building objects displayed at a height (first height) where the height of the distorted ground shape is less than or equal to the preset height (2900).
[0740] Here, the above-mentioned preset height (2900) may be determined according to the position where the 3D map view camera (2400) is placed. For example, the height at which the 3D map view camera (2400) is placed may be determined as the above-mentioned preset height (2900), or the above-mentioned preset height (2900) may be determined according to the upper limit of the field of view provided through the 3D map view camera (2400).
[0741] On the other hand, the processor (1330) may restrict the rendering of building objects that are at a height (second height) exceeding the first height among the building objects included in the distorted ground shape. In this case, for building objects that are at a height (second height) exceeding the first height, only the shape of the area occupied by the building objects, that is, the footprint shape, may be displayed. Therefore, in the distorted ground shape with a height exceeding the first height, only the footprint shape of the building objects and the road objects that are distorted together with the ground shape may be displayed. Accordingly, a map area at a sufficient distance from the vehicle (2410) may be displayed with route guidance information, the shape of the road section, and footprint shapes of building objects displayed similarly to a top-down view.
[0742] Meanwhile, when the processor (1330) completes parsing and modeling of building objects from map data, it can place the modeled building objects below the ground object, that is, on the underside of the area (footprint area) corresponding to each building object on the ground object. Then, when rendering begins, the processor (1330) can smoothly move the modeled building objects from below the ground object toward the sky. Then, it can render the building objects sequentially starting from the part exposed above the ground object. Accordingly, the sense of time axis fragmentation in which the building objects are rendered can be eliminated.
[0743] In this case, when building objects are rendered, the processor (1330) may restrict the rendering of building objects so that the height of the ground shape bent by the distortion is above a certain height from the vehicle (2410), i.e., above a second height. In this case, the building objects parsed and modeled from the map data may not rise to the ground object while placed on the lower surface of the terrain object bent by the distortion. That is, the rendering of the modeled building objects may not be performed.
[0744] Meanwhile, similarly, the processor (1330) may limit the height of building objects displayed on the curved ground shape based on the height of the curved ground shape according to the distance from the vehicle (2410).
[0745] As described above, the processor (1330) can distort the terrain object (2500) so that the ground shape of the second area (2402) that is a first distance away from the self-vehicle (2410) is bent according to a constant curvature. In this case, the processor (1330) can limit the rendering of building objects included in the second area (2402) so that their height decreases according to the distance from the self-vehicle (2410). That is, with models of building objects included in the second area placed on the lower surface of the terrain object bent by distortion, the processor (1330) can limit the height rising above the terrain object (2500) according to the distance from the self-vehicle (2410) at a certain ratio or maximum height. Therefore, the greater the distance from the self-vehicle (2410), the lower the height of the building object rising from the terrain object (2500). And some parts of the building object model placed on the terrain object (2500) may not be rendered.
[0746] The rendering of the above building objects (2412) can be limited by restricting them from rising up.
[0747] That is, the processor (1330) can limit the rendering of the building objects by restricting some of the building objects included in the curved ground shape from rising up from the ground, and accordingly, the height at which the building objects are displayed can be limited according to the distance from the vehicle (2410).
[0748] Here, the processor (1330) may limit the rendering of the building objects so that the height of the building objects decreases as the height of the curved ground shape approaches the first height. Accordingly, building objects included in the distorted ground shape are displayed with their height gradually decreasing as they move away from the vehicle (2410), and when the preset first height is reached, a 3D map view may be provided in which the display of building objects is smoothly transitioned to only the footprint area without building objects.
[0749] Meanwhile, although FIG. 29 described limiting the rendering of building objects included in the distorted ground shape according to the height from the vehicle (2410), it is also possible to limit the rendering of building objects included in the distorted ground shape based on a route guidance location such as a destination or a waypoint.
[0750] First, referring to FIG. 30 which illustrates an example where the path guidance location (2451) is obscured by building objects when the terrain object (2500) is not distorted, there may be multiple building objects (2411, 2412) between the path guidance location (2451) and the vehicle (2410).
[0751] And the 3D map view camera (2400), which provides a view of the 3D map view, can observe the path guidance location (2451) for path guidance. Then, a vector corresponding to the viewing direction of the 3D map view camera (2400) can be aligned toward the path guidance location (2451). In this case, there may be multiple building objects between the 3D map view camera (2400) and the path guidance location (2451), and in this case, even if the 3D map view camera (2400) is placed at a certain height, the path guidance location (2451) may be obscured by building objects colliding with the viewing direction vector (3000), as shown in FIG. 30.
[0752] In order to prevent the path guidance location (2451) from being obscured by building objects, the 3D map view generating device (1300) according to an embodiment of the present invention can distort the terrain object (2500) so that the shape of the ground in a second area separated by a first distance or more from the vehicle (2410) is bent.
[0753] Then, the building objects (2412) included in the second area where the ground shape is distorted can be displayed to protrude above the distorted ground shape. In this case, the building objects (2412) can protrude in a direction perpendicular to the horizontal plane that is parallel to the driving plane of the vehicle (2410), which is the ground shape on which the vehicle (2410) is driving, as disclosed in FIG. 31 (a), according to the ground direction according to the undistorted terrain object. That is, although the ground shape is in a bent, distorted state, the building objects (2412) can protrude in a direction perpendicular to the horizontal plane. In this case, as described above, if a building object model parsed and modeled from map data is placed below a terrain object (2500) and rendering of the building object is performed as the building object model rises from the terrain object (2500), then the building objects (2412) can be placed below the distorted terrain object (2500) in a direction perpendicular to the horizontal plane in the area where each building object (2412) is placed, i.e., the footprint area, as shown in FIG. 31 (a). Then, the processor (1330) can perform rendering sequentially starting from the part exposed above the terrain object (2500) while moving the modeled object in a direction perpendicular to the horizontal plane. Thus, as shown in FIG. 31 (a), the building objects (2412) can be rendered with a curved ground shape and an incline.
[0754] On the other hand, alternatively, the processor (1330) may protrude in a direction perpendicular to the curved ground shape, that is, according to the ground direction according to the distorted terrain object, as disclosed in FIG. 31 (b). In this case, if a building object model parsed and modeled from map data as described above is placed below the terrain object (2500) and rendering of the building object is performed as the building object model rises from the terrain object (2500), then the building objects (2412) may be placed below the distorted terrain object (2500) in a direction perpendicular to the curved ground shape in a footprint area included in the curved ground shape, as shown in FIG. 31 (b). And the processor (1330) may perform rendering sequentially starting from the part exposed above the terrain object (2500) while moving the modeled object in a direction perpendicular to the footprint area of each building object. Therefore, as shown in Fig. 31 (b), each building object (2412) can be rendered rising at different angles depending on the height of the curved ground shape.
[0755] Meanwhile, when building objects (2412) are rendered in this manner, the processor (1330) may limit the height at which building objects (2412) included in the second area (2402) are displayed according to the distance to the path guidance location (2451).
[0756] In this case, the processor (1330) can limit the rendering of the building objects (2412) so that the height displayed decreases as the distance from the vehicle (2410) of the building objects (2412) included in the second area (2402) increases. That is, the processor (1330) can limit the height of the building objects (2412) rising above the terrain object (2500) according to the distance from the vehicle (2410) at a certain ratio or maximum height, while the models of the building objects (2412) included in the second area (2402) are placed on the lower surface of the terrain object bent by distortion.
[0757] Accordingly, the further the distance from the vehicle (2410), the lower the height at which the building object (2412) of the second area (2402) rises from the terrain object (2500). Therefore, some parts of the building object model placed on the underside of the terrain object (2500) may not be rendered. That is, the processor (1330) can limit the rendering of the building objects (2412) by restricting the part of the building objects (2412) included in the curved ground shape from rising from the ground. Accordingly, the height at which the building objects (2412) are displayed may be limited depending on the distance from the vehicle (2410).
[0758] Here, the processor (1330) may limit the rendering of the building objects (2412) such that the height of the building objects (2412) decreases as the height of the curved ground shape approaches the path guidance position (2451). Accordingly, the building objects (2412) included in the distorted ground shape are displayed with a gradually decreasing height as they move away from the vehicle (2410), and the displayed height may be lowest when reaching the path guidance position (2451).
[0759] Here, the height of a building object near the path guidance location (2451) may be limited to the height of the building object. Then, the building object near the path guidance location (2451) may not protrude above the terrain object (2500). In this case, only the shape of the footprint area of the building object near the path guidance location (2451) may be displayed.
[0760] Meanwhile, for building objects (2413) located after the path guidance position (2451), the processor (1330) may not restrict rendering. Accordingly, the processor (1330) may move each building object so that it rises toward the sky by the height of each building object for the building objects (2413) located after the path guidance position (2451), and for the part exposed above the terrain object (2500), rendering through tiling of texture images may be performed according to the processes described in FIGS. 18a to 21 above.
[0761] In this case, as described in (a) and (b) of FIG. 31 above, building objects may protrude over the curved ground shape in a direction perpendicular to the horizontal plane or in a direction perpendicular to the ground bent by distortion.
[0762] FIG. 32(a) illustrates an example in which building objects protrude above a distorted terrain object (2500) in a direction perpendicular to the horizontal plane. In this case, the processor (1330) can place the models of each building object, which have been completed, below the terrain object (2500) in a direction perpendicular to the horizontal plane within the footprint area of each building object included in the distorted ground shape. Then, rendering can be performed while raising each building object model in a direction perpendicular to the horizontal plane. Thus, each building object protruding above the distorted terrain object (2500) can be protruded at an angle in the direction in which the ground shape bends due to the distortion. Thus, as shown in FIG. 32(a), the building objects may be rendered as if lying on the bent ground shape.
[0763] In contrast, FIG. 32 (a) illustrates an example in which building objects protrude in a direction perpendicular to the direction of the distorted ground. In this case, the processor (1330) can place the models of each building object, which have been completed, under the terrain object (2500) in a direction perpendicular to the direction of the distorted ground, within the footprint area of each building object included in the distorted ground shape. Then, rendering can be performed while raising each building object model in a direction perpendicular to the direction of the distorted ground. Thus, each building object protruding above the distorted terrain object (2500) can be rendered as each building object protrudes above the terrain object (2500) at different angles depending on the distance from the vehicle (2410), as shown in FIG. 32 (a).
[0764] Meanwhile, since 3D map views typically display objects using perspective projection, the size may decrease as the distance from the vehicle increases. And as the displayed size becomes smaller, it becomes more difficult to identify, so there is a problem that objects that are farther from the vehicle (2410) are difficult to identify.
[0765] However, as described above, when a terrain object (2500) is distorted, in the case of an object that is far from the vehicle (2410), even if the actual distance is far, the distance to the 3D map view camera (2400) may actually become closer due to the distortion. Accordingly, the 3D map view generating device (1300) according to an embodiment of the present invention may display the size of objects such as terrain, buildings, or roads that are far from the vehicle (2410) larger by reflecting the distance difference caused by such distortion.
[0766] FIG. 33 is an exemplary diagram illustrating an example in which building objects displayed on a distorted terrain object are displayed larger according to distance in a 3D map view generating device (1300) according to an embodiment of the present invention. In FIG. 33, building objects were assumed for convenience, but it is not limited thereto. That is, it can be applied to various objects, such as road objects or static objects like cars, in addition to building objects.
[0767] Referring to FIG. 33, the processor (1330) can distort the terrain object (2500) so that the ground shape is curved in the case of a second area (2402) that is outside the first area (2401), where the distance from the vehicle (2410) is the first distance. In this case, the building objects (2412) placed in the second area (2402) may be displayed with different sizes depending on the distance from the vehicle (2410).
[0768] For example, the processor (1330) can greatly enlarge the size of building objects (2412) placed in the second area (2402) according to the distance from the self-propulsion (2410), as shown in FIG. 33. Thus, the size of the building objects (2412) can be increased while the vertical position, i.e., the height, is changed due to distortion. Accordingly, not only can the size reduced by perspective projection according to the distance from the self-propulsion (2410) be compensated, but the size can also be increased to a size that is easy for the user to identify. Accordingly, the user can observe objects at a distance more easily.
[0769] Meanwhile, according to the above description, the increase in size of the objects is due to the distance between the 3D map view camera (2400) and each of the changed objects being brought closer due to distortion.
[0770] Accordingly, the processor (1330) can determine the rate of increase in the size of the objects based on the straight-line distance between the ground curved by distortion, the 3D map view camera (2400), and each building object. In this case, regardless of the distance from the vehicle (2410), the size of the object may increase as the distance between the object protruding on the ground curved by distortion and the 3D map view camera (2400) is closer. On the other hand, the size of the object may decrease according to the perspective projection principle as the distance between the object protruding on the ground curved by distortion and the 3D map view camera (2400) is farther.
[0771] Alternatively, the processor (1330) may increase the size of building objects placed on the ground shape bent by the distortion, but only for objects that satisfy pre-set conditions. Here, the objects that satisfy the pre-set conditions may be objects corresponding to path guidance, object types, or pre-set POIs (Point Of Interest). For example, the processor (1330) may increase the size displayed only for pre-set objects, such as pre-set path guidance locations like destinations or waypoints, or objects around said path guidance locations (2451). Alternatively, the processor (1330) may increase the size of objects corresponding to pre-set POIs (Point Of Interest). Alternatively, the processor (1330) may increase the size displayed only for pre-set types of building objects, such as landmark objects.
[0772] Meanwhile, in the above description, for convenience, it was assumed that the objects displayed (rendered) on the distorted ground shape are building objects, but it goes without saying that the present invention is not limited thereto. That is, in addition to building objects, various dynamic objects such as vehicles can be displayed on the distorted ground shape.
[0773] FIG. 34 is an example diagram illustrating an example in which dynamic objects are displayed according to a distorted terrain in a 3D map view generating device (1300) according to an embodiment of the present invention.
[0774] The processor (1330) of the above 3D map view generating device (1300) can detect information of fixed objects, such as terrain objects, road objects, and building objects, from map data of at least one map tile included within the field of view. Then, the detected objects can be modeled, and modeled and rendered procedurally according to a preset order. For example, the processor (1330) can first model and render terrain objects and road objects. Then, with terrain and road objects rendered, modeling and rendering of building objects can be performed.
[0775] And the processor (1330) can collect information about movable dynamic objects, such as vehicles, within the field of view. To this end, the processor (1330) can receive information about dynamic objects, such as the location of surrounding vehicles and traf...
Claims
Claim 1 A 3D map view generating device comprises: an interface unit for receiving driving information including the position of a vehicle; and a memory for storing a map object including at least one of a terrain object and a road object included in the map data. A 3D map view generating device comprising a processor that generates a 3D map view including a guidance position for guiding a driving path of the vehicle based on the driving information, wherein the processor determines a distance for securing visibility for the guidance position based on at least one of the distance between the vehicle's position and the guidance position, the viewing condition of the 3D map view, and the driving state of the vehicle, maintains the vertex position of the map object in a first area corresponding to within the distance for securing visibility based on the vehicle's position, changes the position of at least some vertices constituting the map object based on a defined coordinate transformation function in a second area exceeding the distance for securing visibility based on the vehicle's position, generates a modified 3D map view such that the map object of the second area is continuous with the map object of the first area based on the at least some vertices whose positions have been changed, and controls the guidance position to be displayed on the map object of the second area in the modified 3D map view. Claim 2 A 3D map view generating device according to claim 1, wherein the processor determines the visibility securing distance based on at least one of the height of the 3D map view camera providing the field of view of the 3D map view, the gaze direction vector of the 3D map view camera, and the driving state of the vehicle, when at least one of the driving state of the vehicle, including the height of the 3D map view camera providing the field of view of the 3D map view, the gaze direction vector of the 3D map view camera, and the driving state of the vehicle changes. Claim 3 A 3D map view generating device according to claim 1, wherein the processor modifies the location of at least some of the vertices constituting the terrain and road objects detected from the map data based on a vertex coordinate transformation function defined by a distortion model defined in the form of an exponential function and a distortion parameter including an exponential slope value determined according to a scenario in which the 3D map view is provided. Claim 4 A 3D map view generating device according to claim 1, wherein the processor varies the curvature at which the terrain and road objects are distorted according to the distance between a guidance position for guiding the driving path of the vehicle and the vehicle or the driving speed of the vehicle, and the guidance position is at least one of a destination, a waypoint for guiding the path, or a specific location set by a user. Claim 5 A 3D map view generating device according to claim 1, wherein the processor distorts a map object of the second area with a distortion rate greater than or equal to a preset minimum distortion rate, and the minimum distortion rate is a distortion rate that distorts the terrain object such that the guidance position exists within a preset distance from the intersection point of the distorted terrain object and the upper vector of the field of view of the 3D map view camera providing the field of view of the 3D map view. Claim 6 delete Claim 7 A 3D map view generating device according to claim 1, wherein, when there are multiple guidance positions for guiding the driving path of the vehicle, the processor distinguishes a map area according to each of the multiple guidance positions based on the distance from the vehicle, distorts the terrain and road objects so as to be bent according to different curvatures for each distinguished area, and the different curvatures include curvatures having different polarities (positive (+), negative (-)). Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete Claim 21 A 3D map view generating device according to claim 1, wherein the processor performs a tessellation process of adding at least one vertex between the vertices when the interval between the vertices constituting the map object detected from the map data is greater than or equal to a preset distance, and modifies the vertices after the tessellation process is performed based on the defined coordinate transformation function. Claim 22 A 3D map view generating device according to claim 1, wherein the processor determines the visibility-securing distance based on at least one of the road shape included in the driving path of the vehicle or the position information of the guidance position. Claim 23 A 3D map view generating device comprises: an interface unit that receives driving information of a vehicle and receives object information regarding dynamic objects around the vehicle; and a memory that stores terrain objects and static objects included in map data. A 3D map view generating device comprising a processor for generating a 3D map view including a guidance position for guiding a driving path of the vehicle, wherein the processor determines a second area of unobtained visibility where the guidance position is not displayed in a visible state in the 3D map view, and causes the terrain object of the second area to be deformed by changing the position of at least some vertices constituting a terrain object included in the second area based on a defined coordinate transformation function, detects a static object corresponding to the second area from the map data, detects a dynamic object to be located in the second area or displayed corresponding to the second area based on the object information, determines the display position of the static object and the display position of the dynamic object respectively based on a transformation coordinate system corresponding to the deformed terrain object of the second area, and renders the static object and the dynamic object so as to be displayed together on the deformed terrain object to generate a 3D map view including the guidance position, the static object, and the dynamic object. Claim 24 A 3D map view generating device according to claim 23, wherein the processor models building objects that are static objects detected from the map data, places the modeled building object models under the footprint area of each building object included in the modified terrain object of the second area, and displays the building objects on the modified terrain object by rendering at least a part of the building object models protruding from the footprint area according to the driving information of the vehicle. Claim 25 A 3D map view generating device according to claim 23, wherein the processor restricts rendering by restricting the protrusion from the footprint area of a static object among the modified terrain objects of the second area, for a static object included in an area where the height of the modified terrain object exceeds a preset height. Claim 26 A 3D map view generating device according to claim 24, wherein the processor protrudes each building object placed below the footprint area in a direction perpendicular to a horizontal plane parallel to the driving plane of the vehicle, or in a direction perpendicular to the ground shape of the modified terrain object of the second area. Claim 27 A 3D map view generating device according to claim 23, wherein the processor enlarges the size of a static object included in the second area in proportion to the distance from the vehicle, and the enlargement ratio of the static object is determined based on the straight-line distance between the center of the 3D map view camera providing the field of view of the 3D map view and each of the static objects. Claim 28 A 3D map view generating device according to claim 23, wherein the processor sequentially renders the static object and the dynamic object on the modified terrain object after the modified terrain object of the second region is modeled and rendered, and the dynamic object includes surrounding vehicles recognized based on information obtained from an ADAS (Advanced Driver Assistance System). Claim 29 A 3D map view generating device according to claim 23, wherein the processor, when deforming the terrain object of the second region, distorts it into a hemispherical shape such that the height of the ground gradually increases as it moves away from the vehicle centered on the position of the vehicle.
Citation Information
Patent Citations
Display devices and route guidance systems based on mixed reality
KR1020230068439A
3-D map display
US20100305853A1
Method of obtaining a three-dimensional map representation and navigation system
US7002578B1
