A method and apparatus for displaying a 3D map within an augmented reality view and providing content that includes information related to objects within the 3D map.

Abstract

The present invention provides a user terminal control method that displays an AR view including video footage captured by the user terminal's camera, extends the video within the AR view to display a 3D map of an area including the user terminal's location or a location specified by the user, and outputs content containing information related to objects in accordance with the user's interaction with objects in the 3D map.

Description

Technical Field

[0001] The following description relates to a method and apparatus for displaying a three-dimensional map of an area including the position of a user terminal or a position designated by a user through an Augmented Reality (AR) view, and to a method and apparatus for providing content including information related to an object in the three-dimensional map displayed through the AR view.

Background Art

[0002] Augmented Reality (AR) refers to a technology that combines and complements virtual objects and information created by computer technology with the real world. That is, augmented reality is a technology for expanding and displaying virtual content in a real environment, and a user can experience the content expanded corresponding to the real environment through an electronic device.

[0003] A three-dimensional map is constructed by three-dimensionally modeling a real space or a real environment, and is provided to a user through an application or the like that provides a service using such a three-dimensional map. As an example of such a three-dimensional map, "S-Map" constructed as virtual reality by three-dimensionally modeling the entire area of Seoul can be mentioned. A three-dimensional map is used, for example, to provide a user with an experience as if visiting the actual location even without the user going to the actual location shown on the map. A user can, for example, virtually visit a specific area and use the three-dimensional map in work related to policies such as the safety, environment, and urban planning of that area.

[0004] When using a three-dimensional map, there is a need for a technology that effectively provides information about an object (such as a ground feature, etc.) displayed on the three-dimensional map to the user while providing an immersive experience to the user.

[0005] As an example of technologies related to various services utilizing augmented reality technology, Korean Published Patent No. 10-2014-0065963 (published on May 30, 2014) discloses an augmented reality navigator that is installed in a vehicle, displays video of the driving route captured by a camera on a display, and maps virtual display information that guides the driver along the driving route onto the video displayed on the display.

[0006] The information described above is intended solely to aid in understanding the present invention and may include content that is not part of the prior art, and may not include content that would be presented as prior art to a person with ordinary skill in the art to which the present invention belongs. [Overview of the project] [Problems that the invention aims to solve]

[0007] This invention provides a user terminal control method for displaying an AR (augmented reality) view that includes video footage captured by the user terminal's camera. Within the AR view, it displays a 3D map of an area including the user terminal's location and locations specified by the user, augmented from the video footage. In accordance with the user's interaction with objects in the 3D map, it outputs content containing information related to those objects.

[0008] The present invention provides a user terminal control method that outputs audio content containing information describing a first object in a 3D map, and when the audio content outputs information describing a second object on the 3D map, it changes the display of the second object on the 3D map or displays additional AR content associated with the second object.

[0009] The present invention provides a user terminal control method that outputs audio content as content containing information describing a first object in a 3D map, and when the audio content outputs information describing a third object outside the 3D map, it further displays additional AR content showing the third object outside the 3D map. [Means for solving the problem]

[0010] According to one embodiment, a user terminal control method is provided that is executed by a computer system which is a server or a user terminal, and includes the steps of: displaying an augmented reality (AR) view including an image captured by the camera of the user terminal; displaying a three-dimensional map within the AR view, augmented from the image, of an area including the location of the user terminal or a location specified by the user of the user terminal; and outputting content including information related to the first object based on the user's interaction with the first object in the three-dimensional map.

[0011] The three-dimensional map is a three-dimensional model of features actually located in the area, the first object represents at least one of the features, and the three-dimensional map may be extracted from a wider area three-dimensional map that also includes the surrounding area.

[0012] The step of outputting the aforementioned content may involve outputting explanatory content from the user terminal that includes information describing the first object.

[0013] The first object includes real estate properties within the area, and the descriptive content may include information describing the real estate properties.

[0014] The information describing the real estate property may include at least one of the following: information describing a second object in the three-dimensional map in relation to the real estate property, and information describing a third object outside the three-dimensional map in relation to the real estate property.

[0015] The information describing the real estate property includes information describing the second object, and the step of outputting the content may include changing the display of the second object on the 3D map or displaying additional AR content associated with the second object when the information describing the second object of the description content is output from the user terminal.

[0016] The modification of the display of the second object may include at least one of the following: highlighting the second object, and applying motion effects to the second object.

[0017] The additional AR content may include at least one of an image and text describing the second object, distance information from the property to the second object, and route information from the property to the second object.

[0018] The step of changing the display of the second object or displaying additional AR content associated with the second object may include changing the display of the second object or controlling the display of the 3D map so that the additional AR content is displayed in a predetermined area of ​​the AR view.

[0019] The information describing the real estate property includes information describing the third object, and the step of outputting the content may further include displaying additional AR content showing the third object outside the 3D map when the information describing the third object of the description content is output from the user terminal.

[0020] The step of further displaying the additional AR content may include the steps of determining a position in the AR view to display the additional AR content based on the distance between the geographical location of the real estate property and the geographical location of the third object, and displaying the additional AR content at the determined position.

[0021] The step of outputting the content may include adjusting the scale of the three-dimensional map based on the distance between the geographical location of the real estate property and the geographical location of the third object.

[0022] The scale of the 3D map can be automatically adjusted so that the additional AR content and the 3D map are displayed on a single screen of the user's terminal.

[0023] The step of displaying the three-dimensional map may be performed to display the three-dimensional map from a changed viewpoint as the viewpoint of the camera changes.

[0024] Interaction with the first object may include selecting the first object or AR content output in association with the first object, gauging to keep the first object positioned in a certain area of ​​the AR view, or voice commands to invoke information related to the first object.

[0025] The three-dimensional map includes a plurality of objects, and content output is associated with the objects. The method for controlling the user terminal may further include the step of displaying content related to the object indicated by the command in the three-dimensional map when a command for calling information regarding at least one of the objects is received.

[0026] The step of outputting the content may output content including information explaining the first object and other objects around the first object by including azimuth information centered on the first object.

[0027] The first object includes real estate within the area. The method for controlling the user terminal further includes the step of displaying AR content for viewing property information registered for the real estate in association with the first object, and the step of outputting the property information based on an interaction by the user with the AR content. The property information may include audio content and visual content.

[0028] In another embodiment, a computer system constituting a user terminal or a server includes at least one processor configured to execute computer-readable instructions included in a memory. The at least one processor displays an AR (augmented reality) view including video captured by a camera of the user terminal, and in the AR view, expands from the video to display a three-dimensional map of an area including the position of the user terminal or a position designated by a user of the user terminal. A computer system is provided that outputs content including information related to the first object based on an interaction by the user with the first object in the three-dimensional map.

Effects of the Invention

[0029] Within the AR view, a 3D map of the area including the user's device location and user-specified locations is displayed, and content including information about objects within the 3D map is provided. This allows users to easily understand the area through the 3D map, thus providing an immersive experience for users utilizing the 3D map.

[0030] When outputting content that includes information describing objects such as real estate properties within a 3D map, if information describing other objects within the 3D map or even other objects outside the 3D map is displayed, the display of the 3D map can be modified, or additional content can be output within the AR view to provide an intuitive and comprehensive explanation of the objects within the 3D map.

[0031] By displaying a 3D map of a specific area in an AR view and outputting content that includes information describing objects such as real estate within the 3D map, it is possible to provide a way for users to feel as if they have actually visited that area, even without physically going there. [Brief explanation of the drawing]

[0032] [Figure 1] This figure shows a method for displaying a 3D map within an AR view and outputting content related to objects within the 3D map, according to one embodiment. [Figure 2] This figure shows a computer system and server in one embodiment that execute an AR view, display a 3D map within the AR view, and output content. [Figure 3] This flowchart shows a method for controlling a user terminal to display a 3D map within an AR view and output content related to objects within the 3D map, according to one embodiment. [Figure 4]This flowchart shows a method for controlling a user terminal when information describing a second object on a three-dimensional map is displayed among content that includes information describing a first object in a three-dimensional map, according to one embodiment. [Figure 5] This flowchart shows a method for controlling a user terminal when content describing a third object outside a three-dimensional map is displayed, in one embodiment of the content which includes information describing a first object in a three-dimensional map. [Figure 6] This flowchart shows a method for outputting property information registered in association with a property in an AR view, when the object in a 3D map is real estate, according to one embodiment. [Figure 7] This figure shows content containing information describing a first object in a three-dimensional map in one embodiment. [Figure 8] This figure shows a method in one embodiment in which a 3D map is displayed within an AR view and content related to objects within the 3D map is output. [Figure 9] This figure shows a method in one embodiment in which a 3D map is displayed within an AR view and content related to objects within the 3D map is output. [Figure 10] This figure shows a method in one embodiment in which a 3D map is displayed within an AR view and content related to objects within the 3D map is output. [Figure 11] This figure shows a method in one embodiment in which a 3D map is displayed within an AR view and content related to objects within the 3D map is output. [Figure 12] This figure shows a method in one embodiment in which a 3D map is displayed within an AR view and content related to objects within the 3D map is output. [Figure 13] This figure shows a method in one embodiment in which a 3D map is displayed within an AR view and content related to objects within the 3D map is output. [Figure 14]This figure shows a method in one embodiment in which a 3D map is displayed within an AR view and content related to objects within the 3D map is output. [Figure 15] This figure shows a method for outputting property information registered in association with real estate within an AR view, according to one embodiment. [Figure 16] This figure shows objects and points of interest (POIs) included in a 3D map displayed within an AR view in one embodiment. [Figure 17] This figure shows a method in one embodiment for displaying a 3D map within an AR view such that the orientation of the AR view matches the orientation of the 3D map, according to the orientation information of the AR view. [Figure 18] This flowchart shows a method in one embodiment for displaying a 3D map within an AR view so that the orientation of the AR view matches the orientation of the 3D map, and for controlling the user terminal to further display additional information about an external area outside the 3D map. [Figure 19] This flowchart shows a method for displaying additional content or a two-dimensional map as additional information regarding an external area outside of a three-dimensional map, according to one embodiment. [Figure 20] This figure shows a method for matching the coordinates of a 3D map with the coordinates of a 2D map representing the area of ​​the 3D map and its surrounding area, according to one embodiment. [Figure 21] This figure shows a method for displaying a 2D map in an external area outside the 3D map within an AR view, according to one embodiment. [Figure 22] This figure shows a method for displaying a 2D map in an external area outside the 3D map within an AR view, according to one embodiment. [Figure 23] This figure shows a method for displaying a 2D map in an external area outside the 3D map within an AR view, according to one embodiment. [Figure 24] This figure shows a method for displaying POI information as additional content in an external area outside the 3D map within an AR view, according to one embodiment. [Figure 25] This flowchart shows a method for controlling a user terminal in one embodiment to either input first content in an AR view and output first corresponding content on a 3D map, or input second content on a 3D map and output second corresponding content on an AR view. [Figure 26] This flowchart shows a method for controlling a user terminal in one embodiment to input first content in an AR view and output first corresponding content on a 3D map that corresponds to the first content. [Figure 27] This flowchart shows a method for controlling a user terminal in one embodiment to input second content using a 3D map and output a second corresponding content in an AR view. [Figure 28] This figure shows a method in one embodiment for verifying (or confirming) content created at a specific location on a 3D map using an AR view that includes that specific location, and then reflecting the content verified (or confirmed) in the AR view onto the 3D map. [Figure 29] This figure shows a method in one embodiment in which an action to be performed by the user terminal is input as second content at a specific location on a 3D map, and the user terminal performs the action when that specific location is included in the AR view. [Figure 30] This figure shows a method for displaying a 3D map including the current location in an AR view that includes video footage captured at the current location of the user's terminal, according to one embodiment. [Figure 31] This figure shows a method in one embodiment in which a route is input as second content at a specific location on a 3D map, and when that specific location is included in the AR view, the user terminal performs guidance regarding the input route. [Modes for carrying out the invention]

[0033] The embodiments will be described in detail below with reference to the attached drawings.

[0034] Figure 1 shows a method in one embodiment for displaying a 3D map within an AR view and outputting content related to objects within the 3D map.

[0035] In the embodiments described below, "control" of the user terminal 100 may include, for example, using the AR view 10 of the user terminal 100 as shown in Figure 1, displaying content that shows a specific service or information over the image of the AR view 10, displaying a user interface (UI) associated with a specific service on the user terminal 100, displaying a 3D map 20 of an area including the location of the user terminal 100 or a location specified by the user of the user terminal 100 within the AR view 10, and outputting content 35, 42-46 on the 3D map 20, or controlling the settings of the user terminal 100 in order to provide a specific service on the user terminal 100. Furthermore, "control" of the user terminal in the embodiments of this disclosure may include controlling the functions and / or operations of the user terminal 100, which are not limited to the examples described above. At least a portion of such control of the user terminal 100 may be performed by the server 200 shown in the figure.

[0036] In the following detailed explanation, the location of user terminal 100 may refer to the location of the user who possesses user terminal 100. Alternatively, for the sake of clarity, the terms "user" and "user terminal 100" possessed by the user may be used interchangeably.

[0037] In the following detailed description, "extending content and / or information (e.g., indicators) onto the video" may be interpreted, depending on the embodiment, to include overlaying content and / or information onto the video / screen of the AR view 10. The "extended content" displayed in the AR view 10 may also be called "AR content."

[0038] In Figure 1, for the sake of explanation, the screen of user terminal 100 is shown as user terminal 100.

[0039] For example, a user terminal 100, such as a smartphone, may use its camera to take pictures of its surroundings and display an AR view 10 on its screen that includes the images captured by the camera. The user terminal 100 may display a 3D map 20 in at least a portion of the AR view 10, that is, within the AR view 10, extended from the images, which includes the location of the user terminal 100 or a location specified by the user of the user terminal 100.

[0040] In other words, in the AR view 10 of the embodiment, a 3D map 20 of a certain area may be displayed in an expanded form. The 3D map 20 is expanded onto the image of the AR view 10 (the part of the AR view 10 other than the 3D map 20 in Figure 1), as shown in the figure, and may be displayed in a part of the AR view 10. Alternatively, the 3D map 20 may be displayed in the entire area of ​​the AR view 10. The scale of the 3D map 20 may be adjusted (i.e., enlarged or reduced) according to the operation of the user terminal 100. Therefore, the user of the user terminal 100 can obtain information about the area indicated by the 3D map 20 through the 3D map 20.

[0041] The 3D map 20 displayed within the AR view 10 is augmented in space through AR and may provide the user with an immersive experience associated with the area shown by the 3D map 20. For example, the user can experience actually entering the area shown by the 3D map 20 (when the 3D map 20 is sufficiently enlarged and displayed within the AR view 10). The display of the 3D map 20 may also change as the user terminal 100 moves with the user, so that the user can virtually experience and explore the area shown by the 3D map 20. Alternatively, the 3D map 20 may provide a function for the user to virtually experience and explore the area shown by the 3D map 20 in a manner similar to Street View or a Street View service (i.e., a method using screen touch, etc.). For example, the user can use the AR view 10 and the 3D map 20 of this embodiment to virtually visit the area shown by the 3D map 20 or to plan a trip.

[0042] The 3D map 20 may be a 3D model of the features actually located in the area in question. Features are all natural or artificial objects that exist on the ground, and may include, for example, buildings, trees, houses, rivers, roads, etc. The 3D map 20 may contain multiple objects, and each such object may represent at least one or more of the features.

[0043] In one embodiment, the user terminal 100 may output content 35, 42-46 containing information related to the first object 30 in the three-dimensional map 20, based on the user's interaction with the first object 30. The output of content from the user terminal 100 may be in a visual and / or auditory form. For example, if a specific "building" is selected as the first object 30 in the three-dimensional map 20, the user terminal 100 may output content 35 containing information about that building, and further output content 42-46 associated with other objects necessary to describe that building.

[0044] The method by which the user terminal 100 outputs content 35, 42-46 based on its interaction with the first object 30 will be explained in more detail with reference to Figures 3-16.

[0045] On the other hand, the 3D map 20 may be extracted from a wide-area 3D map (of a broad area) that also includes the area surrounding the area shown by the 3D map 20. Such wide-area 3D maps may be provided by external services (e.g., S-Map, Google Earth, Apple Maps, etc.).

[0046] Thus, in this embodiment, the AR view 10 provided through the user terminal 100 can display not only video but also a 3D map 20 of an area including the location of the user terminal 100 or a location specified by the user of the user terminal 100. By interacting with an object 30 included in the 3D map 20, content including information related to that object 30 can be output.

[0047] Figure 2 shows a computer system that performs a method for controlling a user terminal in an embodiment.

[0048] In this embodiment, referring to Figure 1, the user terminal 100 described above is realized by the computer system 2 shown in the figure. too Yes. For example, in computer system 2, a computer program for realizing the method of the embodiment may be installed and executed, and such computer system 2 may perform a method for controlling the user terminal 100 according to the embodiment in accordance with the control of the executed computer program.

[0049] The method for controlling the user terminal 100 according to the embodiment may be executed through a PC-based program or an application specifically for the mobile terminal. For example, the method for controlling the user terminal 100 according to the embodiment may be implemented as an independently operating program, or it may be configured as an in-app application for a specific application and implemented to operate on the specific application. Such a specific application may, for example, be installed on a computer system 2 and execute the method for controlling the user terminal 100 of the embodiment by displaying a 3D map 20 through an AR view 10 and outputting content 35, 42-46.

[0050] Computer system 2 is an electronic device corresponding to the user terminal 100 described above, and may be a smartphone or similar device capable of installing and running applications or programs, as shown in Figure 1. Computer system 2 may also be a PC (personal computer), laptop computer, tablet, IoT (Internet of Things) device, or wearable computer.

[0051] As shown in the figure, the computer system 2 may include a memory 110, a processor 120, a communication interface 130, and an input / output interface 140 as components for performing a method of controlling the user terminal 100.

[0052] Memory 110 is a computer-readable recording medium and may include RAM (random access memory), ROM (read-only memory), and a permanent mass storage device such as a disk drive. Here, the ROM and the permanent mass storage device such as a disk drive may be included in the computer system 2 as separate permanent storage devices distinct from memory 110. Memory 110 may also store an operating system and at least one program code. Such software components may be loaded into memory 110 from a computer-readable recording medium separate from memory 110. Such a separate computer-readable recording medium may include a floppy disk drive, disk, tape, DVD / CD-ROM drive, or memory card. In other embodiments, software components may be loaded into memory 110 via a communication interface 130 that is not a computer-readable recording medium. For example, software components may be loaded into memory 110 of the computer system 2 based on a computer program installed by a file received via a network 160.

[0053] The processor 120 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor 120 by memory 110 or a communication interface 130. For example, the processor 120 may be configured to execute instructions received according to program code stored in a recording device such as memory 110.

[0054] In other words, the processor 120 may manage the components of the computer system 2 and may execute programs or applications used by the computer system 2. For example, the processor 120 may execute an application for performing a method for controlling the user terminal 100 according to the embodiment and may be configured to process data received from the server 200 for controlling the user terminal 100 (for example, data received from the server 200 to display the 3D map 20 through the AR view 10 and output content 35, 42-46). The processor 120 may also process calculations necessary for executing programs or applications and data processing, and may be at least one processor or at least one core within a processor of the computer system 2.

[0055] The communication interface 130 may provide a function for computer system 2 to communicate with other computer systems (not shown) via the network 160. For example, requests, instructions, data, files, etc., generated by the processor 120 of computer system 2 according to program code recorded in a recording device such as memory 110 may be transmitted to other devices via the network 160 under the control of the communication interface 130. Conversely, signals, instructions, data, files, etc., from other devices may be received by computer system 2 via the network 160 through the communication interface 130 of computer system 2. Signals, instructions, data, etc., received via the communication interface 130 may be transmitted to the processor 120 or memory 110, and files, etc., may be recorded on a recording medium (the persistent recording device described above) that computer system 2 may further include. For example, the communication interface 130 may be a hardware module such as a network interface card, network interface chip, and networking interface port of computer system 2, or a software module such as a network device driver or networking program.

[0056] The input / output interface 140 may be a means for interface with the input / output device 150. For example, the input device may include a microphone, keyboard, or mouse, and the output device may include a display or speaker. As another example, the input / output interface 140 may be a means for interface with a device that integrates input and output functions into one, such as a touchscreen. The input / output device 150 may consist of the computer system 2 and one device.

[0057] Furthermore, in other embodiments, the computer system 2 may include more components than those shown in Figure 2. However, it is not necessary to explicitly show most of the conventional components in the figure. For example, the computer system 2 may be implemented to include at least some of the input / output devices connected to the input / output interface 140 described above, or it may further include other components such as transceivers, GPS (Global Positioning System) modules, cameras, various sensors, and databases. As a more specific example, if the computer system 2 is implemented in the form of a mobile device such as a smartphone, the computer system 2 may further include various components commonly found in mobile devices, such as cameras, accelerometers and gyroscopes, various physical buttons, buttons using a touch panel, input / output ports, and vibrators for vibration.

[0058] For example, a computer system 2, which corresponds to a user terminal 100, may include a camera that takes pictures of the surroundings in order to run the AR view 10. The computer system 2 may display the images captured through the camera as the AR view 10. The computer system 2 may extend the images from such an AR view 10 to display a three-dimensional map 20, and may display content 35, 42-46 based on interactions with objects 30 included in the three-dimensional map 20.

[0059] On the other hand, at least some of the operations and calculations performed by the computer system 2 to display the AR view 10 and the 3D map 20 and output content 35, 42-46 may be performed by the server 200 instead of the computer system 2. The server 200 may be an electronic device that communicates with the computer system 2 and provides the computer system 2 with data for controlling the user terminal 100, thereby enabling the user terminal 100 to be controlled. The server 200 may include a database as a device for managing (registering, changing, deleting, recording, maintaining, etc.) the data and content provided to the computer system 2 and the 2D and 3D maps, and may communicate with a database or external service that provides this. The server 200 may also include a map server that provides digital maps (3D maps and / or 2D maps), or may be configured to communicate with such a map server.

[0060] Server 200 may include at least one computer system. The computer system included in Server 200 may have a configuration similar to that of computer system 2 described above, and redundant explanations regarding this will be omitted.

[0061] In this embodiment, the user terminal 100, which is a computer system 2, may communicate with the server 200 and be controlled based on data and / or information provided by the server 200. At this time, the server 200 may receive the location of the user terminal 100 or a location specified by the user terminal, and may control the user terminal 100 to display the 3D map 20 in the AR view 10 based on the received location. Alternatively, the server 200 may recognize interactions with objects 30 in the 3D map 20 at the user terminal 100, and thereby send data to the user terminal 100 for outputting content 35, 42-46.

[0062] In the following detailed explanation, for the sake of clarity, the embodiment will be described primarily from the perspective of the computer system 2, which corresponds to the user terminal 100. Explanations regarding communication with the server 200 and the operation of the server 200 may be simplified or omitted. Furthermore, for the sake of clarity, operations (steps) performed by the configuration of the computer system 2 (or server 200) (e.g., the processor) may be described as being performed by the computer system 2 (or server 200).

[0063] The technical features described above, with reference to Figure 1, can be directly applied to Figure 2 as well; therefore, redundant explanations will be omitted.

[0064] Figure 3 is a flowchart illustrating a method in one embodiment for controlling a user terminal to display a 3D map within an AR view and output objects and related content within the 3D map.

[0065] The method for controlling the user terminal 100, which is performed by the computer system 2, will be described with reference to Figure 3. Since the computer system 2 may be the user terminal 100 described with reference to Figure 1, the following detailed description will use the term "user terminal 100" instead of "computer system 2" to describe the embodiment.

[0066] Furthermore, at least some of the substantial calculations and operations of steps 310 to 620, which will be explained with reference to Figures 3 to 6, may be configured to be performed by the server 200 rather than the user terminal 100. In the following, the execution of each step will be explained focusing on the user terminal 100, and redundant explanations regarding the server 200 will be omitted.

[0067] In step 310, the user terminal 100 may display an augmented reality (AR) view 10 that includes video footage captured by the camera. The AR view 10 may be displayed on the entire screen of the user terminal 100, or in at least a portion of the screen. The AR view 10 may be associated with objects included in the video footage and configured to display augmented content related to those objects.

[0068] In step 320, the user terminal 100 may display a 3D map 20 within the AR view 10, extended from the video, covering an area that includes the location of the user terminal 100 or a location specified by the user of the user terminal 100. The user terminal 100 may display the 3D map 20 in at least a portion of the AR view 10. For example, the user terminal 100 may display the 3D map 20 in the portion of the video in the AR view 10 that corresponds to the ground.

[0069] The 3D map 20 may show an area that includes the current location of the user terminal 100. In this case, the 3D map 20 may act as a minimap for the image of the AR view 10.

[0070] Alternatively, the 3D map 20 may show an area that includes a location specified by the user of the user terminal 100 (regardless of the current location of the user terminal 100). In this case, the user may set the area of ​​the 3D map 20 to be displayed in the AR view 10 through area selection or area search in a separate map application (for example, a map application that provides 2D and / or 3D map services). Alternatively, the user may set the area to be displayed in the AR view 10 on the 3D map 20 through voice search (for example, by speaking an address or administrative district).

[0071] The 3D map 20 may show an administrative area (such as a town or ward) or a block demarcated by roads that includes the current location of the user terminal 100 or a location specified by the user. Alternatively, the extent of the area shown by the 3D map 20 may be set by the user.

[0072] The 3D map 20 may be scaled (i.e., enlarged or reduced) according to operations on the user terminal 100, and the viewpoint may be changed. When displaying the 3D map 20, the user terminal 100 may display the 3D map 20 with a changed viewpoint in accordance with the change in the viewpoint of the camera used to display the AR view 10. Therefore, users using such a 3D map 20 can virtually experience the area shown by the 3D map 20 in a more intuitive way.

[0073] On the other hand, as described above, the 3D map 20 may provide the user with an immersive experience related to the area shown by the 3D map 20. For example, (when the 3D map 20 is sufficiently enlarged and displayed in the AR view 10) the user may be given an experience that makes them feel as if they are actually entering the area shown by the 3D map 20. As an example, the display of the 3D map 20 may change as the user terminal 100 moves with the user, so that the user can virtually experience and explore the area shown by the 3D map 20. Alternatively, the 3D map 20 may provide a function for the user to virtually experience and explore the area displayed by the 3D map 20 using a method similar to Street View or the Street View service (i.e., a method using screen touch, etc.).

[0074] In step 330, the user terminal 100 may output content 35 containing information related to the first object 30 in the 3D map 20, based on the user's interaction with the first object 30. For example, the first object 30 may represent at least one of the features located in the real environment corresponding to the 3D map 20. The first object 30 may be, for example, a POI included in the 3D map 20.

[0075] Alternatively, the first object 30 may include real estate properties within the area indicated by the three-dimensional map 20. Real estate properties may include buildings, apartment complexes, apartment buildings, apartment complexes (including multiple apartment buildings), other houses and land, or parts thereof.

[0076] An interaction with the first object 30 may be selecting the first object 30 from the three-dimensional map 20 displayed in the AR view 10. The selection may be selecting or touching the first object 30 or AR content or icons output in association with the first object 30. The AR content may include the name of the first object 30, such as the name of the apartment complex to which the first object 30 belongs. Alternatively, an interaction with the first object 30 may include a voice command to call up information associated with the first object 30, such as "Tell me about the first object 30." Alternatively, an interaction with the first object 30 may include gazing the first object 30 on the AR view 10. Such gazing may involve positioning and maintaining the first object 30 in a certain area of ​​the AR view (e.g., the central area).

[0077] When an interaction with the first object 30 as described above is recognized, the user terminal 100 may output content 35 containing information related to the first object 30.

[0078] A more specific method for displaying a 3D map 20 within the AR view 10 and providing content 35 containing information related to the first object 30 within the 3D map 20 will be described in more detail with reference to Figures 4-16.

[0079] The technical features described above, with reference to Figures 1 and 2, can be directly applied to Figure 4 as well; therefore, redundant explanations will be omitted.

[0080] Figure 4 is a flowchart illustrating how to control the user terminal when information describing a second object on a 3D map is output from content that includes information describing a first object on a 3D map, in one example.

[0081] Figure 5 is a flowchart illustrating how to control the user terminal when, in content containing information describing a first object within a 3D map, information describing a third object outside the 3D map is displayed.

[0082] When content 35 containing information related to the first object 30 described above is output, such content 35 may include descriptive content containing information describing the first object 30. That is, when the interaction described above is recognized, the user terminal 100 may output such descriptive content. The descriptive content may include text and / or audio content. If the descriptive content includes audio content, such audio content may be played on the user terminal 100 when the interaction is recognized.

[0083] The descriptive content may include information describing the real estate property that falls under the first object 30. The information describing the real estate property may include, for example, basic information about the real estate property, such as information about the surrounding environment of the real estate property, completion date information, actual transaction price information, and information related to the buying / selling / leasing of the real estate property.

[0084] The information describing the above-mentioned real estate property may include at least one of the following: information describing a second object (which is a different object from the first object 30) within the 3D map 20 in association with the real estate property corresponding to the first object 30; and information describing a third object outside the 3D map 20 in association with the real estate property corresponding to the first object 30. Such second and third objects may be objects referenced to describe the surrounding environment of the real estate property corresponding to the first object 30.

[0085] Referring again to Figure 4, we will now describe an example in which the information describing the real estate property includes information describing a second object in the 3D map 20.

[0086] In step 410, when the user terminal 100 outputs content 35 containing information related to the first object 30, it may output information describing the second object.

[0087] In step 420, when information describing the second object of the explanatory content is output from the user terminal 100, the user terminal 100 may change the display of the second object on the 3D map 20 or display additional AR content associated with the second object. For example, when the text of the information describing the second object is displayed on the user terminal 100, or when the audio content of the information describing the second object is played on the user terminal 100, the user terminal 100 may change the display of the second object on the 3D map 20 or display additional AR content associated with the second object.

[0088] In this case, the change in the display of the second object may include at least one of highlighting the second object and applying motion effects to the second object. Therefore, when a user receives information describing a real estate property corresponding to the first object 30, they can visually recognize the second object when information describing the second object, which represents other buildings, rivers, roads, natural objects, convenience facilities, etc., in the vicinity of the property, is displayed.

[0089] On the other hand, the additional AR content displayed in association with the second object may include at least one of the following: an image and text describing the second object; distance information from the real estate property corresponding to the first object 30 to the second object; and route information from the real estate property corresponding to the first object 30 to the second object. The image and text describing the second object may include images, icons, and text describing other buildings, rivers, roads, natural objects, convenience facilities, etc., located around the real estate property corresponding to the first object 30. The distance information may include the distance from the real estate property corresponding to the first object 30 to the other buildings, rivers, roads, natural objects, convenience facilities that are the second object, and the route information may include the travel route from the real estate property corresponding to the first object 30 to the second object.

[0090] In step 430, the user terminal 100 may appropriately control the display of the 3D map 20 when changing the display of the second object or displaying additional AR content associated with the second object in step 420. For example, the user terminal 100 may control the display of the 3D map 20 so that the display of the second object is changed or the additional AR content is displayed in a predetermined area of ​​the AR view. As an example, the display of the 3D map 20 may be controlled so that the display of the second object is changed or the additional AR content is displayed in the center of the AR view 10.

[0091] Thus, in this embodiment, information describing the second object in the 3D map 20 can be output from the user terminal 100 as surrounding information related to the real estate property corresponding to the first object 30, in a way that is both effective and intuitively recognizable to the user.

[0092] Referring again to Figure 5, we will describe an example in which the information describing the real estate includes information describing a third object outside the 3D map 20.

[0093] In step 510, when the user terminal 100 outputs content 35 containing information related to the first object 30, it may output information describing the third object.

[0094] In step 520, the user terminal 100 may further display additional AR content that shows the third object outside the 3D map 20 when information describing the third object of the explanatory content is output from the user terminal 100. For example, as in steps 522 and 524, the user terminal 100 may determine a position to display the additional AR content within the AR view 10 based on the distance between the geographical location of the real estate property corresponding to the first object 30 and the geographical location of the third object, and may display the additional AR content at the determined position.

[0095] On the other hand, the additional AR content displayed in association with the third object may include at least one of the following: an image and text describing the third object; distance information from the property corresponding to the first object 30 to the third object; and route information from the property corresponding to the first object 30 to the third object. The image and text describing the third object may include images, icons, and text describing other buildings, rivers, roads, natural objects, convenience facilities, etc., located around the property corresponding to the first object 30. The distance information may include the distance from the property corresponding to the first object 30 to other buildings, rivers, roads, natural objects, convenience facilities, etc., which are the third object, and the route information may include the travel route F from the property corresponding to the first object 30 to the third object.

[0096] In step 430, the user terminal 100 may appropriately control the display of the 3D map 20 when displaying additional AR content in step 520. For example, the scale of the 3D map 20 may be adjusted based on the distance between the geographical location of the real estate property corresponding to the first object 30 and the geographical location of the third object. At this time, the scale of the 3D map 20 may be automatically adjusted so that the additional AR content and the 3D map 20 are displayed on a single screen of the user terminal 100. Therefore, in this embodiment, the scale of the 3D map 20 can be appropriately adjusted so that the 3D map 20 and the additional AR content are displayed on a single screen.

[0097] Thus, in this embodiment, information describing a third object located outside the 3D map 20 can also be displayed on the user terminal 100 as surrounding information related to the real estate property corresponding to the first object 30, in a way that is both effective and intuitively recognizable to the user.

[0098] The following sections will provide further details, with reference to Figures 7-15, on how the display of the AR view 10 is controlled when content 35 containing information related to the first object is displayed, when information describing the second object is displayed when providing content 35, and when information describing the third object is displayed.

[0099] Figure 7 shows an example of content containing information describing a first object in a 3D map.

[0100] The content 35 shown in the figure may be an example of explanatory content that includes information describing the first object 30 described above. The content 35 may be displayed on the user terminal 100 in text form, or / or additionally played on the user terminal 100 in the form of audio content. Such audio content may be considered to be an "audio guide".

[0101] As shown in the example in the diagram, Content 35 may consist of content such as, "Apartment Complex A is located across the street from Company N and a short distance away from the Tan River, making it a suitable place for walks. On the other side, it is surrounded by Mount Daebong and Mount Jinjae, and you can enjoy a panoramic view of the mountain range from the rooftop. There is also a cafe street nearby where you can enjoy brunch on weekends."

[0102] Here, Apartment Complex A 710 may represent the first object 30 (i.e., the real estate property corresponding to the first object 30). On the other hand, Company N Building 720, Tan River 730, and Cafe Street 750 may each represent the second object corresponding to the surrounding environment of the first object 30. Daebong Mountain and Jinjae Mountain 740 may represent the third object corresponding to the surrounding environment of the first object 30.

[0103] Thus, the content 35 can include information such as the direction the user should move in within the 3D map 20, available roads, destinations corresponding to other objects, and areas of the 3D map 20 to be checked.

[0104] The user can explore the area shown on the 3D map 20 by listening to content 35, which is output in the form of audio content, for example, and controlling the display of the 3D map 20 (for example, by rotating the user terminal 100).

[0105] Figures 8-14 illustrate an example of how a 3D map is displayed within an AR view, and how objects within the 3D map and related content are output.

[0106] As shown in Figure 8, the 3D map 20 may include an icon representing the first object 30 and AR content 810. When the AR content 810 is selected, content 35 may be output to the user terminal 100.

[0107] When, for example, content such as "N Company Building is across the street," is output from the user terminal 100, additional AR content 910 may be displayed on the 3D map 20, as shown in Figure 9. In this case, the building corresponding to N Company Building may correspond to the second object described above. Although not shown in the figure, when content such as "N Company Building is across the street," is displayed on the user terminal 100, the display of the building corresponding to N Company Building, which corresponds to the second object, may be changed, either additionally or alternatively.

[0108] Furthermore, when content 35 such as "The Tan River flows a little further ahead, making this a suitable place for a walk" is displayed on the user terminal 100, additional AR content 1010 may be displayed on the 3D map 20, as shown in Figure 10. In this case, the river corresponding to the Tan River may correspond to the second object described above. Although not shown in the figure, when content such as "The Tan River flows a little further ahead, making this a suitable place for a walk" is displayed on the user terminal 100, the display of the river corresponding to the Tan River, which corresponds to the second object, may be changed additionally or alternatively. For example, a motion effect that simulates a flowing river may be applied to the second object.

[0109] Furthermore, when content 35 such as, "The opposite side is surrounded by Mount Daebong and Mount Jinjae, and you can enjoy a panoramic view of the mountain range from the rooftop," is displayed on the user terminal 100, additional AR content 1110 may be displayed on the 3D map 20, as shown in Figure 11. In this case, the mountains corresponding to Mount Daebong and Mount Jinjae may correspond to the third object described above. Such a third object may not be included within the range of the 3D map 20 and may be displayed outside the 3D map 20.

[0110] Furthermore, when content 35 such as "There is a cafe street nearby, and you can enjoy brunch on weekends" is displayed on the user terminal 100, additional AR content 1410 may be displayed on the 3D map 20, as shown in Figure 14. In this case, the roads and streets corresponding to the cafe street may correspond to the second object described above. Although not shown in the figure, when content such as "There is a cafe street nearby, and you can enjoy brunch on weekends" is displayed on the user terminal 100, the display of objects corresponding to roads and streets that correspond to the second object may be changed, either additionally or alternatively.

[0111] Furthermore, as shown in Figure 14, routes 1420 and / or arrival times 1425 to other buildings, rivers, roads, natural objects, convenience facilities, etc., which correspond to the second object, may be displayed in the AR view 10 as additional AR content. For example, in order to determine accessibility to public transportation (such as subway stations and bus stops) and accessibility to schools, which are important factors in deciding on a real estate transaction, routes and / or arrival times to public transportation stations and routes and / or arrival times to schools may be displayed in the AR view 10 as additional AR content. Such additional AR content may be appropriately output to the user terminal 100 depending on the content of content 35 and the time of output of content 35.

[0112] Furthermore, as shown in Figure 12, information regarding the straight-line distance between the first object 30 and the second object (for example, the distance between buildings, the distance between apartment buildings, etc.) and surrounding roads may be displayed in the AR view 10 as additional AR content 1210. Such additional AR content may be appropriately output to the user terminal 100 depending on the content 35 and the time of output of content 35.

[0113] Furthermore, as shown in Figure 13, information regarding the height and terrain of the first object 30, or hierarchy information (e.g., building height and hierarchy information), may be displayed in the AR view 10 as additional AR content 1310. Such additional AR content may be appropriately output to the user terminal 100 depending on the content of content 35 and the time of output of content 35. Similarly, information regarding the height and terrain of the second object, or hierarchy information (e.g., surrounding building height and hierarchy information), may also be displayed in the AR view 10 as additional AR content.

[0114] As explained with reference to Figures 9-14, the output of additional AR content in the AR view 10 may be synchronized with the output or playback of content 35.

[0115] On the other hand, in this embodiment, when outputting content 35, the user terminal 100 may output content that includes orientation information centered on the first object 30, and information that describes the first object 30 and other objects around the first object (the second and third objects described above).

[0116] For example, content 35 may include cardinal directions (north, south, east, west, or up, down, left, and right) relative to the first object 30 and other objects around the first object (the second and third objects mentioned above). Such cardinal directions may be based on the current location of the user terminal 100.

[0117] For example, among the 35 contents shown in Figure 7, "The N Company building is across the street" could be restructured as "The N Company building is on the right," and "The Tan River flows a little further ahead, making it a suitable place for walks." could be restructured as "The Tan River flows even further to the right, making it a suitable place for walks." and "The opposite side is surrounded by Mount Daebong and Mount Jinjae, and you can enjoy a panoramic view of the mountain range from the rooftop." could be restructured as "The left side is surrounded by Mount Daebong and Mount Jinjae, and you can enjoy a panoramic view of the mountain range from the rooftop."

[0118] Of the additional AR content described with reference to Figures 9-14 above, at least one may be output by the user terminal 100 in response to a command being received by the user terminal 100 to call information corresponding to the additional AR content, regardless of the output of content 35.

[0119] For example, the 3D map 20 may contain multiple objects, and such objects may have associated content outputs. For example, an object may have instructions to cause content to be output, or the content itself may be mapped or tagged. The content may be at least one of text, audio content, and AR content.

[0120] For example, as shown in step 340 of Figure 3, the user terminal 100 may receive a command to retrieve information about at least one of the objects and display the object and related content indicated by the command on the 3D map 20. The displayed content may be at least one of the additional AR content described with reference to Figures 9-14.

[0121] For example, when a command is received requesting route information from a specific object on the 3D map 20 to another object, or when a command is received requesting information about facilities such as schools, additional AR content, as described with reference to Figures 9-14, may be displayed in the AR view 10. The command for displaying the additional AR content may be received by the user terminal 100 with the first object 30 selected. The command may include a voice command (query).

[0122] Thus, the embodiment can be implemented to output additional AR content in the AR view 10 while predetermined explanatory content is being output, and further output additional AR content in the AR view 10 as needed in accordance with user commands.

[0123] The technical features described above, with reference to Figures 1-3, can be directly applied to Figures 4, 5, and 7-14; therefore, redundant explanations will be omitted.

[0124] Figure 6 is a flowchart illustrating a method for outputting property information registered in association with a real estate property in an AR view, in one example where the object in the 3D map is a real estate property.

[0125] In connection with this, Figure 15 is a diagram illustrating how property information registered in association with real estate can be output within an AR view in one example.

[0126] As described above, the first object 30 in the 3D map 20 may include real estate within the area.

[0127] In step 610, the user terminal 100 may display AR content (for example, AR content 1510, or an icon displaying the name of the first object 30) associated with the first object, for viewing property information registered for the real estate property corresponding to the first object 30.

[0128] In step 610, the user terminal 100 may output property information based on the user's interaction with the AR content 1510.

[0129] Property information may include audio and visual content.

[0130] Such property information may be made viewable by touching an icon (for example, AR content 1510) that indicates the property displayed on the real estate corresponding to the first object 30. The property information may include descriptions of the property's area, interior structure, price, and other property features. If the property information is output as audio content, background music may be provided along with it, and the content of the property information provided may differ depending on the characteristics of the property.

[0131] As shown in the figure, the AR content 1510 is an image related to the property, and may include interior photos, etc. The user terminal 100 may select a thumbnail of an image included in the AR content 1510 or enlarge the AR content 1510 to view the image.

[0132] This allows users who need real estate property information to obtain it in an intuitive way, without having to directly visit real estate agents in the area or the property itself.

[0133] The descriptions of the technical features described above, with reference to Figures 1-5 and 7-14, can be directly applied to Figures 6 and 15 as well; therefore, redundant explanations will be omitted.

[0134] Figure 16 shows an example of objects and points of interest (POIs) included in a 3D map displayed within an AR view.

[0135] Figure 16 shows the screen of the user terminal 100 with the aforementioned 3D map 20 enlarged.

[0136] The 3D map 20 may contain multiple objects, and as shown in the figure, POI1610 and the building may each correspond to an object. As shown in the figure, POI1610 may indicate building facilities such as the building entrance and the building's parking lot.

[0137] In this embodiment, when such a POI 1610 is selected, content including information related to that POI 1610 may be displayed on the user terminal 100. Alternatively, even if the POI 1610 is not displayed as an icon, when a specific building is selected in the 3D map 20, content including information related to the object corresponding to the selected building may be displayed on the user terminal 100.

[0138] The technical features described above, with reference to Figures 1-15, can be directly applied to Figure 16 as well; therefore, redundant explanations will be omitted.

[0139] In the embodiment described above, content 35 containing information related to real estate along with the 3D map 10 is output through the AR view 10, and additional AR content is further output in synchronization with the output of content 35, thereby providing users with expert information about real estate and the area shown by the 3D map 10 in an effective manner. For example, a user can virtually visit an area of ​​interest through the 3D map 20 and easily obtain real estate information for that area.

[0140] Figure 17 shows a method in an embodiment for displaying a 3D map within an AR view so that the orientation of the AR view matches the orientation of the 3D map, according to the orientation information of the AR view.

[0141] As explained with reference to Figure 1, a user terminal 100, such as a smartphone, may use its camera to photograph its surroundings and display an AR view 10 on its screen that includes the image captured by the camera. The user terminal 100 may display a 3D map 20 in at least a portion of the AR view 10, that is, within the AR view 10, extending from the image to show an area that includes the location of the user terminal 100 or a location specified by the user of the user terminal 100.

[0142] In this case, the user terminal 100 may extend the image within the AR view 10 and display the 3D map according to the orientation information of the AR view 10 acquired from the server 200 or the user terminal 100. This makes it possible to match the orientation of the image with the orientation of the 3D map 20 within the AR view 10. The orientation information of the AR view 10 may include the cardinal directions (north, south, east, west, or up, down, left, right) of the AR view 10. For example, the north side in the image of the AR view 10 may be the same as north in the 3D map 20 in terms of orientation.

[0143] Thus, in this embodiment, the orientation in the 3D map 20 displayed in the AR view 10 matches the direction in the real world. Therefore, users using the 3D map 20 can clearly recognize the direction of objects within the 3D map 20 based on their current location.

[0144] In this embodiment, as shown in the figure, directional information 1710 may be further displayed near the 3D map 20 within the AR view 10. Therefore, users utilizing the 3D map 20 can more intuitively recognize the orientation of objects within the 3D map 20.

[0145] The technical features described above, with reference to Figures 1-16, are also applicable to Figure 17; therefore, redundant explanations will be omitted.

[0146] Figure 18 is a flowchart illustrating a method in one embodiment for controlling the user terminal to display a 3D map within the AR view so that the orientation of the AR view matches the orientation of the 3D map, and to further display additional information about external areas outside the 3D map.

[0147] In step 1810, the user terminal 100 may display an AR view 10 that includes video footage captured by the camera. The AR view 10 may be displayed on the entire screen of the user terminal 100, or in at least a portion of the screen. The AR view 10 may be associated with objects included in the video footage and configured to display augmented content related to those objects.

[0148] In step 1820, the user terminal 100 may extend the image within the AR view 10 to display a 3D map 20 of the area including the location of the user terminal 100 or a location specified by the user of the user terminal 100, according to the orientation information of the AR view 10 obtained from the server 200 or the user terminal 100.

[0149] Since the same explanations can be applied to 3D map 20 while referring to Figures 1-16, redundant explanations will be omitted.

[0150] Directional information may be received from the server 200 or obtained based on an electronic compass or sensor included in the user terminal 100. The directional information may include east-west-north-south (or up-down-left-right) directional information of the AR view 10.

[0151] In this embodiment, the orientation of the image within the AR view 10 and the orientation of the 3D map 20 may coincide.

[0152] In step 1825, the user terminal 100 may further display directional information 1710 near the 3D map 20 within the AR view 10. The directional information 1710 may include directions indicating the east-west-north-south (or up-down-left-right) directions of the AR view 10. Through the directional information 1710, users utilizing the 3D map 20 can more intuitively recognize the orientation of objects within the 3D map 20.

[0153] When displaying the 3D map 20, the user terminal 100 may change the 3D map 20 to match the changed viewpoint and orientation of the AR view 10's image in response to a change in the viewpoint of the user terminal 100's camera. In this way, the display of the 3D map 20 may be controlled in conjunction with the movement of the user terminal 100 (i.e., the movement of the camera), and therefore, an immersive experience can be provided to the user using the 3D map 20.

[0154] In step 1830, the user terminal 100 may extend the image in the AR view 10 to display additional information about areas outside the 3D map 20 that the 3D map 20 represents. In other words, the user terminal 100 can not only display the 3D map 20 within a certain range of area including the user terminal 100's current location or a location specified by the user, but can also display additional information about areas outside that range in the AR view 10.

[0155] Such additional information may include additional content, such as 2D maps and / or information about external areas.

[0156] In this embodiment, by extending the 3D map 20 using the AR view 10, rather than in a conventional form such as a web or app interface, an immersive experience can be provided to users of the 3D map 20. In other words, users can have an experience as if they were directly entering and inspecting the space shown by the 3D map 20.

[0157] The 3D map 20 in the embodiment may show only a limited area compared to a 3D map provided in the form of a web or app interface. For example, the 3D map 20 extended within the AR view 10 may be provided to show an area that is less than or equal to a pre-set data size or less than or equal to a pre-set area range. Additional information about areas outside the area shown by the 3D map 20 may be mashed up to be provided near the 3D map 20 as data in a lighter and less rendering-intensive format, rather than as part of the 3D map 20 itself.

[0158] Therefore, in this embodiment, depending on the scenario requested by the user (such as obtaining real estate information or travel information), the area corresponding to the user's area of ​​interest is rendered through the 3D map 20. However, information about the surrounding areas outside of that area may be provided near the 3D map 20 by mashing up information such as a 2D map or points of interest extracted from the 2D map, etc. In this way, the user terminal 100 of this embodiment can reduce the load required to render the 3D map 20 and display information about the area of ​​interest and its surrounding areas in an efficient manner.

[0159] For more specific methods on how to display additional information about the external area in AR View 10, please refer to Figures 19-24 for further details.

[0160] The technical features described above with reference to Figures 1-17 can be directly applied to Figure 18 as well; therefore, redundant explanations will be omitted.

[0161] Figure 19 is a flowchart illustrating how to display additional content or a 2D map as additional information about an external area outside the 3D map, in one example.

[0162] In step 1910, the user terminal 100 may obtain additional content and / or a two-dimensional map as additional information regarding the area outside the area shown by the three-dimensional map 20.

[0163] In step 1920, the user terminal 100 may display the additional content and / or a two-dimensional map acquired in step 1910 at the position of the AR view 10 corresponding to the external area. The position of the AR view 10 corresponding to the external area may indicate the relative position of the external area with respect to the three-dimensional map 20 (i.e., the area shown by the three-dimensional map 20).

[0164] To illustrate a more specific example of steps 1910 and 1920, the user terminal 100 may obtain additional content regarding external areas from a map service that provides a two-dimensional map including the external areas, as additional information regarding external areas of the area shown by the three-dimensional map 20. The two-dimensional map may be a commercial map and may be provided by a separate map application (for example, a map application that provides two-dimensional and / or three-dimensional map services).

[0165] The user terminal 100 may display the acquired additional content in an area corresponding to an external area outside the 3D map 20. Such additional content may not be the 2D map itself, but may include information about external areas extracted from the 2D map or obtained from a service that provides the 2D map.

[0166] As an example, additional content may include at least one of the following as real-time information about the external area: traffic information for the external area, air quality information for the external area, and real-time video information for the external area. The real-time video information may include video captured from CCTV or surveillance cameras installed in the external area.

[0167] Such real-time information may be displayed within AR View 10 in the form of numerical data, graphs, charts, etc. Alternatively, such real-time information may be displayed within AR View 10 overlaid on a two-dimensional map showing an external area. In other words, real-time information may be displayed within AR View 10 overlaid on a two-dimensional map, such as traffic information, air quality information, and real-time video information presented on a two-dimensional map. Such real-time information may be obtained from a map service or a separate service that provides real-time information.

[0168] Alternatively, the additional content may include at least one POI located within the external area. Such a POI may be displayed at a location outside the 3D map that corresponds to the POI's location on the 2D map of the external area. In other words, the user terminal 100 may determine the location on the AR view 10 where the POI is displayed, taking into account the relative position of the external area to the 3D map 20 (i.e., the area shown by the 3D map 20), and may display the POI as additional content at the determined location.

[0169] The POIs displayed in AR View 10 may be selected from among the POIs included in the 2D map of the external area. In other words, the user terminal 100 may decide that POIs corresponding to major facilities such as public transport stations (subway stations, bus stops) or schools, or other public facilities, will be displayed in AR View 10. Alternatively, according to the user's selection, the POIs of the selected category may be decided as the POIs displayed in AR View 10. Such categories may include, for example, public facilities category, natural objects category (mountains, rivers, etc.), store category (restaurants, shopping malls, etc.), and housing category (apartments, studio apartments, etc.).

[0170] Next, as a more specific example of steps 1910 and 1920, we will further elaborate on an example of displaying a 2D map in AR view as the aforementioned additional information.

[0171] The user terminal 100 may obtain a two-dimensional map including the external area as additional information regarding the external area of ​​the area shown by the three-dimensional map 20. The two-dimensional map may be a commercial map and may be provided by a separate map application (for example, a map application that provides two-dimensional and / or three-dimensional map services), and the two-dimensional map may be obtained from such a map application or map service.

[0172] The user terminal 100 may display a two-dimensional map including the external area, extended from the video in the AR view 10, as additional information regarding the external area described above. Such a two-dimensional map may be displayed in an area corresponding to the external area outside the three-dimensional map.

[0173] As a result, in AR view 10, a 3D map 20 may be displayed for the user's area of ​​interest, and a 2D map may be displayed for external areas outside the area of ​​interest. The area of ​​interest may refer to the area displayed on the 3D map 20 that the user needs to directly inspect and conduct on-site, while the external area may refer to a certain range of areas surrounding that area.

[0174] The two-dimensional map may include, for example, at least one of the following: a vector map of the external area, a satellite map of the external area, a cadastral map of the external area, a topographic map of the external area, and a schematic diagram of the external area. Such two-dimensional maps may be layered and displayed in AR view 10.

[0175] The 2D map may be selectively displayed in the AR view 10 by being extended from the image, according to the selection from the user terminal 100. In other words, the user terminal 100 may receive a selection from the user for at least one of vector maps, satellite maps, cadastral maps, topographic maps, and sketch maps, or for each of the layers, which the user wants to display in the AR view 10, and may display the 2D map corresponding to the selected layer in the AR view 10.

[0176] The user terminal 100 may display a 2D map overlaid on the 3D map 20. The 3D map 20 may be displayed on top of the 2D map. Therefore, in the area where the 3D map 20 is displayed in the AR view 10, the 2D map does not need to be identified.

[0177] However, depending on the embodiment, a two-dimensional map may be displayed overlaid on the area where the three-dimensional map 20 is displayed. In this case, the two-dimensional map may be displayed in the AR view 10 as semi-transparent.

[0178] On the other hand, the 2D map may be displayed semi-transparently in AR view 10 and configured to minimize interference with the image of AR view 10.

[0179] Furthermore, the 2D map may display additional content as described above.

[0180] As described, in the embodiment, a two-dimensional map and / or additional content may be displayed in the AR view 10 as additional information about the external area.

[0181] The technical features described above, with reference to Figures 1-18, can be directly applied to Figure 19 as well; therefore, redundant explanations will be omitted.

[0182] The following section provides a more detailed explanation of how to display a 2D map and / or additional content in the AR view 10 as additional information regarding the area outside the area shown by the 3D map 20, referring to Figures 20 and 21-24.

[0183] First, Figure 20 shows an example of a method for matching the coordinates of a 3D map with the area of ​​the 3D map and the coordinates of a 2D map showing the area outside of it.

[0184] As shown in the figure, an external area may be identified when the coordinates on the 2D map 2000 and the coordinates on the 3D map 20 match (or sync). In other words, an external area 2020 outside the 3D map 10, which is an external area of ​​the aforementioned area, may be identified when the coordinates of a certain area on the 3D map 20 match the coordinates on the 2D map of the area containing that area 2010. Such coordinate matching and external area identification processing may be performed on the server 200.

[0185] The user terminal 100 may acquire at least one of the following as additional information regarding the external area 2020: a two-dimensional map of the external area 2020 identified by coordinate matching as described above, real-time information regarding the identified external area 2020, and information extracted from the two-dimensional map of the identified external area 2020.

[0186] Once external area 2020 is identified, the position in AR view 10 where additional information about external area 2020 is displayed may also be determined.

[0187] Next, referring to Figures 21-24, we will provide a more detailed explanation of a specific example of displaying additional information regarding the External Area 2020.

[0188] Figures 21-23 illustrate an example of how to display a 2D map in an external area outside the 3D map within an AR view. Figure 24 illustrates an example of how to display POI information as additional content in an external area outside the 3D map within an AR view.

[0189] As shown in Figures 21-23, a two-dimensional map of the external area 2020 identified around the three-dimensional map 20 may be displayed.

[0190] In Figure 21, a regular two-dimensional map (e.g., a vector map) 2100 is displayed around the three-dimensional map 20 within the AR view 10. As shown in the figure, additional information about the external area 2020 may be provided through the two-dimensional map 2100 of the external area 2020 outside the three-dimensional map 20. On the other hand, as shown in the figure, further additional information 2110 may be displayed outside the two-dimensional map 2100. Such further additional information 2110 may include POIs, as shown in the figure. The method for displaying the other additional information 2110 can be similarly applied to the description of the method for displaying additional content described above, so redundant explanations are omitted.

[0191] In Figure 22, a 2D map 2200, which is a satellite map, is displayed around the 3D map 20 within the AR view 10. As shown in the figure, additional information about the external area 2020 may be provided through the satellite map 2200 of the external area 2020 outside the 3D map 20.

[0192] In Figure 23, a 2D map 2300, which is a topographic map (cadastral map), is displayed around the 3D map 20 within the AR view 10. As shown in the figure, additional information about the external area 2020 may be provided through the topographic map 2300 of the external area 2020 outside the 3D map 20. For example, the general topography and land information of the external area 2020 may be understood through the topographic map 2300. As shown in the figure, when displaying the topographic map 2300, the external area 2020 may be set to be wider than when displaying other 2D maps.

[0193] In Figure 24, multiple POIs 2410 are displayed around the 3D map 20 within the AR view 10. POIs 2410 provide additional information about the external area 2020 and may be extracted, for example, from the 2D maps 2100, 2200, and 2300 described above. The method for displaying POIs 2410 is similar to the method for displaying additional content described above, so redundant explanations are omitted.

[0194] The two-dimensional maps 2100, 2200, and 2300 described with reference to Figures 21-23 may be obtained from a map service that provides such two-dimensional maps.

[0195] Furthermore, as mentioned above, additional information regarding the External Area 2020 may be configured to include real-time information. Such real-time information may be obtained from a map service or a separate service that provides real-time information.

[0196] As described with reference to Figures 21-24, in this embodiment, the user terminal 100 may mash up at least one piece of information from a service that provides a 2D map and / or a service that provides real-time information, in addition to the service that provides the AR view 10 or the service that provides the 3D map 20, in order to display additional information about the external area 2020 in the AR view 10, thereby displaying the additional information together with the 3D map 20 in the AR view 10.

[0197] Thus, in this embodiment, additional information such as 2D map data is mashed up, allowing for the provision of richer information about the external area 2020 on the AR view 10. Therefore, users utilizing the AR view 10 and the 3D map 20 can obtain necessary information not only about the area 2010 displayed by the 3D map 20, but also about its external area 2020. This enables them to visit area 2010 more effectively.

[0198] The explanation of the technical features described above with reference to Figures 1-19 can be directly applied to Figures 20-24 as well; therefore, redundant explanations will be omitted.

[0199] The following describes a user interface that allows users to input content for the video / 3D map 20 displayed within the AR view 10, and a method and apparatus for displaying the content input into the user interface on the 3D map 20 / video displayed within the AR view 10 (i.e., displaying the corresponding content), with reference to Figures 25-31.

[0200] In one embodiment, a first UI may be provided for the user to input (i.e., create) content at a selected location within the AR view 10's video, and the created content may be reflected as corresponding content in the 3D map 20 including the selected location. Alternatively, a second UI may be provided for the user to input (i.e., create) content at a selected location within the 3D map 20, and the created content may be reflected as corresponding content in the AR view 10's video displaying the selected location.

[0201] Figure 25 is a flowchart illustrating a method for controlling a user terminal in one embodiment, such as inputting a first content in the AR view and outputting a first corresponding content on a 3D map, or inputting a second content on a 3D map and outputting a second corresponding content on the AR view.

[0202] In step 2510, (1) the user terminal 100 may provide a first user interface (UI) for inputting first content at a selected first position in the image of the AR view 10, which includes the image captured by the camera of the user terminal 100. For example, the user may select (or touch) any position in the image of the AR view 10 for inputting first content. The first UI may be a UI for inputting at least one of the following as first content: audio, favorites, bookmarks, text, and images (including images and / or videos). Alternatively, the first UI may be a UI for inputting an action to be performed on the user terminal 100. Or / additionally, the user terminal 100 may provide a second user interface (UI) for inputting second content at a selected second position in the 3D map 20, which is displayed extended from the image in the AR view 10. For example, the user may select (or touch) any position in the 3D map 20 for inputting second content. The second UI may be a UI for inputting at least one of the following as second content: audio, favorites, bookmarks, text, and images (including images and / or videos). Alternatively, the second UI may be a UI for inputting an action to be performed on the user terminal 100.

[0203] In step 2520, (1) when the user terminal 100 receives the first content via the first UI, it may output the first corresponding content when the 3D map 20 including the first location is displayed in the AR view 10. This ensures that the first content input into the AR view 10's video via the first UI is reflected in the corresponding 3D map 20 and output. (2) When the user terminal 100 receives the second content via the second UI, it may output the second corresponding content in the AR view 10's video including the second location. This ensures that the second content input into the 3D map 20 via the second UI is reflected in the corresponding AR view 10 and output.

[0204] Embodiment (1) of steps 2510 and 2520 described above will be explained in more detail below with reference to Figure 26, and embodiment (2) will be explained in more detail below with reference to Figure 26.

[0205] The technical features described above, with reference to Figures 1-24, can be directly applied to Figure 25 as well; therefore, redundant explanations will be omitted.

[0206] Figure 26 is a flowchart illustrating a method for controlling a user terminal in one example, where the first content is input in the AR view and the corresponding first corresponding content is output on a 3D map.

[0207] Referring to Figure 26, a first UI is provided for inputting first content at a first position within the AR view 10, and an embodiment in which the first content is input at the first position will be described in more detail.

[0208] In step 2610, the user terminal 100 may display an AR view 10 that includes the video captured by the camera. The AR view 10 may include a first position.

[0209] In step 2620, the user terminal 100 may receive input of first content via the first UI at the first position.

[0210] The first position may be, for example, any position within the video of AR view 10. Alternatively, the first position may be a position within the video of AR view 10 that is pre-configured to allow input of the first content. For example, the first position may be at least one of the POIs (Points of Interest) displayed in the video, and the selected first position may correspond to the selected POI. In this case, the first content may be various types of content that are input in association with the selected POI. Alternatively, the first position may correspond to an object displayed in the video, or a selected position on an object. For example, the first position may include a building, a building sign, a selected position on the exterior wall of a building, a building entrance, a building parking lot entrance, etc.

[0211] The user may select (or touch) such a first location within the video, thereby allowing the user terminal 100 to invoke a first UI and provide an environment for the user to input first content. For example, the first UI may be a UI for inputting at least one of the following as first content: audio, favorites, bookmarks, text, and images. Alternatively, the first UI may be a UI for inputting an action to be performed on the user terminal 100.

[0212] In step 2630, the user terminal 100 may display a 3D map 20 including the first location within the AR view 10, depending on the location of the user terminal 100 or a request from the user of the user terminal 100. The request may be for displaying a 3D map 20 of an area including a location specified by the user within the AR view 10.

[0213] In step 2640, the user terminal 100 may output a first corresponding content associated with the first location in the 3D map 20 that includes the first location. For example, the first corresponding content may be output associated with the same first location in the 3D map 20 as the first location into which the first content of the AR view 10's video was input, or with an object in the 3D map 20 that includes the first location. The first corresponding content includes the input first content and may be a modified version of the first content for output associated with the 3D map 20. For example, the first corresponding content may be a modified version of the first content for display on the 3D map 20. Alternatively, the first corresponding content may be a different type of content from the first content. For example, if the first content is text, the first corresponding content may be audio content. Such first corresponding content may be automatically output when the 3D map 20 including the first location is displayed in the AR view 10, or output on the user terminal 100 when the first location or an object including the first location is selected in the 3D map 20, or when an icon or AR content indicating the first corresponding content is selected in the 3D map 20.

[0214] The user terminal 100 may determine whether it is at the second location by positioning. When the user terminal 100 is at the second location, it may output the second corresponding content associated with the second location in the AR view 10's video. If the AR view 10's video includes the second location, the user terminal 100 may be determined to be at the second location according to the positioning.

[0215] Thus, in this embodiment, the location where the second content is input on the 3D map 20 can be linked (or synced) with the location in the AR view 10 determined by positioning. Therefore, the user can load the 3D map 20 including the second location and create the second content at the second location without having to visit the second location directly, and can actually move to the second location and check the corresponding second content (corresponding to the created second content) through the AR view 10.

[0216] For example, the first content may include at least one of audio, favorites, bookmarks, text, and images. For instance, the first content may be at least a portion of notes or checklists entered for a specific property or object. In this case, the output of the first corresponding content may include at least one of the following: playback of the audio, display of the text, and display of the image. In other words, in this embodiment, notes and checklists entered by the user for a specific property or object through the AR view 10 can be viewed through the corresponding 3D map 20.

[0217] As another example, the route that the user terminal 100 travels may be input as first content through the first UI. In such an example, the first position may be the starting point of the route that the user terminal 100 travels. For example, the first UI or the user terminal 100 may record the movement of the user terminal 100 from the first position, which is the starting point of the route. The user terminal 100 may record the position coordinates of the user terminal 100's movement by positioning, or by recording the AR view 10. The first UI may provide such a record. Such a record may be terminated when the user terminal 100 reaches a predetermined destination. For example, the record may be terminated when the user inputs through the first UI that they have arrived at the destination, or when the user inputs a request to stop recording to the first UI. This allows the input first content to include information about the route from the starting point to the destination.

[0218] The user terminal 100 may display the route from the starting point to the destination within a three-dimensional map 20 that includes the first position (starting point). In this embodiment, the user can view the AR view 10 and confirm the route they actually traveled through the corresponding three-dimensional map 20. The route may consist of nodes (indicating points specified by the user terminal 100 during travel or points where the user stayed for a certain period of time) and edges connecting the nodes, and such nodes and edges may be displayed on the three-dimensional map 20.

[0219] In step 2650, the user terminal 100 may further provide a third UI for inputting additional content for the first corresponding content displayed on the 3D map 20. The third UI may be a UI for inputting at least one of the following as additional content: audio, favorites, bookmarks, text, and images. Alternatively, the third UI may be a UI for inputting an action to be performed on the user terminal 100.

[0220] In step 2660, when additional content is input via the third UI, the user terminal 100 may reflect that additional content in the AR view 10 including the first position. This allows both the first content and the additional content to be displayed in the video of the AR view 10 (or associated with the first position in the video).

[0221] For example, a user may move while viewing the image on AR View 10, enter a memo at the first location, then return home and display a 3D map 20 including the first location on AR View 10, and confirm the memo at the first location on the 3D map 20 on AR View 10. Alternatively, the user may enter additional memos to the memo displayed at the first location on such 3D map 20, and then, when the user actually moves to the first location so that the image on AR View 10 includes the first location, the additional memo may be displayed in association with the first location in the image along with the existing memo.

[0222] Therefore, while running the AR view 10, users can organize and review notes they have recorded while moving around in real space through the 3D map 20. In addition, they can display any further items that need to be checked on the 3D map 20 as notes and easily check them in real space through the AR view 10.

[0223] The technical features described above, with reference to Figures 1-25, can be directly applied to Figure 26 as well; therefore, redundant explanations will be omitted.

[0224] Figure 27 is a flowchart illustrating a method for controlling a user terminal in an embodiment to input a second content on a 3D map and output a second corresponding content in an AR view.

[0225] Referring to Figure 27, a second UI is provided for inputting second content at a second location within the 3D map 20, and a specific example of what happens when the second content is input at the second location will be described in more detail.

[0226] In step 2710, the user terminal 100 may display a 3D map 20 including the second location within the AR view 10, depending on the location of the user terminal 100 or a request from the user of the user terminal 100. The area shown by the 3D map 20 may include the second location.

[0227] In step 2720, the user terminal 100 may receive input of second content via the second UI at the first position.

[0228] The second location may be any location within the 3D map 20, for example. Alternatively, the second location may be a location within the 3D map 20 that is pre-configured to allow input of the second content. For example, the second location may be at least one of the POIs displayed on the 3D map 20, and the selected second location may correspond to the selected POI. Alternatively, the second location may correspond to an object displayed on the 3D map 20, or a selected location on an object. For example, the second location may include a building, a building sign, a selected location on the exterior wall of a building, a building entrance, a building parking lot entrance, etc.

[0229] In this case, the second content may be various types of content that are entered in relation to the selected POI or selected location.

[0230] The user may select (or touch) such a second location within the 3D map 20, and accordingly, the user terminal 100 may invoke a second UI to provide an environment for the user to input second content. For example, the second UI may be a UI for inputting at least one of the following as second content: audio, favorites, bookmarks, text, and images. Alternatively, the second UI may be a UI for inputting an action to be performed on the user terminal 100.

[0231] In step 2730, the user terminal 100 may display an AR view 10 that includes video footage captured by the camera. The AR view 10 may actually be run in an environment that includes a second location, and therefore the video footage of the AR view 10 may include the second location.

[0232] In step 2740, if the AR view 10's video contains a second location, the user terminal 100 may output a second corresponding content associated with the second location in the video. For example, the second corresponding content may be output associated with the same second location in the AR view 10's video as the input second location of the 3D map 20, or with an object in the video that contains that second location. The second corresponding content may include the input second content and may be a modified version of the second content for output associated with the video in the AR view 10. For example, the second corresponding content may be a modified version of the second content for display in the video in the AR view 10. Alternatively, the second corresponding content may be a different type of content from the second content. For example, if the second content is text, the second corresponding content may be audio content. Such second-compatible content may be automatically output when it is included in the video of AR view 10 which includes the second position, or it may be output on the user terminal 100 when the second position or an object including the second position is selected in the video of AR view 10, or when an icon or AR content in the video of AR view 10 which indicates the second-compatible content is selected.

[0233] The user terminal 100 may determine whether it is located at the second location by positioning. If the user terminal 100 is located at the second location, it may output the second corresponding content associated with the second location in the AR view 10's video. If the AR view 10's video includes the second location, it may be determined, based on positioning, that the user terminal 100 is located at the second location.

[0234] Thus, in this embodiment, the location where the second content is input on the 3D map 20 can be linked (or synced) with the location in the AR view 10 determined by positioning. Therefore, the user can load the 3D map 20 including the second location and create the second content at the second location without directly visiting the second location, and can actually move to the second location and confirm the corresponding second content (corresponding to the created second content) through the AR view 10.

[0235] For example, the second content may include at least one of audio, favorites, bookmarks, text, and images. For instance, the second content may be at least part of notes or a checklist entered for a specific property or object. In this case, the output of the second corresponding content may include at least one of the following: playback of the audio, display of the text, and display of the image. In other words, in this embodiment, a user can make notes through the 3D map 20 about things to check in the real environment regarding a specific property or object, and when they move to the real environment and run the AR view 10, they can view these notes through the video.

[0236] As another example, the second content may be the viewpoint or perspective that the user is facing in the 3D map 10. An image of the 3D map 10 from such a specific viewpoint or perspective may be captured and input as the second content.

[0237] The 3D map 20, including the second location, may be displayed within the AR view 10 as a minimap of the area including the second location. Such a 3D map 20 may serve as supplementary reference data for the user's movement. Alternatively, as described above, the 3D map 20 may be displayed on the full screen of the user terminal 100 to provide the user with an immersive experience.

[0238] As in step 2742, if the AR view 10's video includes a second location, the user terminal 100 may play or display content, perform a predetermined action, or perform a function such as route guidance (route guidance may be an example of an action).

[0239] The following describes in more detail an example where the second content includes a predetermined action performed by the user terminal 100. In this case, the user terminal 100 may perform a corresponding action if the image of the AR view 10 includes the second location.

[0240] The actions performed on the user terminal 100 may include, for example, i) providing route guidance from the first point to the second point corresponding to the second location, ii) starting recording / recording of the AR view 10, iii) interacting with an IoT device placed in association with the second location, iv) calling a specific contact of the user terminal 100, or v) providing real-time information associated with the second location. Alternatively, the execution of the actions may be at least one of the actions i) to v).

[0241] i) The first point may be the starting point of the route, and the second point may be the destination of the route. For example, if the first point is the entrance to an apartment and the second point is a public transport station or school, in this embodiment, a route guidance action may be entered at the entrance to the apartment in the 3D map 20, and when the AR view 10 is actually executed at the entrance to the apartment, route guidance from the entrance to the apartment to the public transport station or school may be provided.

[0242] ii) The recording of the AR view 10 may be the recording of the video of the AR view 10 or the recording of the movement of the user terminal 100. When the AR view 10 is executed and the video includes the second position, such recording of the AR view 10 may begin.

[0243] iii) The interaction with IoT devices may be interaction with IoT devices located in the real environment around the second location. For example, if the real environment is a construction site or work site, the IoT device may be a device for calling a specific worker. In such an example, when the AR view 10 is executed and the second location is included in the image, the user terminal 100 and the IoT device may interact, thereby performing the task of calling a specific worker. On the other hand, the IoT device may be a lighting device or an audio device, and thereby, when the AR view 10 is executed and the second location is included in the image, the user terminal 100 may be controlled to operate the lighting device or audio device.

[0244] iv) A call to a specific contact on the user terminal 100 may be a request to make a call or send a message to a specific contact, a specific phone number, or a specific user (such as a friend) registered in a messenger application, which are stored on the user terminal 100. For example, a user visiting an area for a real estate transaction can automatically contact a real estate agent or the like through such an embodiment by running AR view 10 when they arrive at a specific second location.

[0245] The real-time information in v) may include at least one of the following: weather information, traffic information, and atmospheric information for the area associated with the second location. The real-time information may be obtained from an external map service or another external service that provides such real-time information. As an example, a user visiting an area can, through such an embodiment, display on the 3D map 20 the location for which they want to check atmospheric information or traffic information, and then, when they actually move to that location, they can check the atmospheric information or traffic information through the AR view 10.

[0246] The action corresponding to the provision of real-time information described above may be executed only if the time in which the second location is included in the AR view 10's video falls within a predetermined time period. Such a time period may be set by the user or may be a pre-set time period. For example, it may be implemented so that when the second location is displayed in the AR view 10's video during a specific time period, atmospheric conditions are checked, and when the second location is displayed in the AR view 10's video during other specific time periods, traffic conditions are checked. On the other hand, the real-time information described with reference to Figures 1 to 26 may also include solar radiation information, noise information, and other information in addition to the examples described above.

[0247] Below, as another example, we will describe in more detail how the route taken by the user terminal 100 is input as second content through a second UI. In such an example, the second location may be the starting point of the route specified on the 3D map 20 which includes the second location. For example, the second UI may be a UI for the user to input the route on the 3D map 20 from the second location, which is the starting point of the route, to the destination. As a result, the input second content will include information about the route from the starting point to the destination.

[0248] At least a portion of the aforementioned route may be specified by the user of the user terminal 100 via a second UI on a three-dimensional map 20 including a second location. For example, the user may specify at least a portion of the route by drawing lines or selecting roads on the three-dimensional map 20, or by selecting or inputting at least one of a starting point and a destination.

[0249] Alternatively, as described above regarding the 3D map 20, the 3D map 20 including the second location may be displayed on the user terminal 100 so that the user of the user terminal 100 can virtually explore the area including the second location. For example, the user may virtually explore the area by moving while the 3D map 20 is zoomed in, or by virtually exploring the area by touching the screen. At least a portion of the aforementioned route may be a route that has been automatically recorded by the user virtually exploring on the 3D map 20 including the second location (i.e., a route that has been tracked and recorded by the user virtually exploring). This ensures that the route virtually explored by the user is input as second content without the user having to directly specify a route on the 3D map.

[0250] If the user terminal 100 includes a second location which is the starting point of the route in the AR view 10, it may provide guidance regarding the route from the starting point corresponding to the second location to the destination in the AR view 10's video. The route guidance may be provided through AR content displayed in the AR view 10's video.

[0251] Next, additional embodiments will be described with reference to steps 2750 and 2760.

[0252] In step 2750, the user terminal 100 may further provide a third UI for inputting additional content for the second corresponding content output associated with the second position in the video of the AR view 10. The third UI may be a UI for inputting at least one of the following as additional content: audio, favorites, bookmarks, text, and images. Alternatively, the third UI may be a UI for inputting an action to be performed on the user terminal 100.

[0253] In step 2760, when additional content is input via the third UI, the user terminal 100 may reflect that additional content on the 3D map 20 including the second location. In other words, when additional content is input via the third UI, the user terminal 100 may output the additional content in association with the second content at the second location of the 3D map 20 including the second location when displaying the 3D map 20 including the second location. This allows both the second content and the additional content to be displayed on the 3D map 20 (or in association with the second location of the 3D map 20).

[0254] For example, a user may virtually visit an area on the 3D map 20 and enter a memo at a second location, and then actually move to that area and run AR View 10 to investigate the area. At this time, the user can check the memo through the second location in the AR View 10 image. Alternatively, the user may enter additional memos on the memo displayed in the AR View 10 image, and then when the user returns home and uses the user terminal 100 to display the 3D map 20 including the second location in AR View 10, the 3D map 20 may display the additional memo in addition to the existing memo.

[0255] Therefore, users can check the notes recorded on the 3D map 20 while moving through the real world by running the AR view 10, and furthermore, they can organize and check the notes recorded in the AR view 10's video while moving through the real world via the 3D map 20.

[0256] The technical features described above, with reference to Figures 1-26, can be directly applied to Figure 27 as well; therefore, redundant explanations will be omitted.

[0257] The first position where the first content is entered and the second position where the second content is entered can, of course, be multiple locations.

[0258] On the other hand, the three-dimensional map 20 of the embodiment (i.e., including a first and / or second location, on which content has been created) may be shared with other user terminals. In other words, such a three-dimensional map 20 may be displayed on other user terminals in response to the location of the other user terminal or a request from the user of the other user terminal. As a result, the first corresponding content may be output in association with the first location in the three-dimensional map 20 displayed on the other user terminal. According to such an embodiment, for example, when a first user inputs first content at a first location in the AR view 10, and a second user displays the three-dimensional map 20 including that first location on their own user terminal, the three-dimensional map 20 may output the first corresponding content corresponding to the first content. A similar explanation may apply to the second content and the second corresponding content. Thus, in this embodiment, content created for the three-dimensional map 20 and / or the image of the AR view 10 can be shared among users (e.g., among friends or family).

[0259] For the sharing of such content and for the content linking between the AR view 10 and the 3D map 20 described above, the user terminal 100 may send the first content input via the first UI to the cloud server along with the 6DoF information of the first position. The user terminal 100 may also send the second content input via the second UI to the cloud server along with the 6DoF information of the second position. The cloud server may be the server 200 described above. In this way, since the first position and the second position can have three-dimensional spatial information (including, for example, height and rotation information), the content between the AR view 10 and the 3D map 20 can be linked more precisely. For example, the first content input to the first position in the image of the AR view 10 may be reflected as the first corresponding content at the (more accurate) first position on the 3D map 20, taking into account the three-dimensional spatial information of the first position. That is, the positioning between the AR view 10 and the 3D map 20 can be linked more accurately.

[0260] On the aforementioned cloud server, a positioning system may be implemented to link (or synchronize) the location within the 3D map 20 with the location in real space. For example, through such a positioning system, the location on the 3D map 20 (e.g., the location where the content was created) may be synchronized with the location in real space that indicates that location (or the location within the AR view 10). Therefore, the location in real space determined by the positioning of the user terminal 100 can be synchronized with the location on the 3D map 20 of the embodiment. In this way, by synchronizing the location on the 3D map 20 with the location in real space, content or actions input at a specific location on the 3D map 20 can be output at the corresponding location in real space.

[0261] The 3D map 20 is a 3D model of the space (environment) and may be reconstructed based on the results of mapping the space. Based on such spatial mapping results, a SLAM map for spatial positioning may be generated. Furthermore, the 3D map 20 may be configured to include positional information in a top-down direction. This makes it possible to realize the 3D map 20 so that users can experience more immersive virtual tours and field surveys. Consequently, the content between the AR view 10 and the 3D map 20 can also be linked more precisely.

[0262] Below, we will explain in more detail specific examples of content linkage between AR view 10 and 3D map 20, referring to Figures 28-31.

[0263] Figure 28 illustrates an example of a method in which content created at a specific location on a 3D map is viewed (or confirmed) in an AR view that includes that specific location, and the content viewed (or confirmed) in the AR view is then reflected on the 3D map.

[0264] The first diagram in Figure 28 illustrates a case where a memo 2810, "Check the slope of the parking lot entrance," is registered as a second content at a second location on the 3D map 20 (for example, the entrance to the parking lot of an apartment complex). In this way, users can identify locations and items that need to be checked at the actual site through the 3D map 20 and leave them as memos on the 3D map 20.

[0265] The second diagram in Figure 28 illustrates a case where the AR view 10's image includes a second location (for example, the entrance to an apartment complex's parking lot). In this case, the AR view 10's image may display the memo "Check the slope of the parking lot entrance" (i.e., the second corresponding content) 2820 entered in the first diagram. Therefore, the user can remember and confirm the actual slope of the parking lot entrance through the memo 2820.

[0266] The third diagram in Figure 28 illustrates the case where additional content is added to the memo "Checked the slope of the parking lot entrance" (i.e., second corresponding content) 2820 output to the AR view 10's video. Specifically, the memo "Checked the slope of the parking lot entrance - OK" (i.e., second corresponding content) 2830 further includes the additional content "- OK" indicating that the slope of the parking lot entrance has actually been checked.

[0267] The fourth figure in Figure 28 illustrates a case where the memo 2840 displayed at the second location on the 3D map 20 (the entrance to the apartment complex parking lot) also includes the aforementioned additional content. In this way, the additional content "-OK," which indicates that the slope of the parking lot entrance has been confirmed, may also be reflected in the 3D map 20.

[0268] As described above, the 3D map 20 may be shared with other users' terminals, and therefore other users can also see the reflection of such additional content and the creation of second content through the 3D map 20.

[0269] FIG. 29 is a diagram showing a method in which an action executed by a user terminal as second content is input at a specific position on a 3D map, and the user terminal executes the action when the specific position is included in an AR view.

[0270] In FIG. 29, a case where the "action" executed by the user terminal 100 is providing route guidance is illustrated. For example, as shown in the figure, an action to execute route guidance from a selected point (second position, parking lot entrance) to a specific destination (Company N building) may be input as second content in the 3D maps 20, 2910. When the user terminal 100 moves to the second position and the second position is included in the AR views 10, 2920, the user terminal 100 may provide route guidance from the second position to the destination.

[0271] FIG. 30 is a diagram showing a method of displaying a 3D map including the current location in an AR view including an image captured at the current location of a user terminal in an example.

[0272] In FIG. 30, a 3D map 20 of an area including the current location of the user terminal 100 is displayed in the AR view 10. As shown in the figure, the same building is displayed in the image of the AR view 10 and the 3D map 20. A note "logo confirmation" (second content) may be input as second content at the position of the signboard of the building in the 3D map 20, and thus, "logo confirmation" (second corresponding content) may be similarly displayed in the same building in the image of the AR view 10. When inputting the second content, the 3D spatial information of the input second position is transmitted to the server 200 together, so that the second corresponding content is displayed at the exact position corresponding to the second position in the image of the AR view 10.

[0273] FIG. 31 is a diagram showing a method in which a route is input as second content at a specific position on a 3D map, and the user terminal executes guidance regarding the input route when the specific position is included in an AR view.

[0274] In the example shown in the figure, in the 3D map 20, a memo "Route check from the entrance of the second housing complex to the bus stop" 3110 and a route 3120 are input as the second content at the starting point (second position) of the route. Such a route and memo may be input before the user actually visits the area. When the user moves to the actual position corresponding to the starting point and executes the AR view 10 on the user terminal 100, as the second corresponding content, a memo "Route check from the entrance of the second housing complex to the bus stop" and route guidance to the destination may be provided.

[0275] On the other hand, the specified route may include a plurality of intermediate stops other than the destination, unlike that shown in the figure. When the user terminal 100 reaches an intermediate stop, AR content indicating that the user has reached the intermediate stop may be displayed in the AR view 10.

[0276] As described while referring to FIGS. 25 to 31, an AR to AR content production system between the AR view 10 and the 3D map 20 can be realized by the content linkage method between the AR view 10 and the 3D map 20.

[0277] As described above, the description of the technical features described while referring to FIGS. 1 to 27 can be directly applied to FIGS. 28 to 31, so overlapping descriptions are omitted.

[0278] The AR view 10 and the 3D map 20 according to the embodiment described while referring to FIGS. 1 to 31 can be used for virtual visits, virtual tours, travel planning, management of construction / work sites, and the like.

[0279] The above-described apparatus may be implemented by hardware components, software components, and / or combinations of hardware and software components. For example, the apparatus and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, ALUs (arithmetic logic units), digital signal processors, microcomputers, FPGAs (field programmable gate arrays), PLUs (programmable logic units), microprocessors, or various devices capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications running on the OS. The processing unit may also respond to software execution, access data, record, manipulate, process, and generate data. For convenience of understanding, it may be described as if a single processing unit is used, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0280] Software may include computer programs, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or which may instruct the processing unit independently or collectively. Software and / or data may be embodied in any kind of machine, component, physical device, computer recording medium, or device for interpretation based on the processing unit or for providing instructions or data to the processing unit. Software may be distributed across a networked computer system, and may be recorded or executed in a distributed manner. Software and data may be recorded on one or more computer-readable recording media.

[0281] The methods according to the embodiment may be implemented in the form of program instructions executable by various computer means and recorded on a computer-readable medium. In this case, the medium may continuously record computer-executable programs or may temporarily record them for execution or download. Furthermore, the medium may be various recording or storage means in the form of a combination of one or more hardware components, and may be a medium directly connected to a computer system or distributed on a network. Examples of mediums include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and devices configured to record program instructions such as ROM, RAM, and flash memory. Other examples of mediums include recording media and storage media managed by app stores that distribute applications, and sites and servers that supply and distribute various other software.

[0282] As described above, embodiments have been explained based on limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations from the above description. For example, the described technique may be performed in a different order than described, and / or the components of the described system, structure, apparatus, circuit, etc. may be combined or assembled in a different manner than described, or opposed or replaced by other components or equivalents, and still achieve suitable results.

[0283] Therefore, even if the embodiment is different, it falls within the scope of the attached claims if it is equivalent to the claims.

Claims

1. A user terminal control method executed by a computer system which is a server or a user terminal, The steps include: displaying an AR view that includes images captured by the camera of the user terminal; Within the AR view, the steps include displaying a three-dimensional map of an area including the location of the user terminal or a location specified by the user of the user terminal, extended from the video; The steps include outputting content containing information related to the first object based on the user's interaction with the first object in the three-dimensional map, Includes, A user terminal control method in which interaction with the first object includes selecting the first object or AR content output in association with the first object, gauging to maintain the first object in a certain area of ​​the AR view, or a voice command to invoke information related to the first object.

2. The three-dimensional map is a three-dimensional model of the features actually located in the area, and the first object represents at least one of the features. The user terminal control method according to claim 1, wherein the three-dimensional map is extracted from a wide-area three-dimensional map that also includes the area surrounding the area.

3. The user terminal control method according to claim 1, wherein the step of outputting the content is to output explanatory content from the user terminal that includes information describing the first object.

4. The user terminal control method according to claim 3, wherein the first object includes real estate properties within the area, and the descriptive content includes information describing the real estate properties.

5. A user terminal control method executed by a computer system which is a server or a user terminal, The steps include: displaying an AR view that includes images captured by the camera of the user terminal; Within the AR view, the steps include displaying a three-dimensional map of an area including the location of the user terminal or a location specified by the user of the user terminal, extended from the video; The steps include outputting content containing information related to the first object based on the user's interaction with the first object in the three-dimensional map, Includes, The step of outputting the content involves outputting explanatory content containing information describing the first object from the user terminal, The first object includes real estate properties within the area, and the descriptive content includes information describing the real estate properties. A user terminal control method wherein the information describing the real estate property includes at least one of the following: information describing a second object in the three-dimensional map in relation to the real estate property, and information describing a third object outside the three-dimensional map in relation to the real estate property.

6. The information describing the aforementioned real estate property includes information describing the second object, The step of outputting the aforementioned content is: The user terminal control method according to claim 5, further comprising the step of changing the display of the second object in the three-dimensional map or displaying additional AR content associated with the second object when information describing the second object of the explanatory content is output from the user terminal.

7. The user terminal control method according to claim 6, wherein the change in the display of the second object includes at least one of highlighting the second object and motion effect processing the second object.

8. The user terminal control method according to claim 6, wherein the additional AR content includes at least one of an image and text describing the second object, distance information from the property to the second object, and route information from the property to the second object.

9. The step of changing the display of the second object or displaying additional AR content associated with the second object is: The user terminal control method according to claim 6, comprising the step of changing the display of the second object or controlling the display of the three-dimensional map so that the additional AR content is displayed in a predetermined area of ​​the AR view.

10. The information describing the aforementioned real estate property includes information describing the third object, The step of outputting the aforementioned content is: The user terminal control method according to claim 5, further comprising the step of displaying additional AR content showing the third object outside the three-dimensional map when information describing the third object of the explanatory content is output from the user terminal.

11. The step of displaying the additional AR content is: The steps include determining the position in the AR view where the additional AR content will be displayed based on the distance between the geographical location of the real estate property and the geographical location of the third object, The steps include displaying the additional AR content at the determined location, The user terminal control method according to claim 10, including the method described in claim 10.

12. The step of outputting the aforementioned content is: The user terminal control method according to claim 11, further comprising the step of adjusting the scale of the three-dimensional map based on the distance between the geographical location of the real estate property and the geographical location of the third object.

13. The user terminal control method according to claim 12, wherein the scale of the three-dimensional map is automatically adjusted so that the additional AR content and the three-dimensional map are displayed on a single screen of the user terminal.

14. The user terminal control method according to claim 1, wherein the step of displaying the three-dimensional map is to display the three-dimensional map from a changed viewpoint as the viewpoint of the camera is changed.

15. A user terminal control method executed by a computer system which is a server or a user terminal, The steps include: displaying an AR view that includes images captured by the camera of the user terminal; Within the AR view, the steps include displaying a three-dimensional map of an area including the location of the user terminal or a location specified by the user of the user terminal, extended from the video; The steps include outputting content containing information related to the first object based on the user's interaction with the first object in the three-dimensional map, Includes, The aforementioned three-dimensional map contains multiple objects, The aforementioned object has content output associated with it. The user terminal control method is as follows: A user terminal control method further comprising the step of receiving a command to retrieve information about at least one of the aforementioned objects, and displaying content in the three-dimensional map related to the object indicated by the command.

16. The step of outputting the aforementioned content is: The user terminal control method according to claim 1, which outputs content that includes information describing the first object and other objects in the vicinity of the first object, by including orientation information centered on the first object.

17. A user terminal control method executed by a computer system which is a server or a user terminal, The steps include: displaying an AR view that includes images captured by the camera of the user terminal; Within the AR view, the steps include displaying a three-dimensional map of an area including the location of the user terminal or a location specified by the user of the user terminal, extended from the video; The steps include outputting content containing information related to the first object based on the user's interaction with the first object in the three-dimensional map, Includes, The first object includes real estate within the area, The user terminal control method is as follows: The steps include: displaying AR content for viewing property information registered in the aforementioned real estate, associated with the first object; The steps include outputting the property information based on the user's interaction with the AR content, It further includes, The aforementioned property information includes audio content and visual content, and is a user terminal control method.

18. A program recorded on a non-temporary computer-readable recording medium for causing the computer system to execute the user terminal control method according to any one of claims 1 to 17.

19. A computer system that constitutes a user terminal or server, It includes at least one processor configured to execute computer-readable instructions contained in memory, The aforementioned at least one processor is The system displays an AR view that includes images captured by the camera of the user terminal. Within the AR view, a three-dimensional map of the area including the location of the user terminal or a location specified by the user of the user terminal is displayed, extended from the video. Based on the user's interaction with the first object in the three-dimensional map, content including information related to the first object is output. A computer system in which interaction with the first object includes selecting the first object or AR content output in association with the first object, gauging to maintain the first object in a certain area of ​​the AR view, or voice commands to invoke information associated with the first object.

Images