Method and device for providing user interface enabling user to input content for image and three-dimensional map displayed in augmented reality view
The user interface system for AR view and 3D maps enables users to input and display content at specific locations, enhancing interaction and information delivery within augmented reality environments, addressing the limitations of current AR technologies.
Patent Information
- Application Number
- PCT/KR2024/005991
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-05-03
- Publication Date
- 2024-11-14
AI Technical Summary
Current augmented reality technologies lack an effective method for users to input and display content directly on 3D maps and images within an AR view, limiting the immersive experience and information provision for users interacting with virtual objects and environments.
A user interface system that allows users to input content at specific locations on 3D maps and images within an AR view, enabling the display of corresponding content, such as audio, text, or images, and executing actions like route guidance or real-time information provision when the user moves to those locations.
Enhances user interaction by allowing content input and display synchronization with location, providing an immersive experience and efficient information delivery, enabling users to organize notes and verify locations without direct visitation, and offering real-time information and action execution.
Smart Images

Figure KR2024005991_14112024_PF_FP_ABST
Abstract
Description
Method and device for providing a user interface that enables a user to input content for an image and a three-dimensional map displayed within an augmented reality view
[0001] The description below relates to a method and device for providing a user interface that allows a user to input content for an image / 3D map displayed within an Augmented Reality (AR) view, and for reflecting the content input through the user interface on the 3D map / image displayed within the AR view and displaying it (i.e., displaying corresponding content).
[0002] Augmented Reality (AR) refers to a technology that combines and supplements the real world with computer-generated virtual objects and information. In other words, AR is a technology that augments the real world with virtual content, allowing users to view the augmented content in response to their actual environment through electronic devices.
[0003] 3D maps are constructed by modeling real-world spaces or environments in three dimensions. They are provided to users through separate applications that provide services utilizing these 3D maps. An example of such a 3D map is S-MAP, which is constructed by modeling the entire Seoul Metropolitan Government in a three-dimensional virtual reality. 3D maps can be used to provide users with an experience similar to visiting the locations depicted on the map, even if they do not actually visit the locations. Users utilize these 3D maps for tasks such as virtually exploring specific areas or establishing policies for safety, the environment, and urban planning.
[0004] In utilizing 3D maps, a technology is required that provides a more immersive experience to users and more effectively provides users with information about objects (e.g., landmarks) displayed through 3D maps.
[0005] As a technology related to various services utilizing augmented reality technology, for example, Korean Patent Publication No. 10-2014-0065963 (published on May 30, 2014) discloses an augmented reality navigator that displays an image of a driving route captured by a camera installed in a vehicle on a display and maps virtual display information that guides the driving route onto the image displayed on the display.
[0006] The information described above is for the purpose of understanding only and may contain matters that do not form part of the prior art and may not contain what the prior art would suggest to a person skilled in the art.
[0007] A method for controlling a user terminal to provide a user interface that enables a user to input content for an image displayed within an AR view and to display corresponding content by reflecting the content input through the user interface on a three-dimensional map displayed within the AR view can be provided.
[0008] A method can be provided for controlling a user terminal to provide a user interface that enables a user to input content for a 3D map displayed within an AR view, and to display corresponding content by reflecting the content input through the user interface on an image of the AR view.
[0009] A method may be provided to input an action to be performed by a user terminal as content at a specific location on a 3D map, and to have the user terminal perform the action when the specific location is included in the AR view (i.e., when the user terminal moves to the specific location).
[0010] According to one aspect, a method for controlling a user terminal performed by a computer system which is a server or a user terminal, the method comprising: providing a first user interface (UI) for inputting first content at a selected first location within an image of an augmented reality (AR) view including an image captured by a camera of the user terminal, or providing a second user interface (UI) for inputting second content at a selected second location within a three-dimensional map displayed by augmenting the image within the AR view; and when the first content is input through the first UI, outputting first corresponding content corresponding to the first content when a three-dimensional map including the first location is displayed within the AR view, or when the second content is input through the second UI, outputting second corresponding content corresponding to the second content in an image of the AR view including the second location.
[0011] In a case where the first UI is provided and the first content is input at the first location, the method for controlling the user terminal further includes a step of displaying a three-dimensional map including the first location within the AR view according to the location of the user terminal or a request by a user of the user terminal, and the outputting step may output the first corresponding content in association with the first location within the three-dimensional map including the first location.
[0012] The first content may include at least one of audio, text, and image, and the output of the first corresponding content may include at least one of playback of the audio, display of the text, and display of the image.
[0013] A three-dimensional map including the first location may be displayed on another user terminal according to the location of the other user terminal or a request by a user of the other user terminal, and the first corresponding content may be output in association with the first location within the three-dimensional map displayed on the other user terminal.
[0014] The first location corresponds to a selected POI among the POIs (Points Of Interest) displayed in the image, and the first content can be input in association with the selected POI.
[0015] The first location may be a starting point of a path along which the user terminal moves, the first content may include information about the path from the starting point to a predetermined destination, and the outputting step may include a step of displaying the path from the starting point to the destination within a three-dimensional map including the first location.
[0016] The method for controlling the user terminal further includes a step of displaying a three-dimensional map including the second location within the AR view according to the location of the user terminal or a request by a user of the user terminal, wherein the second content is input in association with the second location through the second UI, and the outputting step may output the second corresponding content in association with the second location within the image of the AR view if the second location is included in the image of the AR view.
[0017] A three-dimensional map including the second location may be displayed within the AR view as a mini-map of an area including the second location.
[0018] The second content may include at least one of audio, text, and image, and the output of the second corresponding content may include at least one of playback of the audio, display of the text, and display of the image.
[0019] The second content may include a predetermined action to be executed by the user terminal, and the outputting step may include a step of executing the action on the user terminal when the second location is included in the image of the AR view.
[0020] Execution of the above action may include providing route guidance from a first point corresponding to the second location to a second point, starting recording of the AR view, linking with an IoT device placed in association with the second location, calling a specific contact of the user terminal, or providing real-time information associated with the second location.
[0021] The real-time information may include at least one of weather information, traffic information, and air quality information of the area associated with the second location.
[0022] The action corresponding to the provision of the real-time information may be executed only when the time at which the second location is included in the image of the AR view falls within a preset time zone.
[0023] Interoperation with the IoT device may include calling a specific person through the IoT device.
[0024] The second location is a starting point of a route specified on a three-dimensional map including the second location, the second content includes information about the route from the starting point to a predetermined destination, and the outputting step may include a step of providing guidance about the route from the starting point corresponding to the second location to the destination within an image of the AR view.
[0025] At least a portion of the above path may be designated by a user of the user terminal on a three-dimensional map including the second location.
[0026] A three-dimensional map including the second location may be displayed on the user terminal so that a user of the user terminal can virtually explore an area including the second location, and at least a portion of the route may be an automatically recorded route virtually explored by the user on the three-dimensional map including the second location.
[0027] The second location corresponds to a selected POI among POIs displayed on a 3D map including the second location, or the second location corresponds to a selected location on an object displayed on a 3D map including the second location, and the second content can be input in association with the selected POI or the selected location.
[0028] The method for controlling the user terminal may further include: providing a third UI for inputting additional content for the second corresponding content output in association with the second location within the image of the AR view; and, when additional content is input through the third UI, outputting the additional content in association with the second content at the second location of the three-dimensional map including the second location when displaying a three-dimensional map including the second location.
[0029] The method may further include a step of transmitting the first content input through the first UI to a cloud server together with 6dof information of the first location, or transmitting the second content input through the second UI to a cloud server together with 6dof information of the second location.
[0030] In another aspect, a computer system implementing a user terminal or a server is provided, comprising at least one processor configured to execute computer-readable instructions contained in a memory, wherein the at least one processor displays an augmented reality (AR) view including an image captured by a camera of the user terminal, and provides a first user interface (UI) for inputting first content at a first location selected within the image, or provides a second user interface (UI) for inputting second content at a second location selected within a three-dimensional map displayed by augmenting the image within the AR view, and when the first content is input through the first UI, outputs first corresponding content corresponding to the first content when a three-dimensional map including the first location is displayed within the AR view, or outputs second corresponding content corresponding to the second content in an image of the AR view including the second location when the second content is input through the second UI.
[0031] In another aspect, a method for controlling a user terminal performed by a computer system which is a server or a user terminal, the method comprising: displaying a three-dimensional map of an area including a first location which is a location of the user terminal or a location designated by a user; providing a first user interface (UI) for inputting content at a selected second location within the three-dimensional map; when the content is input through the first UI, if the second location is included in an augmented reality (AR) view including an image captured by a camera of the user terminal, outputting corresponding content corresponding to the content in association with the second location within an image of the AR view; providing a second UI for inputting additional content for the outputted corresponding content; and when the additional content is input through the second UI, outputting the additional content in association with the content at the second location of the three-dimensional map when displaying the three-dimensional map.
[0032] By reflecting user-input (authored) content on a 3D map / image displayed within an AR view and displaying it within the AR view, the usability of the AR view and the 3D map displayed therein can be increased. That is, a user using the AR view and 3D map of the embodiment can perform confirmation or verification through the AR view at an actual location for a location requiring confirmation or verification specified on the 3D map. In addition, such a user can organize memos or check items made through the AR view while moving around a certain area through a 3D map representing the area.
[0033] When content including images, text, etc. or an action to be performed by the user terminal is input at a specific location on a 3D map, when the specific location is included in the AR view (i.e., when the user terminal moves to a specific location), the content may be output on the user terminal or various actions such as route guidance, AR view recording, and real-time information provision may be executed. In other words, when content is input at a specific location on a 3D map and the user moves to the specific location and the AR view is executed, the content may be output at the specific location within the AR view (the specific location where the content was input). Accordingly, the user can input content indicating matters to be checked in relation to the specific location without directly visiting the specific location, and when the user actually moves to the location, the content may be output at the location corresponding to the specific location in the AR view based on the positioning of the user terminal. The location on the 3D map where the content is input and the location within the AR view determined based on the positioning may be linked (or synchronized).
[0034] FIG. 1 illustrates a method for displaying a three-dimensional map within an AR view and outputting content related to objects within the three-dimensional map, according to one embodiment.
[0035] FIG. 2 illustrates a computer system and server that execute an AR view, display a three-dimensional map within the AR view, and output content, according to one embodiment.
[0036] FIG. 3 is a flowchart illustrating a method for controlling a user terminal to display a 3D map within an AR view and output content related to an object within the 3D map, according to one embodiment.
[0037] FIG. 4 is a flowchart illustrating a method for controlling a user terminal when information describing a second object on a three-dimensional map is output from among contents including information describing a first object on a three-dimensional map, according to an example.
[0038] FIG. 5 is a flowchart illustrating a method for controlling a user terminal when information describing a third object outside a three-dimensional map is output from among contents including information describing a first object within a three-dimensional map, according to an example.
[0039] FIG. 6 is a flowchart illustrating a method for outputting registered listing information related to real estate within an AR view when an object within a 3D map is real estate, according to an example.
[0040] FIG. 7 illustrates content including information describing a first object within a three-dimensional map, according to an example.
[0041] FIGS. 8 to 14 illustrate a method in which a 3D map is displayed within an AR view and content related to objects within the 3D map is output, according to an example.
[0042] Figure 15 illustrates a method for outputting registered listing information related to real estate within an AR view, according to an example.
[0043] FIG. 16 illustrates objects and POIs included in a 3D map displayed within an AR view, according to an example.
[0044] FIG. 17 illustrates a method for displaying a 3D map within an AR view such that the orientation of the AR view and the orientation of the 3D map are aligned according to orientation information of the AR view, according to one embodiment.
[0045] FIG. 18 is a flowchart illustrating a method for controlling a user terminal to display a 3D map within an AR view so that the orientation of the AR view and the orientation of the 3D map are aligned, and to further display additional information about an external area outside the 3D map, according to one embodiment.
[0046] FIG. 19 is a flowchart illustrating a method for displaying additional content or a two-dimensional map as additional information about an external area outside a three-dimensional map, according to an example.
[0047] FIG. 20 illustrates a method for matching coordinates of a three-dimensional map with coordinates of a two-dimensional map representing an area of the three-dimensional map and an area outside the area, according to an example.
[0048] FIGS. 21 to 23 illustrate a method for displaying a two-dimensional map in an external area outside a three-dimensional map within an AR view, according to an example.
[0049] FIG. 24 illustrates a method for displaying POI information as additional content in an external area outside a 3D map within an AR view, according to an example.
[0050] FIG. 25 is a flowchart illustrating a method for controlling a user terminal to input first content in an AR view and output first corresponding content corresponding to the first content in a 3D map, or to input second content in a 3D map and output second corresponding content corresponding to the second content in an AR view, according to one embodiment.
[0051] FIG. 26 is a flowchart illustrating a method for controlling a user terminal to input first content in an AR view and output first corresponding content corresponding to the first content on a 3D map, according to an example.
[0052] FIG. 27 is a flowchart illustrating a method for controlling a user terminal to input second content on a 3D map and output second corresponding content corresponding to the second content in an AR view, according to an example.
[0053] FIG. 28 illustrates a method for confirming content created at a specific location on a 3D map in an AR view including the specific location, according to an example, and reflecting the content confirmed (or confirmed) in the AR view on the 3D map.
[0054] FIG. 29 illustrates a method for inputting an action to be executed by a user terminal as second content at a specific location on a 3D map according to an example, and for executing the action when the specific location is included in the AR view.
[0055] FIG. 30 illustrates a method for displaying a three-dimensional map including a current location within an AR view including an image captured at the current location of a user terminal according to an embodiment of the present invention.
[0056] FIG. 31 illustrates a method for providing guidance to a user terminal for an input path when a path is input as second content at a specific location on a 3D map and the specific location is included in an AR view, according to an example.
[0057] Hereinafter, examples are described in detail with reference to the attached drawings.
[0058]
[0059] FIG. 1 illustrates a method for displaying a three-dimensional map within an AR view and outputting content related to objects within the three-dimensional map, according to one embodiment.
[0060] In the embodiments described below, the “control” of the user terminal (100) may include at least one of: augmenting and displaying content representing a specific service or information on an image of the AR view (10) using the AR view (10) of the user terminal (100), as illustrated in FIG. 1; displaying a user interface (UI) related to a specific service on the user terminal (100); displaying a three-dimensional map (20) of an area including the location of the user terminal (100) or a location designated by a user of the user terminal (100) within the AR view (10) and outputting content (35, 42 to 46) on the three-dimensional map (20); and controlling the settings of the user terminal (100) to provide a specific service on the user terminal (100). In addition, the “control” of the user terminal in the embodiments of the present disclosure may include controlling a function and / or operation of the user terminal (100), which is not limited to the examples described above. At least part of the control of these user terminals (100) may be performed by the illustrated server (200).
[0061] In the detailed description to be given later, the location of the user terminal (100) may indicate the location of the user carrying the user terminal (100). In addition, for convenience of explanation, the terms 'user' and 'user terminal (100)' carried by the user may be used interchangeably.
[0062] In the detailed description to be given below, "augmenting and displaying" content and / or information (e.g., indicators, etc.) on an image may be interpreted to encompass, depending on the embodiment, displaying the content and / or information by overlapping it on the image / screen of the AR view (10). The "augmented content" displayed on the AR view (10) may also be referred to as "AR content."
[0063] In Fig. 1, the screen of the user terminal (100) is displayed as the user terminal (100) for convenience of explanation.
[0064] For example, a user terminal (100) such as a smart phone can capture the surroundings using a camera and display an AR view (10) including an image captured by the camera on the screen. The user terminal (100) can display a three-dimensional map (20) of an area including the location of the user terminal (100) or a location specified by a user of the user terminal (100) by augmenting the image in at least a part of the AR view (10), i.e., within the AR view (10).
[0065] In other words, in the AR view (10) of the embodiment, a 3D map (20) of a certain area can be augmented and displayed. The 3D map (20) can be displayed in a part of the AR view (10) by being augmented in the image of the AR view (10) as illustrated (a part of the AR view (10) other than the 3D map (20) in FIG. 1). Alternatively, the 3D map (20) can be displayed in the entire area of the AR view (10). The scale of the 3D map (20) can be adjusted (i.e., enlarged or reduced) according to an operation on the user terminal (100). Therefore, a user of the user terminal (100) can obtain information about the area indicated by the 3D map (20) through the 3D map (20).
[0066] The 3D map (20) displayed within the AR view (10) is augmented in space through AR, and can provide the user with an immersive experience in relation to the area indicated by the 3D map (20). For example, the user can have an experience as if he or she is actually in the area indicated by the 3D map (20) through the 3D map (20) (when the 3D map (20) is sufficiently enlarged and displayed within the AR view (10). As the user terminal (100) moves with the user, the display of the 3D map (20) may also change accordingly, and thus, the user can virtually experience or explore the area indicated by the 3D map (20). Alternatively, the 3D map (20) may provide a function that allows the user to virtually experience or explore the area indicated by the 3D map (20) in a manner similar to a street view or street view service (i.e., by using a screen touch, etc.). For example, a user can utilize the AR view (10) and the 3D map (20) of the embodiment to virtually visit an area indicated by the 3D map (20) or to make a travel plan for the area.
[0067] A 3D map (20) may be a 3D model of features actually located in a given area. Features may include any natural or artificial object existing on the ground, such as buildings, trees, houses, rivers, roads, etc. The 3D map (20) may include a plurality of objects, and each of these objects may represent at least one of the features.
[0068] In an embodiment, the user terminal (100) may output content (35, 42 to 46) including information related to the first object (30) based on the user's interaction with the first object (30) within the 3D map (20). The output of the content on the user terminal (100) may be in a visual and / or auditory form. For example, when a specific 'building' is selected as the first object (30) within the 3D map (20), the user terminal (100) may output content (35) including information about the building, and further output content (42 to 46) associated with other objects necessary to describe the building.
[0069] The method by which the user terminal (100) outputs content (35, 42 to 46) based on the interaction with the first object (30) is described in more detail with reference to FIGS. 3 to 16, which will be described later.
[0070] Meanwhile, the 3D map (20) may be extracted from a wide-area 3D map of an environment (wide area) including the area indicated by the 3D map (20). This wide-area 3D map may be provided through an external service (e.g., S-Map, Google Earth, Apple Maps, etc.).
[0071] In this way, in the embodiment, in addition to an image, a 3D map (20) of an area including the location of the user terminal (100) or a location specified by the user of the user terminal (100) can be displayed in the AR view (10) provided through the user terminal (100), and as interaction is made with respect to an object (30) included in the 3D map (20), content including information related to the object (30) can be output.
[0072]
[0073] FIG. 2 illustrates a computer system that performs a method for controlling a user terminal according to one embodiment.
[0074] The user terminal (100) described above with reference to FIG. 1 according to the embodiments may be implemented through the illustrated computer system (2). For example, a computer program for implementing the method of the embodiment may be installed and run on the computer system (2), and the computer system (2) may perform a method for controlling the user terminal (100) according to the embodiments under the control of the run computer program.
[0075] The method for controlling the user terminal (100) according to the embodiments may be performed through a PC-based program or an application dedicated to a mobile terminal. For example, the method for controlling the user terminal (100) according to the embodiments may be implemented in the form of an independently operating program, or may be implemented in the form of an in-app of a specific application so as to be operable on the specific application. Such a specific application may be installed on a computer system (2) to display a 3D map (20) through an AR view (10) and output content (35, 42 to 46), thereby performing the method for controlling the user terminal (100) according to the embodiments.
[0076] The computer system (2) is an electronic device corresponding to the user terminal (100) described above, and may be a smartphone or similar device capable of installing and executing an application or program as illustrated in FIG. 1. In addition, the computer system (2) may be a personal computer (PC), a laptop computer, a tablet, an Internet of Things (IoT) device, a wearable computer, or the like.
[0077] As illustrated, the computer system (2) may include a memory (110), a processor (120), a communication interface (130), and an input / output interface (140) as components for executing a method of controlling a user terminal (100).
[0078] The memory (110) is a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the ROM and the permanent mass storage device such as the disk drive may be included in the computer system (2) as a separate permanent storage device distinct from the memory (110). In addition, the memory (110) may store an operating system and at least one program code. These software components may be loaded into the memory (110) from a computer-readable recording medium separate from the memory (110). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (110) through a communication interface (130) rather than a computer-readable recording medium. For example, software components may be loaded into the memory (110) of the computer system (2) based on a computer program that is installed by files received over a network (160).
[0079] The processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (120) via the memory (110) or the communication interface (130). For example, the processor (120) may be configured to execute instructions received according to program code stored in a storage device such as the memory (110).
[0080] That is, the processor (120) can manage components of the computer system (2) and execute a program or application used by the computer system (2). For example, the processor (120) can be configured to execute an application for performing a method of controlling a user terminal (100) of the embodiment, and process data received from a server (200) to control the user terminal (100) (e.g., data received from the server (200) to display a three-dimensional map (20) through an AR view (10) and output content (35, 42 to 46)). In addition, the processor (120) can process operations necessary for executing a program or application and processing data, and can be at least one processor of the computer system (2) or at least one core within the processor.
[0081] The communication interface (130) may provide a function for the computer system (2) to communicate with another computer system (not shown) via a network (160). For example, requests, commands, data, files, etc. generated by the processor (120) of the computer system (2) according to program codes stored in a recording device such as a memory (110) may be transmitted to another computer system via the network (160) under the control of the communication interface (130). Conversely, signals, commands, data, files, etc. from another computer system may be received by the computer system (2) via the communication interface (130) of the computer system (2) via the network (160). The signals, commands, data, etc. received via the communication interface (130) may be transmitted to the processor (120) or the memory (110), and the files, etc. may be stored in a storage medium (the aforementioned permanent storage device) that the computer system (2) may further include. For example, the communication interface (130) may be a hardware module such as a network interface card, a network interface chip, and a networking interface port of the computer system (2), or a software module such as a network device driver or a networking program.
[0082] The input / output interface (140) may be a means for interfacing with an input / output device (150). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (140) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (150) may also be configured as a single device with the computer system (2).
[0083] Also, in other embodiments, the computer system (2) may include more components than those in FIG. 2. However, it is not necessary to explicitly illustrate most of the conventional components. For example, the computer system (2) may be implemented to include at least some of the input / output devices connected to the above-described input / output interface (140), or may further include other components such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, etc. For a more specific example, when the computer system (2) is implemented in the form of a mobile device such as a smartphone, various components that are generally included in the mobile device, such as a camera, an acceleration sensor or a gyro sensor, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration, may be further implemented to be included in the computer system (2).
[0084] For example, a computer system (2) corresponding to a user terminal (100) may include a camera that captures the surroundings to execute an AR view (10). The computer system (2) may display an image captured by the camera as an AR view (10). The computer system (2) may display a three-dimensional map (20) by augmenting the image of the AR view (10), and may output content (35, 42 to 46) based on an interaction with an object (30) included in the three-dimensional map (20).
[0085] Meanwhile, at least some of the operations or calculations performed by the computer system (2) to display the AR view (10) and the 3D map (20) and output the contents (35, 42 to 46) may be performed by the server (200) rather than the computer system (2). The server (200) may be an electronic device that communicates with the computer system (2) to provide data for controlling the user terminal (100) to the computer system (2), thereby enabling the user terminal (100) to be controlled. The server (200) may include a database as a device that manages (registers, changes, deletes, stores, maintains, etc.) the data and contents to be provided to the computer system (2), and the 2D map and the 3D map, or may communicate with a database or an external service that provides the same. In addition, the server (200) may include a map server that provides a digital map (a 3D map and / or a 2D map) or may be configured to communicate with such a map server.
[0086] The server (200) may include at least one computer system. The computer system included in the server (200) may include configurations similar to those of the computer system (2) described above, and a duplicate description thereof will be omitted.
[0087] In an embodiment, a user terminal (100), which is a computer system (2), may communicate with a server (200) and be controlled based on data and / or information provided from the server (200). At this time, the server (200) may receive the location of the user terminal (100) or a location specified by the user terminal, and may control the user terminal (100) to display a 3D map (20) in an AR view (10) based on the received location. In addition, the server (200) may recognize that an interaction has occurred with an object (30) within the 3D map (20) at the user terminal (100), and accordingly, may transmit data for outputting content (35, 42 to 46) to the user terminal (100).
[0088] In the detailed description to be described below, for the sake of convenience of explanation, embodiments will be described focusing on the computer system (2) corresponding to the user terminal (100), and communication with the server (200) and operations on the server (200) side may be briefly described or omitted. In addition, for the sake of convenience of explanation, operations (steps) performed by the components (e.g., processors, etc.) of the computer system (2) (or server (200)) may be described as being performed by the computer system (2) (or server (200)).
[0089] The description of the technical features described above with reference to FIG. 1 can also be applied to FIG. 2, so any redundant description will be omitted.
[0090]
[0091] FIG. 3 is a flowchart illustrating a method for controlling a user terminal to display a 3D map within an AR view and output content related to an object within the 3D map, according to one embodiment.
[0092] Referring to FIG. 3, a method for controlling a user terminal (100) performed by a computer system (2) will be described. The computer system (2) may be a user terminal (100) with reference to FIG. 1, and in the detailed description to be described later, the term "user terminal (100)" will be used instead of "computer system (2)" to describe an embodiment.
[0093] Additionally, at least some of the actual operations and actions of steps (310 to 620) to be described later with reference to FIGS. 3 to 6 may be configured to be performed by the server (200) rather than the user terminal (100). Below, the performance of the steps will be described with a focus on the user terminal (100), and redundant descriptions regarding the server (200) may be omitted.
[0094] In step (310), the user terminal (100) may display an augmented reality (AR) view (10) including an image captured by a camera. The AR view (10) may be displayed on the user terminal (100) in full screen, or in at least a portion of the screen. The AR view (10) may be configured to display augmented content related to an object included in the image in relation to the object.
[0095] In step (320), the user terminal (100) can display a three-dimensional map (20) of an area including the location of the user terminal (100) or a location designated by a user of the user terminal (100) by augmenting the image within the AR view (10). The user terminal (100) can display the three-dimensional map (20) in at least a portion of the AR view (10). For example, the user terminal (100) can display the three-dimensional map (20) in a portion corresponding to the bottom of the image of the AR view (10).
[0096] The 3D map (20) can represent an area including the current location of the user terminal (100). At this time, the 3D map (20) can serve as a mini-map for the image of the AR view (10).
[0097] Alternatively, the 3D map (20) may represent an area including a location specified by a user of the user terminal (100) (regardless of the current location of the user terminal (100). In this case, the user may set an area to be displayed on the 3D map (20) in the AR view (10) through area selection, area search, etc. in a separate map application (e.g., a map application providing 2D and / or 3D map services). Alternatively, the user may set an area to be displayed on the 3D map (20) in the AR view (10) through voice search (e.g., speaking an address, administrative district, etc.).
[0098] The 3D map (20) may represent administrative districts (dong, gu, etc.) that include the current location of the user terminal (100) or a location designated by the user, or blocks separated by roads. Alternatively, the range of the area represented by the 3D map (20) may be set by the user.
[0099] The scale of the 3D map (20) can be adjusted (i.e., enlarged or reduced) and the viewpoint can be changed according to the operation on the user terminal (100). When displaying the 3D map (20), the user terminal (100) can display the 3D map (20) with a changed viewpoint as the viewpoint of the camera for displaying the AR view (10) is changed. Accordingly, a user using such a 3D map (20) can virtually experience the area indicated by the 3D map (20) in a more intuitive manner.
[0100] Meanwhile, as described above, the 3D map (20) can provide the user with an immersive experience with respect to the area indicated by the 3D map (20), for example, can provide the user with an experience as if he or she is actually in the area indicated by the 3D map (20) (when the 3D map (20) is sufficiently enlarged and displayed within the AR view (10). For example, as the user terminal (100) moves with the user, the display of the 3D map (20) can also change accordingly, and thus, the user can virtually experience or explore the area indicated by the 3D map (20). Alternatively, the 3D map (20) can provide a function that allows the user to virtually experience or explore the area indicated by the 3D map (20) in a manner similar to a street view or street view service (i.e., by using a screen touch, etc.).
[0101] In step (330), the user terminal (100) may output content (35) including information related to the first object (30) based on the user's interaction with the first object (30) within the 3D map (20). For example, the first object (30) may represent at least one of the features located in the actual environment corresponding to the 3D map (20). The first object (30) may be, for example, a POI included in the 3D map (20).
[0102] Alternatively, the first object (30) may include real estate within the area represented by the three-dimensional map (20). The real estate may include buildings, multi-family buildings, apartments, apartment complexes (including multiple apartment buildings), other houses, and land or portions thereof.
[0103] The interaction with the first object (30) may be selecting the first object (30) from a three-dimensional map (20) displayed within the AR view (10). The selection may be selecting or touching the first object (30) or AR content or icon output in association with the first object (30). The AR content may include the name of the first object (30), such as the name of an apartment complex corresponding to the first object (30). Alternatively, the interaction with the first object (30) may include a voice command that calls up information related to the first object (30), such as, for example, “Tell me about the first object (30).” Alternatively, the interaction with the first object (30) may include gazing the first object (30) on the AR view (10). This gauging may be to keep the first object (30) positioned in a certain area of the AR view (e.g., a central area, etc.).
[0104] When an interaction with the first object (30) as described above is recognized, the user terminal (100) can output content (35) including information related to the first object (30).
[0105] A more specific method of displaying a three-dimensional map (20) within an AR view (10) and providing content (35) including information related to a first object (30) within the three-dimensional map (20) will be described in more detail with reference to FIGS. 4 to 16, which will be described later.
[0106] The description of the technical features described above with reference to FIGS. 1 and 2 can also be applied to FIG. 4, so any redundant description will be omitted.
[0107]
[0108] FIG. 4 is a flowchart illustrating a method for controlling a user terminal when information describing a second object on a three-dimensional map is output from among contents including information describing a first object on a three-dimensional map, according to an example.
[0109] FIG. 5 is a flowchart illustrating a method for controlling a user terminal when information describing a third object outside a three-dimensional map is output from among contents including information describing a first object within a three-dimensional map, according to an example.
[0110] When outputting content (35) including information related to the first object (30) described above, such content (35) may include descriptive content including information describing the first object (30). In other words, when the interaction described above is recognized, the user terminal (100) may output such descriptive content. The descriptive content may include text and / or audio content. When the descriptive content includes audio content, such audio content may be played on the user terminal (100) when the interaction is recognized.
[0111] The descriptive content may include information describing the real estate corresponding to the first object (30). The information describing the real estate may include, for example, basic information about the real estate, such as information on the surrounding environment of the real estate, information on the completion date, actual transaction price information, and information related to the purchase / sale / lease of the real estate.
[0112] The information describing the above real estate may include at least one of information describing a second object (an object different from the first object (30)) within the three-dimensional map (20) in relation to the real estate corresponding to the first object (30) and information describing a third object outside the three-dimensional map (20) in relation to the real estate corresponding to the first object (30). These second and third objects may be objects cited to describe the surrounding environment of the real estate corresponding to the first object (30).
[0113] Referring again to FIG. 4, an example is described where the information describing the real estate includes information describing a second object within the three-dimensional map (20).
[0114] In step (410), the user terminal (100) may output information describing the second object when outputting content (35) including information related to the first object (30).
[0115] In step (420), the user terminal (100) can change the display of the second object on the 3D map (20) or display additional AR content in association with the second object when information describing the second object of the description content is output from the user terminal (100). For example, the user terminal (100) can change the display of the second object on the 3D map (20) or display additional AR content in association with the second object when text of the information describing the second object is displayed on the user terminal (100) or audio content of the information describing the second object is played on the user terminal (100).
[0116] At this time, the change in the display of the second object may include at least one of highlight processing for the second object and motion effect processing for the second object. Accordingly, when the user is provided with information describing real estate corresponding to the first object (30), the user can visually recognize the second object when information describing the second object representing other buildings, rivers, roads, natural objects, convenience facilities, etc. around the real estate is output.
[0117] Meanwhile, the additional AR content displayed in connection with the second object may include at least one of an image and text describing the second object; distance information from the real estate corresponding to the first object (30) to the second object; and path information from the real estate to the second object (30). The image and text describing the second object may include images, icons, and text describing other buildings, rivers, roads, natural objects, convenience facilities, etc. around the real estate corresponding to the first object (30). The distance information may include a distance from the real estate corresponding to the first object (30) to other buildings, rivers, roads, natural objects, convenience facilities, etc., which are the second objects, and the path information may include a movement path from the real estate corresponding to the first object (30) to the second object.
[0118] In step (430), the user terminal (100) can appropriately control the display of the 3D map (20) when changing the display of the second object in step (420) or displaying additional AR content in association with the second object. For example, the user terminal (100) can control the display of the 3D map so that the change in the display of the second object or the additional AR content is displayed in a predetermined area of the AR view. For example, the display of the 3D map (20) can be controlled so that the change in the display of the second object or the display of the additional AR content is performed at the center of the AR view (10).
[0119] In this way, in the embodiment, information describing a second object in a three-dimensional map (20) as surrounding information about real estate corresponding to a first object (30) can be output from a user terminal (100) in a more effective and intuitive manner that is recognized by the user.
[0120] Referring again to FIG. 5, an example is described where the information describing the real estate includes information describing a third object outside the three-dimensional map (20).
[0121] In step (510), the user terminal (100) may output information describing a third object when outputting content (35) including information related to the first object (30).
[0122] In step (520), the user terminal (100) may further display additional AR content representing the third object outside the three-dimensional map (20) when information describing the third object of the explanatory content is output from the user terminal (100). For example, as in steps (522 and 524), the user terminal (100) may determine a location to display the additional AR content within the AR view (10) based on the distance between the geographic location of the real estate corresponding to the first object (30) and the geographic location of the third object, and may display the additional AR content at the determined location.
[0123] Meanwhile, the additional AR content displayed in connection with the third object may include at least one of an image and text describing the third object; distance information from the real estate corresponding to the first object (30) to the third object; and path information from the real estate to the third object in the first object (30). The image and text describing the third object may include images, icons, and text describing other buildings, rivers, roads, natural objects, convenience facilities, etc. around the real estate corresponding to the first object (30). The distance information may include a distance from the real estate corresponding to the first object (30) to other buildings, rivers, roads, natural objects, convenience facilities, etc. which are the third objects, and the path information may include a movement path from the real estate corresponding to the first object (30) to the third object.
[0124] In step (430), the user terminal (100) can appropriately control the display of the 3D map (20) when further displaying the additional AR content in step (520). For example, the scale of the 3D map (20) can be adjusted based on the distance between the geographic location of the real estate corresponding to the first object (30) and the geographic location of the third object. At this time, the scale of the 3D map (20) can be automatically adjusted so that the additional AR content and the 3D map (20) can be displayed on one screen of the user terminal (100). Therefore, in the embodiment, the scale of the 3D map (20) can be appropriately adjusted so that the 3D map (20) and the additional AR content can be displayed on one screen.
[0125] In this way, in the embodiment, information describing a third object existing outside the three-dimensional map (20) as surrounding information about real estate corresponding to the first object (30) can also be output from the user terminal (100) in a more effective and intuitive manner for the user.
[0126] Below, with reference to FIGS. 7 to 15, a more detailed description will be given of a method for controlling the display of the AR view (10) when information describing a second object is output and information describing a third object is output when providing content (35) including information related to a first object.
[0127] FIG. 7 illustrates content including information describing a first object within a three-dimensional map, according to an example.
[0128] The content (35) being shown may be an example of descriptive content that includes information describing the first object (30) described above. The content (35) may be output from the user terminal (100) in text form, and / or may additionally be played from the user terminal (100) in the form of audio content. Such audio content may be considered to correspond to an "audio guide."
[0129] As in the example shown, content (35) can be structured as something like, "Apartment complex A is located in a good location for a walk, with N building across the street and Tancheon flowing a little further. On the other side are Taebongsan Mountain and Jinjaesan Mountain, so you can enjoy an unobstructed view of the mountains from the top floor penthouse. It is also close to a cafe alley, so it is good for enjoying brunch on the weekend."
[0130] Here, the A apartment complex (710) may represent the first object (30) (i.e., real estate corresponding to the first object (30)). Meanwhile, the N office building (720), Tancheon (730), and cafe alley (750) may each represent second objects corresponding to the surrounding environment of the first object (30). Taebongsan Mountain and Jinjaesan Mountain (740) may represent third objects corresponding to the surrounding environment of the first object (30).
[0131] In this way, the content (35) may include information about the direction in which the user should move within the 3D map (20), available roads, destinations corresponding to other objects, parts of the 3D map (20) to be viewed, etc.
[0132] A user can explore an area indicated by a 3D map (20) by controlling the display of the 3D map (20) (e.g., by rotating the user terminal (100)) while listening to content (35) output in the form of, for example, audio content.
[0133]
[0134] FIGS. 8 to 14 illustrate a method in which a 3D map is displayed within an AR view and content related to objects within the 3D map is output, according to an example.
[0135] As illustrated in FIG. 8, the 3D map (20) may include an icon representing a first object (30) or AR content (810). As the AR content (810) is selected, the content (35) may be output on the user terminal (100).
[0136] Among the contents (35), when content such as "N building is across the street," is output from the user terminal (100), additional AR content (910) may be displayed on the 3D map (20), as illustrated in FIG. 9. At this time, the building corresponding to the N building may correspond to the second object described above. In addition, although not illustrated, when content such as "N building is across the street," is output from the user terminal (100), the display of the building corresponding to the N building, which corresponds to the second object, may additionally or alternatively be changed.
[0137] In addition, when content (35), such as, for example, "If you go a little further, the Tancheon River flows, making it a good location for a walk," is output on the user terminal (100), additional AR content (1010) may be displayed on the 3D map (20), as illustrated in FIG. 10. At this time, the river corresponding to the Tancheon River may correspond to the second object described above. In addition, although not illustrated, when content such as, "If you go a little further, the Tancheon River flows, making it a good location for a walk," is output on the user terminal (100), the display of the river corresponding to the Tancheon River, which corresponds to the second object, may additionally or alternatively be changed. For example, a motion effect, such as a flowing river, may be applied to the second object.
[0138] In addition, when content (35), for example, "Since Taebongsan Mountain and Jinjaesan Mountain are surrounded on the opposite side, you can view the mountain scenery without any obstruction from the top floor penthouse" is output on the user terminal (100), additional AR content (1110) may be displayed on the 3D map (20), as illustrated in FIG. 11. In this case, the mountains corresponding to Taebongsan Mountain and Jinjaesan Mountain may correspond to the third object described above. This third object is not included in the scope of the 3D map (20) and may be displayed outside the 3D map (20).
[0139] In addition, when content (35), such as, for example, "It's close to the cafe alley, so it's great for enjoying brunch on the weekend," is output on the user terminal (100), additional AR content (1410) may be displayed on the 3D map (20), as illustrated in FIG. 14. At this time, a street or road corresponding to the cafe alley may correspond to the second object described above. In addition, although not illustrated, when content such as "It's close to the cafe alley, so it's great for enjoying brunch on the weekend," is output on the user terminal (100), the display of an object corresponding to a street or road corresponding to the second object may be changed additionally or alternatively.
[0140] In addition, as illustrated in FIG. 14, the path (1420) and / or arrival time (1425) to other buildings, rivers, roads, natural objects, convenience facilities, etc. corresponding to the second object may be displayed as additional AR content in the AR view (10). For example, in order to determine accessibility to public transportation (subway stations, bus stops, etc.) and accessibility to schools, which may be important factors in determining real estate transactions, the path and / or arrival time to a public transportation station, or the path and / or arrival time to a school may be displayed as the additional AR content in the AR view (10). Such additional AR content may be appropriately output from the user terminal (100) depending on the content of the content (35) and the output timing of the content (35).
[0141] Additionally, as illustrated in FIG. 12, information regarding the straight-line distance between the first object (30) and the second object (e.g., distance between buildings, distance between apartment buildings, etc.) or surrounding roads may be displayed as additional AR content (1210) in the AR view (10). Such additional AR content may be appropriately output from the user terminal (100) depending on the content of the content (35) and the output timing of the content (35).
[0142] In addition, as illustrated in FIG. 13, information about the height or terrain of the first object (30) or floor information (e.g., building height, floor information) may be displayed as additional AR content (1310) in the AR view (10). This additional AR content may be appropriately output from the user terminal (100) depending on the content of the content (35) and the output time of the content (35). Similarly, information about the height or terrain of the second object or floor information (i.e., surrounding building height, floor information) may also be displayed as additional AR content in the AR view (10).
[0143] As described with reference to FIGS. 9 to 14, output of additional AR content in the AR view (10) can be performed in synchronization with the output or playback of the content (35).
[0144] Meanwhile, in the embodiment, the user terminal (100) may output content (35) including information describing the first object (30) and other objects (the second object and the third object described above) surrounding the first object, including azimuth information centered on the first object (30).
[0145] For example, the content (35) may include directional information in the east, west, south, north, up, down, left, and right directions for the first object (30) and other objects surrounding the first object (the second object and the third object described above). This directional information may be based on the current location of the user terminal (100).
[0146] For example, among the contents (35) described above in FIG. 7, “N building is across the street” can be configured as “If you look to the right, you will see N building,” and “Tancheon flows a little further, so it is a good location for a walk.” can be configured as “Tancheon flows further to the right, so it is a good location for a walk.” and “Taebongsan Mountain and Jinjaesan Mountain are surrounding on the opposite side, so you can see the mountain scenery without obstruction from the top floor penthouse.” can be configured as “Taebongsan Mountain and Jinjaesan Mountain are surrounding on the left, so you can see the mountain scenery without obstruction from the top floor penthouse.”
[0147] At least one of the additional AR contents described with reference to the aforementioned FIGS. 9 to 14 may be output from the user terminal (100) when a command for calling up information corresponding to the additional AR contents is received by the user terminal (100), regardless of the output of the content (35).
[0148] For example, a 3D map (20) may include multiple objects, and these objects may be associated with content output. For example, an object may have a command to output content, or the content itself may be mapped or tagged. The content may be at least one of text, audio content, and AR content.
[0149] For example, as in step (340) of FIG. 3, when a command for calling up information about at least one of the objects is received, the user terminal (100) may display content related to the object indicated by the command on the 3D map (20). The displayed content may be at least one of the additional AR contents described with reference to FIGS. 9 to 14 described above.
[0150] For example, when a command is received requesting path information from a specific object to another object on a 3D map (20), or a command is received requesting information on a facility such as a school, additional AR content such as that described with reference to FIGS. 9 to 14 may be displayed in the AR view (10). The command for displaying the additional AR content may be received by the user terminal (100) while the first object (30) is selected. The command may include a voice command (query).
[0151] In this way, the embodiment can be implemented to output additional AR content from the AR view (10) while a predetermined description content is output, and also to output additional AR content from the AR view (10) when necessary according to a command from the user.
[0152] The description of the technical features described above with reference to FIGS. 1 to 3 can also be applied to FIGS. 4, 5, and 7 to 14, so redundant descriptions are omitted.
[0153]
[0154] FIG. 6 is a flowchart illustrating a method for outputting registered listing information related to real estate within an AR view when an object within a 3D map is real estate, according to an example.
[0155] In relation to this, FIG. 15 illustrates a method of outputting registered listing information related to real estate within an AR view according to an example.
[0156] As described above, the first object (30) within the 3D map (20) may include real estate within the area.
[0157] In step (610), the user terminal (100) can display AR content (e.g., AR content (1510) or an icon indicating the name of the first object (30)) for viewing the property information registered in the real estate corresponding to the first object (30) in association with the first object.
[0158] In step (610), the user terminal (100) can output product information based on the user's interaction with AR content (1510).
[0159] Listing information may include audio content and visual content.
[0160] Such listing information can be viewed by touching an icon (e.g., AR content (1510)) representing a listing displayed on the real estate corresponding to the first object (30). The listing information may include a description of the area, internal structure, etc. of the listing, a description of the price, and a description of other features of the listing. When the listing information is output as audio content, background music may be provided, and the content of the listing information provided may be configured differently depending on the characteristics of the listing.
[0161] As illustrated, AR content (1510) is an image of a property, which may include an interior photo, etc., and the image may be viewed by selecting a thumbnail of the image included in the AR content (1510) by the user terminal (100) or by enlarging and displaying the AR content (1510).
[0162] Accordingly, users who wish to obtain real estate listing information can obtain such information in an intuitive manner without having to visit a real estate agency or the location of the property in question.
[0163] The description of the technical features described above with reference to FIGS. 1 to 5 and 7 to 14 can also be applied to FIGS. 6 and 15, so redundant descriptions are omitted.
[0164]
[0165] FIG. 16 illustrates objects and POIs included in a 3D map displayed within an AR view, according to an example.
[0166] In Fig. 16, the screen of the user terminal (100) with the 3D map (20) described above enlarged is shown.
[0167] A 3D map (20) may include multiple objects, and each of the POIs (1610) and buildings may correspond to an object, as illustrated. As illustrated, the POIs (1610) may represent building facilities, such as a building entrance or a building parking lot.
[0168] In an embodiment, when such a POI (1610) is selected, content including information related to the POI (1610) may be output from the user terminal (100). In addition, even if the POI (1610) is not displayed as an icon or the like, when a specific building is selected on the 3D map (20), content including information related to an object corresponding to the selected building may be output from the user terminal (100).
[0169] The description of the technical features described above with reference to FIGS. 1 to 15 can also be applied to FIG. 16, so redundant descriptions are omitted.
[0170]
[0171] In the above-described embodiments, content (35) including real estate-related information is output together with a 3D map (10) via an AR view (10), and additional AR content is further output in synchronization with the output of the content (35), thereby effectively providing specialized information about the real estate or the area indicated by the 3D map (10) to the user. For example, the user can virtually envision an area of interest via the 3D map (20) and easily obtain real estate information, etc., of the area.
[0172]
[0173] FIG. 17 illustrates a method for displaying a 3D map within an AR view such that the orientation of the AR view and the orientation of the 3D map are aligned according to orientation information of the AR view, according to one embodiment.
[0174] As described above with reference to FIG. 1, a user terminal (100), such as a smart phone, can capture a surrounding area using a camera and display an AR view (10) including an image captured by the camera on a screen. The user terminal (100) can display a three-dimensional map (20) of an area including the location of the user terminal (100) or a location designated by a user of the user terminal (100) by augmenting the image in at least a portion of the AR view (10), i.e., within the AR view (10).
[0175] At this time, the user terminal (100) can display the 3D map according to the direction information of the AR view (10) obtained from the server (200) or the user terminal (100) by augmenting the image within the AR view (10). Accordingly, the direction of the image within the AR view (10) and the direction of the 3D map (20) can be matched. The direction information of the AR view (10) can include the east, west, south, north (or up, down, left, right) direction information of the AR view (10). For example, the north in the image of the AR view (10) can be the same as the direction and the north in the 3D map (20).
[0176] In this way, in the embodiment, since the direction in the 3D map (20) displayed in the AR view (10) matches the direction in the actual environment (real world), a user using the 3D map (20) can clearly recognize the direction of objects in the 3D map (20) based on the current location.
[0177] In an embodiment, as illustrated, direction information (1710) may be further displayed around a 3D map (20) within an AR view (10). Accordingly, a user using the 3D map (20) can more intuitively recognize the directions of objects within the 3D map (20).
[0178] The description of the technical features described above with reference to FIGS. 1 to 16 can also be applied to FIG. 17, so any duplicate description will be omitted.
[0179]
[0180] FIG. 18 is a flowchart illustrating a method for controlling a user terminal to display a 3D map within an AR view so that the orientation of the AR view and the orientation of the 3D map are aligned, and to further display additional information about an external area outside the 3D map, according to one embodiment.
[0181] In step (1810), the user terminal (100) may display an AR view (10) including an image captured by a camera. The AR view (10) may be displayed on the user terminal (100) in full screen, or in at least a portion of the screen. The AR view (10) may be configured to display augmented content related to an object included in the image in association with the object.
[0182] In step (1820), the user terminal (100) can display a 3D map (20) of an area including the location of the user terminal (100) or a location specified by the user of the user terminal (100) according to the direction information of the AR view (10) obtained from the server (200) or the user terminal (100) by augmenting the image within the AR view (10).
[0183] As for the 3D map (20), the above-described contents can be similarly applied with reference to FIGS. 1 to 16, so redundant descriptions are omitted.
[0184] The direction information may be received from a server (200) or acquired based on an electronically implemented compass or sensor included in a user terminal (100). The direction information may include east, west, south, north (or up, down, left, right) direction information of the AR view (10).
[0185] In an embodiment, the orientation of the image within the AR view (10) and the orientation of the 3D map (20) can be matched.
[0186] In step (1825), the user terminal (100) can further display direction information (1710) around the 3D map (20) within the AR view (10). The direction information (1710) can include a direction indicating the east, west, south, north (or up, down, left, right) directions of the AR view (10). Through the direction information (1710), a user using the 3D map (20) can more intuitively recognize the directions of objects within the 3D map (20).
[0187] When the viewpoint of the camera of the user terminal (100) changes when displaying a 3D map (20), if the viewpoint and direction of the image of the AR view (10) change, the user terminal (100) can change and display the 3D map (20) to match the changed viewpoint and direction. In this way, the display of the 3D map (20) can be controlled in conjunction with the movement of the user terminal (100) (i.e., the movement of the camera), and thus, can provide an immersive experience to a user using the 3D map (20).
[0188] In step (1830), the user terminal (100) can display additional information about an area outside the area indicated by the 3D map (20) outside the 3D map (20) by augmenting the image within the AR view (10). In other words, the user terminal (100) can display the 3D map (20) in a certain range of areas including the current location of the user terminal (100) or a location designated by the user, and can further display additional information about an area outside the area in a wider range than the area in the AR view (10).
[0189] This additional information may include additional content including two-dimensional maps and / or information about external areas.
[0190] In an embodiment, a more immersive experience can be provided to users of the 3D map (20) by augmenting the 3D map (20) using an AR view (10) rather than using a conventional form, such as a web or app interface. In other words, the user can have an experience similar to directly entering and exploring the space indicated by the 3D map (20).
[0191] The 3D map (20) in the embodiment may only display a limited area compared to a 3D map provided via a web or app interface. For example, the 3D map (20) augmented within the AR view (10) may be provided to display an area that falls within a preset data size or a preset area range. Additional information about an area outside of the area indicated by the 3D map (20) may be mashed up to be provided around the 3D map (20) as data in a format that is lighter and requires less load for rendering, rather than the 3D map (20).
[0192] Accordingly, in the embodiment, depending on the scenario requested by the user (obtaining real estate information, travel information, etc.), the area corresponding to the area of interest of the user is rendered through a 3D map (20), and the surrounding information of the external area may be provided around the 3D map (20) by mashing up information such as a 2D map or POI extracted from a 2D map. In this way, the user terminal (100) in the embodiment can output information on the area of interest and its surrounding area in an efficient manner while reducing the load required to render the 3D map (20).
[0193] A more specific method of displaying additional information about an external area in the AR view (10) is described in more detail with reference to FIGS. 19 to 24, which will be described later.
[0194] The description of the technical features described above with reference to FIGS. 1 to 17 can also be applied to FIG. 18, so redundant descriptions are omitted.
[0195]
[0196] FIG. 19 is a flowchart illustrating a method for displaying additional content or a two-dimensional map as additional information about an external area outside a three-dimensional map, according to an example.
[0197] In step (1910), the user terminal (100) can obtain additional content and / or a two-dimensional map as additional information about an area outside the area indicated by the three-dimensional map (20).
[0198] In step (1920), the user terminal (100) may display additional content and / or a two-dimensional map acquired in step (1910) at the location of the AR view (10) corresponding to the external area. The location of the AR view (10) corresponding to the external area may indicate the relative location of the external area with respect to the three-dimensional map (20) (i.e., the area indicated by the three-dimensional map (20)).
[0199] To first explain a more specific example of steps (1910 and 1920), the user terminal (100) can obtain additional content about an external area from a map service that provides a 2D map including the external area as additional information about the external area indicated by the 3D map (20). The 2D map may be provided as a commercial map through a separate map application (e.g., a map application that provides 2D and / or 3D map services).
[0200] The user terminal (100) can display the acquired additional content in an area corresponding to an external area outside the 3D map (20). This additional content is not the 2D map itself, but may include information about the external area extracted from the 2D map or acquired from a service providing the 2D map.
[0201] For example, the additional content may include real-time information about an external area, such as traffic information, air quality information, and real-time video information about the external area. The real-time video information may include video captured from CCTV or surveillance cameras installed in the external area.
[0202] This real-time information can be displayed within the AR view (10) in the form of numbers, graphs, diagrams, etc. Alternatively, this real-time information can be displayed within the AR view (10) by overlaying it on a two-dimensional map representing an external area. In other words, real-time information can be displayed within the AR view (10) by overlaying it on a two-dimensional map in the form of traffic information, air quality information, real-time image information, etc. presented on a two-dimensional map. This real-time information can be obtained from a map service or a separate service providing real-time information.
[0203] Alternatively / additionally, the additional content may include at least one POI included in the external area. Such a POI may be displayed at a location outside the 3D map corresponding to the location of the POI on the 2D map of the external area. In other words, the user terminal (100) may determine a location on the AR view (10) where the POI is to be displayed by considering the relative location of the external area with respect to the 3D map (20) (i.e., the area indicated by the 3D map (20)), and may display the POI as additional content at the determined location.
[0204] The POI displayed in the AR view (10) may be selected from among the POIs included in the two-dimensional map of the external area. That is, the user terminal (100) may determine POIs corresponding to public transportation stations (subway stations, bus stops), major facilities such as schools, or public facilities as POIs to be displayed in the AR view (10). Alternatively, the user terminal (100) may determine POIs of a category selected according to the user's selection as POIs to be displayed in the AR view (10). Such categories may include, for example, a public facility category / a natural object category (mountains, rivers, etc.) / a store category (restaurants, shopping malls, etc.) / a house category (apartments, officetels, etc.).
[0205] Next, as a more specific example of steps (1910 and 1920), an example of displaying a two-dimensional map in an AR view as additional information is described in more detail.
[0206] The user terminal (100) can obtain a two-dimensional map including the external area as additional information about the external area indicated by the three-dimensional map (20). The two-dimensional map may be provided as a commercial map through a separate map application (e.g., a map application providing two-dimensional and / or three-dimensional map services), and the two-dimensional map may be obtained from such a map application or map service.
[0207] The user terminal (100) can display a two-dimensional map including the external area by augmenting the image within the AR view (10) with additional information about the external area described above. This two-dimensional map can be displayed in an area corresponding to the external area outside the three-dimensional map.
[0208] Accordingly, a 3D map (20) may be displayed for the user's area of interest in the AR view (10), and a 2D map may be displayed for an external area outside the area of interest. The area of interest may be indicated by a 3D map (20) and may refer to an area that the user must directly check with their own eyes and conduct a field survey, and the external area may refer to a certain range of areas surrounding the area.
[0209] The two-dimensional map may include, for example, at least one of a vector map of the external area, a satellite map of the external area, a cadastral map of the external area, a topographic map of the external area, and a schematic map of the external area. These two-dimensional maps may be layered and displayed in the AR view (10).
[0210] A two-dimensional map can be selectively displayed in an augmented form on an image within an AR view (10) according to a selection from a user terminal (100). In other words, the user terminal (100) can receive a user's selection of a layer to be displayed in the AR view (10) among at least one or each of a vector map, a satellite map, a cadastral map, a topographic map, and a schematic map, and can display a two-dimensional map corresponding to the selected layer in the AR view (10).
[0211] The user terminal (100) can display a two-dimensional map by overlaying it on a three-dimensional map (20). The three-dimensional map (20) can be displayed on top of the two-dimensional map. Accordingly, the two-dimensional map may not be discernible in the area where the three-dimensional map (20) of the AR view (10) is displayed.
[0212] However, depending on the embodiment, a 2D map may be displayed overlapping an area where a 3D map (20) is displayed. In this case, the 2D map may be displayed semi-transparently in the AR view (10).
[0213] Meanwhile, the two-dimensional map can be configured to be displayed semi-transparently in the AR view (10) to minimize interference with the image of the AR view (10).
[0214] Additionally, the 2D map may display additional content as described above.
[0215] As described, in embodiments, a two-dimensional map and / or additional content may be displayed in the AR view (10) as additional information about the external area.
[0216] The description of the technical features described above with reference to FIGS. 1 to 18 can also be applied to FIG. 19, so any duplicate description will be omitted.
[0217]
[0218] Below, with reference to FIGS. 20 and 21 to 24, a method of displaying a two-dimensional map and / or additional content as additional information about an area outside of the area indicated by the three-dimensional map (20) in the AR view (10) is described in more detail.
[0219] First, FIG. 20 illustrates a method for matching coordinates of a 3D map with coordinates of a 2D map representing an area of the 3D map and an area outside the area, according to an example.
[0220] As illustrated, an external region can be identified by matching (or syncing) the coordinates on the two-dimensional map (2000) with the coordinates on the three-dimensional map (20). That is, by matching the coordinates of a certain region on the three-dimensional map (20) with the coordinates on the two-dimensional map of an area including the region (2010), an external region (2020) outside the three-dimensional map (10), which is an external region of the certain region, can be identified. Such coordinate matching and external region identification processing can be performed on the server (200).
[0221] The user terminal (100) can obtain at least one piece of information from among a two-dimensional map of an external area (2020) identified according to the coordinate matching as described above, real-time information on the identified external area (2020), and information extracted from the two-dimensional map of the identified external area (2020) as additional information on the aforementioned external area (2020).
[0222] As the external area (2020) is identified, the location where additional information about the external area (2020) will be displayed in the AR view (10) can also be determined.
[0223] Next, specific examples of displaying additional information about an external area (2020) will be described in more detail with reference to FIGS. 21 to 24.
[0224] Figures 21 to 23 illustrate a method for displaying a two-dimensional map in an external area outside a three-dimensional map within an AR view, according to an example. Figure 24 illustrates a method for displaying POI information as additional content in an external area outside a three-dimensional map within an AR view, according to an example.
[0225] As illustrated in FIGS. 21 to 23, a two-dimensional map of an external area (2020) identified around a three-dimensional map (20) can be displayed.
[0226] In FIG. 21, within the AR view (10), a general two-dimensional map (e.g., a vector map) (2100) is displayed around a three-dimensional map (20). As illustrated, additional information about an external area (2020) outside the three-dimensional map (20) may be provided through the two-dimensional map (2100) of the external area (2020). Meanwhile, as illustrated, additional information (2110) may be displayed outside the two-dimensional map (2100). This additional information (2110) may include POI(s), as illustrated. As for a method for displaying additional information (2110), the description of the method for displaying additional content described above may be similarly applied, and thus, a redundant description thereof will be omitted.
[0227] In Fig. 22, within the AR view (10), a two-dimensional map (2200), which is a satellite map, is displayed around the three-dimensional map (20). As illustrated, additional information about the external area (2020) outside the three-dimensional map (20) can be provided through the satellite map (2200) of the external area (2020).
[0228] In Fig. 23, a two-dimensional map (2300), which is a topographic map (cadastral editing map), is displayed around a three-dimensional map (20) within the AR view (10). As illustrated, additional information about the external area (2020) outside the three-dimensional map (20) can be provided through the topographic map (2300) of the external area (2020). For example, a rough topography or land information of the external area (2020) can be identified through the topographic map (2300). As illustrated, when displaying the topographic map (2300), the external area (2020) may be set wider than when displaying other two-dimensional maps.
[0229] In FIG. 24, within the AR view (10), a plurality of POIs (2410) are displayed around a three-dimensional map (20). The POIs (2410) provide additional information about an external area (2020), and may be extracted from, for example, the two-dimensional maps (2100, 2200, 2300) described above. The description of the method for displaying additional content described above may be similarly applied to the method for displaying the POIs (2410), and thus, a redundant description will be omitted.
[0230] The two-dimensional maps (2100, 2200, 2300) described with reference to FIGS. 21 to 23 may be obtained from a map service that provides such two-dimensional maps.
[0231] Additionally, as described above, additional information about the external area (2020) may be configured to include real-time information. This real-time information may be obtained from a map service or a separate service providing real-time information.
[0232] As described with reference to FIGS. 21 to 24, in an embodiment, the user terminal (100) may mashup at least one piece of information from a service providing a two-dimensional map and / or a service providing real-time information, in addition to a service providing an AR view (10) or a service providing a three-dimensional map (20), in order to display additional information about an external area (2020) in the AR view (10), and thus display the additional information together with the three-dimensional map (20) in the AR view (10).
[0233] In this way, in the embodiment, additional information such as data from a two-dimensional map can be mashed up to provide richer information about an external area (2020) on the AR view (10). Accordingly, a user using the AR view (10) and the three-dimensional map (20) can obtain necessary information not only about the area (2010) indicated by the three-dimensional map (20) but also about the external area (2020). Accordingly, more effective virtual forest management of the area (2010) can be enabled.
[0234] The description of the technical features described above with reference to FIGS. 1 to 19 can also be applied to FIGS. 20 to 24, so redundant descriptions are omitted.
[0235]
[0236] Below, with reference to FIGS. 25 to 31, a method and device for providing a user interface that allows a user to input content for an image / 3D map (20) displayed within an AR view (10), and reflecting the content input through the user interface on the 3D map (20) / image displayed within the AR view (10) and displaying it (i.e., displaying corresponding content) are described.
[0237] In embodiments, a first UI may be provided that allows a user to input (i.e., author) content at a selected location within an image of an AR view (10), and the authored content may be reflected as corresponding content in a three-dimensional map (20) that includes the selected location. In addition, a second UI may be provided that allows a user to input (i.e., author) content at a selected location within the three-dimensional map (20), and the authored content may be reflected as corresponding content in an image of an AR view (10) that displays the selected location.
[0238]
[0239] FIG. 25 is a flowchart illustrating a method for controlling a user terminal to input first content in an AR view and output first corresponding content corresponding to the first content in a 3D map, or to input second content in a 3D map and output second corresponding content corresponding to the second content in an AR view, according to one embodiment.
[0240] In step (2510), (1) the user terminal (100) may provide a first user interface (UI) for inputting first content at a selected first location within an image of an AR view (10) including an image captured by a camera of the user terminal (100). For example, the user may select (or touch) any location for inputting the first content within the image of the AR view (10). The first UI may be a UI for inputting at least one of audio, favorites, bookmarks, text, and images (including images and / or videos) as the first content. Alternatively, the first UI may be a UI for inputting an action to be executed on the user terminal (100). Alternatively, the user terminal (100) may provide a second user interface (UI) for inputting second content at a selected second location within a three-dimensional map (20) displayed by augmenting the image within the AR view (10). For example, a user may select (or touch) any location on a 3D map (20) to input second content. The second UI may be a UI for inputting at least one of audio, favorites, bookmarks, text, and images (including images and / or videos) as the second content. Alternatively, the second UI may be a UI for inputting an action to be executed on the user terminal (100).
[0241] In step (2520), (1) when first content is input through the first UI, the user terminal (100) can output first corresponding content corresponding to the first content when a three-dimensional map (20) including a first location is displayed in the AR view (10). Accordingly, the first content input in the image of the AR view (10) through the first UI can be reflected and output in the corresponding three-dimensional map (20). (2) When the second content is input through the second UI, the user terminal (100) can output second corresponding content corresponding to the second content in the image of the AR view (10) including the second location. Accordingly, the second content input in the three-dimensional map (20) through the second UI can be reflected and output in the corresponding AR view (10).
[0242] The embodiment of (1) of the above-described steps (2510, 2520) is described in more detail with reference to FIG. 26, which will be described later, and the embodiment of (2) is described in more detail with reference to FIG. 26, which will be described later.
[0243] The description of the technical features described above with reference to FIGS. 1 to 24 can also be applied to FIG. 25, so any duplicate description will be omitted.
[0244]
[0245] FIG. 26 is a flowchart illustrating a method for controlling a user terminal to input first content in an AR view and output first corresponding content corresponding to the first content on a 3D map, according to an example.
[0246] Referring to FIG. 26, an embodiment is described in more detail in which a first UI is provided for inputting first content at a first location within an AR view (10), and the first content is input at the first location.
[0247] In step (2610), the user terminal (100) may display an AR view (10) including an image captured by a camera. The AR view (10) may include a first location.
[0248] In step (2620), the user terminal (100) can input first content through the first UI at the first location.
[0249] The first position may be, for example, any location within the image of the AR view (10). Alternatively, the first position may be a preset location within the image of the AR view (10) to enable input of the first content. For example, the first position may be at least one of the POIs (Points of Interest) displayed in the image, and the selected first position may correspond to the selected POI. In this case, the first content may be various types of content input in relation to the selected POI. Alternatively, the first position may correspond to an object displayed in the image or a selected location on an object. For example, the first position may include a building, a signboard of the building, a selected location on the exterior wall of the building, an entrance to the building, an entrance to the building's parking lot, etc.
[0250] A user can select (or touch) this first location within the video, and accordingly, the user terminal (100) can call the first UI to provide an environment for the user to input first content. For example, the first UI may be a UI for inputting at least one of audio, favorites, bookmarks, text, and images as the first content. Alternatively, the first UI may be a UI for inputting an action to be executed on the user terminal (100).
[0251] In step (2630), the user terminal (100) may display a three-dimensional map (20) including a first location within the AR view (10) according to the location of the user terminal (100) or a request by a user of the user terminal (100). The request may be for displaying a three-dimensional map (20) of an area including a location designated by the user within the AR view (10).
[0252] In step (2640), the user terminal (100) may output first corresponding content in association with a first location within a three-dimensional map (20) including a first location. For example, the first corresponding content may be output in association with a first location within a three-dimensional map (20) that is identical to the first location where the first content of the image of the AR view (10) is input, or with an object within the three-dimensional map (20) including the first location. The first corresponding content may include the input first content and may be processed from the first content for output associated with the three-dimensional map (20). For example, the first corresponding content may be processed from the first content for displaying on the three-dimensional map (20). Alternatively, the first corresponding content may be a different type of content from the first content. For example, if the first content is text, the first corresponding content may be audio content. This first corresponding content may be automatically output when a 3D map (20) including a first location is displayed in an AR view (10), or may be output from a user terminal (100) when a first location within the 3D map (20) or an object including the first location is selected, or when an icon or AR content within the 3D map (20) representing the first corresponding content is selected.
[0253] The user terminal (100) can determine whether the user terminal (100) is located at a second location through positioning. When the user terminal (100) is located at the second location, the user terminal (100) can output second corresponding content in association with the second location within the image of the AR view (10). If the second location is included in the image of the AR view (10), the user terminal (100) can be determined to be located at the second location based on positioning.
[0254] In this way, in the embodiment, the location where the second content is input on the 3D map (20) and the location within the AR view (10) determined based on the positioning can be linked (or synchronized), and thus, the user can load the 3D map (20) including the second location without directly visiting the second location, create the second content at the second location, and move to the actual second location to check the second corresponding content (corresponding to the created second content) through the AR view (10).
[0255] For example, the first content may include at least one of audio, favorites, bookmarks, text, and images. For example, the first content may be at least a portion of a memo or checklist entered for a specific property or object. In this case, the output of the first corresponding content may include at least one of the following: playback of the audio, display of the text, and display of the image. In other words, in the embodiment, the user can confirm the memo or checklist entered for a specific property or object through the image of the AR view (10) on a corresponding 3D map (20).
[0256] As another example, a path along which a user terminal (100) moves may be input as first content through the first UI. In this example, the first location may be the starting point of the path along which the user terminal (100) moves. For example, the first UI or the user terminal (100) may record the movement of the user terminal (100) from the first location, which is the starting point of the path. The user terminal (100) may record the location coordinates along which the user terminal (100) moves through positioning, or may record the movement of the user terminal (100) by recording an AR view (10). The first UI may provide such a record. The record may end when the user terminal (100) reaches a predetermined destination. For example, the record may end when the user inputs through the first UI that the destination has been reached, or when the user inputs a request to stop recording through the first UI. Accordingly, the input first content may include information about the path from the starting point to the destination.
[0257] The user terminal (100) can display a path from a starting point to a destination within a three-dimensional map (20) including a first location (which is a starting point). Through this embodiment, the user can check the path actually traveled through a corresponding three-dimensional map (20) while viewing the AR view (10). The path can be composed of nodes (which represent points designated by the user terminal (100) during movement or where the user terminal (100) stayed for a certain period of time) and edges connecting the nodes, and these nodes and edges can be displayed on the three-dimensional map (20).
[0258] In step (2650), the user terminal (100) may further provide a third UI for inputting additional content for the first corresponding content displayed on the 3D map (20). The third UI may be a UI for inputting at least one of audio, favorites, bookmarks, text, and images as additional content. Alternatively, the third UI may be a UI for inputting an action to be executed on the user terminal (100).
[0259] In step (2660), when additional content is input through the third UI, the user terminal (100) can reflect the additional content in the AR view (10) including the first location. Accordingly, the first content and the additional content can be output together in the image of the AR view (10) (or in relation to the first location of the image).
[0260] For example, a user can input a memo at a first location while moving while viewing an image of an AR view (10), and then when returning home, a 3D map (20) including the first location can be displayed on the AR view (10) to check the memo at the first location of the 3D map (20) on the AR view (10). In addition, the user can input an additional memo for the memo displayed at the first location of the 3D map (20), and when the user then moves to an actual first location and includes the first location in the image of the AR view (10), the additional memo can be displayed in association with the first location in the image to the existing memo.
[0261] Accordingly, the user can organize and check the notes recorded while moving through the actual space while running the AR view (10) through the 3D map (20), and can also mark the items that need to be additionally checked on the 3D map (20) as notes and conveniently check them through the AR view (10) in the actual space.
[0262] The description of the technical features described above with reference to FIGS. 1 to 25 can also be applied to FIG. 26, so redundant descriptions are omitted.
[0263]
[0264] FIG. 27 is a flowchart illustrating a method for controlling a user terminal to input second content on a 3D map and output second corresponding content corresponding to the second content in an AR view, according to an example.
[0265] Referring to FIG. 27, an embodiment in which a second UI is provided for inputting second content at a second location within a three-dimensional map (20), and the second content is input at the second location, will be described in more detail.
[0266] In step (2710), the user terminal (100) may display a three-dimensional map (20) including a second location within the AR view (10) according to the location of the user terminal (100) or a request by a user of the user terminal (100). The area indicated by the three-dimensional map (20) may include the second location.
[0267] In step (2720), the user terminal (100) can input second content through the second UI at the first location.
[0268] The second location may be, for example, any location within the 3D map (20). Alternatively, the second location may be a preset location within the 3D map (20) that allows input of second content. For example, the second location may be at least one of the POIs displayed on the 3D map (20), and the selected second location may correspond to the selected POI. Alternatively, the second location may correspond to an object displayed on the 3D map (20) or a selected location on an object. For example, the second location may include a building, a signboard of the building, a selected location on the exterior wall of the building, an entrance to the building, an entrance to the building's parking lot, etc.
[0269] At this time, the second content may be various types of content entered in relation to the selected POI or selected location.
[0270] The user can select (or touch) this second location within the 3D map (20), and accordingly, the user terminal (100) can call up the second UI to provide an environment for the user to input second content. For example, the second UI may be a UI for inputting at least one of audio, favorites, bookmarks, text, and images as the second content. Alternatively, the second UI may be a UI for inputting an action to be executed on the user terminal (100).
[0271] In step (2730), the user terminal (100) may display an AR view (10) including an image captured by a camera. The AR view (10) may be executed in an environment including an actual second location, and thus, the image of the AR view (10) may include the second location.
[0272] In step (2740), if a second location is included in the image of the AR view (10), the user terminal (100) may output second corresponding content in association with the second location within the image. For example, the second corresponding content may be output in association with a second location within the image of the AR view (10) that is identical to the second location where the second content of the 3D map (20) is input, or an object within the 3D image that includes the second location. The second corresponding content may include the input second content, and may be processed second content for output in association with the image within the AR view (10). For example, the second corresponding content may be processed second content for display in the image within the AR view (10). Alternatively, the second corresponding content may be a different type of content from the second content. For example, if the second content is text, the second corresponding content may be audio content. This second corresponding content may be automatically output when included in an image of an AR view (10) including a second location, or may be output from a user terminal (100) when a second location or an object including the second location is selected in an image of an AR view (10), or when an icon or AR content within an image of an AR view (10) representing the second corresponding content is selected.
[0273] The user terminal (100) can determine whether the user terminal (100) is located at a second location through positioning. When the user terminal (100) is located at the second location, the user terminal (100) can output second corresponding content in association with the second location within the image of the AR view (10). If the second location is included in the image of the AR view (10), the user terminal (100) can be determined to be located at the second location based on positioning.
[0274] In this way, in the embodiment, the location where the second content is input on the 3D map (20) and the location within the AR view (10) determined based on the positioning can be linked (or synchronized), and thus, the user can load the 3D map (20) including the second location without directly visiting the second location, create the second content at the second location, and move to the actual second location to check the second corresponding content (corresponding to the created second content) through the AR view (10).
[0275] For example, the second content may include at least one of audio, favorites, bookmarks, text, and images. For example, the second content may be at least a portion of a memo or checklist entered for a specific property or object. In this case, the output of the second corresponding content may include at least one of the following: playback of the audio, display of the text, and display of the image. In other words, in the embodiment, a user can note down items to be checked in a real-world environment for a specific property or object through a 3D map (20), and when moving to the real-world environment and executing the AR view (10), the user can confirm these notes through an image.
[0276] As another example, the second content may be a viewpoint or perspective viewed by the user on a 3D map (10). An image of the 3D map (10) from this specific viewpoint or perspective may be captured and input as the second content.
[0277] A 3D map (20) including a second location can be displayed within the AR view (10) as a mini-map of the area including the second location. This 3D map (20) can serve as auxiliary reference data for the user's movement. Furthermore, as described above, the 3D map (20) can be displayed on the user terminal (100) in full screen to provide a more immersive experience to the user.
[0278] As in step (2742), if the second location is included in the image of the AR view (10), the user terminal (100) can play or display content, execute a predetermined action, or execute a function such as route guidance (route guidance may be an example of an action).
[0279] Below, an example is described in more detail where the second content includes a predetermined action to be executed by the user terminal (100). In this case, the user terminal (100) can execute the corresponding action if the second location is included in the image of the AR view (10).
[0280] Execution of an action in the user terminal (100) may include, for example, i) providing route guidance from a first point corresponding to a second location to a second point, ii) starting recording / recording of an AR view (10), iii) linking with an IoT device placed in association with the second location, iv) calling a specific contact of the user terminal (100), or v) providing real-time information associated with the second location. Alternatively, execution of the action may be execution of at least one of the actions i) to v) described above.
[0281] i) The first point may be the starting point of the route, and the second point may be the destination of the route. For example, if the first point is an apartment entrance and the second point is a public transportation station or a school, in the embodiment, a route guidance action may be input at the apartment entrance in the 3D map (20), and when the AR view (10) is executed at the actual apartment entrance, route guidance from the apartment entrance to the public transportation station or school may be provided.
[0282] ii) Recording / recording of the AR view (10) may be recording / recording of an image of the AR view (10) or recording the movement of the user terminal (100). When the AR view (10) is executed and a second location is included in the image, recording / recording of the AR view (10) may begin.
[0283] iii) The linkage with the IoT device may be a linkage with an IoT device deployed in a real environment around the second location. For example, if the real environment is a construction site or a work site, the IoT device may be a device for calling a specific person. In this example, when the AR view (10) is executed and the second location is included in the image, the user terminal (100) may be linked with the IoT device, and thus the task of calling a specific person may be performed. Meanwhile, the IoT device may be a lighting device or an audio device, and thus, when the AR view (10) is executed and the second location is included in the image, the user terminal (100) may control the lighting device or the audio device to operate.
[0284] iv) A call to a specific contact on the user terminal (100) may be a call request or message transmission to a specific contact stored on the user terminal (100), a specific phone number, or a specific user (e.g., a friend) registered in a messenger application. For example, a user visiting an area for a real estate transaction can, through this embodiment, automatically contact a real estate agency, etc. by executing the AR view (10) upon arrival at a specific second location.
[0285] v) Real-time information may include at least one of weather information, traffic information, and air quality information for the area associated with the second location. The real-time information may be obtained from an external map service or a separate external service that provides such real-time information. For example, a user occupying an area may, through this embodiment, mark a point on a 3D map (20) where air quality or traffic information should be checked, and when actually moving to that point, check the air quality or traffic information through the AR view (10).
[0286] The action corresponding to the provision of the above real-time information can be executed only when the time at which the second location is included in the image of the AR view (10) falls within a preset time zone. This time zone may be set by the user or may be a preset time zone. For example, if the second location is displayed in the image of the AR view (10) at a specific time zone, it may be implemented to check air quality, and if the second location is displayed in the image of the AR view (10) at another specific time zone, it may be implemented to check traffic volume. Meanwhile, the real-time information described with reference to FIGS. 1 to 26 may further include information on the amount of sunlight, noise information, etc., in addition to the examples described above.
[0287] Below, as another example, a method of inputting a path along which a user terminal (100) moves as second content through a second UI is described in more detail. In this example, the second location may be the starting point of a path specified on a three-dimensional map (20) including the second location. For example, the second UI may be a UI for a user to input a path on the three-dimensional map (20) from the second location, which is the starting point of the path, to the destination. Accordingly, the input second content may include information about the path from the starting point to the destination.
[0288] At least a portion of the above-described route may be designated by a user of the user terminal (100) via a second UI on a three-dimensional map (20) including a second location. For example, the user may designate at least a portion of the route by actions such as drawing a line or selecting a road on the three-dimensional map (20), or by selecting or inputting at least one of a starting point and a destination.
[0289] Alternatively, as described above with respect to the 3D map (20), the 3D map (20) including the second location may be displayed on the user terminal (100) so that the user of the user terminal (20) can virtually explore the area including the second location. For example, the user may virtually explore the area by moving the 3D map (20) in an enlarged state or by using a screen touch, etc. At least a part of the path may be an automatically recorded path that the user virtually explored on the 3D map (20) including the second location (i.e., a path that the user virtually explored was tracked and stored). Accordingly, even if the user does not directly designate the path on the 3D map, the path that the user virtually explored may be input as the second content.
[0290] When a second location, which is the starting point of a route, is included in the image of the AR view (10), the user terminal (100) can provide guidance for the route from the starting point corresponding to the second location in the image of the AR view (10) to the destination. The route guidance can be provided through AR content displayed in the image of the AR view (10).
[0291] Next, additional embodiments are described with reference to steps (2750, 2760).
[0292] In step (2750), the user terminal (100) may further provide a third UI for inputting additional content for the second corresponding content output in association with the second location within the image of the AR view (10). The third UI may be a UI for inputting at least one of audio, favorites, bookmarks, text, and images as additional content. Alternatively, the third UI may be a UI for inputting an action to be executed on the user terminal (100).
[0293] In step (2760), when additional content is input through the third UI, the user terminal (100) can reflect the additional content on the three-dimensional map (20) including the second location. In other words, when the user terminal (100) displays the three-dimensional map (20) including the second location, when additional content is input through the third UI, the user terminal (100) can output the additional content in association with the second content at the second location of the three-dimensional map (20) including the second location. Accordingly, the second content and the additional content can be output together on the three-dimensional map (20) (or in association with the second location of the three-dimensional map (20).
[0294] For example, a user can virtually explore an area through a 3D map (20) and input a memo at a second location, and then actually move to the area, execute an AR view (10), and explore the area. At this time, the memo can be confirmed through the second location in the image of the AR view (10). In addition, the user can further input an additional memo for the memo displayed through the image of the AR view (10), and when the user returns home and displays a 3D map (20) including the second location in the AR view (10) on the user terminal (100), the 3D map (20) can display the additional memo in addition to the existing memo.
[0295] Accordingly, the user can check the notes recorded on the 3D map (20) while moving through the actual space by executing the AR view (10), and can also organize and check the notes recorded in the image of the AR view (10) while moving through the actual space through the 3D map (20).
[0296] The description of the technical features described above with reference to FIGS. 1 to 26 can also be applied to FIG. 27, so any duplicate description will be omitted.
[0297]
[0298] It goes without saying that there may be multiple first locations where the first content described above is input and multiple second locations where the second content is input.
[0299] Meanwhile, the 3D map (20) of the embodiment (i.e., including the first location and / or the second location and having the content authored thereon) can be shared with other user terminals. In other words, such a 3D map (20) can be displayed on another user terminal according to the location of the other user terminal or a request by a user of the other user terminal. Accordingly, the first corresponding content can be output in association with the first location within the 3D map (20) displayed on the other user terminal. Through this embodiment, for example, when a first user inputs the first content at the first location of the AR view (10), when a second user displays the 3D map (20) including the first location on his / her user terminal, the first corresponding content corresponding to the first content can be output on the 3D map (20). A similar description can be applied to the second content and the second corresponding content. In this way, in the embodiment, content created for the image of the 3D map (20) and / or AR view (10) can be shared between users (i.e., between friends or family).
[0300] In order to share such content and link content between the aforementioned AR view (10) and the 3D map (20), the user terminal (100) may transmit the first content input through the first UI together with 6dof information of the first location to a cloud server. In addition, the user terminal (100) may transmit the second content input through the second UI together with 6dof information of the second location to a cloud server. The cloud server may be the aforementioned server (200). In this way, since the first location and the second location may have 3D spatial information (e.g., including height, rotation information, etc.), the content link between the AR view (10) and the 3D map (20) may be performed more precisely. For example, the first content input at the first location of the image of the AR view (10) may be reflected as the first corresponding content at the (more accurate) first location of the 3D map (20) in consideration of the 3D spatial information of the first location. That is, the positioning between the AR view (10) and the 3D map (20) can be linked more accurately.
[0301] On the above cloud server, a positioning system may be implemented to link (or connect) a location in the aforementioned 3D map (20) with a location in an actual space. For example, through this positioning system, a location on the 3D map (20) (e.g., a location where content is created, etc.) may be linked with a location in an actual space representing the location (or a location in the AR view (10)). Accordingly, a location in an actual space according to the positioning of the user terminal (100) may be linked with a location on the 3D map (20) of the embodiment. In this way, by linking a location on the 3D map (20) with a location in an actual space, content or actions input at a location on the 3D map (20) may be output at a corresponding location in an actual space.
[0302] A 3D map (20) is a 3D model of a space (environment) and can be reconstructed based on the results of mapping the space. A slam map for positioning the space can also be generated based on the results of mapping the space. In addition, the 3D map (20) can be configured to include positioning information in a top-down direction. Accordingly, the 3D map (20) can be implemented to allow the user to experience a virtual tour or forest more realistically. Accordingly, the content linkage between the AR view (10) and the 3D map (20) can also be more sophisticated.
[0303]
[0304] Below, a concrete example of content linkage between an AR view (10) and a 3D map (20) is described in more detail with reference to FIGS. 28 to 31.
[0305] FIG. 28 illustrates a method for confirming content created at a specific location on a 3D map in an AR view including the specific location, according to an example, and reflecting the content confirmed (or confirmed) in the AR view on the 3D map.
[0306] The first drawing of Fig. 28 illustrates a case where a note (2810) titled "Check parking lot entrance slope" is registered as second content at a second location (e.g., the parking lot entrance of an apartment complex) on a 3D map (20). In this way, a user can identify a location and items to be checked in an actual field through the 3D map (20) and leave a note on the 3D map (20).
[0307] The second drawing of FIG. 28 illustrates a case where a second location (e.g., the entrance to an apartment complex's parking lot) is included in the image of the AR view (10). In this case, the memo "Check parking lot entrance slope" (i.e., second corresponding content) (2820) entered in the first drawing may be displayed in the image of the AR view (10). Accordingly, the user can check the actual slope of the parking lot entrance without forgetting it through the memo (2820).
[0308] The third picture of FIG. 28 illustrates an example where additional content is input to the memo “Check parking lot entrance slope” (i.e., second corresponding content) (2820) output from the image of the AR view (10). That is, the memo “Check parking lot entrance slope - OK” (i.e., second corresponding content) (2830) further includes the additional content “- OK” indicating that the slope of the parking lot entrance has actually been checked.
[0309] The fourth drawing of Figure 28 illustrates a case where the above-mentioned additional content is further included in a note (2840) displayed at a second location (the entrance to the parking lot of an apartment complex) on the 3D map (20). In this way, the additional content "- OK," indicating that the actual slope of the parking lot entrance has been confirmed, can also be reflected on the 3D map (20).
[0310] As explained above, the 3D map (20) can be shared with other user terminals of other users, and thus, other users can also confirm the reflection of such additional content or the creation of secondary content through the 3D map (20).
[0311] FIG. 29 illustrates a method for inputting an action to be executed by a user terminal as second content at a specific location on a 3D map according to an example, and for executing the action when the specific location is included in the AR view.
[0312] In Fig. 29, an example is provided where the "action" performed by the user terminal (100) is the provision of route guidance. For example, as illustrated, an action that performs route guidance from a selected point (second location, parking lot entrance) to a specific destination (N building) can be input as second content on a 3D map (20, 2910). When the user terminal (100) moves to the second location and the second location is included in the AR view (10, 2920), the user terminal (100) can provide route guidance from the second location to the destination.
[0313] FIG. 30 illustrates a method for displaying a three-dimensional map including a current location within an AR view including an image captured at the current location of a user terminal according to an embodiment of the present invention.
[0314] In Fig. 30, a 3D map (20) of an area including the current location of the user terminal (100) is displayed within the AR view (10). As illustrated, the same building is displayed in the image of the AR view (10) and the 3D map (20). A note (second content) titled "Logo Confirmation" may be input as second content at the location of the building signboard in the 3D map (20), and thus, "Logo Confirmation" (second corresponding content) may be displayed identically on the same building within the image of the AR view (10). When the second content is input, the 3D spatial information of the input second location is also transmitted to the server (200), and thus, the second corresponding content may be displayed at the exact location corresponding to the second location in the image of the AR view (10).
[0315] FIG. 31 illustrates a method for providing guidance to a user terminal for an input path when a path is input as second content at a specific location on a 3D map and the specific location is included in an AR view, according to an example.
[0316] In the illustrated example, the note "Check the route from the entrance of complex 2 to the bus stop" (3110) and the route (3120) are input as second content at the starting point (second location) of the route on the 3D map (20). The input of the route and the note can be performed before the user actually goes to the area. When the user moves to the actual location corresponding to the starting point and executes the AR view (10) on the user terminal (100), the note "Check the route from the entrance of complex 2 to the bus stop" and the route guidance to the destination can be provided as second corresponding content.
[0317] Meanwhile, the designated route may include multiple stopovers in addition to the destination, unlike the one shown. When the user terminal (100) reaches a stopover, AR content indicating that the user has reached the stopover may be displayed on the AR view (10).
[0318] According to the content linkage method between the AR view (10) and the 3D map (20) described with reference to FIGS. 25 to 31, an AR to AR content authoring system between the AR view (10) and the 3D map (20) can be implemented.
[0319] The description of the technical features described above with reference to FIGS. 1 to 27 can also be applied to FIGS. 28 to 31, so any duplicate description will be omitted.
[0320] The AR view (10) and 3D map (20) of the embodiments described above with reference to FIGS. 1 to 31 can be utilized for virtual forestry, virtual tours, travel planning, construction / work site management, etc.
[0321] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.
[0322] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.
[0323] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.
[0324] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.
[0325] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.
Claims
1. A method for controlling a user terminal performed by a computer system which is a server or a user terminal, A step of providing a first user interface (UI) for inputting first content at a selected first location within an image of an augmented reality (AR) view including an image captured by a camera of the user terminal, or providing a second user interface (UI) for inputting second content at a selected second location within a three-dimensional map displayed by augmenting the image within the AR view; and When the first content is input through the first UI, a step of outputting first corresponding content corresponding to the first content when a 3D map including the first location is displayed within the AR view, or when the second content is input through the second UI, a step of outputting second corresponding content corresponding to the second content in an image of the AR view including the second location A method for controlling a user terminal, comprising:
2. In paragraph 1, In the case where the first UI is provided and the first content is input at the first location, A step of displaying a 3D map including the first location within the AR view according to the location of the user terminal or a request by a user of the user terminal. Including more, The above outputting step is: A method for controlling a user terminal to output the first corresponding content in association with the first location within a three-dimensional map including the first location.
3. In paragraph 2, The first content includes at least one of audio, text, and image, A method for controlling a user terminal, wherein the output of the first corresponding content includes at least one of playback of the audio, display of the text, and display of the image.
4. In paragraph 2, A 3D map including the first location is displayed on another user terminal according to the location of the other user terminal or a request by a user of the other user terminal, A method for controlling a user terminal, wherein the first corresponding content is output in association with the first location within a three-dimensional map displayed on the other user terminal.
5. In paragraph 2, The above first location corresponds to a POI selected from among the POIs (Points Of Interest) displayed in the image, A method for controlling a user terminal, wherein the first content is input in association with the selected POI.
6. In paragraph 2, The above first location is the starting point of the path along which the user terminal moves, The above first content includes information about the route from the starting point to the predetermined destination, The above outputting step is: A method for controlling a user terminal, comprising the step of displaying the route from the starting point to the destination within a three-dimensional map including the first location.
7. In paragraph 1, A step of displaying a 3D map including the second location within the AR view according to the location of the user terminal or a request by a user of the user terminal. Including more, The second content is input in connection with the second location through the second UI, The above outputting step is: A method for controlling a user terminal to output the second corresponding content in association with the second location within the image of the AR view, if the second location is included in the image of the AR view.
8. In paragraph 7, A method for controlling a user terminal, wherein a three-dimensional map including the second location is displayed as a mini-map of an area including the second location within the AR view.
9. In paragraph 7, The second content includes at least one of audio, text, and image, A method for controlling a user terminal, wherein the output of the second corresponding content includes at least one of playback of the audio, display of the text, and display of the image.
10. In paragraph 7, The second content includes a predetermined action to be executed by the user terminal, The above outputting step is: A method for controlling a user terminal, comprising the step of executing the action on the user terminal when the second location is included in the image of the AR view.
11. In paragraph 10, Execution of the above action is: Providing route guidance from the first point corresponding to the second location to the second point, Start recording the above AR view, Interoperability with IoT devices placed in connection with the above second location, A call to a specific contact of the above user terminal, or Provision of real-time information related to the above second location A method for controlling a user terminal, comprising:
12. In paragraph 11, The above real-time information is, A method for controlling a user terminal, the method comprising at least one of weather information, traffic information, and air quality information of a region associated with the second location.
13. In paragraph 11, A method for controlling a user terminal, wherein the action corresponding to the provision of the real-time information is executed only when the time at which the second location is included in the image of the AR view falls within a preset time zone.
14. In paragraph 7, The second location is the starting point of a path specified on a three-dimensional map including the second location, The second content includes information about the route from the starting point to the destination, The above outputting step is: A method for controlling a user terminal, comprising a step of providing guidance for the route from the starting point corresponding to the second location to the destination within the image of the AR view.
15. In paragraph 14, A three-dimensional map including the second location is displayed on the user terminal so that the user of the user terminal can virtually explore an area including the second location, A method for controlling a user terminal, wherein at least a portion of the above path is automatically recorded as a path virtually explored by the user on a three-dimensional map including the second location.
16. In paragraph 7, The second location corresponds to a selected POI among POIs displayed on a 3D map including the second location, or the second location corresponds to a selected location on an object displayed on a 3D map including the second location, A method for controlling a user terminal, wherein the second content is input in association with the selected POI or the selected location.
17. In paragraph 9, A step of providing a third UI for inputting additional content for the second corresponding content output in relation to the second location within the image of the AR view; and When additional content is input through the third UI, when displaying a three-dimensional map including the second location, a step of outputting the additional content in association with the second content at the second location of the three-dimensional map including the second location A method for controlling a user terminal, further comprising:
18. In paragraph 1, A step of transmitting the first content input through the first UI to a cloud server together with 6dof information of the first location, or transmitting the second content input through the second UI to a cloud server together with 6dof information of the second location. A method for controlling a user terminal, further comprising:
19. In a computer system implementing a user terminal or server, At least one processor configured to execute computer-readable instructions contained in memory Including, At least one processor, A computer system that displays an augmented reality (AR) view including an image captured by a camera of the user terminal, provides a first user interface (UI) for inputting first content at a first location selected within the image, or provides a second user interface (UI) for inputting second content at a second location selected within a three-dimensional map displayed by augmenting the image within the AR view, and when the first content is input through the first UI, outputs first corresponding content corresponding to the first content when a three-dimensional map including the first location is displayed within the AR view, or outputs second corresponding content corresponding to the second content in an image of the AR view including the second location when the second content is input through the second UI.
20. A method for controlling a user terminal performed by a computer system which is a server or a user terminal, A step of displaying a three-dimensional map of an area including a first location, which is a location of the user terminal or a location specified by the user; A step of providing a first user interface (UI) for inputting content into a selected second location within the three-dimensional map; When the content is input through the first UI, if the second location is included in an AR (augmented reality) view including an image captured by a camera of the user terminal, a step of outputting corresponding content corresponding to the content in association with the second location within the image of the AR view; A step of providing a second UI for inputting additional content for the output corresponding content; and When additional content is input through the second UI, a step of outputting the additional content in association with the content at the second location of the three-dimensional map when displaying the three-dimensional map A method for controlling a user terminal, comprising:
Citation Information
Patent Citations
Touchscreen hover detection in an augmented and / or virtual reality environment
KR1020170130602A
Push-open type tumbler cap
KR1020240010295A
Plasma nitrogen oxide control method and apparatus for ships using SNCR
KR102350269B1
Augmented reality system with an active portable anchor
KR102566700B1
KR20200109812A
