A method and apparatus for displaying virtual reality content on a user terminal based on the user terminal's position within a pre-configured custom area.

Abstract

Provided is a user terminal control method that determines, based on the position of the user terminal, whether the user terminal is located in a preset custom area in space, determines whether the user terminal is located in a recognition area where the user terminal can recognize an object in the custom area, and, when the user terminal is located in the recognition area, recognizes the object and displays, in an AR view, virtual reality (VR) content that is preset in association with the recognized object.

Description

Technical Field

[0001] The following description relates to a method and apparatus for controlling a user terminal based on the position of the user terminal, and to a method and apparatus for displaying virtual reality (VR) content associated with a recognized object on the user terminal when the user terminal located in a preset custom area in space recognizes an object within the custom area.

Background Art

[0002] Augmented Reality (AR) means a technology that combines and complements virtual objects and information created by computer technology with the real world. That is, augmented reality is a technology that expands and displays virtual content in the actual environment, and a user can view the content expanded corresponding to the actual environment through an electronic device.

[0003] On a user terminal, which is a user's electronic device, content associated with an object existing within a predetermined distance or within a predetermined radius from the position of the user terminal is expanded and provided in the video of the user terminal. When the augmented content is provided only based on the physical distance from the user terminal like this, more content than necessary is expanded, which not only leads to an excessive cognitive load on the user but also an excessive load on the user terminal. Furthermore, it is also possible that content not suitable for the service required by the user is expanded on the user terminal.

[0004] In order to solve such problems, when a user is located in a specific area in space, a method is required that can display only highly useful content on the user terminal and provide a high level of immersion to the user.

[0005] As an example of technologies related to various services utilizing augmented reality technology, Korean Published Patent No. 10-2014-0065963 (published May 30, 2014, Patent Document 1) discloses an augmented reality navigator that is installed in a vehicle, displays video of the driving route captured by a camera on a display, and maps virtual display information that guides the driver along the driving route onto the video displayed on the display.

[0006] The information described above is provided solely to aid in understanding the present invention and may include content that is not part of the prior art, or may not include anything that the prior art could present to a person of ordinary skill. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Korean Published Patent No. 10-2014-0065963 (Publication Date: May 30, 2014) [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] This provides a method for displaying pre-configured VR content associated with a recognized object by a user terminal located in a pre-defined custom area within a space, upon recognizing the object within the custom area through an AR view.

[0009] To provide users with more detailed and comprehensive information about objects, VR content associated with objects within a custom region can be displayed using an AR viewer.

[0010] When a user terminal is located in a pre-configured custom area, the settings of the user terminal, including the execution of a module for object recognition, can be controlled according to the scenario conditions defined for that custom area.

[0011] By setting various scenario conditions for a custom area, different VR content can be provided to the user's device depending on whether the scenario conditions are met, and additional content can also be provided to the user's device. [Means for solving the problem]

[0012] In one aspect, the present invention provides a user terminal control method executed by a computer system which is a server or a user terminal, comprising the steps of: determining whether the user terminal is located in a pre-configured custom area in space based on the position of the user terminal; determining whether the user terminal is located in a recognition area in which objects within the custom area can be recognized; recognizing the object from an AR view including images captured by the user terminal's camera when the user terminal is located in the recognition area; and displaying pre-configured virtual reality (VR) content associated with the recognized object using the AR view.

[0013] The custom region is associated with at least one service element, the service element includes data for displaying the VR content related to the service associated with the space, and the custom region may define scenario conditions for controlling the user terminal based on the service element when the user terminal is located in the custom region.

[0014] The service element includes data for controlling the settings of the user terminal to provide the user terminal with the services associated with the space, and the user terminal control method may further include a step of controlling the settings of the user terminal, including the execution of a module for recognizing the object, in accordance with the scenario conditions, when the user terminal enters the custom area.

[0015] The step of determining whether the user terminal is located in the recognition area may be performed if the object is displayed in the AR view, in which case it is determined that the user terminal is located in the recognition area.

[0016] The step of determining whether the user terminal is located in the recognition area may be performed if at least one of the following conditions is met: the distance between the object and the user terminal is less than or equal to a predetermined distance; the object is displayed in a specific area within the AR view; and the object is displayed in the AR view at a predetermined rate or higher.

[0017] The recognition area is a predefined area within the custom area, and the step of determining whether the user terminal is located in the recognition area may be to determine that the user terminal is located in the recognition area if the user terminal is located in the predefined recognition area.

[0018] The user terminal control method may further include a step of controlling the settings of the user terminal, including the execution of a module for recognizing an object, when the user terminal enters the recognition area.

[0019] The step of displaying the VR content may further include changing the viewpoint in accordance with a change in the viewpoint of the camera and displaying the VR content, and the user terminal control method may further include ending the display of the VR content when the user terminal is no longer located in the recognition area.

[0020] The step of displaying the VR content is to display the VR content on the entire screen of the AR view, and the VR content may depict the virtual space around the object.

[0021] The virtual space may be represented by a virtual spherical space centered on the object.

[0022] The virtual space may be represented by a virtual three-dimensional space defined by reflecting the geometry of the surrounding space where the object is located.

[0023] When the user terminal is located in the custom area, the scenario condition is set such that, as the VR content, VR content determined based on other custom areas that the user terminal has entered before entering the custom area is displayed by the AR view. The step of displaying the VR content may display the VR content determined based on the other custom areas by the AR view.

[0024] When the user terminal is located in the custom area, the scenario condition is set such that, as the VR content, VR content determined based on the points To that the user terminal has passed through is displayed by the AR view. The step of displaying the VR content may display the VR content determined based on the points To that the user terminal has passed through by the AR view.

[0025] Inside or outside the custom area, a plurality of points To are defined. The step of displaying the VR content may display different VR content by the AR view when the user terminal passes through a first point to among the plurality of points To and when the user terminal passes through a second point to among the plurality of points To of the plurality of points.

[0026] Multiple custom regions are set in the space, and the scenario condition includes at least one of the following: i) a first scenario condition indicating whether the user terminal has passed through a predetermined other custom region that is not the custom region, and ii) a second scenario condition indicating whether the number of user terminals located in the custom region falls within a predetermined range, and the scenario condition is set to provide the user terminal with different services or information associated with the space based on the service elements, and / or to control the settings of the user terminal differently, depending on whether the scenario condition is met, and the step of displaying the VR content may display the VR content by the AR view only if the scenario condition is met, or display different VR content depending on whether the scenario condition is met or not.

[0027] The user terminal control method further includes the steps of receiving a request for route guidance to a destination at the user terminal, and providing route guidance to the user terminal in accordance with the request for route guidance, wherein the steps of determining whether the user terminal is located in the custom area, determining whether the user terminal is located in the recognition area, recognizing the object, and displaying the VR content may be performed when the user terminal moves to the destination in accordance with the route guidance.

[0028] The scenario conditions are set such that VR content determined based on the number of times the user terminal enters the custom area is displayed in the AR view, and the step of displaying the VR content may display the VR content determined based on the number of times the user terminal enters the custom area in the AR view.

[0029] In another aspect, the present invention provides a computer system that realizes a user terminal or a server, comprising at least one processor configured to execute computer-readable instructions contained in memory, wherein the at least one processor determines, based on the location of the user terminal, whether the user terminal is located in a pre-defined custom area in space, whether the user terminal is located in a recognition area capable of recognizing objects within the custom area, and, when the user terminal is located in the recognition area, recognizes the object from an AR view including images captured by the user terminal's camera, and displays pre-defined virtual reality (VR) content associated with the recognized object in the AR view. [Effects of the Invention]

[0030] Custom regions for controlling user terminals can be generated by defining scenarios for controlling user terminals according to a scenario, such as setting custom regions in 2D or 3D space on a spatial map, associating service elements with the custom regions, providing services or information associated with the space based on the service elements, and / or controlling the settings of user terminals.

[0031] When a user terminal is located in a pre-configured custom area, a module for object recognition is executed according to the scenario conditions defined for that custom area. By recognizing objects within the custom area, resource waste that could occur not only on the user terminal but also on the server communicating with the user terminal can be prevented.

[0032] By displaying VR content associated with objects within a custom domain through an AR view, more detailed and richer information about the objects can be provided to the user's device, offering a highly immersive experience.

[0033] By setting various scenario conditions for a custom area and providing different VR content to the user's device depending on whether the scenario conditions are met, as well as providing additional content to the user's device, the efficiency of content delivery can be improved. [Brief explanation of the drawing]

[0034] [Figure 1A] This figure illustrates a method for controlling a user terminal based on its location within a pre-configured custom area, according to one embodiment. [Figure 1B] This figure illustrates a method of controlling a user terminal so that VR content is displayed on the user terminal when the user terminal is positioned in a pre-set custom area and an object within the custom area is recognized, in one embodiment. [Figure 2] This figure shows a computer system that performs a user terminal control method in one embodiment. [Figure 3] This flowchart illustrates a method for controlling a user terminal so that VR content is displayed on the user terminal when the user terminal is positioned in a pre-defined custom area and an object within the custom area is recognized, in one embodiment. [Figure 4] This flowchart illustrates a method for controlling the settings of a user terminal when the user terminal enters a recognition area capable of recognizing an object within a custom area, in one embodiment. [Figure 5A] This figure illustrates a method for controlling the settings of a user terminal based on the location of the user terminal in a pre-configured custom area, according to one embodiment. [Figure 5B] This figure illustrates a method for providing VR content to a user terminal based on the user terminal being positioned in a pre-defined custom area, as described in one embodiment. [Figure 6] This figure illustrates a method for defining AR content associated with an object within a custom region, according to one embodiment. [Figure 7] This figure illustrates a method for defining VR content associated with an object within a custom region, according to one embodiment. [Figure 8] This figure illustrates a method for defining AR and VR content associated with objects within a custom region in one embodiment. [Figure 9] This figure shows a custom region defined as a three-dimensional space in one embodiment. [Figure 10] This figure shows the Degree of Freedom (DoF) of a user terminal in one embodiment. [Figure 11] This figure shows a custom region containing multiple points in one embodiment. [Figure 12] This figure illustrates a method in which, in one embodiment, multiple custom regions are set in a space, and scenario conditions for controlling a user terminal are set for each custom region. [Modes for carrying out the invention]

[0035] The embodiments will be described in detail below with reference to the attached drawings.

[0036] Figures 1A and 1B illustrate a method for controlling a user terminal based on its position within a pre-defined custom area, in one embodiment.

[0037] In the following embodiments, "control" of the user terminal 100 may include, for example, using the AR view 10 of the user terminal 100 to display content (e.g., AR content 30) that shows services and information associated with space 5 on the image of the AR view 10, as shown in Figures 1A and 1B; displaying a user interface (UI) associated with services associated with space 5 on the user terminal 100; displaying map element content 35 associated with services associated with space 5 on the map (or map view 20) of the user terminal 100; or controlling the settings of the user terminal 100 to provide services associated with space 5 to the user terminal 100. Furthermore, in embodiments of this disclosure, "control" of the user terminal is not limited to the examples described above and may include controlling the functions and / or operations of the user terminal.

[0038] In the embodiment, “Space 5” related to the provision of extended content may refer to an indoor space and / or an outdoor space. In other words, the definition of Space 5 described below may be a space that includes both indoor and outdoor locations, or it may be a composite space that combines indoor and outdoor spaces. Space 5 may be at least a part of the service area for providing services that include the provision of content.

[0039] In the following detailed explanation, the location of the user possessing user terminal 100 may be described as the location of user terminal 100 for the sake of clarity. Also, for the sake of clarity, the terms "user" and "user terminal 100" possessed by the user may be used interchangeably.

[0040] In the following detailed description, "extending content and / or information (e.g., indicators) onto the image" may, depending on the embodiment, be interpreted to include overlaying content and / or information onto the image / screen of the AR view.

[0041] As shown in the figure, a user terminal 100, such as a smartphone, may be capable of taking pictures of its surroundings using a camera, and may display AR content 30 using an AR view 10 that includes the images captured by the camera. The AR content 30 may be associated with an object existing in space 5. Such AR content 30 may be mapped to that object (or a node that represents the object).

[0042] At least one custom region may be defined on the map of space 5 shown in the figure. In the example shown in the figure, two custom regions are defined: a first custom region 60 and a second custom region 70. At least one service element may be associated with each custom region 60, 70. Furthermore, each custom region 60, 70 may define a scenario for providing services or information associated with space 5 based on the service element, or for controlling the settings of the user terminal 100. The services or information may include information for advertising or marketing. Such a scenario may include conditions for controlling the user terminal 100 (i.e., scenario conditions). The "custom regions 60, 70" may be pre-configured on the server 200. For example, at least a portion of space 5 or a sub-region of space 5 may be configured as "custom regions 60, 70" on the server 200.

[0043] The detailed method for defining each custom area 60 and 70, and setting up (generating) custom areas 60 and 70 by associating service elements and scenarios with them, is described below.

[0044] In this embodiment, based on the location of the user terminal 100, it is determined whether the user terminal 100 is located in a first custom area 60 within space 5. If it is determined that the user terminal 100 is located in the first custom area 60, AR content 30 can be displayed in the AR view 10 of the user terminal 100 based on the service elements associated with the custom area 60. For example, when the user terminal 100 enters the first custom area 60, AR content 30 associated with the object "metal type" may be displayed in the AR view 10. At this time, the current location 50 of the user terminal 100 may be displayed in the map view 20. The current location 50 may correspond to a position 50' in space 5. In the embodiment shown in the figure, when the user terminal 100 enters the first custom area 60, an example is shown in which AR content 30 associated with "metal type," an object existing outside the first custom area 60, is displayed. However, depending on the service element associated with the first custom area 60, when the user terminal 100 enters the first custom area 60, other AR content associated with an object existing inside the first custom area 60, such as "the epitaph of Princess Shuning," may be displayed.

[0045] On the other hand, in other embodiments, based on the position of the user terminal 100, other forms of content may be displayed in the AR view 10, appropriate UI may be provided, appropriate map element content may be displayed, or the settings of the user terminal 100 may be appropriately changed according to scenario conditions defined in the first custom area 60, such as when the user terminal 100 moves out of the first custom area 60.

[0046] The map view 20 displayed on the screen of the user terminal 100 along with the AR view 10 may include a map that matches the image of the AR view 10. The map view 20 may include the current location 50 of the user terminal 100. The map view 20 may include a two-dimensional map or a three-dimensional map. The map view 20 may include objects in the space 5 in which the user terminal 100 is moving. The map view 20 may display map element content 35 corresponding to the AR content 30. The map element content 35 may be displayed when the AR content 30 is displayed in the AR view 10, or when an object associated with the AR content 30 is included in the map view 20.

[0047] The user terminal 100 may communicate with the server 200 to control its entry into and exit from custom areas 60 and 70. For example, the server 200 may control the user terminal 100 based on the service elements associated with the custom areas 60 and 70 in which the user terminal 100 is located.

[0048] Server 200 is a computer system that manages custom regions 60 and 70 defined for space 5, and may be a server that manages the association of service elements and the definition of scenarios for custom regions 60 and 70. Server 200 may also be a map server that provides digital maps (3D maps and / or 2D maps).

[0049] The following describes a more specific method of controlling the user terminal 100 by positioning it in custom regions 60 and 70, referring to Figures 1B to 12, which involves providing virtual reality (VR) content to the user terminal 100.

[0050] Figure 1B shows a method in one embodiment for controlling a user terminal so that VR content is displayed on the user terminal when the user terminal is positioned in a pre-set custom area and an object within the custom area is recognized.

[0051] Figure 1B shows how, when a user terminal 100 enters a custom area 60 or 70 while moving according to route guidance and recognizes an object 105, VR content 107 is displayed in the AR view 10 in relation to the recognized object 105.

[0052] Maps 101 and 102 shown in the figure may correspond to the map view 20 described above. Map view 101 may display a route 132 for route guidance to a predetermined destination. Assume that the user terminal 100 moves along the route 132 and enters custom areas 60 and 70, and then recognizes an object 105. When object 105 is recognized, VR content 107 may be displayed in AR view 10 in relation to the recognized object 105.

[0053] Furthermore, Figure 1B shows a method for displaying VR content 107 in the AR view 10 in relation to the recognized object 105 when the user terminal 100 enters a custom area 60 or 70 while moving and recognizes an object 105, regardless of route guidance.

[0054] Object 105 is something placed within space 5, and for example, if space 5 is a museum, it may be an exhibit displayed in the museum. In the example shown in the figure, object 105 may be the "Monument to King Jinheung's Tour".

[0055] Unlike Figure 1A, the user terminal 100 may not only display AR content 30 associated with object 105, but also display VR content 107 via AR view 10 to provide the user with a high level of immersion. The VR content 107 can provide richer information about the associated object 105 compared to the AR content 30. For example, the VR content 107 may include AR content and environment content that depicts a background associated with object 105, such as the background around the object.

[0056] The VR content 107 may be displayed in the AR view 10 based on the service elements associated with the custom regions 60 and 70 and the scenario conditions defined for the custom regions 60 and 70.

[0057] The details of the VR content 107 and the specific method of controlling the user terminal 100 to display the VR content 107 in the AR view 10 will be explained with reference to Figures 2 to 12.

[0058] Figure 2 shows a computer system and server that perform a method for controlling a user terminal in one embodiment.

[0059] The user terminal 100, described with reference to Figures 1A and 1B, may be implemented by the computer system 2 shown in the figures. For example, a computer program for carrying out the method of the embodiment may be installed and executed in the computer system 2, and such a computer system 2 may perform the extended content provision method according to the embodiment in accordance with the control of the executed computer program.

[0060] The method for controlling the user terminal 100 according to the embodiment may be executed through a PC-based program or an application specifically for the mobile terminal. For example, the method for controlling the user terminal 100 according to the embodiment may be implemented as an independently operating program, or it may be implemented to be operable on the specific application which is configured as an in-app version of a specific program. Such a specific application may be installed on a computer system 2 and perform the method for controlling the user terminal 100 according to the embodiment by providing VR content 107 through the AR view 10.

[0061] Computer system 2 is an electronic device corresponding to the user terminal 100 described above, and may be a smartphone or similar device that can install and run applications or programs, as shown in Figures 1A and 1B. Computer system 2 may also be a PC (personal computer), laptop computer, tablet, Internet of Things device, or wearable computer.

[0062] As shown in the figure, the computer system 2 may include a memory 110, a processor 120, a communication interface 130, and an input / output interface 140 as components for performing a method of controlling the user terminal 100.

[0063] Memory 110 is a computer-readable recording medium and may include RAM (random access memory), ROM (read-only memory), and persistent mass storage devices such as disk drives. Here, persistent mass storage devices such as ROM and disk drives may be included in the computer system 2 as separate persistent storage devices distinct from memory 110. Memory 110 may also store an operating system and at least one program code. Such software components may be loaded into memory 110 from a computer-readable recording medium separate from memory 110. Such a separate computer-readable recording medium may include computer-readable recording media such as floppy disks, disks, tapes, DVD / CD-ROM drives, and memory cards. In other embodiments, software components may be loaded into memory 110 through a communication interface 130 that is not a computer-readable recording medium. For example, software components may be loaded into memory 110 of the computer system 2 based on a computer program installed by a file received via a network 160.

[0064] The processor 120 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor 120 by memory 110 or a communication interface 130. For example, the processor 120 may be configured to execute instructions received according to program code stored in a recording device such as memory 110.

[0065] In other words, the processor 120 may manage the components of the computer system 2 and may execute programs or applications used by the computer system 2. For example, the processor 120 may execute an application for performing a method for controlling the user terminal 100 of the embodiment and may be configured to process data received from the server 200 to control the user terminal 100 (for example, data received from the server 200 to provide AR content 30 by the AR view 10). The processor 120 may also process calculations necessary for executing programs or applications and processing data, and may be at least one processor or at least one core within a processor of the computer system 2.

[0066] The communication interface 130 may provide a function for computer system 2 to communicate with other computer systems (not shown) via the network 160. For example, requests, instructions, data, files, etc., generated by the processor 120 of computer system 2 according to program code recorded in a recording device such as memory 110 may be transmitted to other computer systems via the network 160 under the control of the communication interface 130. Conversely, signals, instructions, data, files, etc., from other computer systems may be received by computer system 2 via the network 160 and through the communication interface 130 of computer system 2. Signals, instructions, data, etc., received through the communication interface 130 may be transmitted to the processor 120 and memory 110, and files, etc., may be recorded on a recording medium (the persistent recording device described above) that computer system 2 may further include. For example, the communication interface 130 may be a hardware module such as a network interface card, network interface chip, and networking interface port of computer system 2, or a software module such as a network device driver or networking program.

[0067] The input / output interface 140 may be a means for interface with the input / output device 150. For example, the input device may include a microphone, keyboard, or mouse, and the output device may include a display or speaker. As another example, the input / output interface 140 may be a means for interface with a device that integrates input and output functions into one, such as a touchscreen. The input / output device 150 may consist of the computer system 2 and one device.

[0068] Furthermore, in other embodiments, the computer system 2 may include more components than those shown in Figure 2. However, it is not necessary to explicitly show most of the conventional components in the figure. For example, the computer system 2 may be implemented to include at least some of the input / output devices that connect to the input / output interface 140 described above, and may further include other components such as transceivers, GPS (Global Positioning System) modules, cameras, various sensors, and databases. As a more specific example, if the computer system 2 is implemented in the form of a mobile device such as a smartphone, the computer system 2 may further include various components commonly found in mobile devices, such as cameras, accelerometers and gyroscopes, various physical buttons, buttons using touch panels, input / output ports, and vibrators for vibration.

[0069] For example, a computer system 2 corresponding to a user terminal 100 may include a camera that takes pictures of the surroundings in order to run the AR view 10. The computer system 2 may display the images captured by the camera as the AR view 10, and may extend and display AR content 30 and VR content 107 on the images of the AR view 10. For example, the computer system 2 may determine, based on positioning, whether the user terminal 100 is located in pre-set custom areas 60 and 70, and if the user terminal 100 recognizes an object 105 within the custom areas 60 and 70, it may display VR content 107 in the AR view 10 in relation to the recognized object 105. Visual localization or a Visual Positioning System (VPS) may be used to position the user terminal 100. Alternatively, methods such as recognizing markers, using GPS, using a Wi-Fi Positioning System (WPS), using a Bluetooth module, or using odometry may be used to position the user terminal 100.

[0070] On the other hand, the substantial operation of determining whether computer system 2 is located in custom areas 60 and 70 and whether it recognizes object 105 may be performed by server 200 rather than computer system 2. Server 200 may be an electronic device that communicates with computer system 2 and provides computer system 2 with data for controlling user terminal 100, thereby enabling control of user terminal 100. Server 200 may include a database as a device for managing (registering, changing, deleting, recording, maintaining, etc.) the data and content provided to computer system 2 and custom areas 60 and 70, and may communicate with such a database. Server 200 may also be a map server that provides digital maps (3D maps and / or 2D maps), and may include such a map server. Server 200 may include at least one computer system. The computer system included in server 200 has a configuration similar to that of computer system 2 described above, so redundant explanations regarding it are omitted.

[0071] In the following detailed explanation, for the sake of clarity, the embodiment will be described primarily from the perspective of the computer system 2, which corresponds to the user terminal 100, and explanations regarding communication with the server 200 and the operation of the server 200 may be simplified or omitted. Also, for the sake of clarity, operations (steps) performed by the configuration of the computer system 2 (or server 200) (e.g., the processor) may be described as being performed by the computer system 2 (or server 200).

[0072] The technical features described above, with reference to Figures 1A and 1B, can be directly applied to Figure 2 as well; therefore, redundant explanations will be omitted.

[0073] Figure 3 is a flowchart illustrating a method in one embodiment for controlling a user terminal so that VR content is displayed on the user terminal when the user terminal is positioned in a pre-defined custom area and an object within the custom area is recognized.

[0074] Referring to Figure 3, the method for controlling the user terminal 100, which is run by computer system 2, will be described. Since computer system 2 corresponds to the user terminal 100 described with reference to Figures 1A and 1B, the terms user terminal 100 and server 200 will be used instead of computer system 2 in the following detailed description.

[0075] Furthermore, steps 302 to 350, and at least some of the steps described with reference to Figure 4, may be configured to be executed by the server 200 rather than the user terminal 100. In the following, the execution of the steps will be described focusing on the user terminal 100, and redundant explanations regarding the server 200 may be omitted.

[0076] In step 310, the user terminal 100 or the server 200 may determine, based on the user terminal 100's location, whether the user terminal 100 is located in a pre-configured custom area 60 or 70 within space 5. The custom areas 60 or 70 may be areas defined within space 5 based on an absolute coordinate system. The custom areas 60 or 70 may be associated with at least one service element. The user terminal 100 may determine its location by, for example, a method based on at least one of GPS, VPS (Visual Positioning System), and WPS (Wi-Fi Positioning System), and based on the determined location, it may determine whether the user terminal 100 is located in the custom areas 60 or 70, or whether it has entered or left the custom areas 60 or 70. Alternatively, the server 200 may receive the determined location of the user terminal 100 and, based on the received location, determine whether the user terminal 100 is located in the custom areas 60 and 70, or whether it has entered / entered the custom areas 60 and 70.

[0077] In step 320, the user terminal 100 or the server 200 may determine whether the user terminal 100 is located in a recognition area where it can recognize the object 105 within the custom areas 60, 70.

[0078] In step 330, the user terminal 100 may recognize the object 105 within the custom areas 60 and 70. In other words, the user terminal 100 may be made capable of recognizing the object 105 within the recognition area. The object 105 may be recognized by a module for object recognition. The module for object recognition may be, for example, a module for VOT (Vision Object Tracking). Alternatively, the module for object recognition may be configured to be used for object recognition that utilizes the smart lens of the user terminal 100. Such a module may be part of the processor 120, or it may be implemented separately within the user terminal 100. Furthermore, such a module may be a software module executed by the processor 120.

[0079] Object 105 may be recognized by the AR view 10. For example, object 105 may be recognized if it is included in the video footage captured by the camera included in the AR view 10.

[0080] In step 340, the user terminal 100 may display pre-configured VR content 107 associated with the object 105 recognized in step 330 using the AR view 10. The VR content 107 may be 360-degree content. For example, the VR content 107 may include a panoramic image. The VR content 107 may include AR content depicting the object 105 and environment content depicting a background associated with the object 105, such as the background around the object. The VR content 107 may be associated with a node defined at the position corresponding to the object 105 in space 5.

[0081] To enhance user immersion, the user terminal 100 may display VR content 107 on the entire screen of the AR view 10. Such VR content 107 may depict a virtual space around object 105. For example, if object 105 is the "Monument to King Jinheung's Tour," the environment content depicting the virtual space around object 105 may depict the geographical environment in which the Monument to King Jinheung's Tour is located. The environment content may further include text describing object 105, as shown in Figure 1B. Depending on the type of object 105, the environment content may also be depicted in virtual space for purposes such as recreating historical spaces, transforming themed spaces, or simulating future spaces.

[0082] The user terminal 100 may change its viewpoint in response to changes in the camera's viewpoint in the AR view 10 and display the VR content 107. This allows the user to experience an immersive experience as if they were actually exploring the virtual space where the object 105 is located.

[0083] Furthermore, the VR content 107 may include animation effects. For example, when object 105 is recognized and the VR content 107 is displayed in the AR view 10, the VR content 107 may be represented with an animation that novelly and gradually expands to fill the entire screen of the AR view 10. Alternatively, after the AR content corresponding to object 105 is displayed, the VR content 107 may be displayed by overlaying the aforementioned environment content onto the space surrounding the AR content.

[0084] Such VR content 107 may be displayed while the object 105 is being recognized by the user terminal 100.

[0085] In step 350, the display of the VR content 107 may be terminated when the user terminal 100 is no longer located in the recognition area. For example, the display of the VR content 107 may be terminated when the camera's viewpoint in the AR view 10 changes and the user terminal 100 moves out of the recognition area, or when the user terminal 100 moves out of the recognition area.

[0086] The end of the display of the VR content 107 may also include an animation effect. For example, when the user terminal 100 moves out of the recognition area of ​​object 105, the VR content 107 may disappear from the AR view 10 in a novel way, and the original AR view 10 may be displayed.

[0087] In one embodiment, each custom area 60, 70 is associated with at least one service element. Furthermore, the custom areas may have scenario conditions set for controlling the user terminal 100 based on the service element, depending on the user terminal 100's location within the custom areas 60, 70. These scenario conditions may include conditions for providing services or information associated with space 5, or for controlling the settings of the user terminal 100, based on the service element associated with the custom areas 60, 70.

[0088] The display of VR content 107 through the AR view 10 described above may also be based on service elements associated with custom regions 60 and 70 and scenario conditions set in custom regions 60 and 70.

[0089] The user terminal 100 or the server 200 may control the user terminal 100 based on the service elements associated with the custom areas 60 and 70, once it is determined that the user terminal 100 is located in a custom area. In other words, the user terminal 100 may be controlled to perform specific actions based on the data contained in the service elements associated with the custom areas 60 and 70.

[0090] Here, "control" of the user terminal 100 may include at least one of the following: using the AR view 10 of the user terminal 100 to extend and display content (e.g., AR content 30 and / or VR content 107, etc.) that shows services and information associated with space 5 in the image of the AR view 10; displaying a user interface (UI) associated with services associated with space 5 on the user terminal 100; displaying map element content 35 associated with services associated with space 5 on the map of the user terminal 100 (or on the map view 20); or controlling the settings of the user terminal 100 to provide services associated with space 5 to the user terminal 100.

[0091] Service elements associated with custom regions 60 and 70 may include data for displaying related VR content 107 associated with the service in space 5.

[0092] Furthermore, service elements associated with custom regions 60 and 70 may include data for displaying the space 5 and related VR content 107 associated with the service in the space. Thus, in this embodiment, the VR content 107 can be displayed in the AR view 10 based on such service elements and scenario conditions defined for the custom regions 60 and 70.

[0093] The method for recognizing the object 105 in order to display the VR content 107, and the method for displaying the VR content 107 in the AR view 10 according to specific scenario conditions, will be explained in more detail with reference to Figures 4-12.

[0094] On the other hand, the method for controlling the user terminal 100 in the embodiment may be performed in combination with a route guidance service. For example, the method for controlling the user terminal 100 in the embodiment may be performed in combination with AR navigation that provides route guidance indoors or outdoors.

[0095] In step 302, the user terminal 100 may receive a request for route guidance to a destination. For example, the user of the user terminal 100 may request route guidance from the user terminal 100 by using the user terminal 100 to launch an application or program that provides an AR navigation service or other route guidance service and entering a destination. The route guidance request may be transmitted to the server 200 that provides the route guidance service.

[0096] In step 304, the user terminal 100 may provide route guidance in accordance with the route guidance request received in step 302. For example, the user terminal 100 may receive data for route guidance to the destination from the server 200 that provides route guidance services, and may display information for route guidance to the destination (e.g., guide information) based on the received data.

[0097] The user of user terminal 100 may move to the destination according to the route guidance. In this case, steps 310 to 340 described above may be executed when user terminal 100 moves to the destination according to the route guidance.

[0098] For example, when the user terminal 100 moves to a destination according to route guidance, it may be determined whether the user terminal 100 is located in custom areas 60 and 70, and whether object 105 has been recognized. Based on the results of these determinations, the user terminal 100 may display VR content 107 in the AR view 10. An example of this has been explained with reference to Figure 1B, so redundant explanations will be omitted.

[0099] The technical features described above, with reference to Figures 1A to 2, can be directly applied to Figure 3 as well; therefore, redundant explanations will be omitted.

[0100] Figure 4 is a flowchart illustrating a method for controlling the settings of a user terminal by having the user terminal enter a recognition area where it can recognize objects within a custom area, as in one example.

[0101] Referring to Figure 4, we will now further explain how to determine whether the user terminal 100 is located in the recognition area for recognizing object 105 and how to recognize object 105.

[0102] As described above, the service elements associated with custom areas 60 and 70 may include data for controlling the settings of the user terminal 100 in order to provide the services associated with space 5 to the user terminal 100.

[0103] In step 410, the user terminal 100 or the server 200 may control the settings of the user terminal 100 in accordance with the scenario conditions set for the custom areas 60 and 70 if the user terminal 100 enters a custom area 60 or 70, or if the user terminal 100 enters a pre-configured recognition area.

[0104] Controlling the settings of user terminal 100 may include controlling the functions and / or operations of user terminal 100 in relation to the services provided to user terminal 100 within space 5. Controlling functions and / or operations may include turning functions and / or operations on / off or activating / deactivating them.

[0105] For example, when user terminal 100 enters custom areas 60 and 70, server 200 may control the user terminal's settings, including the execution of a module for recognizing object 105, according to the scenario conditions set in custom areas 60 and 70. In other words, when user terminal 100 enters custom areas 60 and 70, the module for recognizing object 105 may be executed. The module for recognizing object 105 may be, for example, the VOT module described above. When user terminal 100 enters custom areas 60 and 70, the module for recognizing object 105 in user terminal 100 is turned on or initiated, allowing user terminal 100 to recognize object 105 within custom areas 60 and 70. On the other hand, when user terminal 100 leaves custom areas 60 and 70, the module for recognizing object 105 in user terminal 100 may be turned off or deactivated again. However, depending on the embodiment, it is also possible to ensure that controlled settings are maintained even when the user terminal 100 leaves the custom areas 60 and 70.

[0106] In this embodiment, the user terminal 100 can recognize and detect objects only in specific custom areas, or only when it enters a specific custom area, without recognizing or detecting objects unnecessarily in all areas of space 5. Therefore, services associated with space 5 can be efficiently provided to the user terminal 100 while taking into account the user's movement within space 5 and its field of view or vision (which may be a three-dimensional space with a custom area height range).

[0107] With the module for recognizing object 105 running, the user terminal 100 may recognize object 105 through the AR view 10. For example, when object 105 is displayed in the AR view 10, the user terminal 100 may determine that it is located in the recognition area for object 105, and thereby recognize object 105. Alternatively, the user terminal 100 may recognize object 105 if the distance between object 105 and the user terminal 100 is less than or equal to a predetermined distance, if object 105 is displayed in a specific area within the AR view 10 (e.g., the central area or its vicinity), and if object 105 is displayed in the AR view 10 at a rate greater than or equal to a predetermined percentage (i.e., If at least one of the following conditions (conditions in which object 105 occupies a predetermined proportion or more in AR view 10) is met, it may be determined that the user terminal 100 is located in the recognition area for recognizing object 105, and thus object 105 may be recognized. The above-mentioned conditions for recognizing object 105 may be included in the scenario conditions set for custom areas 60 and 70.

[0108] In other embodiments, the recognition area for recognizing the object 105 described above may be a predefined area within the custom areas 60 and 70. Such a predefined area may be a predetermined range of area around the object 105. The user terminal 100 or server 200 may determine that the user terminal 100 is located in the recognition area for recognizing the object 105 if the user terminal 100 is located within such a predefined recognition area. In this case, when it is determined that the user terminal 100 has entered the recognition area, the user terminal 100 or server 200 may control the settings of the user terminal 100, including the execution of a module for recognizing the object 105. In other words, if a recognition area that is a predefined area exists within the custom areas 60 and 70, when the user terminal 100 comes to be located within the recognition area, a module for recognizing the object 105 may be executed, thereby recognizing the object 105. The predefined recognition area described above may be included in the scenario conditions set in the custom areas 60 and 70.

[0109] When the user terminal 100 recognizes an object 105, the VR content 107 associated with the recognized object 105 may be displayed by the AR view 10.

[0110] The technical features described above, with reference to Figures 1A to 3, can be directly applied to Figure 4 as well; therefore, redundant explanations will be omitted.

[0111] Figure 5A illustrates a method for controlling user terminal settings based on the location of the user terminal in a pre-configured custom area, as in one example.

[0112] Service elements associated with custom areas 60 and 70 may include data for controlling the settings of user terminal 100 in order to provide services associated with space 5.

[0113] The user terminal 100 or server 200 may, when the user terminal 100 enters a custom area 60 or 70, execute a module for object recognition on the user terminal 100 based on the service elements associated with the custom areas 60 or 70.

[0114] As shown in the figure, a custom region 510-1 is set in space 500-1, and when the user terminal 100 enters the custom region 510-1 from position 505, the settings of the user terminal 100 are controlled, and the module for object recognition of the user terminal 100 may be turned on or initiated.

[0115] After the object recognition module is executed, when the user terminal 100 recognizes the object 105, which is the Jinheung King's Tour Monument, the AR view 10 may display VR content 107 in relation to the recognized object 105.

[0116] According to this embodiment, the object recognition module is executed only when the user terminal 100 enters the custom area 510-1, thereby eliminating the complexity of the user interface caused by more content than necessary that is extended through the AR view 10, as well as the increased cognitive load on the user and the increased load on the user terminal 100.

[0117] On the other hand, Figure 5B is a diagram illustrating a method for providing VR content to a user terminal based on the user terminal being positioned in a pre-defined custom area, as in one example.

[0118] As shown in the figure, a custom region 510-2 is set in space 500-2. When the user terminal 100 enters the custom region 510-2, the settings of the user terminal 100 are controlled, and the object recognition module of the user terminal 100 may be turned on or initiated.

[0119] On the other hand, the custom region 510-2 may have multiple points A and B defined. Each of the multiple points A and B may indicate an entrance to the space corresponding to the custom region 510-2.

[0120] The scenario conditions for custom region 510-2 may be set so that when the user terminal 100 is located in custom region 510-2, the VR content 107 determined based on the points the user terminal 100 has passed through is displayed via the AR view 10. Therefore, the user terminal 100 can display such VR content 107 determined based on the points the user terminal 100 has passed through via the AR view 10. For example, the user terminal 100 may display different VR content 107 via the AR view 10 depending on whether the user terminal 100 entered custom region 510-2 via point A or via point B.

[0121] On the other hand, the "points that the user terminal 100 passed through" may be a predetermined point outside the custom area 510-2 that the user terminal 100 passed through, or it may be a predetermined point defined within the custom area 510-2.

[0122] The "determined VR content" mentioned above may include existing VR content displayed in a different form, with the application of other animations or effects, VR content with different content, or existing VR content with additional content (such as audio, text, images, or videos) added to it.

[0123] Other examples of displaying VR content in AR View 10 according to scenario conditions set for a custom region are described in more detail below, with reference to the diagrams.

[0124] The technical features described above, with reference to Figures 1A to 4, can be directly applied to Figures 5A and 5B as well; therefore, redundant explanations will be omitted.

[0125] Figure 6 shows an example of how to define AR content associated with an object in a custom region, and Figure 7 shows an example of how to define VR content associated with an object in a custom region.

[0126] The AR content 610 shown in the figure may be defined to be displayed within the AR view 10 in relation to object 105, the "Monument to King Jinheung's Tour."

[0127] The AR content 610 may be placed in a UI 600 on which data representing space 5 has been loaded or rendered, or it may be placed in custom regions 60, 70 set up within space 5. The AR content 610 may be included in service elements associated with the custom regions 60, 70.

[0128] The VR content 710 shown in Figure 7 may be defined to be displayed within the AR view 10 in relation to object 105, the "Monument to King Jinheung's Tour." The VR content 710 may include AR content 610 and environment content that depicts the virtual space around object 105. As shown in the figure, the virtual space around object 105 depicted by the VR content 710 may be represented (environment content) as a virtual spherical or hemispherical space centered on object 105. By including a virtual space represented as a virtual spherical space centered on object 105 in the VR content 710, the user can experience immersion through the AR view 10 that displays the VR content 710.

[0129] UI600, 700 may include, or be part of, an indoor map generation UI for creating an indoor map of space 5 if space 5 is indoors. Such an indoor map generation UI may provide the functionality to set up detailed spaces (such as exhibition rooms), means of transportation such as stairs, escalators, and elevators, facilities such as toilets, signs, notices, and information, other obstacles such as pillars, route information within space 5, nodes, and links connecting nodes within space 5. For example, a user of an administrator terminal to generate an indoor map and generate custom regions 60, 70 on the indoor map can associate the custom regions 60, 70 with the AR content 610 and / or VR content 710 (i.e., service elements containing them) by setting the regions corresponding to the custom regions 60, 70 through UI600, 700, and by directly placing the AR content 610 and / or VR content 710 in the set regions, or by selecting the AR content 610 and / or VR content 710 to place in the set regions. UI600 and 700 may provide a separate UI for selecting and placing service elements associated with the custom areas 60 and 70. Users may select and place service elements by methods such as clicking or dragging and dropping.

[0130] In the UI600 and 700 examples shown in the figure, it is assumed that a map corresponding to space 5 is displayed, and that custom regions 60 and 70 are generated on the map.

[0131] On the other hand, unlike what is shown in the figure, UI600 and 700 may render at least one of the following as data representing space 5: a map of space 5, point cloud data (PCD) collected from space 5, and an image of space 5, and custom regions 60 and 70, which are two-dimensional and / or three-dimensional spaces, may be generated on the rendered data.

[0132] If the indoor map generation UI includes UI600 and 700, the creation of custom regions 60 and 70 may be performed along with the construction of the indoor map for space 5, and the placement of content 610 and / or VR content 710 in the custom regions 60 and 70 may also be performed at the same time.

[0133] The technical features described above, with reference to Figures 1A to 5B, can be directly applied to Figures 6 and 7 as well; therefore, redundant explanations will be omitted.

[0134] Figure 8 illustrates how to define AR and VR content associated with objects within a custom region in one example.

[0135] As described above, in order to enhance user immersion, the user terminal 100 may display VR content 107 on the full screen of the AR view 10. Here, the VR content 107 may depict the virtual space around the object 105.

[0136] In this case, the virtual space depicted by the VR content 107 may be represented as a virtual three-dimensional space determined by reflecting the geometry of the surrounding space in which the object 105 is located. In other words, the virtual space depicted by the VR content 107 may reflect the form of the surrounding real space in which the object 105 is located.

[0137] For example, as shown in the figure, if object 810 is placed in exhibition space 820 and exhibition space 820 has a hexahedral geometry, the VR content displayed in AR view 10 in relation to object 810 may include AR content 830 representing object 810 and environment content 840 representing exhibition space 820. Thus, the environment content 840 may have a hexahedral shape, just like the exhibition space 820, as it reflects the geometry of the exhibition space 820.

[0138] By reflecting the geometry of the actual exhibition room space 820 in which object 810 is placed in the VR content displayed in AR view 10, the immersion for users viewing the VR content can be further enhanced.

[0139] On the other hand, the geometry of the exhibition space 820 may be base data acquired when creating an indoor map of space 5 containing the exhibition space 820, or when collecting data (PCD, photographic images, etc.) representing space 5. The geometry of the exhibition space 820 may be recorded and managed on server 200.

[0140] Therefore, as explained with reference to Figure 7, when VR content 710 is placed in custom regions 60 and 70 through UI 600 and 700, the virtual space around object 105 depicted by VR content 710 is not a virtual spherical or hemispherical space centered on object 105, but a three-dimensional space that reflects the geometry of the real space in which object 105 is placed. Such a three-dimensional space may be automatically displayed on UI 600 and 700 based on the geometry of the real space of the base.

[0141] The technical features described above, with reference to Figures 1A to 7, can be directly applied to Figure 8 as well; therefore, redundant explanations will be omitted.

[0142] The following sections provide a more detailed explanation of the custom regions 60 and 70 defined and generated for space 5, and the scenario conditions set for custom regions 60 and 70.

[0143] Figure 9 shows a custom region defined as a three-dimensional space in one example.

[0144] Figure 9 provides a more detailed explanation of the custom region 900, defined as a three-dimensional space, and the scenario conditions defined within it.

[0145] The custom regions 60 and 70 described above may be defined on the map of space 5 to include at least one of a two-dimensional space and a three-dimensional space.

[0146] The user terminal 100 or server 200 may control the user terminal 100 based on the service elements associated with custom areas 60 and 70 if the user terminal 100 is located in custom areas 60 and 70 on the map (or if it is determined that the user has moved out of custom areas 60 and 70).

[0147] The custom region 900 may be defined to include a two-dimensional region and a three-dimensional space having a set height range. Alternatively, the custom region 900 may be defined as a composite-dimensional space including both a two-dimensional space and a three-dimensional space.

[0148] The two-dimensional region constituting custom region 900 may include a two-dimensional coordinate space (x, z). The height range constituting custom region 900 may include the y-axis coordinate space. The height range may have upper and lower limits, as shown in the figure (H), or it may be a range without upper or lower limits, as shown in the figure.

[0149] The user terminal 100 or server 200 may determine that the user terminal 100 is located in the custom region 900 if the user terminal 100 is located in the two-dimensional region that constitutes the custom region 900 and the height of the user terminal 100 falls within the set height range that constitutes the custom region 900, and may control the user terminal 100 based on the service elements associated with the custom region 900. Therefore, when it is determined that the user terminal 100 is located in the custom region 900, a module for recognizing object 105 is executed, and upon recognition of object 105, the VR content 107 may be displayed in the AR view 10.

[0150] The height of the user terminal 100 may be obtained from a sensor included in the user terminal 100. The height of the user terminal 100 may, for example, indicate the height of the camera of the user terminal 100.

[0151] Figure 9 shows an example of one height range (H), but a custom region 900 may have multiple height ranges defined, and a service element may be associated with each height range to define a scenario.

[0152] In this way, by setting a custom region 900 in a three-dimensional space, even if the user terminal 100 is located in the same two-dimensional space, the user terminal 100 can be controlled by different scenarios depending on the height of the user terminal 100.

[0153] The following provides a more detailed explanation of the scenarios defined for custom areas 60 and 70, referring to Figure 10.

[0154] Figure 10 shows the Degree of Freedom (DoF) of a user terminal in one example.

[0155] Custom regions 60 and 70 may define 3DoF or 6DoF attitude conditions for the user terminal 100. 6DoF may represent motion in the x, y, and z axes, and rotation (roll, yaw, and pitch) in the x, y, and z axes.

[0156] Such attitude conditions may be included in the scenario conditions defined for custom areas 60 and 70, and as described above, such scenario conditions may be included in the service elements associated with custom areas 60 and 70.

[0157] When the user terminal 100 or server 200 determines that the user terminal 100 is located in custom regions 60 or 70, and the user terminal 100's attitude satisfies the attitude conditions defined for custom regions 60 or 70, the user terminal 100 may be controlled based on the service elements associated with custom regions 60 or 70. The user terminal 100's attitude may be obtained from a sensor provided in the user terminal 100, such as a gyroscope sensor.

[0158] By defining such posture conditions in custom regions 60 and 70, the user terminal 100 may be controlled based on the service element only when the user terminal 100 enters a custom region 60 or 70 and the AR view 10 maintains a specific angle or a specific range of angles. For example, VR content that should be displayed in a specific region within custom regions 60 or 70 at a position above or below a predetermined height may be displayed in the AR view 10 of the user terminal 100 only when the user terminal 100 enters a custom region 60 or 70 and the AR view 10 maintains an angle or range of angles that displays that specific region above or below the predetermined height.

[0159] Therefore, in this embodiment, AR content can be displayed at a more appropriate position and viewpoint of the AR view 10.

[0160] The following describes other examples of scenario conditions defined for custom regions 60 and 70.

[0161] Custom areas 60 and 70 may contain scenario conditions that include at least one of the following conditions: a first condition indicating a time condition, a second condition indicating the movement of the user terminal 100, and a third condition indicating an operation on the user terminal 100.

[0162] The first condition may be, for example, a condition requiring the user terminal 100 to remain in the custom areas 60 and 70 for a predetermined time or longer (e.g., 3 seconds or more), or to remain in the custom areas 60 and 70 for less than a predetermined time. Alternatively, the first condition may be a condition requiring the user terminal 100 to enter the custom areas 60 and 70 during a predetermined time period, or to enter the custom areas 60 and 70 at a time other than the predetermined time period. The second condition may indicate a physical operation of the user terminal 100, and may be a condition requiring the user terminal 100 to be shaken a predetermined number of times or with a predetermined force or longer. Alternatively, the second condition may also require the user terminal 100 to be moved into a specific shape (e.g., a figure eight). The third condition may be an electronic operation of the user terminal 100, and may be a condition requiring, for example, to press a specific button or key, to input a specific gesture, to input a specific voice or audio, to photograph a specific object, etc.

[0163] When the user terminal 100 or server 200 determines that the user terminal 100 is located in a custom area 60 or 70, and the user terminal 100 satisfies at least one of the first to third scenario conditions, it may control the user terminal 100 based on the service elements associated with the custom areas 60 or 70. For example, depending on whether the scenario condition is met, the user terminal 100 may display different AR or VR content, display a different UI, display different map element content, or perform different settings controls on the user terminal 100.

[0164] The technical features described above, with reference to Figures 1A to 8, can be directly applied to Figure 9 as well; therefore, redundant explanations will be omitted.

[0165] Figure 11 shows a custom region containing multiple points in one example.

[0166] The custom area 1100 shown in the figure may be an area defined on the map of space 5. The custom area 1100 may include a point indicating entrance / exit A and a point indicating entrance / exit B. The custom area 1100 may further include a point indicating object A and a point indicating object B. The user terminal 100 or the server 200 may determine whether the user terminal 100 has entered or exited the custom area 1100 by tracking the location of the user terminal 100.

[0167] Furthermore, multiple points may be defined not only inside the custom region 1100, but also outside of the custom region 1100.

[0168] A custom area 1100 may have multiple service elements associated with it. In this case, each of the multiple service elements may be associated with a point within the custom area 1100. The scenario or scenario conditions defined in the custom area may be defined such that the services or information associated with space 5 are provided differently depending on the order in which the user terminal 100 passes through the points associated with each service element. The services or information provided differently may be AR content or VR content, or other forms of content such as audio, text, images, or videos, which are in a form that can be displayed on the user terminal 100.

[0169] As an example, scenario conditions may be defined in the custom area 1100 so that different forms of content are displayed to the user terminal 100 when the user terminal 100 enters the custom area 1100 and when it leaves the custom area 1100.

[0170] Alternatively, scenario conditions may be defined in the custom area 1100 to distinguish between cases where the user terminal 100 enters the custom area 1100 via a specific point within the custom area 1100 (for example, at least one of the points of entrance / exit A, entrance / exit B, object A, and object B shown in the figure), or where the user enters the custom area 1100 after passing through a specific point outside the custom area 1100, and to ensure that different forms of content are displayed to the user terminal 100. For example, if the user terminal 100 enters the custom area 1100 via the point indicating entrance / exit A, general welcome content may be displayed to the user terminal 100, and if the user terminal 100 enters the custom area 1100 via the point indicating entrance / exit B, other content, such as coupon content, may be displayed to the user terminal 100. The coupon content may include discount coupons or offers usable within space 5.

[0171] Thus, when multiple points are defined inside or outside the custom region 1100, the AR view 10 can display different VR content depending on whether the user terminal 100 passes through the first point or the second point. In other words, even if the same object is recognized, different VR content can be displayed by the AR view 10 depending on which point the user terminal 100 passed through to enter the custom region 1100, or which point it passed through within the custom region 1100.

[0172] Furthermore, when the user terminal 100 is located in a specific custom area 60 or 70, it may be determined whether the user has entered or exited that custom area 60 or 70. Based on the determination result and the set scenario conditions, the user terminal 100 may be controlled to display different VR content when entering and exiting the custom areas 60 or 70, respectively. At this time, based on the current location of the user terminal 100 and the path the user terminal 100 has already moved or will move, it may be determined that the user terminal 100 has entered the custom areas 60 or 70 ([1]) or exited the custom areas 60 or 70 ([2]). The user terminal 100 or the server 200 may know, based on the path, which nodes the user terminal 100 has already passed through or will pass through, and accordingly determine whether the user terminal 100 has entered or exited the custom area where it is currently located. In the cases of [1] and [2] described above, the user terminal 100 may be controlled to display different VR content. That is, even if the same object is recognized, different VR content can be displayed by the AR view 10 depending on whether the user terminal 100 enters or exits the custom regions 60 and 70.

[0173] Alternatively, a scenario condition may be defined in the custom area 1100 that, when the user terminal 100 enters the custom area 1100, requests additional actions from the user terminal 100, such as requesting it to move along the edge of the custom area 1100, and that, upon execution of the action, other forms of content are displayed to the user terminal 100. The request for the additional action may also be provided to the user terminal 100 in the form of content or UI.

[0174] Alternatively, the custom area 1100 may define scenario conditions such as counting the number of times the user terminal 100 enters the custom area 1100 and displaying specific content to the user terminal 100 if it enters the custom area 1100 n times (where n is a natural number greater than or equal to 2) or more (the number of times the user terminal 100 enters the custom area 1100 may be counted within a predetermined time (e.g., 1 hour or 1 day)), or displaying specific content to the user terminal 100 if the user terminal 100 stays in the custom area 1100 for a predetermined time or longer. For example, suppose a scenario condition is set so that VR content determined based on the number of times the user terminal 100 enters the custom area 1100 is displayed by the AR view. In this case, the VR content determined based on the number of times the user terminal 100 enters the custom area 1100 may be displayed by the AR view 10. In other words, even if the same object is recognized, different VR content can be displayed by the AR view 10 depending on the number of times the user terminal 100 has entered the custom area 1100.

[0175] On the other hand, if the user terminal 100 leaves the custom area 1100, all content displayed on the user terminal 100 may be hidden, i.e., the display may be canceled. When the user terminal 100 leaves the custom area 1100, a predetermined notification sound may be displayed on the user terminal 100. Alternatively, when the user terminal 100 leaves the custom area 1100, AR content or VR content that informs the user terminal 100 that it has left the custom area 1100 may be displayed on the AR view 10.

[0176] In this embodiment, it is possible to determine whether the user terminal 100 has entered or left the custom area 1100, and depending on the determination result, different AR or VR content can be displayed (for example, an expansion of entry guide AR or VR content when entering, or an expansion of exit guide AR or VR content when leaving), different UI can be displayed, different map element content can be displayed, or different settings can be controlled for the user terminal 100.

[0177] The technical features described above, with reference to Figures 1A to 10, can be directly applied to Figure 11 as well; therefore, redundant explanations will be omitted.

[0178] Figure 12 illustrates an example in which multiple custom regions are set in a space, and scenario conditions for controlling the user terminal are set for each custom region.

[0179] Below, we will elaborate further on other examples of scenario conditions defined for each of the multiple custom regions 1210-1240 set in space 1200.

[0180] The aforementioned custom areas 60 and 70 may be any one of the custom areas 1210 to 1240.

[0181] In any one of the custom areas 1210 to 1240, custom areas 60 and 70 may have at least one scenario condition defined: i) a first scenario condition indicating whether the user terminal 100 has passed through a predetermined other custom area other than custom areas 60 and 70, and ii) a second scenario condition indicating whether the number of user terminals located in custom areas 60 and 70 falls within a predetermined range. Depending on whether the scenario condition is met, the user terminal 100 may be controlled to receive different services or information associated with spaces 5 and 1200, or to have different settings made for the user terminal 100, based on the service elements associated with custom areas 60 and 70.

[0182] The first scenario condition may be a condition that requires a user terminal 100 that has entered custom areas 60 and 70 to enter custom areas 60 and 70 after entering another specific custom area. For example, the first scenario condition may be satisfied if the user terminal 100 enters a specific custom area among custom areas 1220 to 1240 and then enters custom area 1210 (within a predetermined time). Alternatively, the first scenario condition may be satisfied if the user terminal 100 enters at least a portion of custom areas 1210 to 1240 in a specific order and then enters custom area 1210 (within a predetermined time). Depending on whether the first scenario condition is satisfied, the user terminal 100 may display different AR or VR content, display a different UI, display different map element content, or perform different settings controls on the user terminal 100.

[0183] For example, the scenario conditions may be set so that when the user terminal 100 is located in custom areas 60 and 70, the VR content determined based on other custom areas that the user terminal 100 has entered before entering the custom area is displayed by the AR view 10. In this case, the AR view 10 of the user terminal 100 may display the VR content determined based on the other custom areas. In other words, even if the same object is recognized, different VR content may be displayed by the AR view 10 depending on which other custom areas the user terminal 100 entered before entering the custom area, or which specific other custom area it entered.

[0184] The second scenario condition may be a condition requiring that the number of user terminals located in custom areas 60 and 70, or the cumulative number of user terminals that have entered custom areas 60 and 70 (within a predetermined time), is greater than or equal to a predetermined number, or less than a predetermined number. Alternatively, the second scenario condition may be a condition requiring that the number of user terminals located in custom areas other than custom areas 60 and 70 into which user terminal 100 enters, or the cumulative number of user terminals that have entered other custom areas (within a predetermined time), is greater than or equal to a predetermined number, or less than a predetermined number. Alternatively, the second scenario condition may be a condition requiring that the number of user terminals located in both custom areas 60 and 70 and the other custom areas, or the cumulative number of user terminals that have entered both custom areas (within a predetermined time), is greater than or equal to a predetermined number, or less than a predetermined number. Depending on whether the second scenario condition is met, user terminal 100 may display different AR or VR content, display a different UI, display different map element content, or perform different settings controls on user terminal 100.

[0185] The first and second scenario conditions described above may be used in combination with each other. For example, when the user terminal 100 enters a custom area 60 or 70, the user terminal 100 may determine whether the first and second scenario conditions are met, and only if both scenario conditions are met may the user terminal 100 be controlled based on the service element. Alternatively, different control of the user terminal 100 based on the service element may be performed depending on which of the first and second scenario conditions is met.

[0186] For example, the user terminal 100 may display VR content using the AR view 10 only when the scenario conditions are met, or it may display different VR content using the AR view 10 depending on whether the scenario conditions are met or not.

[0187] The "determined VR content" mentioned above may include existing VR content displayed in a different form, with different animations or effects, or VR content with different content, or existing VR content with additional content (such as audio, text, images, or videos) added to it.

[0188] The technical features described above, with reference to Figures 1A to 11, can be directly applied to Figure 12 as well; therefore, redundant explanations will be omitted.

[0189] Thus, in this embodiment, a "custom range" method can be utilized, which is easier to handle and allows for the display of AR / VR content on the user terminal 100 and control of the user terminal 100 in a natural way, compared to a "rendering range" based method that displays AR / VR content or controls the user terminal 100 based on the distance or radius to the position of an object located in space, or a "semantic range" based method that divides space into semantic units and displays AR / VR content or controls the user terminal 100 when the user terminal 100 enters a unit space. The custom range may be the same as the "custom area" described with reference to the embodiment. The "custom range" method may utilize a custom area in which service elements for displaying AR / VR content and controlling the user terminal 100 are associated and scenario conditions are defined.

[0190] When using rendering range-based or semantic range-based methods, the user terminal 100 may display AR / VR content inappropriately depending on its location, and in some cases, the AR / VR content may appear in the occlusion region. The custom region method, described with reference to the embodiment, can solve these problems by utilizing a custom region.

[0191] The above-described apparatus may be implemented by hardware components, software components, and / or combinations of hardware and software components. For example, the apparatus and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, ALUs (arithmetic logic units), digital signal processors, microcomputers, FPGAs (field programmable gate arrays), PLUs (programmable logic units), microprocessors, or various devices capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications running on the OS. The processing unit may also respond to software execution, access data, record, manipulate, process, and generate data. For convenience of understanding, it may be described as if a single processing unit is used, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0192] Software may include computer programs, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or which may instruct the processing unit independently or collectively. Software and / or data may be embodied in any kind of machine, component, physical device, computer recording medium, or device for interpretation based on the processing unit or for providing instructions or data to the processing unit. Software may be distributed across a networked computer system, and may be recorded or executed in a distributed manner. Software and data may be recorded on one or more computer-readable recording media.

[0193] The methods according to the embodiment may be implemented in the form of program instructions executable by various computer means and recorded on a computer-readable medium. In this case, the medium may continuously record computer-executable programs or may temporarily record them for execution or download. Furthermore, the medium may be various recording or storage means in the form of a combination of one or more hardware components, and may be a medium directly connected to a computer system or distributed on a network. Examples of mediums include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and devices configured to record program instructions such as ROM, RAM, and flash memory. Other examples of mediums include recording media and storage media managed by app stores that distribute applications, and sites and servers that supply and distribute various other software.

[0194] As described above, embodiments have been explained based on limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations from the above description. For example, the described technique may be performed in a different order than described, and / or the components of the described system, structure, apparatus, circuit, etc. may be combined or assembled in a different manner than described, or opposed or replaced by other components or equivalents, and still achieve suitable results.

[0195] Therefore, even if the embodiment is different, it falls within the scope of the attached claims if it is equivalent to the claims.

Claims

1. A user terminal control method executed by a computer system which is a server or a user terminal, The steps include determining whether the user terminal is located in a pre-defined custom area within the space based on the location of the user terminal, The steps include determining whether the user terminal is located in a recognition area where it can recognize objects within the custom area, When the user terminal is positioned in the recognition area, the user terminal recognizes the object from an AR view including an image captured by the camera of the user terminal. The process includes the step of displaying pre-configured VR content associated with the recognized object using the AR view, The aforementioned custom area is associated with at least one service element, The service element includes data for displaying the VR content associated with the service associated with the space, The custom area defines scenario conditions for controlling the user terminal based on the service element, based on the user terminal's location within the custom area. The scenario conditions include at least one of the following: i) a posture condition indicating whether the AR view of the user terminal maintains a predetermined angular range, and ii) a terminal number condition indicating whether the number of user terminals located in the custom region falls within a predetermined range. The scenario conditions are configured to provide the user terminal with different services or information associated with the space based on the service element, and / or to control the user terminal's settings differently, depending on whether the scenario conditions are met. The step of displaying the VR content is: A user terminal control method that either displays the VR content using the AR viewer only when the aforementioned scenario conditions are met, or displays different VR content depending on whether the scenario conditions are met or not.

2. The scenario conditions include the attitude conditions, The step of displaying the VR content is: The VR content is displayed by the AR view only when the aforementioned posture conditions are met, or different VR content is displayed depending on whether the aforementioned posture conditions are met or not. The user terminal control method according to claim 1.

3. The service element includes data for controlling the settings of the user terminal in order to provide the service associated with the space to the user terminal. The user terminal control method according to claim 1, further comprising the step of controlling the settings of the user terminal, including the execution of a module for recognizing an object, in accordance with the scenario conditions, when the user terminal enters the custom area.

4. The step of determining whether the user terminal is located in the recognition area is: When the object is displayed within the AR view, it is determined that the user terminal is located in the recognition area. The user terminal control method according to claim 1.

5. The step of determining whether the user terminal is located in the recognition area is: The user terminal is determined to be located in the recognition area if at least one of the following conditions is met: the distance between the object and the user terminal is less than or equal to a predetermined distance; the object is displayed in a specific area within the AR view; and the object is displayed in the AR view at a predetermined rate or higher. The user terminal control method according to claim 1.

6. The aforementioned recognition area is a predefined area within the aforementioned custom area. The step of determining whether the user terminal is located in the recognition area is: When the user terminal is located in the predefined recognition area, it is determined that the user terminal is located in the recognition area. The user terminal control method according to claim 1.

7. The user terminal control method according to claim 6, further comprising the step of controlling the settings of the user terminal, including the execution of a module for recognizing an object, when the user terminal enters the recognition area.

8. The step of displaying the VR content involves changing the viewpoint in accordance with the change in the viewpoint of the camera and displaying the VR content accordingly. The user terminal control method according to claim 1, further comprising the step of terminating the display of the VR content when the user terminal is no longer located in the recognition area.

9. The step of displaying the VR content involves displaying the VR content on the entire screen of the AR view, The VR content depicts the virtual space surrounding the object. The user terminal control method according to claim 1.

10. The aforementioned virtual space is represented as a virtual spherical space centered on the object. The user terminal control method according to claim 9.

11. The aforementioned virtual space is represented in a virtual three-dimensional space defined by reflecting the geometry of the surrounding space in which the object is located. The user terminal control method according to claim 9.

12. The scenario conditions are set such that, when the user terminal is located in the custom area, the VR content displayed in the AR view is determined based on other custom areas that the user terminal entered before entering the custom area. The step of displaying the VR content is: The VR content determined based on the aforementioned other custom areas is displayed by the AR view. The user terminal control method according to claim 1.

13. The aforementioned scenario conditions are set such that, when the user terminal is located in the custom area, the VR content determined based on the points the user terminal has passed through is displayed in the AR view. The step of displaying the VR content is: The VR content determined based on the points the user terminal has passed through is displayed using the AR viewer. The user terminal control method according to claim 1.

14. Multiple points are defined inside or outside the custom region. The step of displaying the VR content is: When the user terminal passes through a first point among the plurality of points, and when the user terminal passes through a second point among the plurality of points, different VR content is displayed in the AR view. The user terminal control method according to claim 13.

15. The steps include: receiving a request for route guidance to the destination at the user terminal; The steps include providing route guidance to the user terminal in accordance with the aforementioned request for route guidance, It further includes, The steps of determining whether the user terminal is located in the custom area, determining whether the user terminal is located in the recognition area, recognizing the object, and displaying the VR content are performed when the user terminal moves to the destination according to the route guidance. The user terminal control method according to claim 1.

16. The aforementioned scenario conditions are set such that VR content determined based on the number of times the user terminal enters the custom area is displayed in the AR view. The step of displaying the VR content is: The VR content determined based on the number of times the user terminal has entered the custom area is displayed using the AR view. The user terminal control method according to claim 1.

17. A computer system that implements a user terminal or server, It includes at least one processor configured to execute computer-readable instructions contained in memory, The aforementioned at least one processor is Based on the location of the user terminal, it is determined whether the user terminal is located in a pre-defined custom area within the space. The system determines whether the user terminal is located in a recognition area where it can recognize objects within the custom area. When the user terminal is positioned in the recognition area, the object is recognized from the AR view which includes the image captured by the camera of the user terminal. The VR content pre-configured in association with the recognized object is displayed using the AR viewer. The aforementioned custom area is associated with at least one service element, The service element includes data for displaying the VR content associated with the service associated with the space, The custom area defines scenario conditions for controlling the user terminal based on the service element, based on the user terminal's location within the custom area. The scenario conditions include at least one of the following: i) a posture condition indicating whether the AR view of the user terminal maintains a predetermined angular range, and ii) a terminal number condition indicating whether the number of user terminals located in the custom region falls within a predetermined range. The scenario conditions are configured to provide the user terminal with different services or information associated with the space based on the service element, and / or to control the user terminal's settings differently, depending on whether the scenario conditions are met. The step of displaying the VR content is: A computer system that displays the VR content using the AR viewer only when the aforementioned scenario conditions are met, or displays different VR content depending on whether the aforementioned scenario conditions are met or not.

Images