Method and apparatus for displaying virtual reality content on a user terminal based on the user terminal being located in a predefined custom area

Abstract

Provided is a user terminal control method that determines, based on the position of the user terminal, whether the user terminal is located in a preset custom area in space, determines whether the user terminal is located in a recognition area where the user terminal can recognize an object in the custom area, and, when the user terminal is located in the recognition area, recognizes the object and displays, in an AR view, virtual reality (VR) content that is preset in association with the recognized object.

Description

[Technical Field]

[0001] The following description relates to a method and apparatus for controlling a user terminal based on the position of the user terminal, and relates to a method and apparatus for a user terminal located in a pre-defined custom area in a space to recognize an object within the custom area, thereby displaying virtual reality (VR) content associated with the recognized object on the user terminal. [Background technology]

[0002] Augmented reality (AR) is a technology that blends virtual objects and information created by computer technology into the real world to complement it. In other words, AR is a technology that displays virtual content in an augmented real-world environment, allowing users to view augmented content that corresponds to the real-world environment through electronic devices.

[0003] The user terminal, which is a user's electronic device, is provided with content associated with objects located within a predetermined distance or radius from the user terminal's location, augmented to the user terminal's video. If augmented content is provided based solely on the physical distance from the user terminal, more content than necessary may be augmented, leading to cognitive strain on the user and excessive strain on the user terminal. Furthermore, content that does not match the service desired by the user may be augmented to the user terminal.

[0004] To solve this problem, a method is needed that can display only highly useful content on the user's terminal when the user is located in a specific area within a space, while providing the user with a highly immersive experience.

[0005] As an example of technology relating to various services that utilize augmented reality technology, Korean Patent Publication No. 10-2014-0065963 (published on May 30, 2014, Patent Document 1) discloses an augmented reality navigator that is installed in a vehicle, displays an image of the driving route captured by a camera on a display, and maps virtual display information that guides the driving route onto the image displayed on the display.

[0006] The information provided above is merely to aid in the understanding of the present invention and may contain material that does not form part of the prior art, or that the prior art would suggest to one of ordinary skill in the art. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Korean Patent Publication No. 10-2014-0065963 (Publication date: May 30, 2014) Summary of the Invention [Problem to be solved by the invention]

[0008] A method can be provided in which a user terminal located in a pre-defined custom area in a space recognizes an object in the custom area through an AR view, and displays pre-defined VR content associated with the recognized object.

[0009] To provide the user terminal with more detailed and richer information about the object, VR content associated with the object within the custom region can be displayed by the AR view.

[0010] When a user terminal is positioned in a pre-defined custom area, the settings of the user terminal, including the execution of a module for recognizing objects, can be controlled according to the scenario conditions defined for that custom area.

[0011] Various scenario conditions can be set for the custom area, and different VR content can be provided to the user terminal depending on whether the scenario conditions are met, and additional content can also be provided to the user terminal. [Means for solving the problem]

[0012] According to one aspect, there is provided a user terminal control method executed by a computer system that is a server or a user terminal, the user terminal control method including the steps of: determining, based on the position of the user terminal, whether the user terminal is located in a pre-set custom area in space; determining whether the user terminal is located in a recognition area where the user terminal can recognize an object in the custom area; when the user terminal is located in the recognition area, recognizing the object from an AR view including an image captured by a camera of the user terminal; and displaying pre-set virtual reality (VR) content associated with the recognized object using the AR view.

[0013] At least one service element is associated with the custom area, and the service element includes data for displaying the VR content related to a service associated with the space, and the custom area may define scenario conditions for controlling the user terminal based on the service element when the user terminal is located in the custom area.

[0014] The service element includes data for controlling settings of the user terminal to provide services associated with the space to the user terminal, and the user terminal control method may further include a step of controlling settings of the user terminal, including executing a module for recognizing the object, in accordance with the scenario conditions when the user terminal enters the custom area.

[0015] The step of determining whether the user terminal is located in the recognition area may determine that the user terminal is located in the recognition area if the object is displayed within the AR view.

[0016] The step of determining whether the user terminal is located in the recognition area may determine that the user terminal is located in the recognition area when at least one of the following conditions is satisfied: the distance between the object and the user terminal is less than a predetermined distance; the object is displayed in a specific area within the AR view; and the object is displayed in the AR view at a predetermined rate or more.

[0017] The recognition area is a predefined area within the custom area, and the step of determining whether the user terminal is located in the recognition area may determine that the user terminal is located in the recognition area if the position of the user terminal is located in the predefined recognition area.

[0018] The user terminal control method may further include a step of controlling settings of the user terminal, including execution of a module for recognizing the object, when the user terminal enters the recognition area.

[0019] The step of displaying the VR content may display the VR content by changing the viewpoint in accordance with a change in the viewpoint of the camera, and the user terminal control method may further include a step of terminating the display of the VR content when the user terminal is no longer located in the recognition area.

[0020] The step of displaying the VR content may be a step of displaying the VR content on the entire screen of the AR view, and the VR content may depict a virtual space around the object.

[0021] The virtual space may be expressed as a virtual spherical space centered on the object.

[0022] The virtual space may be expressed as a virtual three-dimensional space that is defined to reflect the geometry of the surrounding space in which the object is located.

[0023] The scenario conditions may be set so that, when the user terminal is located in the custom area, the VR content to be displayed by the AR view is VR content determined based on another custom area into which the user terminal entered before the user terminal entered the custom area, and the step of displaying the VR content may include displaying the VR content determined based on the other custom area by the AR view.

[0024] The scenario conditions may be set so that when the user terminal is located in the custom area, the VR content is displayed by the AR view as VR content determined based on points passed by the user terminal, and the step of displaying the VR content may include displaying by the AR view the VR content determined based on points passed by the user terminal.

[0025] A plurality of points may be defined inside or outside the custom area, and the step of displaying the VR content may display different VR content using the AR view when the user terminal passes through a first point among the plurality of points and when the user terminal passes through a second point among the plurality of points.

[0026] A plurality of custom areas are set in the space, and the scenario conditions include at least one of i) a first scenario condition indicating whether the user terminal has passed through a predetermined custom area other than the custom area, and ii) a second scenario condition indicating whether the number of user terminals located in the custom area falls within a predetermined range. The scenario conditions are set to provide the user terminal with different services or information associated with the space based on the service elements and / or control the settings of the user terminal to be different depending on whether the scenario conditions are met. The step of displaying the VR content may display the VR content using the AR view only when the scenario conditions are met, or may display different VR content when the scenario conditions are met and when the scenario conditions are not met.

[0027] The user terminal control method further includes the steps of receiving a route guidance request to a destination in the user terminal, and providing route guidance to the user terminal according to the route guidance request, and the steps of determining whether the user terminal is located in the custom area, determining whether the user terminal is located in the recognition area, recognizing the object, and displaying the VR content may be executed when the user terminal moves to the destination according to the route guidance.

[0028] The scenario conditions may be set so that VR content determined based on the number of times the user terminal enters the custom area is displayed by the AR view, and the step of displaying the VR content may include displaying VR content determined based on the number of times the user terminal enters the custom area by the AR view.

[0029] According to another aspect, there is provided a computer system for realizing a user terminal or a server, the computer system including at least one processor configured to execute computer-readable instructions contained in a memory, wherein the at least one processor determines, based on the position of the user terminal, whether the user terminal is located in a pre-set custom area in space, determines whether the user terminal is located in a recognition area in which the user terminal can recognize an object in the custom area, and, when the user terminal is located in the recognition area, recognizes the object from an AR view including an image captured by a camera of the user terminal, and displays pre-set virtual reality (VR) content associated with the recognized object through the AR view. [Effects of the Invention]

[0030] By setting a custom area in a two-dimensional or three-dimensional space on a map of the space, associating a service element with the custom area, and defining a scenario for providing a service or information associated with the space based on the service element and / or controlling the settings of the user terminal, a custom area can be generated for controlling the user terminal according to the scenario.

[0031] When a user terminal is positioned in a pre-defined custom area, a module for recognizing objects is executed according to the scenario conditions defined for that custom area, thereby recognizing objects within the custom area and preventing the waste of resources that may occur not only on the user terminal but also on the server communicating with the user terminal.

[0032] By displaying VR content associated with an object within the custom area through the AR view, more detailed and richer information about the object can be provided to the user terminal, providing a high level of immersion to the user of the user terminal.

[0033] By setting various scenario conditions for a custom area and providing different VR content to a user terminal depending on whether the scenario conditions are met, and also by providing additional content to the user terminal, the efficiency of content provision can be improved. [Brief explanation of the drawings]

[0034] [Figure 1A] FIG. 10 illustrates a method for controlling a user terminal based on the user terminal being located in a pre-defined custom area in one embodiment. [Figure 1B] FIG. 10 is a diagram illustrating a method for controlling a user terminal so that VR content is displayed on the user terminal when the user terminal is positioned in a pre-defined custom area and an object in the custom area is recognized in one embodiment. [Figure 2] 1 illustrates a computer system for implementing a user terminal control method in one embodiment. [Figure 3] 10 is a flowchart illustrating a method for controlling a user terminal so that VR content is displayed on the user terminal when the user terminal is positioned in a pre-defined custom area and an object in the custom area is recognized, in one embodiment. [Figure 4] 10 is a flowchart illustrating a method for controlling settings of a user terminal in response to the user terminal entering a recognition area in which an object in a custom area can be recognized, in one embodiment. [Figure 5A] FIG. 10 illustrates a method for controlling user terminal settings based on the user terminal's location in a pre-defined custom area in one embodiment. [Figure 5B] 10 is a diagram illustrating a method for providing VR content to a user terminal based on the user terminal being located in a pre-defined custom area in one embodiment. [Figure 6] FIG. 10 illustrates a method for defining AR content associated with objects within a custom region in one embodiment. [Figure 7]FIG. 10 illustrates a method for defining VR content associated with objects within a custom region in one embodiment. [Figure 8] FIG. 10 illustrates a method for defining AR and VR content associated with objects within a custom region in one embodiment. [Figure 9] FIG. 1 illustrates a custom region defined as a three-dimensional space in one embodiment. [Figure 10] FIG. 2 is a diagram illustrating the degree of freedom (DoF) of a user terminal in one embodiment. [Figure 11] FIG. 10 illustrates a custom region including multiple points, in one embodiment. [Figure 12] FIG. 10 is a diagram illustrating a method in which multiple custom areas are set in a space and scenario conditions for controlling a user terminal are set for each custom area in an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0035] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0036] 1A and 1B illustrate a method for controlling a user terminal based on the user terminal's location in a pre-defined custom area in one embodiment.

[0037] In the following embodiments, "control" of the user terminal 100 may include at least one of, for example, using the AR view 10 of the user terminal 100 to augment and display content (e.g., AR content 30) indicating services or information associated with the space 5 in the image of the AR view 10, as shown in FIGS. 1A and 1B ; displaying a user interface (UI) associated with the services associated with the space 5 on the user terminal 100; displaying map element content 35 associated with the services associated with the space 5 on a map (or map view 20) of the user terminal 100; and controlling settings of the user terminal 100 to provide the services associated with the space 5 to the user terminal 100. Furthermore, in the embodiments of the present disclosure, "control" of the user terminal is not limited to the above-described examples and may include controlling the functions and / or operations of the user terminal.

[0038] In an embodiment, the "space 5" associated with providing augmented content may refer to an indoor space and / or an outdoor space. In other words, the definition of space 5 described below may refer to a space including an indoor location and an outdoor location, or may be a composite space combining an indoor space and an outdoor space. Space 5 may be at least a portion of a service target area for providing services including content provision.

[0039] In the detailed description below, for convenience of explanation, the location of a user who possesses the user terminal 100 may be described as the location of the user terminal 100. Also, for convenience of explanation, the terms "user" and "user terminal 100" possessed by the user may be used interchangeably.

[0040] In the detailed description below, "extending" content and / or information (e.g., indicators, etc.) onto the video may, in some embodiments, be interpreted to encompass overlaying the content and / or information onto the video / screen of the AR view.

[0041] As shown in the figure, a user terminal 100 such as a smartphone may be capable of capturing images of the surroundings using a camera, and may display AR content 30 using an AR view 10 including images captured by the camera. The AR content 30 may be associated with an object present in a space 5. Such AR content 30 may be mapped to the object (or a node representing the object).

[0042] At least one custom area may be defined on the map of the space 5 shown in the figure. In the example shown in the figure, two custom areas, a first custom area 60 and a second custom area 70, are defined. At least one service element may be associated with each custom area 60, 70. Furthermore, a scenario may be defined for each custom area 60, 70 to provide a service or information associated with the space 5 based on the service element or to control the settings of the user terminal 100. The service or information may include information for advertising or marketing. Such a scenario may include conditions for controlling the user terminal 100 (i.e., scenario conditions). The "custom areas 60, 70" may be preset on the server 200. For example, at least a portion of the space 5 or a partial area obtained by dividing the space 5 may be set as the "custom areas 60, 70" on the server 200.

[0043] A detailed method for defining each custom area 60, 70 and associating service elements and scenarios with the custom areas 60, 70 to set (create) the custom areas 60, 70 will be described below.

[0044] In the embodiment, it is determined whether the user terminal 100 is located in a first custom area 60 within the space 5 based on the location of the user terminal 100. If it is determined that the user terminal 100 is located in the first custom area 60, AR content 30 can be displayed in the AR view 10 of the user terminal 100 based on the service element associated with the custom area 60. For example, when the user terminal 100 enters the first custom area 60, the AR content 30 associated with the object "metal type" may be displayed in the AR view 10. At this time, the current location 50 of the user terminal 100 may be displayed in the map view 20. The current location 50 may correspond to a position 50' in the space 5. In the embodiment shown in the figure, when the user terminal 100 enters the first custom area 60, an AR content 30 associated with "metal type," which is an object located outside the first custom area 60, is displayed. However, depending on the service element associated with the first custom area 60, when the user terminal 100 enters the first custom area 60, other AR content associated with an object located inside the first custom area 60, such as the "Epitaph of Princess Suyeong," may be displayed.

[0045] Meanwhile, in other embodiments, based on the position of the user terminal 100, other types of content may be displayed in the AR view 10, an appropriate UI may be provided, appropriate map element content may be displayed, or the settings of the user terminal 100 may be appropriately changed according to the scenario conditions defined in the first custom area 60, such as when the user terminal 100 leaves the first custom area 60.

[0046] The map view 20 displayed together with the AR view 10 on the screen of the user terminal 100 may include a map that matches the image of the AR view 10. The map view 20 may include a current location 50 of the user terminal 100. The map view 20 may include a two-dimensional map or a three-dimensional map. The map view 20 may include objects in the space 5 through which the user terminal 100 moves. The map view 20 may display map element content 35 corresponding to the AR content 30. The map element content 35 may be displayed when the AR content 30 is displayed in the AR view 10 or when an object associated with the AR content 30 is included in the map view 20.

[0047] The user terminal 100 may communicate with the server 200 to control entry into and exit from the custom area 60, 70. For example, the server 200 may cause the user terminal 100 to be controlled based on a service element associated with the custom area 60, 70 in which the user terminal 100 is located.

[0048] The server 200 may be a computer system that manages the custom areas 60, 70 defined for the space 5, and may be a server that manages the association of service elements with the custom areas 60, 70, the definition of scenarios, etc. The server 200 may also be a map server that provides digital maps (3D maps and / or 2D maps).

[0049] Hereinafter, with reference to Figures 1B to 12, a more specific method of controlling the user terminal 100 by positioning the user terminal 100 in the custom areas 60 and 70 will be described, which is a method of providing virtual reality (VR) content to the user terminal 100.

[0050] FIG. 1B is a diagram illustrating a method for controlling a user terminal in one embodiment so that VR content is displayed on the user terminal when the user terminal is positioned in a pre-defined custom area and an object in the custom area is recognized.

[0051] FIG. 1B shows a method for displaying VR content 107 in the AR view 10 in association with the recognized object 105 when the user terminal 100 enters a custom area 60, 70 while moving according to route guidance and recognizes an object 105.

[0052] The maps 101 and 102 shown in the figure may correspond to the map view 20 described above. The map view 101 may display a route 132 for route guidance to a predetermined destination. Assume that the user terminal 100 recognizes an object 105 after moving along the route 132 and entering a custom area 60 or 70. When the object 105 is recognized, VR content 107 may be displayed in the AR view 10 in association with the recognized object 105.

[0053] FIG. 1B also shows a method for displaying VR content 107 in the AR view 10 in relation to the recognized object 105 when the user terminal 100 enters a custom area 60, 70 while moving and recognizes an object 105, regardless of route guidance.

[0054] The object 105 is placed in the space 5, and may be an exhibit displayed in the museum if the space 5 is a museum. In the example shown in the figure, the object 105 may be the "Monument to King Jinheung's Hunt."

[0055] 1A , the user terminal 100 may not only display the AR content 30 associated with the object 105, but also display the VR content 107 in the AR view 10 to provide a highly immersive experience to the user. The VR content 107 may provide richer information about the associated object 105 than the AR content 30. For example, the VR content 107 may include AR content and environmental content depicting a background associated with the object 105, such as the background around the object.

[0056] The VR content 107 may be displayed in the AR view 10 based on the service elements associated with the custom region 60, 70 and the scenario conditions defined for the custom region 60, 70.

[0057] The details of the VR content 107 and a specific method for controlling the user terminal 100 to display the VR content 107 on the AR view 10 will be described with reference to FIGS.

[0058] FIG. 2 illustrates a computer system and a server that executes a method for controlling a user terminal in one embodiment.

[0059] 1A and 1B may be realized by the computer system 2 shown in the figure. For example, a computer program for implementing the method of the embodiment may be installed and executed in the computer system 2, and such computer system 2 may execute the enhanced content providing method according to the embodiment under the control of the executed computer program.

[0060] The method for controlling the user terminal 100 according to the embodiment may be executed through a PC-based program or a dedicated application for a mobile terminal. For example, the method for controlling the user terminal 100 according to the embodiment may be implemented in the form of an independently operating program, or may be implemented so as to be operable on a specific application configured in the form of an in-app of a specific program. For example, such a specific application may be installed on a computer system 2 and may execute the method for controlling the user terminal 100 according to the embodiment by providing VR content 107 through the AR view 10.

[0061] The computer system 2 is an electronic device corresponding to the above-described user terminal 100, and may be a smartphone or similar device capable of installing and executing an application or program, as shown in Figures 1A and 1B. The computer system 2 may also be a personal computer (PC), a laptop computer, a tablet, an Internet of Things device, a wearable computer, or the like.

[0062] As shown in the figure, the computer system 2 may include a memory 110, a processor 120, a communication interface 130, and an input / output interface 140 as components for executing the method for controlling the user terminal 100.

[0063] Memory 110 is a computer-readable recording medium and may include random access memory (RAM), read-only memory (ROM), and a persistent mass storage device such as a disk drive. Here, a persistent mass storage device such as a ROM or a disk drive may be included in computer system 2 as a separate persistent storage device distinct from memory 110. Memory 110 may also store an operating system and at least one program code. Such software components may be loaded into memory 110 from a computer-readable recording medium separate from memory 110. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. In other embodiments, software components may be loaded into memory 110 through communication interface 130, which is not a computer-readable recording medium. For example, software components may be loaded into memory 110 of computer system 2 based on a computer program installed by a file received over network 160.

[0064] Processor 120 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to processor 120 by memory 110 or by communication interface 130. For example, processor 120 may be configured to execute instructions received according to program code stored in a storage device such as memory 110.

[0065] That is, the processor 120 may manage the components of the computer system 2 and may execute programs or applications used by the computer system 2. For example, the processor 120 may be configured to execute an application for executing a method for controlling the user terminal 100 of the embodiment and process data received from the server 200 for controlling the user terminal 100 (e.g., data received from the server 200 for providing the AR content 30 via the AR view 10). The processor 120 may also process calculations required for executing programs or applications and processing data, and may be at least one processor of the computer system 2 or at least one core within a processor.

[0066] The communication interface 130 may provide a function for the computer system 2 to communicate with other computer systems (not shown) via the network 160. For example, requests, instructions, data, files, etc. generated by the processor 120 of the computer system 2 in accordance with program code stored in a storage device such as the memory 110 may be transmitted to the other computer systems via the network 160 under the control of the communication interface 130. Conversely, signals, instructions, data, files, etc. from the other computer systems may be received by the computer system 2 via the communication interface 130 of the computer system 2 via the network 160. The signals, instructions, data, etc. received via the communication interface 130 may be transmitted to the processor 120 or the memory 110, and files, etc. may be recorded on a storage device (the above-mentioned permanent storage device) that the computer system 2 may further include. For example, the communication interface 130 may be a hardware module such as a network interface card, a network interface chip, or a networking interface port of the computer system 2, or a software module such as a network device driver or a networking program.

[0067] The input / output interface 140 may be a means for interfacing with the input / output device 150. For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface 140 may be a means for interfacing with a device that integrates input and output functions into one, such as a touch screen. The input / output device 150 may be configured as a single device together with the computer system 2.

[0068] Also, in other embodiments, computer system 2 may include more components than those shown in FIG. 2 . However, it is not necessary to explicitly illustrate most of the conventional components. For example, computer system 2 may be implemented to include at least some of the input / output devices connected to input / output interface 140 described above, and may further include other components such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, etc. As a more specific example, if computer system 2 is implemented in the form of a mobile device such as a smartphone, computer system 2 may further include various components typically found in mobile devices, such as a camera, an acceleration sensor, a gyro sensor, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration.

[0069] For example, the computer system 2 corresponding to the user terminal 100 may include a camera that captures images of the surroundings to execute the AR view 10. The computer system 2 may display an image captured by the camera as the AR view 10 and may augment and display the AR content 30 and the VR content 107 on the image of the AR view 10. For example, the computer system 2 may determine whether the user terminal 100 is located in a pre-defined custom area 60 or 70 based on positioning, and if the user terminal 100 recognizes an object 105 within the custom area 60 or 70, may display the VR content 107 in the AR view 10 in association with the recognized object 105. The positioning of the user terminal 100 may be performed using a visual localization or a Visual Positioning System (VPS) method. Alternatively, the positioning of the user terminal 100 may be performed using a marker recognition method, a GPS method, a Wi-Fi Positioning System (WPS) method, a Bluetooth module method, an odometry method, or the like.

[0070] Meanwhile, the substantial operation of determining whether the computer system 2 is located in the custom area 60, 70 and whether it recognizes the object 105 may be performed by the server 200, rather than the computer system 2. The server 200 may be an electronic device that communicates with the computer system 2 and provides the computer system 2 with data for controlling the user terminal 100, thereby controlling the user terminal 100. The server 200 may include a database as a device for managing (registering, changing, deleting, recording, maintaining, etc.) the data and content provided to the computer system 2 and the custom areas 60, 70, and may communicate with such a database. The server 200 may also be a map server that provides digital maps (3D maps and / or 2D maps), or may include such a map server. The server 200 may include at least one computer system. The computer system included in the server 200 has a configuration similar to that of the computer system 2 described above, and therefore, a redundant description thereof will be omitted.

[0071] In the following detailed description, for convenience of explanation, the embodiment will be described focusing on the computer system 2 corresponding to the user terminal 100, and explanations regarding communication with the server 200 and operations on the server 200 side may be simplified or omitted. Also, for convenience of explanation, operations (steps) performed by the components (e.g., processor, etc.) of the computer system 2 (or the server 200) may be described as being performed by the computer system 2 (or the server 200).

[0072] The above description of the technical features given with reference to FIGS. 1A and 1B can be applied to FIG. 2 as well, so duplicated description will be omitted.

[0073] FIG. 3 is a flowchart illustrating a method for controlling a user terminal in one embodiment so that VR content is displayed on the user terminal when the user terminal is positioned in a pre-defined custom area and an object in the custom area is recognized.

[0074] 3, a method for controlling a user terminal 100 executed by a computer system 2 will be described. Since the computer system 2 corresponds to the user terminal 100 described with reference to FIGS. 1A and 1B, the following detailed description will use the terms user terminal 100 and server 200 instead of computer system 2.

[0075] 4 may be configured to be executed by the server 200 instead of the user terminal 100. In the following, the execution of the steps will be described mainly with respect to the user terminal 100, and redundant description of the server 200 may be omitted.

[0076] In step 310, the user terminal 100 or the server 200 may determine, based on the location of the user terminal 100, whether the user terminal 100 is located in a pre-defined custom area 60, 70 within the space 5. The custom area 60, 70 may be an area defined within the space 5 based on an absolute coordinate system. The custom area 60, 70 may be associated with at least one service element. The user terminal 100 may determine the location of the user terminal 100 by a method based on at least one of GPS, a Visual Positioning System (VPS), and a Wi-Fi Positioning System (WPS), for example, and may determine, based on the determined location, whether the user terminal 100 is located in the custom area 60, 70 or whether it has entered or exited the custom area 60, 70. Alternatively, the server 200 may receive the determined location of the user terminal 100 and, based on the received location, determine whether the user terminal 100 is located in the custom area 60, 70 or has entered / exited the custom area 60, 70.

[0077] In step 320, the user terminal 100 or the server 200 may determine whether the user terminal 100 is located in a recognition area where the user terminal 100 can recognize the object 105 in the custom area 60, 70.

[0078] In step 330, the user terminal 100 may recognize the object 105 within the custom area 60, 70. In other words, the user terminal 100 may be able to recognize the object 105 in the recognition area. The object 105 may be recognized by a module for recognizing the object 105. The module for recognizing the object 105 may be, for example, a module for Vision Object Tracking (VOT). Alternatively, the module for recognizing the object 105 may be configured to be used for recognizing the object 105 using a smart lens of the user terminal 100. Such a module may be part of the processor 120 or may be implemented separately within the user terminal 100. Alternatively, such a module may be a software module executed by the processor 120.

[0079] The object 105 may be recognized by the AR view 10. For example, when the object 105 is included in an image captured by a camera included in the AR view 10, the object 105 may be recognized.

[0080] In step 340, the user terminal 100 may display, via the AR view 10, VR content 107 that has been previously set in association with the object 105 recognized in step 330. The VR content 107 may be 360-degree content. For example, the VR content 107 may include a panoramic image. The VR content 107 may include AR content depicting the object 105 and environmental content depicting a background related to the object 105, for example, a background around the object. The VR content 107 may be associated with a node defined at a position corresponding to the object 105 in the space 5.

[0081] To enhance the user's immersion, the user terminal 100 may display VR content 107 on the entire screen of the AR view 10. Such VR content 107 may depict a virtual space around the object 105. For example, if the object 105 is the "Monument to King Jinheung's Invasion," the environmental content depicting the virtual space around the object 105 may depict the geographical environment in which the Monument to King Jinheung's Invasion is located. The environmental content may further include text describing the object 105, as shown in FIG. 1B . Furthermore, depending on the type of object 105, the environmental content may be depicted in a virtual space to recreate a historical space, transform it into a themed space, simulate a future space, or the like.

[0082] The user terminal 100 may display the VR content 107 by changing the viewpoint in response to a change in the viewpoint of the camera in the AR view 10. This allows the user to have an immersive experience as if they were actually exploring a virtual space in which the object 105 is located.

[0083] The VR content 107 may also include an animation effect. For example, when the object 105 is recognized and the VR content 107 is displayed on the AR view 10, the VR content 107 may be expressed with an animation that gradually and innovatively spreads across the entire screen of the AR view 10. Alternatively, after the AR content corresponding to the object 105 is displayed, the VR content 107 may be displayed by overlaying (overlaying) the above-described environmental content in the space surrounding the AR content.

[0084] Such VR content 107 may be displayed while the object 105 is being recognized by the user terminal 100.

[0085] In step 350, when the user terminal 100 is no longer located in the recognition area, the display of the VR content 107 may be terminated. For example, the user terminal 100 may terminate the display of the VR content 107 when the viewpoint of the camera in the AR view 10 changes and the user terminal 100 leaves the recognition area, or when the user terminal 100 moves and leaves the recognition area.

[0086] An animation effect may also be included when the display of the VR content 107 ends. For example, when the user terminal 100 moves out of the recognition area of ​​the object 105, the VR content 107 may disappear progressively from the AR view 10, and the original AR view 10 may be displayed.

[0087] In an embodiment, at least one service element is associated with the custom area 60, 70. Furthermore, the custom area may have scenario conditions set therein for controlling the user terminal 100 based on the service element when the user terminal 100 is located in the custom area 60, 70. The scenario conditions may include conditions for providing services or information associated with the space 5 or controlling the settings of the user terminal 100 based on the service element associated with the custom area 60, 70.

[0088] The display of the VR content 107 through the AR view 10 described above may also be based on the service elements associated with the custom areas 60, 70 and the scenario conditions set in the custom areas 60, 70.

[0089] When the user terminal 100 is determined to be located in the custom area, the user terminal 100 or the server 200 may control the user terminal 100 based on the service elements associated with the custom area 60, 70. In other words, the user terminal 100 may be controlled to perform a specific operation based on the data included in the service elements associated with the custom area 60, 70.

[0090] Here, "controlling" the user terminal 100 may include at least one of using the AR view 10 of the user terminal 100 to augment and display content (e.g., AR content 30 and / or VR content 107) that shows services and information associated with the space 5 in the image of the AR view 10; displaying a user interface (UI) associated with services associated with the space 5 on the user terminal 100; displaying map element content 35 associated with services associated with the space 5 on the map of the user terminal 100 (or on the map view 20); and controlling the settings of the user terminal 100 to provide services associated with the space 5 to the user terminal 100.

[0091] The service elements associated with the custom regions 60, 70 may include data for displaying VR content 107 related to the service associated with the space 5.

[0092] Additionally, the service elements associated with the custom regions 60, 70 may include data for displaying VR content 107 related to the space 5 and the service associated with the space. Thus, in embodiments, the VR content 107 may be displayed in the AR view 10 based on scenario conditions defined for such service elements and custom regions 60, 70.

[0093] How to recognize the object 105 for displaying the VR content 107 and how to display the VR content 107 on the AR view 10 according to specific scenario conditions will be described in more detail with reference to FIGS.

[0094] Meanwhile, the method for controlling the user terminal 100 according to the embodiment may be executed in combination with a route guidance service. For example, the method for controlling the user terminal 100 according to the embodiment may be executed in combination with AR navigation that provides route guidance indoors or outdoors.

[0095] In step 302, the user terminal 100 may receive a request for route guidance to a destination. For example, a user of the user terminal 100 may request route guidance from the user terminal 100 by using the user terminal 100 to launch an application or program that provides an AR navigation service or other route guidance service and inputting a destination. The route guidance request may be transmitted to the server 200 that provides the route guidance service.

[0096] In step 304, the user terminal 100 may provide route guidance according to the route guidance request received in step 302. For example, the user terminal 100 may receive data for route guidance to a destination from the server 200 that provides a route guidance service, and may display information for route guidance to a destination (e.g., guide information) based on the received data.

[0097] The user of the user terminal 100 may move to the destination according to the route guidance. At this time, the above-described steps 310 to 340 may be executed when the user terminal 100 moves to the destination according to the route guidance.

[0098] For example, when the user terminal 100 moves to a destination according to route guidance, it may determine whether the user terminal 100 is located in the custom area 60, 70 and whether an object 105 has been recognized, and based on the determination results, the user terminal 100 may display the VR content 107 in the AR view 10. An example of this has been described with reference to FIG. 1B , so a duplicate description will be omitted.

[0099] The explanation of the technical features given above with reference to FIGS. 1A to 2 can be applied to FIG. 3 as well, so duplicated explanations will be omitted.

[0100] FIG. 4 is a flow chart illustrating a method for controlling settings of a user terminal by entering a recognition area in which the user terminal can recognize an object in a custom area, in one example.

[0101] A method for determining whether the user terminal 100 is located in a recognition area for recognizing the object 105 and recognizing the object 105 will be described in more detail with reference to FIG.

[0102] As described above, the service elements associated with the custom areas 60, 70 may include data for controlling the configuration of the user terminal 100 to provide the services associated with the space 5 to the user terminal 100.

[0103] In step 410, when the user terminal 100 enters a custom area 60, 70, or when the user terminal 100 enters a pre-set recognition area, the user terminal 100 or the server 200 may control the settings of the user terminal 100 according to the scenario conditions set for the custom area 60, 70.

[0104] Controlling the settings of the user terminal 100 may include controlling the functions and / or operations of the user terminal 100 in relation to the services provided to the user terminal 100 within the space 5. Controlling the functions and / or operations may include turning the functions and / or operations on / off or activating / deactivating the functions and / or operations.

[0105] For example, when the user terminal 100 enters the custom area 60, 70, the user terminal 100 or the server 200 may control the settings of the user terminal, including the execution of a module for recognizing the object 105, in accordance with the scenario conditions set in the custom area 60, 70. In other words, when the user terminal 100 enters the custom area 60, 70, the module for recognizing the object 105 may be executed. The module for recognizing the object 105 may be, for example, the VOT module described above. When the user terminal 100 enters the custom area 60, 70, the module for recognizing the object 105 in the user terminal 100 is turned on or initiated, thereby enabling the user terminal 100 to recognize the object 105 in the custom area 60, 70. On the other hand, when the user terminal 100 leaves the custom area 60, 70, the module for recognizing the object 105 in the user terminal 100 may be turned off or deactivated again. However, in some embodiments, the controlled settings may be maintained even when the user terminal 100 moves away from the custom area 60, 70.

[0106] Such an embodiment allows the user terminal 100 to recognize and detect objects only in a specific custom area or only when the user enters a specific custom area, without recognizing or detecting objects more than necessary in all areas within the space 5. Therefore, services associated with the space 5 can be efficiently provided to the user terminal 100 while taking into consideration the user's movement within the space 5 and their field of view or vision (in the sense that the space 5 may be a three-dimensional space having a custom area height range).

[0107] With the module for recognizing the object 105 running, the user terminal 100 may recognize the object 105 through the AR view 10. For example, when the object 105 is displayed in the AR view 10, the user terminal 100 may determine that the user terminal 100 is located in a recognition area for recognizing the object 105, and may thereby recognize the object 105. Alternatively, the user terminal 100 may determine a condition that the distance between the object 105 and the user terminal 100 is equal to or less than a predetermined distance, a condition that the object 105 is displayed in a specific area (e.g., a central area or its vicinity) in the AR view 10, and a condition that the object 105 is displayed in a predetermined proportion or more in the AR view 10 (i.e., When at least one of the conditions (conditions that the object 105 occupies a predetermined percentage or more in the AR view 10) is satisfied, it may be determined that the user terminal 100 is located in a recognition area for recognizing the object 105, and thus the object 105 may be recognized. The above-described conditions for recognizing the object 105 may be included in the scenario conditions set for the custom areas 60 and 70.

[0108] In another embodiment, the recognition area for recognizing the object 105 described above may be a predefined area within the custom area 60, 70. Such a predefined area may be a predetermined range area around the object 105. The user terminal 100 or the server 200 may determine that the user terminal 100 is located in the recognition area for recognizing the object 105 when the user terminal 100 is located within such a predefined recognition area. In this case, when it is determined that the user terminal 100 has entered the recognition area, the user terminal 100 or the server 200 may control the settings of the user terminal 100, including the execution of a module for recognizing the object 105. In other words, if a predefined recognition area exists within the custom area 60, 70, when the user terminal 100 is located in the recognition area, a module for recognizing the object 105 may be executed, thereby recognizing the object 105. The predefined recognition area described above may be included in the scenario conditions set in the custom area 60, 70.

[0109] When the user terminal 100 recognizes the object 105, the VR content 107 associated with the recognized object 105 may be displayed by the AR view 10.

[0110] The explanation of the technical features given above with reference to FIGS. 1A to 3 can be applied to FIG. 4 as well, so duplicated explanations will be omitted.

[0111] FIG. 5A illustrates a method for controlling settings of a user terminal based on the user terminal's location in a pre-defined custom area, in one example.

[0112] The service elements associated with the custom areas 60 , 70 may include data for controlling the configuration of the user terminal 100 to provide the services associated with the space 5 .

[0113] When the user terminal 100 enters the custom area 60, 70, the user terminal 100 or the server 200 may execute a module for object recognition of the user terminal 100 based on the service elements associated with the custom area 60, 70.

[0114] As shown in the figure, a custom area 510-1 is set in space 500-1, and when user terminal 100 enters custom area 510-1 from position 505, the settings of user terminal 100 may be controlled to turn on or initiate a module for object recognition in user terminal 100.

[0115] After the module for object recognition is executed, when the user terminal 100 recognizes the object 105, namely, the Jinheung King's Monument, the AR view 10 may display the VR content 107 in association with the recognized object 105.

[0116] According to this embodiment, the object recognition module is executed only when the user terminal 100 enters the custom area 510-1, thereby eliminating the need for excessive content to be expanded through the AR view 10, which can complicate the user interface, increase the cognitive load on the user, and increase the load on the user terminal 100.

[0117] Meanwhile, FIG. 5B is a diagram illustrating a method for providing VR content to a user terminal based on the user terminal being located in a pre-defined custom area, in one example.

[0118] As shown in the figure, a custom area 510-2 is set in the space 500-2. When the user terminal 100 enters the custom area 510-2, the settings of the user terminal 100 may be controlled to turn on or initiate an object recognition module of the user terminal 100.

[0119] Meanwhile, a plurality of points A and B may be defined in the custom area 510-2. Each of the plurality of points A and B may indicate an entrance for entering the space corresponding to the custom area 510-2.

[0120] The scenario conditions for the custom area 510-2 may be set so that, when the user terminal 100 is located in the custom area 510-2, the VR content 107 determined based on the points passed by the user terminal 100 is displayed as the VR content 107 through the AR view 10. Therefore, the user terminal 100 can display, through the AR view 10, the VR content 107 determined based on the points passed by the user terminal 100. For example, the user terminal 100 may display, through the AR view 10, different VR content 107 when the user terminal 100 enters the custom area 510-2 through point A and when the user terminal 100 enters the custom area 510-2 through point B.

[0121] On the other hand, the "point passed through" by the user terminal 100 may be a predetermined point outside the custom area 510-2 that the user terminal 100 passed through, or may be a predetermined point defined within the custom area 510-2.

[0122] The above-mentioned "determined VR content" may be existing VR content displayed in a different form, with other animations or effects applied, or may be VR content with different content displayed, or existing VR content with additional content (audio, text, images, video, etc.) added.

[0123] Other examples of displaying VR content in the AR view 10 according to scenario conditions set for a custom region will be described in more detail below with reference to the drawings.

[0124] The above description of the technical features given with reference to FIGS. 1A to 4 can also be applied to FIGS. 5A and 5B, and therefore a duplicated description will be omitted.

[0125] FIG. 6 illustrates an example method for defining AR content associated with an object within a custom region, and FIG. 7 illustrates an example method for defining VR content associated with an object within a custom region.

[0126] The illustrated AR content 610 may be defined to be displayed in the AR view 10 in association with the object 105, 'King Jinheung's Monument to the Hunt'.

[0127] The AR content 610 may be placed in the UI 600 where data representing the space 5 has been loaded or rendered, or may be placed in a custom region 60, 70 set within the space 5. The AR content 610 may be included in a service element associated with the custom region 60, 70.

[0128] 7 may be defined to be displayed in the AR view 10 in association with the object 105, the "Monument to King Jinheung's Hunt," and the like. The VR content 710 may include the AR content 610 and environmental content depicting a virtual space around the object 105. As shown in the figure, the virtual space around the object 105 depicted by the VR content 710 may be expressed (environment content) as a virtual spherical space or a hemispherical space centered on the object 105. By including a virtual space represented as a virtual spherical space centered on the object 105 in the VR content 710, the user can feel immersive in the AR view 10 displaying the VR content 710.

[0129] If the space 5 is indoors, the UI 600, 700 may include an indoor map generation UI for creating an indoor map for the space 5 or may be a part of the indoor map generation UI. Such an indoor map generation UI may provide a function for setting detailed spaces (such as exhibition rooms), means of transportation such as stairs, escalators, and elevators, facilities such as restrooms, signs, road signs, and information displays, other obstacles such as pillars, route information within the space 5, nodes, and links connecting the nodes. For example, a user of an administrator terminal for generating an indoor map and creating custom areas 60, 70 on the indoor map can set areas corresponding to the custom areas 60, 70 through the UI 600, 700 and associate the AR content 610 and / or VR content 710 (i.e., service elements including the same) with the custom areas 60, 70 by directly placing the AR content 610 and / or VR content 710 in the set areas or by selecting the AR content 610 and / or VR content 710 to be placed in the set areas. The UIs 600 and 700 may provide a separate UI for selecting and arranging service elements to be associated with the custom areas 60 and 70. The user may select and arrange the service elements by a method such as clicking or dragging and dropping.

[0130] In the examples of UIs 600 and 700 shown in the figures, it is assumed that a map corresponding to space 5 is displayed and custom regions 60 and 70 are generated on the map.

[0131] On the other hand, unlike what is shown in the figure, UI600, 700 may render at least one of a map of space 5, point cloud data (PCD) collected from space 5, and an image captured of space 5 as data showing space 5, and custom areas 60, 70, which are two-dimensional and / or three-dimensional spaces, may be generated on the rendered data.

[0132] When the indoor map generation UI includes UI 600, 700, the custom areas 60, 70 may be generated along with the construction of an indoor map for the space 5, and the placement of content 610 and / or VR content 710 in the custom areas 60, 70 may also be performed.

[0133] The above description of the technical features given with reference to FIGS. 1A to 5B can also be applied to FIGS. 6 and 7, so duplicated description will be omitted.

[0134] FIG. 8 illustrates a method for defining AR and VR content associated with objects within a custom region in one example.

[0135] As described above, to enhance the user's immersion, the user terminal 100 may display the VR content 107 on the full screen of the AR view 10. Here, the VR content 107 may be a depiction of a virtual space around the object 105.

[0136] In this case, the virtual space depicted by the VR content 107 may be expressed as a virtual three-dimensional space determined by reflecting the geometry of the surrounding space in which the object 105 is located. In other words, the virtual space depicted by the VR content 107 may reflect the shape of the surrounding real space in which the object 105 is located.

[0137] For example, as shown in the figure, if an object 810 is placed in an exhibition room space 820, and the exhibition room space 820 has a hexahedral geometry, the VR content displayed in the AR view 10 in association with the object 810 may include AR content 830 showing the object 810 and environmental content 840 showing the exhibition room space 820. In this way, the environmental content 840 may have a hexahedral shape like the exhibition room space 820, reflecting the geometry of the exhibition room space 820.

[0138] By reflecting the geometry of the actual exhibition space 820 in which the object 810 is placed in the VR content displayed on the AR view 10, the immersive experience of the user viewing the VR content can be further enhanced.

[0139] Meanwhile, the geometry of the exhibition room space 820 may be base data acquired when creating an indoor map of the space 5 including the exhibition room space 820 or when collecting data (PCD, photographed images, etc.) showing the space 5. The geometry of the exhibition room space 820 may be recorded and managed in the server 200.

[0140] 7, when the VR content 710 is placed in the custom area 60, 70 through the UI 600, 700, the virtual space around the object 105 depicted by the VR content 710 is not a virtual spherical or hemispherical space centered on the object 105, but a three-dimensional space that reflects the geometry of the real space in which the object 105 is placed. Such a three-dimensional space may be automatically displayed on the UI 600, 700 according to the geometry of the real space of the base.

[0141] The technical features described above with reference to FIGS. 1A to 7 can be applied to FIG. 8 as well, so duplicated explanations will be omitted.

[0142] The custom areas 60, 70 that are defined and generated for the space 5 and the scenario conditions that are set for the custom areas 60, 70 will be described in more detail below.

[0143] FIG. 9 illustrates a custom region defined in three-dimensional space in one example.

[0144] FIG. 9 provides a more detailed explanation of the custom region 900 defined as a three-dimensional space and the scenario conditions defined therein.

[0145] The above-mentioned custom regions 60, 70 may be defined on the map of the space 5 to include at least one of two-dimensional space and three-dimensional space.

[0146] When the user terminal 100 is located in a custom area 60, 70 on the map (or when it is determined that the user terminal 100 has exited the custom area 60, 70), the user terminal 100 or the server 200 may control the user terminal 100 based on the service elements associated with the custom area 60, 70.

[0147] The custom region 900 may be defined to include a two-dimensional area and a three-dimensional space with a set height range, or the custom region 900 may be defined as a multi-dimensional space including two-dimensional and three-dimensional spaces.

[0148] The two-dimensional region that makes up custom region 900 may include a two-dimensional coordinate space (x, z). The height range that makes up custom region 900 may include a y-axis coordinate space. The height range may be a range with an upper and lower bound, as shown in the figure (H), or may be a range without an upper or lower bound, as shown in the figure.

[0149] When the user terminal 100 is located in a two-dimensional area that constitutes the custom area 900 and the height of the user terminal 100 falls within a set height range that constitutes the custom area 900, the user terminal 100 or the server 200 may determine that the user terminal 100 is located in the custom area 900 and control the user terminal 100 based on a service element associated with the custom area 900. Thus, when it is determined that the user terminal 100 is located in the custom area 900, a module for recognizing the object 105 may be executed, and the object 105 may be recognized, causing the VR content 107 to be displayed in the AR view 100.

[0150] The height of the user terminal 100 may be acquired from a sensor included in the user terminal 100. The height of the user terminal 100 may indicate, for example, the height of the camera of the user terminal 100.

[0151] Although one height range (H) is illustrated in FIG. 9, multiple height ranges may be defined in the custom area 900, and a service element may be associated with each height range to define a scenario.

[0152] In this way, by setting the custom area 900 in a stereoscopic three-dimensional space, even if the user terminal 100 is located in the same two-dimensional space, the user terminal 100 can be controlled using different scenarios depending on the height of the user terminal 100.

[0153] The scenarios defined for the custom regions 60, 70 are described in more detail below with reference to FIG.

[0154] FIG. 10 is a diagram showing the DoF (Degree Of Freedom) of a user terminal in one example.

[0155] The custom regions 60, 70 may define 3DoF or 6DoF attitude conditions for the user terminal 100. 6DoF may refer to movement along the x-, y-, and z-axes, and rotation along the x-, y-, and z-axes (roll, yaw, and pitch).

[0156] Such pose conditions may be included in the scenario conditions defined for the custom region 60, 70, and as described above, such scenario conditions may be included in the service elements associated with the custom region 60, 70.

[0157] When the user terminal 100 is determined to be located in the custom area 60, 70, if the attitude of the user terminal 100 satisfies the attitude conditions defined in the custom area 60, 70, the user terminal 100 or the server 200 may control the user terminal 100 based on the service elements associated with the custom area 60, 70. The attitude of the user terminal 100 may be acquired from a sensor provided in the user terminal 100, for example, a gyro sensor.

[0158] By defining such posture conditions in the custom regions 60, 70, the user terminal 100 may be controlled based on the service element only when, for example, the user terminal 100 enters the custom region 60, 70 and the AR view 10 maintains a specific angle or a specific angle range. As an example, VR content that should be displayed in a specific region within the custom region 60, 70, for example, at a position above or below a predetermined height, may be displayed in the AR view 10 of the user terminal 100 only when the user terminal 100 enters the custom region 60, 70 and the AR view 10 maintains an angle or angle range that reflects the specific region above or below the predetermined height.

[0159] Therefore, in the embodiment, the AR content can be displayed at a more appropriate position and viewpoint of the AR view 10.

[0160] Other examples of scenario conditions defined for the custom regions 60 and 70 are described below.

[0161] Scenario conditions may be set in the custom areas 60, 70, including at least one of a first condition indicating a time condition, a second condition indicating a movement of the user terminal 100, and a third condition indicating an operation on the user terminal 100.

[0162] The first condition may be, for example, a condition requiring the user terminal 100 to stay in the custom area 60, 70 for a predetermined time or more (e.g., three seconds or more), or for a predetermined time or less. Alternatively, the first condition may be a condition requiring the user terminal 100 to enter the custom area 60, 70 during a predetermined time period, or to enter the custom area 60, 70 at a time other than the predetermined time period. The second condition may be, for example, a condition indicating a physical operation of the user terminal 100, such as a condition requiring the user to shake the user terminal 100 a predetermined number of times or with a predetermined strength or more. Alternatively, the second condition may be a condition requiring the user terminal 100 to move in a specific shape (e.g., a figure eight). The third condition may be a condition requiring the user terminal 100 to be electronically operated, such as pressing a specific button or key, making a specific gesture, inputting a specific voice or audio, or photographing a specific object.

[0163] When the user terminal 100 is determined to be located in the custom area 60, 70, if the user terminal 100 satisfies at least one scenario condition among the first to third conditions, the user terminal 100 or the server 200 may control the user terminal 100 based on the service elements associated with the custom area 60, 70. For example, the user terminal 100 may display different AR content or VR content, display a different UI, display different map element content, or perform different setting control of the user terminal 100, depending on whether the scenario condition is satisfied.

[0164] The technical features described above with reference to FIGS. 1A to 8 can be applied to FIG. 9 as well, so duplicated explanations will be omitted.

[0165] FIG. 11 illustrates a custom region containing multiple points in one example.

[0166] The illustrated custom area 1100 may be an area defined on a map of the space 5. The custom area 1100 may include a point indicating an entrance / exit A and a point indicating an entrance / exit B. The custom area 1100 may further include a point indicating an object A and a point indicating an object B. The user terminal 100 or the server 200 may determine whether the user terminal 100 has entered or exited the custom area 1100 by tracking the location of the user terminal 100.

[0167] Furthermore, multiple points may be defined not only inside the custom area 1100 but also outside the custom area 1100 .

[0168] A plurality of service elements may be associated with the custom area 1100. In this case, each of the plurality of service elements may be associated with a point within the custom area 1100. A scenario or scenario condition defined in the custom area may be defined such that services or information associated with the space 5 are provided differently depending on the order in which the user terminal 100 passes through the points associated with each service element. The services or information provided differently may be AR content or VR content, or may be other types of content such as audio, text, images, or video that can be displayed on the user terminal 100.

[0169] As an example, a scenario condition may be defined in the custom area 1100 such that different types of content are displayed from the user terminal 100 when the user terminal 100 enters the custom area 1100 and when it leaves the custom area 1100.

[0170] Alternatively, scenario conditions may be defined in the custom area 1100 to distinguish between cases where the user terminal 100 enters through a specific point within the custom area 1100 (e.g., at least one of the points indicated by Entrance A, Entrance B, Object A, and Object B in the figure) and cases where the user terminal 100 enters the custom area 1100 after passing through a specific point outside the custom area 1100, and to display different types of content from the user terminal 100. For example, when the user terminal 100 enters the custom area 1100 through a point indicated by Entrance A, general welcome content may be displayed on the user terminal 100, and when the user terminal 100 enters the custom area 1100 through a point indicated by Entrance B, other content, such as coupon content, may be displayed on the user terminal 100. The coupon content may include a discount coupon or a special offer that can be used within the space 5.

[0171] In this way, when multiple points are defined inside or outside the custom region 1100, different VR content can be displayed by the AR view 10 when the user terminal 100 passes through a first point among the multiple points and when the user terminal 100 passes through a second point among the multiple points. In other words, even when the same object is recognized, different VR content can be displayed by the AR view 10 depending on the point through which the user terminal 100 entered the custom region 1100 or the point through which the user terminal 100 passed within the custom region 1100.

[0172] Furthermore, when the user terminal 100 is located in a specific custom area 60, 70, it may determine whether the user terminal 100 has entered or exited the custom area 60, 70. Based on the determination result and set scenario conditions, the user terminal 100 may be controlled to display different VR content when entering and exiting the custom area 60, 70. In this case, based on the current location of the user terminal 100 and the route the user terminal 100 has already traveled or will travel, it may be determined that the user terminal 100 has entered the custom area 60, 70 ([1]) or will exit the custom area 60, 70 ([2]). The user terminal 100 or the server 200 may determine, based on the route, which nodes the user terminal 100 has already passed through or which nodes the user terminal 100 will pass through, and may determine accordingly whether the user terminal 100 has entered or exited the custom area in which it is currently located. In the cases of [1] and [2], the user terminal 100 may be controlled to display different VR content. That is, even if the same object is recognized, different VR content may be displayed by the AR view 10 depending on whether the user terminal 100 enters or leaves the custom areas 60 and 70.

[0173] Alternatively, a scenario condition may be defined in the custom area 1100 such that, when the user terminal 100 enters the custom area 1100, the user terminal 100 is requested to perform an additional action, such as requesting movement along the edge of the custom area 1100, and a different type of content is displayed on the user terminal 100 as the action is performed. The request for the additional action may also be provided to the user terminal 100 in the form of content or a UI.

[0174] Alternatively, a scenario condition may be defined in the custom area 1100 such that the number of times the user terminal 100 enters the custom area 1100 is counted and specific content is displayed on the user terminal 100 when the user terminal 100 enters the custom area 1100 n times or more (n is a natural number greater than or equal to 2) (the number of times the user terminal 100 enters the custom area 1100 may be counted within a predetermined time period (e.g., one hour or one day)), or a scenario condition may be defined in the custom area 1100 such that specific content is displayed on the user terminal 100 when the user terminal 100 stays in the custom area 1100 for a predetermined time period or more. For example, suppose that a scenario condition is set such that VR content determined based on the number of times the user terminal 100 enters the custom area 1100 is displayed by the AR view. In this case, the VR content determined based on the number of times the user terminal 100 enters the custom area 1100 may be displayed by the AR view 10. That is, even if the same object is recognized, different VR content can be displayed by the AR view 10 depending on the number of times the user terminal 100 enters the custom area 1100.

[0175] Meanwhile, when the user terminal 100 exits (leaves) the custom area 1100, all content displayed on the user terminal 100 may be hidden, i.e., the display may be canceled. When the user terminal 100 exits the custom area 1100, a predetermined notification sound may be displayed from the user terminal 100. Alternatively, when the user terminal 100 exits the custom area 1100, AR content or VR content notifying the user that the user terminal 100 has exited the custom area 1100 may be displayed on the AR view 10.

[0176] In this way, in the embodiment, it is possible to determine whether the user terminal 100 has entered or exited the custom area 1100, and depending on the determination result, different AR content or VR content can be displayed (for example, expanded entry guide AR content or VR content when entering, and expanded exit guide AR content or VR content when exiting), a different UI can be displayed, different map element content can be displayed, or different setting control of the user terminal 100 can be performed.

[0177] The technical features described above with reference to FIGS. 1A to 10 can also be applied to FIG. 11, so duplicated explanations will be omitted.

[0178] FIG. 12 is a diagram showing a method in which a plurality of custom areas are set in a space and a scenario condition for controlling a user terminal is set for each custom area in one example.

[0179] Other examples of scenario conditions defined for each of the multiple custom areas 1210 to 1240 set in the space 1200 will be described in more detail below.

[0180] The custom areas 60 and 70 may be any one of the custom areas 1210 to 1240.

[0181] Any one of the custom areas 60, 70 among the custom areas 1210 to 1240 may have at least one of the following scenario conditions defined: i) a first scenario condition indicating whether the user terminal 100 has passed through a predetermined custom area other than the custom area 60, 70; and ii) a second scenario condition indicating whether the number of user terminals located in the custom area 60, 70 falls within a predetermined range. Depending on whether the scenario condition is satisfied, the user terminal 100 may be controlled to receive different services or information associated with the space 5, 1200 or to perform different settings on the user terminal 100, based on the service elements associated with the custom area 60, 70.

[0182] The first scenario condition may be a condition that requires the user terminal 100, having entered the custom area 60, 70, to enter another specific custom area before entering the custom area 60, 70. For example, the first scenario condition may be satisfied if the user terminal 100 enters the custom area 1210 (within a predetermined time) after entering a specific custom area among the custom areas 1220 to 1240. Alternatively, the first scenario condition may be satisfied if the user terminal 100 enters the custom area 1210 (within a predetermined time) after entering at least some of the custom areas 1210 to 1240 in a specific order. Depending on whether the first scenario condition is satisfied, the user terminal 100 may display different AR content or VR content, display a different UI, display different map element content, or perform different setting control of the user terminal 100.

[0183] For example, the scenario conditions may be set so that, when the user terminal 100 is located in the custom areas 60 and 70, VR content determined based on another custom area into which the user terminal 100 entered before the user terminal 100 entered the custom area is displayed by the AR view 10. In this case, the VR content determined based on the other custom area may be displayed on the AR view 10 of the user terminal 100. In other words, even if the same object is recognized, different VR content may be displayed by the AR view 10 depending on which other custom area the user terminal 100 entered before entering the custom area, or whether the user terminal 100 entered a specific other custom area.

[0184] The second scenario condition may be a condition that requires the number of user terminals located in the custom areas 60, 70, or that the cumulative number of user terminals that have entered the custom areas 60, 70 (within a predetermined time period) be greater than or equal to a predetermined number, or less than a predetermined number. Alternatively, the second scenario condition may be a condition that requires the number of user terminals located in other custom areas other than the custom areas 60, 70 into which the user terminal 100 has entered, or that the cumulative number of user terminals that have entered other custom areas (within a predetermined time period) be greater than or equal to a predetermined number, or less than a predetermined number. Alternatively, the second scenario condition may be a condition that requires the number of user terminals located in both the custom areas 60, 70 and the other custom areas, or that the cumulative number of user terminals that have entered both custom areas (within a predetermined time period) be greater than or equal to a predetermined number, or less than a predetermined number. Depending on whether the second scenario condition is satisfied, the user terminal 100 may display different AR content or VR content, display a different UI, display different map element content, or perform different setting control of the user terminal 100.

[0185] The first and second scenario conditions may be used in combination. For example, when the user terminal 100 enters the custom area 60, 70, the user terminal 100 or the server 200 may determine whether the first and second scenario conditions are met, and may control the user terminal 100 based on service elements only if both scenario conditions are met. Alternatively, the user terminal 100 may be controlled differently based on service elements depending on whether the first or second scenario condition is met.

[0186] For example, the user terminal 100 may display VR content using the AR view 10 only when the scenario conditions are met, or may display different VR content using the AR view 10 when the scenario conditions are met and when the scenario conditions are not met.

[0187] The above-mentioned "determined VR content" may be existing VR content displayed in a different form, with different animations or effects applied, or may be VR content with different content displayed, or existing VR content with additional content (audio, text, images, video, etc.) added.

[0188] The technical features described above with reference to FIGS. 1A to 11 can be applied to FIG. 12 as well, so duplicated explanations will be omitted.

[0189] As such, in the embodiment, a "custom range" scheme may be utilized that is easier to handle and more natural than a "rendering range"-based scheme that displays AR / VR content and controls the user terminal 100 based on the distance or radius to the position of an object located in a space, or a "semantic range"-based scheme that divides a space into semantic unit regions and displays AR / VR content when the user terminal 100 enters a unit space and controls the user terminal 100. The custom range may use a "custom region" described with reference to the embodiment. The "custom range" scheme may utilize a custom region in which service elements for displaying AR / VR content and controlling the user terminal 100 are associated and scenario conditions are defined.

[0190] When using a "rendering range" based method or a "semantic range" based method, the AR / VR content may be inappropriately displayed at the position of an object in the AR view 10 depending on the position of the user terminal 100, and in some cases, the AR / VR content may be displayed in an occlusion area. The "custom area" method described with reference to the embodiments uses a custom area, thereby solving such problems.

[0191] The above-described devices may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the OS. The processing device may also access, record, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, a single processing device may be described. However, those skilled in the art will understand that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0192] Software may include computer programs, codes, instructions, or a combination of one or more of these, which may configure a processing device to operate as desired or may independently or collectively instruct the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored and executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0193] Methods according to embodiments may be implemented in the form of program instructions executable by various computer means and recorded on a computer-readable medium. In this case, the medium may continuously record a computer-executable program or may temporarily record the program for execution or download. The medium may be various recording or storage means in the form of a single piece of hardware or multiple pieces of hardware combined together. It may be a medium directly connected to a computer system or distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to record program instructions, such as ROMs, RAMs, and flash memories. Other examples of media include recording media or storage media managed by app stores that distribute applications, or by websites or servers that provide or distribute various software.

[0194] Although the embodiments have been described above based on limited examples and drawings, those skilled in the art will appreciate that various modifications and variations may be made from the above description. For example, the described techniques may be performed in an order different from that described, and / or the described system, structure, device, circuit, or other element may be coupled or combined in a manner different from that described, or may be substituted or replaced by other elements or equivalents, and still achieve suitable results.

[0195] Therefore, different embodiments are within the scope of the appended claims, provided that they are equivalent to the claims.

Claims

1. A user terminal control method executed by a computer system that is a server or a user terminal, comprising: determining whether the user terminal is located in a preset custom area in space based on the location of the user terminal; determining whether the user terminal is located in a recognition area in which the user terminal can recognize an object in the custom area; When the user terminal is located in the recognition area, recognizing the object from an AR view including an image captured by a camera of the user terminal; displaying, through the AR view, VR content that is preset in association with the recognized object; A user terminal control method, comprising:

2. the custom area has at least one service element associated with it; the service element includes data for displaying the VR content related to a service associated with the space; A scenario condition is defined in the custom area for controlling the user terminal based on the service element when the user terminal is located in the custom area. The user terminal control method according to claim 1 .

3. the service element includes data for controlling a setting of the user terminal to provide the service associated with the space to the user terminal; The user terminal control method of claim 2, further comprising a step of controlling settings of the user terminal, including execution of a module for recognizing the object, in accordance with the scenario conditions when the user terminal enters the custom area.

4. The step of determining whether the user terminal is located in the recognition area includes: When the object is displayed within the AR view, it is determined that the user terminal is located in the recognition area. The user terminal control method according to claim 1 .

5. The step of determining whether the user terminal is located in the recognition area includes: determining that the user terminal is located in the recognition area when at least one of the following conditions is satisfied: a condition that the distance between the object and the user terminal is equal to or less than a predetermined distance; a condition that the object is displayed in a specific area within the AR view; and a condition that the object is displayed in the AR view at a rate equal to or greater than a predetermined rate; The user terminal control method according to claim 1 .

6. the recognition area is a predefined area within the custom area, The step of determining whether the user terminal is located in the recognition area includes: When the user terminal is located in the predefined recognition area, the user terminal is determined to be located in the recognition area. The user terminal control method according to claim 1 .

7. The user terminal control method according to claim 6, further comprising the step of controlling settings of the user terminal, including execution of a module for recognizing the object, when the user terminal enters the recognition area.

8. the step of displaying the VR content includes displaying the VR content while changing a viewpoint in response to a change in the viewpoint of the camera; The user terminal control method according to claim 1 , further comprising the step of terminating the display of the VR content when the user terminal is no longer located in the recognition area.

9. The step of displaying the VR content includes displaying the VR content on a full screen of the AR view, The VR content depicts a virtual space around the object. The user terminal control method according to claim 1 .

10. The virtual space is expressed as a virtual spherical space centered on the object. The user terminal control method according to claim 9.

11. The virtual space is expressed as a virtual three-dimensional space defined by reflecting the geometry of the surrounding space in which the object is located. The user terminal control method according to claim 9.

12. the scenario condition is set so that, when the user terminal is located in the custom area, VR content determined based on another custom area into which the user terminal has entered before the user terminal has entered the custom area is displayed by the AR view as the VR content; The step of displaying the VR content includes: displaying, through the AR view, VR content determined based on the other custom region; The user terminal control method according to claim 2.

13. the scenario conditions are set so that, when the user terminal is located in the custom area, VR content determined based on points passed by the user terminal is displayed by the AR view as the VR content; The step of displaying the VR content includes: Displaying VR content determined based on points traveled by the user terminal using the AR view. The user terminal control method according to claim 2.

14. A plurality of points are defined inside or outside the custom area, The step of displaying the VR content includes: Displaying different VR content by the AR view when the user terminal passes through a first point among the plurality of points and when the user terminal passes through a second point among the plurality of points. The user terminal control method according to claim 13.

15. A plurality of custom areas are set in the space, The scenario conditions include at least one of: i) a first scenario condition indicating whether the user terminal has passed through a predetermined custom area other than the custom area; and ii) a second scenario condition indicating whether the number of user terminals located in the custom area falls within a predetermined range; The scenario conditions are set to provide the user terminal with different services or information associated with the space based on the service elements and / or control the user terminal to be configured differently depending on whether the scenario conditions are met; The step of displaying the VR content includes: Displaying the VR content using the AR view only when the scenario conditions are satisfied, or displaying different VR content when the scenario conditions are satisfied and when the scenario conditions are not satisfied. The user terminal control method according to claim 2.

16. receiving a route guidance request to a destination at the user terminal; providing route guidance to the user terminal according to the route guidance request; Further comprising: the step of determining whether the user terminal is located in the custom area, the step of determining whether the user terminal is located in the recognition area, the step of recognizing the object, and the step of displaying the VR content are executed when the user terminal moves to the destination according to the route guidance; The user terminal control method according to claim 1 .

17. the scenario condition is set so that VR content determined based on the number of times the user terminal enters the custom area is displayed by the AR view; The step of displaying the VR content includes: displaying, through the AR view, VR content determined based on the number of times the user terminal has entered the custom area; The user terminal control method according to claim 2.

18. A computer system realizing a user terminal or a server, at least one processor configured to execute computer-readable instructions contained in a memory; The at least one processor Determine whether the user terminal is located in a preset custom area in space based on the location of the user terminal; determining whether the user terminal is located in a recognition area in which the user terminal can recognize an object in the custom area; When the user terminal is located in the recognition area, the object is recognized from an AR view including an image captured by a camera of the user terminal; A computer system that displays, via the AR view, virtual reality content that is pre-set in association with the recognized object.

Images