Electronic device for providing augmented reality user interface and operating method thereof

Abstract

Various embodiments of the present disclosure relate to an electronic device and an operation method thereof for providing a user interface, the user interface being used to change a display mode associated with the display of a virtual reality object. The electronic device according to various embodiments of the present disclosure may include a display, a processor operably coupled to the display, and a memory operably coupled to the processor. The memory may store instructions that, when executed, cause the processor to control the electronic device: display at least one virtual object in a first display mode, in which the at least one virtual object is superimposed on a real object image corresponding to at least one real object, and the at least one virtual object is displayed at a fixed position on the screen regardless of the user's point of view (POV); receive a first input for changing the display mode of the at least one virtual object; and in response to receiving the first input, display the at least one virtual object in a second display mode, wherein the second position changes according to the user's POV.

Description

Technical Field

[0001] The present disclosure relates to an electronic device for providing a user interface for changing a display mode related to display of a virtual reality object and an operating method thereof. Background Art

[0002] Portable electronic devices such as smartphones not only provide users with basic services such as calls, text messages, etc., but also provide users with various types of services such as financial services (e.g., purchasing goods), entertainment services (e.g., games), etc. in various ways (e.g., virtual reality (VR) solutions, augmented reality (AR) solutions, etc.).

[0003] In addition, with the continuous development of wearable technology, head mounted display (HMD) type electronic devices are gradually popularized. Wearable devices such as HMD type devices use the above-mentioned VR and AR schemes to provide users with amazing experiences that are not easy to encounter in reality.

[0004] By superimposing a virtual object on an image (hereinafter referred to as a real object image) corresponding to a real object (e.g., a building sign, a road, a street tree, etc.), the virtual object can be provided to a user using a device (e.g., an HMD device) capable of displaying a virtual object image on a screen.

[0005] When an electronic device provides a virtual object, the electronic device may display the virtual object image at a fixed position on the screen. For example, if a user wears an HMD device and rides a bicycle, a virtual object indicating the current movement speed, accumulated distance, etc. may be provided to the user by superimposing the virtual object on the real object image. However, in this case, since the virtual object image displayed on the screen is displayed at a fixed position on the screen regardless of the user's head movement, the user may feel inconvenienced when interacting with the virtual object provided on the screen because inconvenient and complicated operations are required instead of simple operations (e.g., head movements).

[0006] In addition, when the electronic device provides a virtual object, the electronic device may display the virtual object image at a variable position on the screen. However, in this case, the position of the virtual object image displayed on the screen changes according to the change of the user's point of view (POV) or viewing area. Therefore, in many cases, the display position of the virtual object image may change unintentionally, making it difficult to continuously recognize the information.

[0007] Therefore, there is a need for a method in which a display mode for displaying a virtual object image is switched based on a user's input to provide a virtual object in the switched display mode. Summary of the invention

[0008] Solution to the problem

[0009] An embodiment of the present disclosure provides an electronic device capable of switching a display mode for displaying a virtual object image based on various types of user inputs and capable of providing a virtual object in the switched display mode, and an operating method thereof.

[0010] According to various exemplary embodiments of the present disclosure, an electronic device may include a display, a processor operably coupled to the display, and a memory operably coupled to the processor. The memory may store instructions that, when executed, cause the processor to control the electronic device: display at least one virtual object in a first display mode, in which the at least one virtual object is superimposed on a real object image corresponding to at least one real object, and the at least one virtual object is displayed at a fixed position on the screen regardless of the user's point of view (POV); receive a first input for changing the display mode of the at least one virtual object; and in response to receiving the first input, display the at least one virtual object in a second display mode, in which the second position changes according to the user's POV.

[0011] According to various example embodiments of the present disclosure, a method for operating an electronic device may include: displaying at least one virtual object in a first display mode, in which the at least one virtual object is superimposed on a real object image corresponding to at least one real object, and the at least one virtual object is displayed at a fixed position on the screen regardless of the user's POV; receiving a first input for changing the display mode of the at least one virtual object; and in response to receiving the first input, displaying the at least one virtual object in a second display mode, in which the screen display position changes according to the user's POV. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 is a block diagram illustrating an electronic device in a network environment according to various embodiments;

[0014] Figure 2A and Figure 2B is a diagram illustrating an example relationship between a user, an electronic device, and a virtual object provided via the electronic device according to various embodiments of the present disclosure;

[0015] Figure 3A and Figure 3Bis a diagram illustrating an example first display mode according to various embodiments of the present disclosure;

[0016] Figure 4A and Figure 4B is a diagram illustrating an example second display mode according to various embodiments of the present disclosure;

[0017] Figure 5 is a block diagram illustrating an example electronic device according to various embodiments of the present disclosure;

[0018] Figure 6 is a flowchart illustrating an example operation of an electronic device for changing a display mode based on a user input according to various embodiments of the present disclosure;

[0019] Fig. 7A , Figure 7B , Figure 7C and Fig.7D is a diagram showing an example of changing a display mode according to various embodiments of the present disclosure;

[0020] Fig. 8A , Figure 8B , Figure 8C and Fig.8D is a diagram illustrating an example of a first user input for changing a display mode according to various embodiments of the present disclosure;

[0021] Fig. 9 is a flowchart illustrating an example operation of an electronic device for displaying at least one virtual object in a second display mode according to various embodiments of the present disclosure;

[0022] Fig.10 is a flowchart illustrating an example operation of an electronic device for displaying at least one virtual object according to various embodiments of the present disclosure;

[0023] Fig.11 is a diagram showing an example of one first screen and at least one second screen according to various embodiments of the present disclosure;

[0024] Fig.12 is a flowchart illustrating an example operation of an electronic device for executing a function corresponding to a selected virtual object based on a user input for selecting a virtual object according to various embodiments of the present disclosure;

[0025] Fig.13 is a flowchart illustrating example operations of an electronic device for determining real-space coordinates corresponding to at least one virtual object according to various embodiments of the present disclosure;

[0026] Fig.14A , Fig. 14B , Fig. 14C and Fig.14D is a diagram illustrating an example operation of an electronic device for determining real-space coordinates corresponding to at least one virtual object based on a user's gesture according to various embodiments of the present disclosure;

[0027] Fig.15 is a flowchart illustrating example operations of an electronic device for determining real-space coordinates corresponding to at least one virtual object according to various embodiments of the present disclosure;

[0028] Fig.16A , Fig. 16B , Fig. 16C and Fig.16D is a diagram illustrating an example operation of an electronic device for determining real space coordinates corresponding to at least one virtual object based on a user's voice input according to various embodiments of the present disclosure;

[0029] Fig.17 is a flowchart illustrating example operations of an electronic device for determining real-space coordinates corresponding to at least one virtual object according to various embodiments of the present disclosure;

[0030] Fig.18A , Fig.18B and Fig.18C is a diagram illustrating an example operation of an electronic device for determining real-space coordinates corresponding to at least one virtual object based on a user's gaze input according to various embodiments of the present disclosure;

[0031] Fig.19 is a flowchart illustrating an example operation of an electronic device receiving a second user input for selecting at least one virtual object according to various embodiments of the present disclosure;

[0032] Fig. 20A and Fig. 20B is a diagram illustrating an example second user input for selecting at least one virtual object according to various embodiments of the present disclosure;

[0033] Fig.21A and Fig.21B is a diagram illustrating an example operation of an electronic device for changing display positions of multiple screens according to various embodiments of the present disclosure;

[0034] Fig.22A and Fig. 22B is a diagram illustrating an example operation of an electronic device for executing a function corresponding to a selected virtual object according to various embodiments of the present disclosure;

[0035] Fig.23A , Fig. 23B , Fig.23C and Fig.23Dis a diagram illustrating an example operation of an electronic device for executing a function corresponding to a selected virtual object according to various embodiments of the present disclosure;

[0036] Fig.24A , Fig. 24B and Fig.24C is a diagram showing an example in which the electronic device according to various embodiments of the present disclosure is a mobile portable terminal;

[0037] Fig.25 is a flowchart illustrating an example operation of an electronic device for combining data related to a virtual object previously displayed in a first display mode and storing the data by converting it into a form suitable for a space according to various embodiments of the present disclosure;

[0038] Fig.26A , Fig.26B , Fig.26C , Fig.26D , Fig.26E , Fig.26F and Figure 26G is a diagram illustrating an example operation of an electronic device for combining data related to a virtual object previously displayed in a first display mode and storing the data by converting it into a form suitable for a space according to various embodiments of the present disclosure;

[0039] Fig. 27 is a flowchart illustrating an example operation of an electronic device for storing at least one virtual object by mapping the at least one virtual object to specific real space coordinates according to various embodiments of the present disclosure;

[0040] Fig.28A , Fig.28B and Fig.28C is a diagram illustrating an example operation of an electronic device for storing at least one virtual object by mapping the at least one virtual object to specific real space coordinates according to various embodiments of the present disclosure;

[0041] Fig.29A , Fig.29B , Fig.29C , Fig.29D and Fig.29E is a diagram illustrating an example user input for returning from a second display mode to a first display mode according to various embodiments of the present disclosure; and

[0042] Fig.30 is a flowchart illustrating an example operation of an electronic device for changing a display mode based on a user input according to various embodiments of the present disclosure. DETAILED DESCRIPTION

[0043] Figure 11 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Figure 1 , the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit.For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented to be embedded in the display device 160 (eg, a display).

[0044] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to an example embodiment, as at least part of the data processing or calculation, the processor 120 may load a command or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a specified function. The auxiliary processor 123 may be implemented separately from the main processor 121 , or as part of the main processor 121 .

[0045] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 (not the main processor 121), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123.

[0046] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0047] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0048] The input device 150 may receive commands or data from outside the electronic device 101 (eg, a user) to be used by other components of the electronic device 101 (eg, the processor 120). The input 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus).

[0049] The sound output device 155 can output a sound signal to the outside of the electronic device 101. The sound output 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as playing multimedia or playing records, and the receiver may be used for incoming calls. Depending on the embodiment, the receiver may be implemented as a separate part from the speaker, or as part of the speaker.

[0050] The display device 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.

[0051] The audio module 170 may convert sound into an electrical signal, and vice versa. According to an embodiment, the audio module 170 may obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) connected to the electronic device 101 or wirelessly connected.

[0052] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0053] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0054] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0055] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0056] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0057] The power management module 188 may manage power supply to the electronic device 101. According to example embodiments, the power management module 188 may be implemented as, for example, at least a portion of a power management integrated circuit (PMIC).

[0058] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0059] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0060] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiating element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a PCB). According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as a part of the antenna module 197.

[0061] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0062] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 and the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute the requested at least part of the function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used.

[0063] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0064] It should be understood that the various exemplary embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but rather include various changes, equivalents, or alternatives to the corresponding embodiments.

[0065] For the description of the drawings, like reference numerals may be used to refer to like or related elements. It will be understood that nouns in the singular form corresponding to a term may include one or more items unless the relevant context clearly indicates otherwise.

[0066] As used herein, each of phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases.

[0067] As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish a corresponding component from another component and do not limit the components in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as "combined with another element (e.g., a second element)", "combined to another element (e.g., a second element)", "connected to another element (e.g., a second element)" or "connected to another element (e.g., a second element)" when the terms "operably" or "communicatively" are used or when the terms "operably" or "communicatively" are not used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0068] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0069] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, a "non-transitory" storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being semi-permanently stored in a storage medium and data being temporarily stored in a storage medium.

[0070] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be published in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be published online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM), or may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a manufacturer's server, an application store's server, or a memory of a forwarding server).

[0071] According to various embodiments, each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities. According to various embodiments, one or more components in the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as a corresponding component in the multiple components before integration. According to various embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operations may be run or omitted in different orders, or one or more other operations may be added.

[0072] Figure 2A and Figure 2B is a diagram illustrating an example relationship between a user, an electronic device, and a virtual object provided via the electronic device according to various embodiments of the present disclosure.

[0073] Reference Figure 2A , discloses an electronic device 200. In an embodiment, the electronic device 200 may correspond to Figure 1 For example, the electronic device 200 may be a portable electronic device (e.g., a glasses-type wearable device, a mobile terminal such as a smart phone, etc.) capable of providing a user with an interface related to a virtual object or a virtual environment. However, it should be understood that the present disclosure is not limited thereto.

[0074] According to an embodiment, the electronic device 200 may display a virtual object in the form of an image or text on the screen.

[0075] For example, the electronic device 200 may use an optical perspective scheme so that a virtual object image superimposed on a real object image may be provided to the user. The real object image visible to the user together with the virtual object image may vary according to the user's point of view (POV) 205 .

[0076] As another example, the electronic device 200 may use a video see-through solution so that a virtual object image superimposed on a real object image may be provided to the user. In this case, the virtual object image may be provided by a camera (not shown) of the electronic device 200, for example Figure 5The electronic device 200 may obtain a real object image by using a camera module 560 of the user, and the electronic device 200 may display the real object image on the screen. For example, the electronic device 200 may display a real object image corresponding to a real object existing in the real world 210 on the screen. The real object image displayed on the screen of the electronic device 200 may change according to the user's POV 205.

[0077] According to an embodiment, the electronic device 200 may display a virtual object image corresponding to a virtual element 220 or a virtual object (not existing in the real world but implemented as if it exists in the real world). By interacting between virtual elements or virtual objects displayed on the screen, the user can obtain a unique experience that is not easily obtained in the real world.

[0078] Reference Figure 2B , the difference between the point of view (POV) and the field of view (FOV) will be described below according to various embodiments of the present disclosure.

[0079] In an embodiment, POV may imply a three-dimensional space corresponding to the user's field of view or field of view range, and FOV may imply a two-dimensional area image on a screen or display of the electronic device 200 that provides a virtual object image to the user.

[0080] In an embodiment, the FOV may be determined according to the user's POV. For example, when the user's field of view corresponds to the first POV 260, the user may recognize the real object 250 included in the first POV 260 from the first FOV 265 (e.g., the object 255 corresponding to the real object 250 may be provided in the FOV 265). For another example, if the user's POV changes from the first POV 260 to the second POV 270 (e.g., if the user turns the user's head to the left), the user may not recognize the real object 250 from the second FOV 275 because the real object 250 is no longer included in the user's POV.

[0081] Figure 3A and Figure 3B is a diagram illustrating an example first display mode according to various embodiments of the present disclosure.

[0082] In an embodiment, the first display mode may be a display mode that displays at least one virtual object at a fixed position of the screen regardless of the user's POV. Figure 3A and Figure 3B The first display mode is described. Figure 3A and Figure 3B Any of them may be a screen of an electronic device (not shown, such as the electronic device 200 of FIG. 2 ) or a display of the electronic device 200 (eg, Figure 5Figure 5 is a diagram of a screen displayed on a display 520.

[0083] Reference Figure 3A and Figure 3B , the electronic device 200 may display at least one virtual object (e.g., virtual object A 310) in the first display mode. The first display mode may be a display mode in which the position at which the at least one virtual object is displayed is determined independently of (independently of) the user's POV. Therefore, even if the user's POV changes, the position on the screen at which the at least one virtual object is displayed may not change. When the user is looking at the kitchen in front ( Figure 3A ), the virtual object A 310 may be displayed at the first position 320 of the screen. Even if the user turns the user's head to the left ( Figure 3B ), the virtual object A 310 may also be continuously displayed at the first position 320 of the screen.

[0084] In an embodiment, the electronic device 200 may display a virtual object (eg, cursor 330 ) corresponding to the user's field of view.

[0085] In an embodiment, the first display mode may be effective when the user repeats the same action, in the sense that information may be persistently conveyed at a fixed location without requiring the display of virtual objects to reflect the user's movements.

[0086] Although not shown, when the user rides a bicycle, the electronic device 200 can display at least one object indicating at least one virtual object (e.g., an exercise path, cumulative exercise time, cumulative exercise distance, current movement speed, average movement speed, etc.) in the first display mode.

[0087] According to an embodiment, the virtual object displayed by the electronic device 200 in the first display mode may be a virtual object (or a virtual reality user interface) that can interact with the user. Although not shown, the electronic device 200 may display a virtual reality user interface including a plurality of virtual icons for executing a plurality of applications at a fixed position (e.g., the center of the screen).

[0088] In an embodiment, the first display mode may be referred to as a device-oriented display mode or a screen-anchored display mode.

[0089] Figure 4A and Figure 4B is a diagram illustrating an example second display mode according to various embodiments of the present disclosure.

[0090] In an embodiment, the second display mode may be a display mode for displaying at least one virtual object at a position on the screen that varies according to the user's POV. Figure 4A and Figure 4BDescribe the second display mode. Figure 4A and Figure 4B Any of them may be a screen of an electronic device (not shown, such as the electronic device 200 of FIG. 2 ) or a display of the electronic device 200 (eg, Figure 5 Figure 5 is a diagram of a screen displayed on the display 520).

[0091] Reference Figure 4A , the electronic device 200 may display at least one virtual object (e.g., virtual object B 420) in the second display mode. In the second display mode, the at least one virtual object may be mapped to a specified coordinate on the real space (or a specific real object on the real space). Figure 4A , the virtual object B 420 for providing TV program information may be mapped to the real space coordinates (three-dimensional coordinates with a specified position as the origin, for example, (x1, y1, z1)) where the real object TV 410 is located. Figure 4B , a virtual object C 440 for providing specific information about a real object (e.g., a beer glass 430) may be mapped to the beer glass 430. In this case, even if the user's POV does not change, when the position of the beer glass 430 changes, the position where the virtual object C 440 is displayed on the screen may also change.

[0092] In an embodiment, the electronic device 200 may change the display position of at least one virtual object on the screen in the second display mode according to the change of the user's POV. For example, if the coordinates mapped to the real space of a specific virtual object are included in the user's POV, the electronic device 200 may display the specific virtual object on the screen. When a user movement (e.g., head movement) that causes a change in the user's POV is detected, the display position of the specific virtual object may be changed, or whether the specific virtual object is displayed may be changed.

[0093] In an embodiment, the electronic device 200 may display a virtual object (eg, cursor 450 ) corresponding to the user's field of view.

[0094] In an embodiment, the second display mode may be referred to as a spatially oriented display mode or a spatially anchored display mode.

[0095] Figure 5 is a block diagram illustrating an example electronic device according to various embodiments of the present disclosure.

[0096] In an embodiment, the electronic device 200 may include a processor (e.g., including a processing circuit) 510, a display 520, a sensor module 530, a communication interface (e.g., including a communication circuit) 540, a user interface (e.g., including a user interface circuit) 550, and a memory 570. In an embodiment, some of the components (e.g., a camera module (e.g., including a camera) 560) may not be included in the electronic device 200.

[0097] In an embodiment, the processor 510 may include various processing circuits and may be electrically coupled and / or operably coupled to the display 520, the sensor module 530, the communication interface 540, the user interface 550, the camera module 560, and the memory 570. The processor 510 may provide overall control to the electronic device 200 by controlling the operation of the display 520, the sensor module 530, the communication interface 540, the user interface 550, the camera module 560, and the memory 570. The processor 510 may process data or information received from the display 520, the sensor module 530, the communication interface 540, the user interface 550, the camera module 560, and the memory 570 (e.g., sensor information received from the sensor module 530), and may generate a control command (or output data) based on the result of the processing. For example, the processor 510 may generate screen data to be output to the display 520 via the user interface 550 in response to receiving a first user input for changing the display mode of at least one object (e.g., from a first display mode to a second display mode).

[0098] In an embodiment, the display 520 may display at least one content. The content displayed on the display 520 may include at least one of a real object image and / or a virtual object image.

[0099] In an embodiment, the real object image may be an image obtained by the camera module 560, and the virtual object image may be an image generated by the processor 510. The display 520 may display at least one content in a specified mode under the control of the processor 510. For example, the display 520 may display at least one virtual object in a first display mode in which the object is displayed at a fixed position of the screen.

[0100] In an embodiment, the sensor module 520 may include at least one sensor capable of obtaining sensor information related to the electronic device (or user) at a specified time interval. Figure 1For example, the sensor module 530 may include at least one of the following: a first sensor that detects movement of the electronic device 200 associated with a user's head movement, a second sensor that detects a user's pupil movement, and a third sensor that detects a user's movement or gesture (e.g., a gesture associated with a movement of touching at least one virtual object displayed on the screen). For example, the sensor included in the sensor module 530 may correspond to at least one of the following: a motion sensor, a camera sensor (e.g., a pupil tracking camera, a motion recognition camera), an iris sensor, a gyroscope sensor, and a direction sensor.

[0101] In an embodiment, the communication interface 540 may include various communication circuits and perform communication with an external device (eg, Figure 1 The communication interface 540 may correspond to Figure 1 The communication module 190.

[0102] In an embodiment, the user interface 550 may include various user interface circuits and is an interface provided to the electronic device 200 to receive user input. For example, if the electronic device 200 is in an HMD device, the user interface 550 may include a button interface associated with at least one button provided to the surface of the electronic device 200.

[0103] In an embodiment, the camera module 560 can capture real object images. The camera module 560 can correspond to Figure 1 Camera module 180.

[0104] In an embodiment, the memory 570 may at least temporarily store commands defining the operation of the processor 510, or at least temporarily store data generated in at least one of the processor 510, the display 520, the sensor module 530, the communication interface or information 540, the user interface 550, and the camera module 560. For example, the memory 570 may store information about at least one virtual object. For example, the memory 570 may store identification information of at least one virtual object based on real space coordinates mapped to the at least one virtual object.

[0105] Figure 6 is a flowchart illustrating an example operation of an electronic device for changing a display mode based on a user input according to various embodiments of the present disclosure.

[0106] In an embodiment, Figure 6 The operations disclosed in the embodiment may be performed by an electronic device (eg, electronic device 200 of FIG. 2 ) or a processor (eg, Figure 5 510) to execute. In the following, for the convenience of description, Figure 6The operations disclosed in are performed by the processor 510, but those skilled in the art will appreciate that various operations may be performed by the electronic device 200 under the control of the processor 510.

[0107] In an embodiment, in operation 610, the processor 510 may display at least one virtual object in a first display mode in which the object is displayed at a fixed position of the screen regardless of the user's POV.

[0108] In an embodiment, the processor 510 may display at least one virtual object in the first display mode by superimposing the object on a real object image corresponding to at least one real object obtained through a camera module (eg, the camera module 180).

[0109] In another embodiment, the processor 510 may display at least one virtual object projected onto an optical combiner made of a semi-transparent mirror or prism in a first display mode.

[0110] Reference Fig. 7A , the processor 510 may display at least one virtual object (eg, a plurality of virtual icons 710) at a fixed position (eg, the first area 720) of the screen. Figure 7B , even if the user turns the user's head to the left to change the user's POV, the at least one virtual object 710 displayed in the first area 720 may be fixedly displayed in the first area 720 .

[0111] In an embodiment, when a virtual object is displayed in the first display mode, since at least one object is conveniently displayed at a fixed position of the screen regardless of the user's POV, information to be continuously updated can be identified regardless of the user's head movement. However, it may be inconvenient for the user to interact with the virtual object. For example, referring to Fig. 7A , when the user intends to interact with (e.g., select) the virtual object A 711 located at the upper left end, typically, the user turns the user's head to the left. However, in this case, due to the change in the user's POV, there is only a change in the real object image that appears to be superimposed on at least one virtual object, and the position of the virtual object A 711 does not change. Therefore, the user may not be able to interact with the desired object, or may only be able to interact with the desired object through a more complicated process (e.g., by performing a gesture that appears to touch the virtual object A 711). Therefore, when the electronic device 200 changes the display mode of at least one virtual object in response to receiving a user input for changing the display mode of the at least one virtual object, the user can interact with the desired object through a simpler process (e.g., head movement or gaze).

[0112] In an embodiment, at least one virtual object may correspond to at least one function. For example, at least one virtual object may correspond to at least one function, such as executing an application installed in the electronic device 200 or providing (or outputting) content (e.g., a song file or notification information).

[0113] In an embodiment, at operation 620, the processor 510 may receive a first user input for changing a display mode of at least one virtual object. In an embodiment, the processor 510 may receive various types of first user input. For example, referring to Fig. 8A , the processor 510 may receive a first user input associated with the user interface 550 (eg, a user input for pressing a button designated to change the display mode). Figure 8B , the processor 510 may use a third sensor to detect a user's motion or gesture to receive a first user input to perform a predetermined action (e.g., seemingly touching (or grabbing) at least one virtual object displayed on the screen). Figure 8C , the processor 510 may receive a voice input including a predetermined voice keyword (e.g., "Notification") as a first user input. In this case, a notification 810 may be displayed on the screen to indicate that the user's voice input is being received. For another example, referring to Fig.8D , the processor 510 may receive, as a first user input, a gaze input for gazing at a designated portion (or designated area, for example, an edge portion of the screen) for at least a designated time.

[0114] In an embodiment, at operation 630, the processor 510 may display at least one virtual object in the second display mode for displaying the object at a position that varies according to the user's POV. Figure 7C , the processor 510 may display at least one virtual object (eg, a plurality of icons 710) at a designated position (eg, the first area 720) of the screen. Fig.7D , if the user turns the user's head to the left to change the user's POV, at least one virtual object previously displayed in the first area 720 may be displayed in a second area 730 of the screen that is different from the first area 720 .

[0115] In an embodiment, the processor 510 may display at least one virtual object different from the at least one virtual object in the second display mode. For example, the processor 510 may display a new virtual object that was not previously displayed on the screen when the first user input is received in the second display mode.

[0116] Fig. 9is a flowchart illustrating example operations for displaying at least one virtual object in a second display mode using an electronic device according to various embodiments of the present disclosure.

[0117] Fig. 9 A flowchart can be, for example, Figure 6 Detailed flow chart of operation 630 of .

[0118] In an embodiment, the processor 510 may determine real space coordinates corresponding to at least one virtual object in operation 910. In an embodiment, the processor 510 may determine real space coordinates corresponding to one or more virtual objects displayed on the screen in the first display mode.

[0119] For example, for a virtual object that was not previously mapped to a specific real space coordinate, the processor 510 may newly generate the corresponding real space coordinate. For a virtual object that was previously mapped to a specific real space coordinate, the processor 510 may identify the mapped real space coordinate.

[0120] In an embodiment, the processor 510 may determine the real space coordinates corresponding to at least one virtual object by using the real space coordinates corresponding to the position of the user or the position of a part of the user's body (e.g., the position of the user's eyes) and the position where at least one virtual object is displayed within the screen. For example, the processor 510 may determine the real space coordinates corresponding to at least one virtual object as: the coordinates ((x0+x1) / 2, (y0+y1) / 2, (z0+z1) / 2) of the point obtained by internally dividing the real space coordinates (x0, y0, z0) corresponding to the user's position and the real space coordinates (x1, y1, z1) corresponding to the position where at least one virtual object is displayed within the screen at a specified ratio (e.g., 5:5) .

[0121] In an embodiment, when the processor 510 may determine the real space coordinates corresponding to the at least one virtual object, the time when the first user input is received may be referenced.

[0122] In an embodiment, the processor 510 may map at least one virtual object to the determined real space coordinates in operation 920. For example, the processor 510 may store identification information corresponding to the at least one virtual object in the memory 570 according to the determined real space coordinates.

[0123] In an embodiment, in operation 930, the processor 510 may display at least one virtual object based on the determined real space coordinates. Based on the determined real space coordinates, the processor 510 may display at least one virtual object to a display (e.g., display 520). The processor 510 may display the virtual object located at the determined real space coordinates to the display 520 as if it were a real object located at a specific real space coordinate (e.g., a building located at a specific latitude and longitude). For example, at least one virtual object may be displayed on the display 520 only when the determined real space coordinates are included in the user's POV. For example, if the user turns around, the at least one virtual object may no longer be displayed on the display 520. For another example, when the user turns the user's head to the left, the at least one virtual object may be displayed by further moving to the right on the screen.

[0124] Fig.10 is a flowchart illustrating example operations of an electronic device for displaying at least one virtual object according to various embodiments of the present disclosure.

[0125] Fig.10 A flowchart can be, for example, Fig. 9 Detailed flow chart of operation 930 of .

[0126] In an embodiment, in operation 1010, the processor 510 may display a first screen on the display 520. When a first user input is received in operation 620, the first screen may include at least one virtual object previously displayed in the first display mode. Fig.11 , upon receiving the first user input, the processor 510 may display a first screen 1110 including all virtual objects previously displayed on the display 520 in the first display mode.

[0127] In an embodiment, the processor 510 may display the first screen 1110 in the same size as a size obtained by reducing the size of the full screen 1150 displayed on the display 520 by a specified ratio. For example, if the size of the full screen displayed on the display 520 is a first size, the processor 510 may display the first screen 1110 in a second size that is the same as the size obtained by reducing the first size by a specified ratio.

[0128] In an embodiment, the processor 510 may display the first screen 1110 on the display 520 based on the range of the designated real space coordinates.

[0129] For example, the processor 510 can determine the range of real space coordinates corresponding to the first screen 1110 by using at least one real space coordinate corresponding to at least one virtual object included in the first screen 1110, and can display the first screen 1110 on the display 520 based on the determined real space coordinate range.

[0130] For another example, the processor 510 may determine a range of real space coordinates corresponding to the first screen 1110 using real space coordinates corresponding to the position of a body part of the user (e.g., coordinates corresponding to the user's eyes), real space coordinates corresponding to an edge or border of the screen, and a specified scale, and may display the first screen 1110 on the display 520 based on the determined range of real space coordinates.

[0131] In an embodiment, the processor 510 may display the first screen 1110 in a designated portion of the display 520. For example, the processor 510 may display the first screen in the center of the display 520.

[0132] In an embodiment, in operation 1020, the processor 510 may display at least a portion of one or more second screens 1112, 1114, 1116, and 1118 (hereinafter may be referred to as 1112 to 1118) near the first screen 1110. For example, referring to Fig.11 , the processor 510 can display at least a portion of the four second screens 1112 , 1114 , 1116 , and 1118 on the upper side, the lower side, the left side, and the right side of the first screen 1110 .

[0133] In an embodiment, similar to the first screen 1110, the processor 510 may display at least one of the second screens 1112 to 1118 on the display 520 based on a specified real space coordinate range. The processor 510 may determine the real space coordinate range corresponding to at least one of the second screens 1112 to 1118 using at least one of the following: the real space coordinate range corresponding to the first screen 1110, the positional relationship between the first screen 1110 and at least one of the second screens 1112 to 1118, and the interval between the first screen 1110 and at least one of the second screens 1112 to 1118, and the processor 510 may display at least one of the second screens 1112 to 1118 on the display 520 based on the determined real space coordinate range.

[0134] In an embodiment, each of at least one second screen among the second screens 1112 to 1118 may include at least one virtual object, and the at least one virtual object may correspond to at least one function. At least one virtual object included in one second screen (any second screen among the second screens 1112 to 1118) may share the same (or similar) attributes, characters or display forms. For example, at least one virtual object included in the second screen 1114 displayed on the upper side of the first screen may include at least one virtual object related to a notification (or providing a notification) generated within a specified time from the current moment. For another example, at least one virtual object included in the second screen 1118 displayed on the lower side of the first screen may include a virtual object for executing at least one application installed in the electronic device 200. For another example, at least one virtual object included in the second screen 1116 displayed on the left side of the first screen may be displayed in the form of a widget. For another example, at least one virtual object included in the second screen 1112 displayed on the right side of the first screen may include a virtual object related to a task manager.

[0135] In an embodiment, the arrangement of at least one of the second screens 1112 to 1118 may vary according to the user's configuration. For example, at least one virtual object included in the second screen 1112 displayed on the right side of the first screen may include at least one virtual object that is not related to the task manager but is related to a notification generated within a specified time from the current moment.

[0136] Users can Fig.10 An embodiment of the present invention expands the search area for virtual objects. In response to receiving a first user input for changing the display mode, the electronic device 200 may display at least one virtual object (the at least one virtual object is displayed on the screen in the first display mode) in the second display mode as described above. Since the second display mode is a display mode in which the display position of the virtual object changes according to the change of the user's POV, when the user turns the user's head (for example, to the left), the display position of the virtual object moves relatively to the right within the screen, and if it exceeds the right boundary of the screen, the virtual object is no longer displayed. In this way, if the display position of the virtual object or whether the virtual object is displayed changes with the change of the user's POV, the search area for the virtual object is not limited to the inside of the screen, but can also be extended to the outside of the screen. This is because the user can access virtual objects that are not displayed on the current screen by simply turning the user's head, because the virtual object is fixed to the mapped real space coordinates rather than to the coordinates within the screen.

[0137] Fig.12is a flowchart illustrating example operations of an electronic device for executing a function corresponding to a selected virtual object based on a user input for selecting the virtual object according to various embodiments of the present disclosure.

[0138] Fig.12 The operations disclosed in may be performed, for example, after operation 630 is performed.

[0139] In an embodiment, in operation 1210, the processor 510 may receive a second user input for selecting any one of at least one virtual object displayed on the screen. In an embodiment, the at least one virtual object displayed on the screen may be a virtual object previously displayed in the first display mode when the first user input is received, or may be a virtual object not previously displayed in the first mode when the first user input is received.

[0140] In an embodiment, similar to the first user input, the second user input for selecting any one of the virtual objects displayed on the screen may include, for example but not limited to, at least one of the following: Figure 5 The processor 510 may include an input related to a user interface 550 of the embodiment of the present invention, an input related to a user's movement or gesture, a voice input including a predetermined voice keyword, and a gaze input of gazing at a specified portion of the screen (or any one of the virtual objects). For example, the processor 510 may receive a second user input of gazing at any one of the virtual objects displayed on the screen for at least a specified time.

[0141] In an embodiment, the processor 510 may receive a second user input for selecting a virtual object previously displayed on the screen within a specified time from the current moment. For example, the processor 510 may receive a voice input for selecting a virtual object previously displayed on the screen within a specified time from the current moment.

[0142] In an embodiment, in operation 1220, the processor 510 may execute a function corresponding to the selected virtual object. For example, when a virtual object corresponding to a function of executing application A is selected, the processor 510 may execute application A, and may display an execution screen of application A on the display 520. For another example, when a virtual object corresponding to a function of providing notification A is selected, the content of notification A may be displayed on the display 520.

[0143] In an embodiment, the processor 510 may display at least one virtual object in the first display mode in operation 1230. For example, the processor 510 may change the display mode from the second display mode to the first display mode after executing a function corresponding to the selected virtual object.

[0144] Although not shown, the processor 510 may maintain the display mode to the second display mode after executing the function corresponding to the selected virtual object.

[0145] Fig.13 is a flowchart illustrating example operations of an electronic device for determining real-space coordinates corresponding to at least one virtual object according to various embodiments of the present disclosure.

[0146] Fig.13 A flowchart can be, for example, Fig. 9 Detailed flow chart of operation 910.

[0147] When the first user input is a gesture input for performing a predetermined action or an input related to the user interface 550, the Fig.13 For example, refer to Fig.14A The first user input may be a gesture input for performing a predetermined action (eg, an action of grabbing at least one virtual object displayed on the screen).

[0148] In an embodiment, in operation 1310, the processor 510 may determine the real space coordinates corresponding to at least one virtual object previously displayed on the display 520 in the first display mode based on the user's position and the user's POV. For example, if the real space coordinates corresponding to the user's position are (x0, y0, z0), and the real space coordinates corresponding to the position where the virtual object A was previously displayed in the screen in the first display mode are (x1, y1, z1), the real space coordinates corresponding to the virtual object A may be determined to be (xa, ya, za). (xa, ya, za) may be the coordinates of a point obtained by dividing (x0, y0, z0) and (x1, y1, z1) by a specified ratio.

[0149] In an embodiment, the processor 510 may use the real space coordinates corresponding to the determined at least one virtual object to determine a real space coordinate range corresponding to a first screen and at least one second screen. The processor 510 may use the determined real space coordinate range to display the first screen and the second screen in the second display mode. For example, referring to Fig. 14B , the processor 510 may display one first screen 1410 and four second screens 1412, 1414, 1416, and 1418 in the second display mode. For example, each of the four second screens 1412, 1414, 1416, and 1418 may correspond to a specified attribute, role, or specified display form related to at least one virtual object included in the screen.

[0150] In an embodiment, in operation 1320, the processor 510 may identify whether an additional user input for changing the display position is received. The additional user input for changing the display position may be a user input of the same type as the first user input, or may be a user input of a different type than the first user input. Fig. 14C , if the first user input is a gesture input for performing a predetermined action (e.g., an action of grabbing at least one virtual object displayed on the screen), the processor 510 may receive an additional gesture that looks like moving the first screen 1410 to the left. For another example, if the first user input is an input related to the user interface 550 (e.g., an input for pressing a button provided on the surface of the electronic device 200), the processor 510 may also receive an additional gesture that looks like moving the first screen 1410 to the left.

[0151] In an embodiment, the additional user input for changing the display position may be a user input for updating real space coordinates corresponding to the first screen 1410 and the second screens 1412 , 1414 , 1416 , and 1418 .

[0152] In an embodiment, if a user input for changing the display positions of the first screen 1410 and the second screens 1412, 1414, 1416, and 1418 is not received, the process may proceed to operation 920. The processor 510 may perform operation 920 using the real space coordinates (e.g., (xa, ya, za)) determined in operation 1310.

[0153] In an embodiment, at operation 1330, the processor 510 may update the real space coordinates of the at least one virtual object based on the additional user input. Fig.14D For example, upon receiving an additional user input that appears to move the first screen 1410 to the left, the real space coordinates corresponding to the specific virtual object may be updated from (xa, ya, za) to (xa-k, ya, za), so that the specific virtual object (or the first screen 1410 including the specific virtual object) is visible on the left side of the user's field of view rather than the front side. Therefore, the second screen 1412 located to the right of the first screen is visible on the front side of the user's field of view. Although only the x-axis coordinate is reduced by k here, this is only for illustrative purposes, and it is obvious to those skilled in the art that a specified value in one or more coordinates may be passed according to the configuration of the coordinate axis.

[0154] In an embodiment, the real space coordinates to be updated may be determined based on the attributes of the additional user input (e.g., based on the direction of the user gesture). For example, if the additional user input is a user gesture that appears to move the first screen 1410 to the left, the real space coordinates corresponding to the specific virtual object may be updated to (xa-k, ya, za), and if the additional user input is a user gesture that appears to move the first screen 1410 to the right, the real space coordinates corresponding to the specific virtual object may be updated to (xa+k, ya, za).

[0155] In an embodiment, the real space coordinates of the virtual object included in the first screen may be updated so that the specific second screen is arranged in the middle of the display 520. For example, if the additional user input is a user gesture that appears to move the first screen to the left, the real space coordinates corresponding to the virtual object included in at least one of the first screen 1410 and the second screens 1412 to 1418 may be updated so that the second screen 1412 arranged on the right side of the first screen 1410 is arranged in the middle of the screen. For another example, if the additional user input is a user gesture that appears to move the first screen 1410 to the right, the real space coordinates corresponding to the virtual object included in at least one of the first screen 1410 and the second screens 1412 to 1418 may be updated so that the second screen 1416 arranged on the left side of the first screen 1410 is arranged in the middle of the screen.

[0156] In an embodiment, the real space coordinates of the virtual object included in the first screen may be updated so that the first screen 1410 is arranged by enlarging or reducing the size thereof.

[0157] For example, if the additional user input is a user gesture that appears to pull the first screen 1410 in the direction of the user, the processor 510 may update the real-space coordinates corresponding to the virtual object included in the first screen 1410 so that the first screen 1410 is displayed by enlarging its size (for example, the processor 510 may determine a point obtained by dividing the real-space coordinates (xa, ya, za) corresponding to the virtual object included in the first screen 1410 and the real-space coordinates (x1, y1, z1) corresponding to the user position by a specified ratio as a new real-space coordinate corresponding to the virtual object).

[0158] For another example, if the additional user input is a user gesture that appears to push the first screen 1410 in a direction opposite to the user direction, the processor 510 may update the real-space coordinates corresponding to the virtual object included in the first screen, thereby displaying the first screen 1410 by reducing its size (for example, the processor 510 may determine the point obtained by internally dividing the real-space coordinates (xa, ya, za) corresponding to the virtual object included in the first screen by the real-space coordinates (x1, y1, z1) corresponding to the position where the virtual object A was previously displayed in the screen in the first display mode).

[0159] In an embodiment, the processor 510 may proceed to operation 920. For example, the processor 510 may perform operation 920 using the updated real space coordinates of the virtual object.

[0160] Fig.15 is a flowchart illustrating example operations of an electronic device for determining real-space coordinates corresponding to at least one virtual object according to various embodiments of the present disclosure.

[0161] Fig.15 A flowchart can be, for example, Fig. 9 Detailed flow chart of operation 910.

[0162] When the first user input is a voice input including a predetermined voice keyword, the Fig.15 The operations disclosed in . Fig.16A , the first user input may be a voice input including a predetermined voice keyword “Grab”.

[0163] In an embodiment, in operation 1510, the processor 510 may determine the real space coordinates corresponding to at least one virtual object previously displayed on the display 520 in the first display mode based on the user's position and the user's POV. For example, if the real space coordinates corresponding to the user's position are (x0, y0, z0), and the real space coordinates corresponding to the position where the virtual object A was previously displayed in the screen in the first display mode are (x1, y1, z1), the real space coordinates corresponding to the virtual object A may be determined to be (xa, ya, za). (xa, ya, za) may be the coordinates of a point obtained by dividing (x0, y0, z0) and (x1, y1, z1) by a specified ratio.

[0164] In an embodiment, the processor 510 may use the real space coordinates corresponding to the determined at least one virtual object to determine a real space coordinate range corresponding to a first screen and at least one second screen. The processor 510 may use the determined real space coordinate range to display the first screen and the second screen in the second display mode. For example, referring to Fig. 16B , the processor 510 can display one first screen 1610 and four second screens 1612, 1614, 1616 and 1618 in the second display mode.

[0165] In an embodiment, in operation 1520, the processor 510 may identify whether an additional voice input is received. If the additional voice input is received at the specified time, the processor 510 may perform operation 1530, and if the additional voice input is not received, the processor 510 may perform operation 920. Fig. 16C , the processor 510 can receive additional voice input such as "Task Manager" ("TaskManager").

[0166] In an embodiment, in operation 1530, the processor 510 may identify whether the content of the additional voice input is related to the change of the display position. For example, whether the additional voice input includes the voice keyword "task manager" indicating the second screen 1612 displayed on the right side of the first screen 1610. For another example, if the additional voice input includes a voice keyword indicating a direction (e.g., up, down, left, right), the processor 510 may determine that the content of the additional voice input is related to the change of the display position. If the content of the additional voice input is related to the change of the display position, the processor 510 may perform operation 1540, and if the content is not related to the change of the display position, the processor 510 may perform operation 920. For example, if the content of the additional voice input is not related to the change of the display position, the processor 510 may perform operation 920 using the real space coordinates determined in operation 1510.

[0167] In an embodiment, in operation 1540, the processor 510 may update the real space coordinates of the at least one virtual object based on the content of the additional voice input.

[0168] For example, when an additional voice input including the voice keyword “task manager” indicating the second screen 1612 displayed on the right side of the first screen 1610 is received, the real space coordinates corresponding to the specific virtual object may be updated from (xa, ya, za) to (xa-k, ya, za), so that the specific virtual object (or the first screen 1610 including the specific virtual object) is visible on the left side rather than the front side within the user's field of view. Fig.16D , the second screen 1612 located on the right side of the first screen 1610 is visible at the front side within the user's field of view.

[0169] Fig.17 is a flowchart illustrating example operations of an electronic device for determining real-space coordinates corresponding to at least one virtual object according to various embodiments of the present disclosure.

[0170] Fig.17 A flowchart can be, for example, Fig. 9 Detailed flow chart of operation 910 of .

[0171] When the first user input is a gaze input for gazing at a designated portion (or designated area) of the screen, the Fig.17 The operations disclosed in . Fig.18A , the first user input may be a gaze input for gazing at the right edge (or right edge area) of the screen for a specified time (eg, 1.5 seconds).

[0172] In an embodiment, in operation 1710, the processor 510 may determine the real space coordinates corresponding to at least one virtual object previously displayed on the display 520 in the first display mode based on the user's position and the user's POV. For example, if the real space coordinates corresponding to the user's position are (x0, y0, z0), and the real space coordinates corresponding to the position where the virtual object A was previously displayed in the screen in the first display mode are (x1, y1, z1), the real space coordinates corresponding to the virtual object A may be determined to be (xa, ya, za). (xa, ya, za) may be the coordinates of a point obtained by dividing (x0, y0, z0) and (x1, y1, z1) by a specified ratio.

[0173] In an embodiment, at operation 1720, the processor 510 may display a guide object. Fig.18B , the processor 510 may display a guide object 1820 for guiding the user to switch the display mode of at least one virtual object, or for notifying the user that the display mode is about to be switched. The processor 510 may display the guide object based on any one of the first display mode and the second display mode.

[0174] In an embodiment, at operation 1730, the processor 510 may identify whether the user's gaze on the edge region (or edge) is maintained for at least a set time. Fig.18B , after displaying the guide object 1820 , the processor 510 may identify whether the user's gaze on the right edge area 1830 associated with the first user input is maintained for at least a set time.

[0175] In an embodiment, if the user's gaze on the edge area is not maintained for at least a set time (No in operation 1730), the processor 510 may identify whether the guide object 1820 is selected in operation 1740. For example, the processor 510 may detect a user gesture for selecting the guide object 1820, or may identify whether the user's gaze on the guide object 1820 is maintained for at least a set time.

[0176] In an embodiment, if the guide object 1820 is not selected ("No" in operation 1740), the processor 510 may proceed to operation 610. For example, if the user's intention to change the display mode is not double-confirmed, the display mode of at least one virtual object may not be changed to the second display mode, but may remain in the first display mode.

[0177] According to an embodiment, if the guide object 1820 is selected (Yes in operation 1740), the processor 510 may proceed to operation 920. Since the user's intention to change the display mode is doubly confirmed, the processor 510 may perform operation 920 based on the real space coordinates determined in operation 1710.

[0178] In an embodiment, if the user's gaze on the edge area is maintained for at least a set time (yes in operation 1730), in operation 1750, the processor 510 may update the real space coordinates of at least one virtual object based on the position of the edge area. For example, if the user's gaze on the edge area is continuously maintained even if the guide object 1820 is displayed, it is determined that the user wants to see a different screen (e.g., the second screen 1612 of FIG. 16) that is not currently displayed on the display in the front, and the processor 510 may update the real space coordinates of at least one virtual object (or a virtual object included in a different screen that is not currently displayed on the display). For example, if the user's gaze on the right edge area 1830 is maintained for a set time, the processor 510 may update the real space coordinates of at least one virtual object so that at least one virtual object (or the first screen 1810 including at least one virtual object) is seen on the left side rather than the front side in the user's field of view, and the second screen 1812 located on the right side of the first screen 1810 among the screens 1812, 1814, 1816, 1818 is in the front of the user's field of view. For example, the processor 510 may update the real space coordinates of the at least one virtual object by subtracting a specific value from the x coordinate of the real space coordinates of the at least one virtual object (e.g., updating (xa, ya, za) to (xa-k, ya, za)). After updating the real space coordinates of the at least one virtual object, the processor 510 may perform operation 920. The processor 510 may perform operation 920 using the updated real space coordinates.

[0179] Fig.19 is a flowchart illustrating example operations of an electronic device receiving a second user input for selecting at least one virtual object according to various embodiments of the present disclosure.

[0180] Fig.19 A flowchart can be, for example, Fig.12 Detailed flow chart of operation 1210.

[0181] Fig.19 The operations disclosed in may be performed in a state where at least a portion of at least one second screen is displayed on and near the first screen.

[0182] In an embodiment, the processor 510 may detect a user's head movement in operation 1910. The processor 510 may detect the user's head movement (e.g., movement degree or movement direction) using sensor information of a first sensor that detects movement of the electronic device 200 corresponding to the user's head movement.

[0183] In an embodiment, in operation 1920, the processor 510 may change the display position of the first screen and at least one second screen. The processor 510 may change the display position of the first screen and at least one second screen based on the detected user head movement. For example, if the user moves the head to the right by at least a specified value, the processor 510 may move the first screen located in the middle of the screen to the left side of the screen (depending on the degree of head movement, at least a portion of the first screen may not be displayed on the screen), and may move the second screen located on the right side of the screen to the middle of the screen. In operation 1920, the real space coordinates corresponding to the first screen and at least one second screen may not be changed. For example, the change in the display position in operation 1920 may represent a change in the position of at least one screen corresponding to the fixed real space coordinates displayed on the screen according to the user's head movement.

[0184] Although not shown, the real space coordinates corresponding to the first screen and the at least one second screen may be changed in operation 1920. For example, if the degree of movement (e.g., the rotation angle) of the user turning the head to the left is greater than or equal to a specified value and / or if the movement speed is greater than or equal to a specified value, the real space coordinates corresponding to the first screen and the at least one second screen may be changed so that the second screen located to the left of the first screen is located in the middle of the screen.

[0185] In an embodiment, in operation 1930, the processor 510 may identify whether a second user input is detected to select any one virtual object from the virtual objects included in any one screen. For example, the processor 510 may identify whether a gaze input for gazing at any one virtual object for at least a specified time is detected. The processor 510 may use sensor information of a second sensor (e.g., a pupil tracking camera) for detecting movement of a user's pupil to detect a gaze input for gazing at any one virtual object for at least a specified time as the second user input. When a gaze input for gazing at any one virtual object for at least a specified time is detected, the processor 510 may provide a graphic effect to the virtual object. For example, referring to Fig. 20A, the processor 510 may not provide additional graphic effects when detecting a gaze input of gazing at the virtual object A 2010 for less than or equal to a specified time. Fig. 20B , upon detecting a gaze input for gazing at the virtual object B 2020 for at least a specified time, the processor 510 may provide an additional graphic effect (e.g., addition of a specified color, or a highlight effect such as a shadow pattern) to a designated area 2025 including the virtual object B 2020.

[0186] Upon detecting a second user input for selecting any one virtual object, the processor 510 may perform operation 1220. For example, the processor 510 may execute a function corresponding to the selected virtual object.

[0187] When the second user input for selecting any virtual object is not detected, the processor 510 may perform operation 1940. For example, the processor 510 may identify whether the display positions of the first screen and the at least one second screen need to be changed. In an embodiment, the processor 510 may identify whether the real space coordinates corresponding to the first screen and the at least one second screen need to be changed.

[0188] For example, when user input for gazing at a specific area of ​​the screen for at least a specified time is detected, if the specific area of ​​the screen is one of four edge areas corresponding to four directions (up, down, left, and right), and if there is another screen in the specific area of ​​the screen, the processor 510 can determine that the display positions of the first screen and at least one second screen need to be changed.

[0189] For example, when user input of gazing at a specific area of ​​the screen for at least a specified time is detected, if the virtual object is not included in the specific area of ​​the screen and if the virtual object is located within a specified range of distance from the specific area of ​​the screen, the processor 510 can determine that the display position of the first screen and at least one second screen needs to be changed.

[0190] In an embodiment, in operation 1950, the processor 510 may change display positions of the first screen and at least one second screen.

[0191] For example, refer to Fig.21A , when a user input for gazing at a specific area of ​​the screen for at least a specified time is detected, if the specific area of ​​the screen is a right edge area corresponding to the right direction and a different screen (e.g., the second screen 2112) also exists on the right side of the right edge area, the processor 510 may change the real space coordinates corresponding to the first screen 2110 and at least one of the second screens 2112, 2114, 2116, and 2118 so that the second screen 2112 is located in front of the user's field of view.

[0192] For example, refer to Fig.21B , the real space coordinates corresponding to the first screen and the at least one second screen may be changed so that the first screen and the at least one second screen move in parallel in a left direction (perpendicular to the user's line of sight in the real space). The processor 510 may change the display positions of the first screen and the at least one second screen based on the changed real space coordinates.

[0193] For another example, when a user input of looking at a specific area of ​​the screen for at least a specified time is detected, if the virtual object is not included in the specific area of ​​the screen and if the virtual object A (not shown) is located within a specified distance from the specific area of ​​the screen, the processor 510 may change the real space coordinates corresponding to the first screen 2110, thereby displaying the screen (e.g., the first screen 2110) including the virtual object A by enlarging its size within the user's field of view. For example, the real space coordinates may be changed so that the first screen 2110 moves parallel to the user direction (or in a direction opposite to the user direction) in the real space. In this case, not only the real space coordinates corresponding to the first screen 2110 but also the real space coordinates corresponding to at least one of the second screens 2112, 2114, 2116, and 2118 may be changed. The display position within the first screen may be changed based on the changed real space coordinates.

[0194] Therefore, the processor 510 can provide the user with an interesting experience of moving a virtual object using only eye movement by changing the real space coordinates corresponding to at least one virtual object (or a screen including at least one virtual object).

[0195] In an embodiment, the processor 510 may perform operation 1910 after operation 1950. In another embodiment, the processor 510 may proceed to operation 1930 after operation 1950.

[0196] Fig.22A and Fig. 22B is a diagram illustrating an example operation of an electronic device for executing a function corresponding to a selected virtual object according to various embodiments of the present disclosure.

[0197] Reference Fig.22A , a first screen 2210 and a portion of four second screens 2212, 2214, 2216, and 2218 located near the first screen 2210 can be displayed on the screen based on the second display mode. The first screen 2210 may include a plurality of virtual objects 2205 (e.g., a plurality of virtual icons). In an embodiment, the plurality of virtual objects 2205 may be virtual objects previously displayed on the screen when the first user input is received. In another embodiment, the plurality of virtual objects 2205 may be virtual objects that were not previously displayed on the screen when the first user input was received.

[0198] In an embodiment, the plurality of virtual objects 2205 included in the first screen 2210 may correspond to respective different functions. For example, the virtual object A 2202 included in the first screen 2210 may correspond to a point of interest (POI) browsing function. The POI browsing function may be a function for identifying (or searching for) a point of interest on a real space at a current location (or associated with the current location or the user's POV), and for displaying a virtual object (including information corresponding to the point of interest on the identified real space) on a screen. Fig.22A , the processor 510 may receive a user gesture for selecting a virtual object A 2202 corresponding to a POI browsing function from among a plurality of objects 2205 .

[0199] In an embodiment, points of interest in real space at the current location (or associated with the current location or the user's POV) may be identified. Fig. 22B , the processor 510 may display a virtual object (eg, virtual object B 2220) on the screen, the virtual object including information corresponding to a point of interest recognized on the real space (eg, business information of a store).

[0200] In an embodiment, when virtual object A 2202 is selected, the processor 510 may limit the display of the first screen 2210 and the four second screens 2212, 2214, 2216, and 2218 on the screen, and may display the selected virtual object A 2202 and unselected virtual objects differently while maintaining the display of multiple virtual objects 2205 previously included in the first screen 2210.

[0201] In an embodiment, the processor 510 may use different display modes according to the properties of the virtual object. For example, the processor 510 may display a plurality of virtual objects 2205 previously included in the first screen 2210 based on the first display mode. For example, even if the user's POV changes as the user walks forward, the plurality of virtual objects 2205 previously included in the first screen 2210 may be displayed at a fixed position of the screen. In addition, the processor 510 may display a virtual object (e.g., virtual object B 2220) based on a second display mode, which includes information corresponding to the identified point of interest in the real space. For example, if the user's POV changes as the user walks forward, the processor 510 may no longer display the virtual object previously displayed on the screen on the screen, or may redisplay the virtual object previously not displayed on the screen on the screen.

[0202] In an embodiment, the processor 510 may receive a user input for selecting a virtual object that includes information corresponding to a point of interest identified in real space (e.g., business information of a store). For example, upon receiving a user input of gazing at a virtual object corresponding to a specific store for at least a specified time, the processor 510 may provide a user interface for receiving a user's comment on the specific store while providing detailed information about the specific store. The processor 510 may store the received user comment based on the location information of the specific store.

[0203] Fig.23A , Fig. 23B , Fig.23C and Fig.23D is a diagram illustrating an example operation of an electronic device for selecting any one virtual object and performing a function corresponding to the selected virtual object according to various embodiments of the present disclosure.

[0204] Reference Fig.23A , the processor 510 may display at least a portion of the first screen 2310 and four second screens 2312, 2314, 2316, 2318 (hereinafter may be referred to as second screens 2312 to 2318). Each of at least a portion of the four second screens 2312 to 2318 may include information (e.g., text information) indicating at least one of the attributes, characters, and display forms of the virtual object (which is included in the corresponding screen). The four second screens may correspond to a specified real space coordinate range.

[0205] The processor 510 may receive a user input for displaying the second screen.

[0206] In an embodiment, the processor 510 may receive a user input that is independent of a change in the real space coordinates corresponding to the second screen as a user input for displaying a second screen. For example, the processor 510 may receive a user's head motion (e.g., a user's motion of turning the user's head to the left).

[0207] In another embodiment, the processor 510 may receive a user input related to a change in the real space coordinates corresponding to the second screen as a user input for displaying a second screen. For example, the processor 510 may receive a user input including a specified voice keyword "widget". The processor 510 may change the real space coordinates of the first screen 2310 and at least one additional screen 2312 to 2318 so that the second screen 2316 corresponding to the specified voice keyword "widget" is visible on the front side within the user's field of view.

[0208] In an embodiment, the processor 510 may receive a user input for selecting any one virtual object from a plurality of virtual objects in a widget form and included in the second screen 2316. The user input for selecting any one virtual object may be any one of a gaze input and a gesture input. Fig. 23B , the processor 510 may receive a user gesture for selecting a virtual object A 2322 from among a plurality of virtual objects 2320 having a widget form and included in the second screen 2316 .

[0209] In an embodiment, the processor 510 may map the selected virtual object A 2322 to a specific real object. Fig.23C , while maintaining the user gesture for selecting the virtual object A 2322 , the processor 510 may display an additional virtual object B 2325 for guiding the mapping of a specific real object to the virtual object A 2322 .

[0210] In an embodiment, when the user gesture for selecting the virtual object A 2322 is released, the processor 510 may map the virtual object A 2322 to a real object corresponding to the release time of the user gesture. Referring to FIG. 24D , the processor 510 may map the virtual object A 2322 to a window as a real object, and may display the virtual object A 2330 mapped to the window.

[0211] Fig.24A , Fig. 24B and Fig.24C is a diagram showing an example in which the electronic device according to various embodiments of the present disclosure is a mobile portable terminal.

[0212] Reference Fig.24A , Fig. 24B and Fig.24C The embodiments disclosed below are directed to the case where the electronic device 200 is a mobile portable terminal.

[0213] Reference Fig.24A and Fig. 24B , the processor 510 may superimpose at least one virtual object 2410 on the image acquired through the camera module (not shown, for example, Figure 6 At least one virtual object 2410 is displayed in a first display mode on a real object image obtained by a camera module 560). The first display mode may be a display mode in which at least one virtual object is displayed at a fixed position of the screen regardless of the user's POV. Fig.24A , at least one virtual object 2410 may be displayed in a first area 2420 of the screen 2405. Fig. 24B, even when the real object image displayed on the screen changes when the user lifts the electronic device 200 upward, at least one virtual object 2410 may be fixedly displayed in the first area 2420 .

[0214] Reference Fig.24C In response to receiving a first user input for changing the display mode, the processor 510 may display at least one virtual object 2410 previously displayed in the first display mode in a second display mode. The processor 510 may determine the real space coordinates corresponding to the at least one virtual object 2410, and may display the at least one virtual object 2410 in the second display mode based on the determined real space coordinates. In an embodiment, the processor 510 may display a first screen including at least one virtual object and at least one second screen located near the first screen. In another embodiment, the processor 510 may display a guide object for indicating that there is a first screen including at least one virtual object and another screen (e.g., a second screen).

[0215] Reference Fig.24C , the processor 510 may change the display position of at least one virtual object 2410 based on the movement of the electronic device 200 .

[0216] For example, upon detecting the motion of the electronic device 200 moving downward, the processor 510 may change the display position of at least one virtual object 2410 from the first area 2420 to the second area 2430. Upon detecting the motion of the electronic device 200 moving downward, the processor 510 may newly display at least one virtual object 2440 that was not previously displayed.

[0217] For another example, when detecting the downward movement of the electronic device 200, the processor 510 can prevent at least a portion of the first screen from being displayed on the screen because the display position of the first screen including at least one virtual object is moved. The processor 510 can newly display at least one virtual object included in the second screen located below the first screen.

[0218] Fig.25 is a flowchart illustrating an example operation of an electronic device for combining data related to a virtual object previously displayed in a first display mode and storing the data by converting it into a form suitable for a space according to various embodiments of the present disclosure.

[0219] In an embodiment, Fig.25 The operations disclosed in the embodiment may be performed by an electronic device (eg, electronic device 200 of FIG. 2 ) or a processor (eg, Figure 5 510) to execute. In the following, for the convenience of description, Fig.25The operations disclosed in are performed by the processor 510, but those skilled in the art will appreciate that various operations may be performed by the electronic device 200 under the control of the processor 510.

[0220] In an embodiment, in operation 2510, the processor 510 may display at least one virtual object in a first display mode, and the virtual object will be displayed at a fixed position of the screen regardless of the user's POV.

[0221] For example, refer to Fig.26A and Fig.26B In response to receiving a user input for activating an exercise application, the processor 510 may provide the user with exercise information (e.g., cumulative exercise time, current movement speed, calorie consumption, and information about competitors (if any)), which is provided from the exercise application in the form of virtual objects. When providing exercise information in the form of virtual objects, the processor 510 may display at least one virtual object in a first display mode. For example, the processor 510 may fixedly display virtual object A 2610 (indicating information about cumulative exercise time, cumulative exercise distance, and calorie consumption) and virtual object B 2615 (indicating information about competitors) in a designated area of ​​the screen.

[0222] In an embodiment, in operation 2520, the processor 510 may receive a third user input for storing information related to at least one virtual object as a virtual object mapped to specific real-space coordinates (e.g., real-space coordinates corresponding to a current location or real-space coordinates corresponding to a landmark located within a specified distance from the current location).

[0223] In an embodiment, the third user input may be a user input of the same type as the first user input or the second user input. For example, the third user input may be at least one of the following inputs: a user interface (e.g., Figure 5 ), a gesture input for performing a predetermined action, a voice input including a predetermined voice keyword "store here", and a gaze input of gazing at a designated portion of the screen for at least a designated period of time. Fig.26D and Fig.26E , the processor 510 can display a virtual object C 2620 (for storing at least one virtual object (or information related to at least one virtual object such as accumulated exercise time) as a virtual object mapped to specific real space coordinates), and the processor 510 can receive a user gesture for selecting the virtual object C 2620.

[0224] In an embodiment, before receiving the third user input, the processor 510 may receive a user input (eg, a first user input) for changing the display mode. Fig.26C and Fig.26D , upon receiving the first user input, the processor 510 may display at least one virtual object 2620 (and / or at least one newly generated virtual object (e.g., virtual object D 2640)) in the second display mode. In an embodiment, when the display mode is changed from the first display mode to the second display mode, the processor 510 may change the shape of the cursor. For example, referring to Fig.26C and Fig.26D , the processor 510 may change the shape of the cursor from an inverted triangle to a cross. When displaying at least one virtual object based on the second display mode, the processor 510 may change the display of at least one virtual object based on the user's motion. Fig.26D and Fig.26E , if the user moves forward, the processor 510 may display at least one virtual object (eg, virtual object C 2620) by relatively enlarging its size. Fig.26D and Fig.26E , when the user moves to the right, the processor 510 may display at least one virtual object by relatively moving at least one virtual object (e.g., virtual object C 2620) to the left on the screen. In this case, the user may select a desired virtual object by positioning the virtual object (e.g., virtual object C 2620) at the cursor 2622 without moving the cursor 2622.

[0225] In an embodiment, the processor 510 may store information related to at least one virtual object by mapping the information to specific real space coordinates in operation 2530. For example, the processor 510 may generate a virtual object including information related to at least one virtual object (e.g., cumulative exercise time, cumulative exercise distance, calorie consumption, competitor information, etc.), and may store the generated virtual object by mapping it to specific real space coordinates.

[0226] In the embodiments, reference Fig.26F , the processor 510 may display a virtual object mapped to a specific real space coordinate in the second display mode. The processor 510 may display a virtual object mapped to a specific real space coordinate in the second display mode as if a virtual object including information corresponding to a point of interest on the real space is displayed.

[0227] According to an embodiment, if the location of the electronic device 200 is within a specified distance from a specific real space coordinate, the processor 510 may display one virtual object mapped to the specific real space coordinate on the screen.

[0228] For example, refer to Figure 26G , when at least one virtual object is displayed in the first display mode (for example, when virtual object E 2650 is displayed in the first display mode), if the position of the electronic device 200 is within a specified distance from specific real space coordinates, the processor 510 can display a virtual object 2630 mapped to specific real space coordinates on the screen based on any display mode (for example, the first display mode or the second display mode).

[0229] For another example, when at least one virtual object is displayed in the second display mode, if the position of the electronic device 200 is within a specified distance from specific real space coordinates, the processor 510 can display a virtual object 2630 mapped to the specific real space coordinates on the screen based on a display mode (i.e., the first display mode or the second display mode).

[0230] Fig. 27 is a flowchart illustrating example operations of an electronic device for storing at least one virtual object by mapping the at least one virtual object to specific real-space coordinates according to various embodiments of the present disclosure.

[0231] Fig. 27 A flowchart can be, for example, Fig.25 Detailed flowchart of operation 2520 and operation 2530.

[0232] In an embodiment, in operation 2710, the processor 510 may identify information of at least one virtual object. Fig.26B , the processor 510 can identify information of all virtual objects displayed on the screen in the first display mode.

[0233] In an embodiment, in operation 2720, the processor 510 may extract (or determine) a combination of virtual objects based on an information classification standard. In an embodiment, the processor 510 may use the information of at least one identified virtual object to classify all virtual objects displayed in the first display mode according to a predetermined standard, and may determine the combination based on this. For example, the processor 510 may classify all virtual objects based on the application information used when generating the virtual object, the generation time of the virtual object, and the attributes of the data related to the virtual object (e.g., information corresponding to a specific time or information corresponding to a specific duration). The processor 510 may determine at least one object of all virtual objects classified as the same category as a combination of virtual objects to be reconfigured as one virtual object. For example, the processor 510 may determine at least one virtual object generated based on the same application (e.g., an exercise application) as a combination of virtual objects to be reconfigured as one virtual object.

[0234] In an embodiment, when determining a combination of virtual objects, the processor 510 may use information about a combination selected by a user from among a plurality of combinations and combinations that the user has previously frequently used (eg, combinations that have been used at least a specified number of times).

[0235] In an embodiment, the processor 510 may reconfigure the determined configuration of the virtual object into one virtual object in operation 2730. For example, when a combination of virtual object A 2610 and virtual object B 2615 is determined based on generation of the virtual object, application information used in generation of the virtual object, and related data attributes in at least one virtual object, the processor 510 may reconfigure data related to virtual object A 2610 and data related to virtual object B 2615 into one virtual object.

[0236] In an embodiment, the processor 510 may map the reconfigured virtual object to real space coordinates in operation 2740. For example, the processor 510 may map the reconfigured virtual object to real space coordinates corresponding to the position of the electronic device 200 when the third user input is received or to real space coordinates corresponding to a landmark located within a specified distance of the position.

[0237] In an embodiment, in operation 2750, the processor 510 may display the reconfigured virtual object based on the mapped real-space coordinates. For example, if the distance between the coordinates corresponding to the current position of the electronic device 200 and the mapped real-space coordinates, and the direction faced by the screen of the electronic device 200 (or, the POV corresponding to the screen of the electronic device 200 based on the direction faced by the electronic device 200) meet the specified conditions, the processor 510 may display the reconfigured virtual object on the screen. For example, if the distance between the coordinates corresponding to the current position of the electronic device 200 and the mapped real-space coordinates is less than the specified distance and if the mapped real-space coordinates are included in the POV corresponding to the screen 200 of the electronic device, the processor 510 may display the reconfigured virtual object on the screen.

[0238] In an embodiment, in operation 2760, the processor 510 may determine whether a third user input for storage is received. The processor 510 may display an additional virtual object for receiving the third user input. For example, referring to Fig.28B , the processor 510 can display the reconfigured virtual object 2825 and the additional virtual object 2820 for receiving the third user input on the screen.

[0239] In an embodiment, upon receiving the third user input, in operation 2780, the processor 510 may store the identification information of the reconfigured virtual object in a memory (eg, Figure 5 in the memory 570).

[0240] In an embodiment, when the third user input is not received, the processor 510 may determine whether a specific distance has been reached or a specific time has passed in operation 2770. For example, the processor 510 may determine whether a specific distance has been reached or a specific time has passed for the time or distance when the additional virtual object 2820 for receiving the third user input is displayed on the display, or the processor 510 may determine whether a specific distance has been reached or a specific time has passed for the time or distance when the first user input for switching the display mode from the first display mode to the second display mode is received.

[0241] In an embodiment, if the specific distance is not reached and the specific time has not passed, operation 2760 may be performed again. For example, the processor 510 may repeatedly perform operations 2760 and 2770.

[0242] In an embodiment, when a specific distance is reached or a specific time has passed, the processor may not store the reconfigured virtual object according to the mapped real space coordinates (or release or delete the mapping relationship between the reconfigured virtual object and the real space coordinates) to return to the first display mode. Fig.28C , if the third user input is not received and the specific distance is reached, the processor 510 may display the virtual object A 2810 based on the first display mode.

[0243] Fig.29A , Fig.29B , Fig.29C , Fig.29D and Fig.29E is a diagram illustrating an example user input for returning to a first display mode from a second display mode according to various embodiments of the present disclosure.

[0244] In an embodiment, the processor 510 may receive a fourth user input for returning from the second display mode to the first display mode. In an embodiment, the fourth user input may be a user input of the same type as the first user input. For example, referring to Fig.29A , the fourth user input may be a gesture input for performing a predetermined action (eg, an action of grabbing at least one virtual object displayed on the screen).

[0245] In another embodiment, the fourth user input may be a selection of a user interface that clearly indicates ending the second display mode or returning to the first display mode. In an embodiment, the user interface may be a user interface (e.g., user interface 550) provided to a surface of the electronic device (e.g., electronic device 200) or may be a virtual object displayed on the screen. For example, referring to Fig.29B , the processor 510 may receive a user input (eg, a gaze input for gazing for at least a specified time) for selecting the virtual object A 2910 (indicating the end of the second display mode).

[0246] In another embodiment, the fourth user input may be a user input for selecting a virtual object corresponding to a specific function. Fig.29C , upon receiving a user input for selecting the virtual object B 2920 (corresponding to the execution of the application B), the processor 510 may execute the application B and thereafter return to the first display mode even without additional user input.

[0247] In another embodiment, if the user moves at least a specified distance, the processor 510 may display at least one virtual object based on the first display mode. Fig.29D When displaying at least one virtual object based on the second display mode, if the user moves at least a specified distance or moves beyond a specified range 2930, the processor 510 may display at least one virtual object based on the first display mode. The specified range 2930 may be a range within a specific distance from the real space coordinates of the at least one virtual object.

[0248] In another embodiment, if a specified time has passed without user input, the processor 510 may display at least one virtual object based on the first display mode. Fig.29E , when displaying at least one virtual object based on the second display mode, if a specified time has passed without user input, even if the user is within the specified range 2930, the processor 510 may display at least one virtual object based on the first display mode.

[0249] In an embodiment, the processor 510 may display at least one virtual object or at least one screen (e.g., a first screen or a second screen) including at least one virtual object based on the first display mode in response to detecting a fourth user input. For example, when a fourth user input is detected while at least a portion of a first screen and at least one second screen are displayed, the processor 510 may enlarge a first screen at a specified ratio (e.g., enlarge it to a full screen), and may display at least one virtual object included in a first screen based on the first display mode.

[0250] Fig.30 is a flowchart illustrating an example operation of an electronic device for changing a display mode based on a user input according to various embodiments of the present disclosure.

[0251] In an embodiment, Fig.30 The operations disclosed in the embodiment may be performed by an electronic device (eg, electronic device 200 of FIG. 2 ) or a processor (eg, Figure 5 510) to execute. In the following, for the convenience of description, Fig.25 The operations disclosed in are performed by the processor 510, but those skilled in the art will appreciate that various operations may be performed by the electronic device 200 under the control of the processor 510.

[0252] In an embodiment, in operation 3010, the processor 510 may display a virtual object at a first position of the screen in response to a first POV.

[0253] In an embodiment, in operation 3020, the processor 510 may maintain the display position of the virtual object at the first position of the screen in response to the second POV. In an embodiment, the second POV may be different from the first POV. For example, even if the POV changes from the first POV to the second POV, the processor 510 may maintain the display position of the virtual object at the first position of the screen.

[0254] In an embodiment, at operation 3030, the processor 510 may receive a user input for changing a display mode of a virtual object. In an embodiment, the user input for changing a display mode of a virtual object may correspond to Figure 1 The first user input disclosed in.

[0255] In an embodiment, in operation 3040, the processor 510 may display a virtual object at a second position of the screen in response to the third POV. In an embodiment, the third POV may be different from the second POV. The third POV may be different from or the same as the first POV. The second position may be different from the first position.

[0256] Computer-readable storage media may include hard disks, floppy disks, magnetic media (e.g., magnetic tapes), optical media (e.g., compact disk-ROMs (CD-ROMs), digital versatile disks (DVDs), magneto-optical media (e.g., magneto-optical disks)), internal memory, etc. Instructions may include codes created by a compiler or codes executed by an interpreter. Modules or programming modules according to various embodiments may also include at least one or more of the above-mentioned elements, or some of them may be omitted, or additional other elements may be included. According to various embodiments, the operations performed by modules, programming modules or other elements may be performed sequentially, in parallel, repeatedly, or in a heuristic manner. At least some operations may be performed in a different order or may be omitted, or other operations may be added.

[0257] Although various exemplary embodiments have been illustrated and described, it should be understood that the various exemplary embodiments are intended to be illustrative rather than restrictive. Those skilled in the art will appreciate that various changes in form and details may be made without departing from the scope of the present disclosure, including the appended claims and their equivalents.

[0258] According to various exemplary embodiments of the present disclosure, an electronic device opens a limited modality so that a user can easily and intuitively access a virtual object interface (e.g., a virtual object image) displayed on a screen, and can search for a virtual space capable of displaying a virtual object by expanding the virtual space.

Claims

1. An electronic device, comprising: monitor; a processor operably coupled to the display; A memory operably coupled to the processor, wherein the memory is configured to store instructions that, when executed, cause the processor to control the electronic device: Displaying at least one virtual object in a first display mode, wherein the at least one virtual object is displayed at a first position of the screen, wherein the at least one virtual object is superimposed on a real object image corresponding to at least one real object, and the first position is fixed and independent of the user's point of view (POV); receiving a first input for changing a display mode of the at least one virtual object; In response to receiving the first input, displaying the at least one virtual object in a second display mode, in which the at least one virtual object is displayed at a second position of the screen, wherein the second position changes according to the POV of the user; receiving a second input for selecting one of the at least one virtual object; In response to receiving the second input, the at least one virtual object is simultaneously displayed using the first display mode and multiple second virtual objects are displayed using the second display mode, wherein the multiple second virtual objects include information corresponding to the identified points of interest in the real space in the field of view FOV and are displayed at the following positions on the screen: the positions correspond to the positions of the corresponding identified points of interest on the screen.

2. The electronic device according to claim 1, wherein: The first input includes at least one of: input for a user interface set on the surface of the electronic device, gesture input for performing a specified action, voice input including a specified keyword, and gaze input based on gazing at a specified area of ​​the screen for at least a specified time.

3. The electronic device according to claim 1, wherein: The instructions, when executed, cause the processor to control the electronic device as at least part of displaying the at least one virtual object in the second display mode: determining real-space coordinates corresponding to the at least one virtual object; mapping the determined real-space coordinates to the at least one virtual object; as well as The at least one virtual object is displayed based on the determined real-space coordinates.

4. The electronic device according to claim 3, wherein: As at least part of an operation to determine real-space coordinates corresponding to the at least one virtual object, the instructions, when executed, cause the processor to control the electronic device to determine the real-space coordinates corresponding to the at least one virtual object using first real-space coordinates corresponding to the user's position and second real-space coordinates in the screen corresponding to the position of the at least one virtual object previously displayed in the first display mode.

5. The electronic device according to claim 4, wherein: As at least part of an operation of determining a real-space coordinate corresponding to the at least one virtual object using the first real-space coordinate and the second real-space coordinate, the instructions, when executed, cause the processor to control the electronic device: determine the real-space coordinate corresponding to the at least one virtual object by dividing the first real-space coordinate by the second real-space coordinate at a specified ratio.

6. The electronic device according to claim 5, wherein: As at least part of an operation to display the at least one virtual object based on the determined real-space coordinates, the instructions, when executed, cause the processor to control the electronic device to display a first screen and at least a portion of at least one second screen, wherein the first screen includes at least one virtual object, and Wherein, the at least one second screen is located within a specified distance from the first screen.

7. The electronic device according to claim 6, wherein: As at least part of receiving the second input, the instructions, when executed, cause the processor to control the electronic device: detecting head movement that results in a change in the user's POV; as well as changing the display positions of the first screen and the at least one second screen based on the detected head movement, Wherein, the real space coordinates corresponding to the first screen and the at least one second screen do not change.

8. The electronic device according to claim 6, wherein: As at least part of the operation of receiving the second input, the instructions, when executed, cause the processor to control the electronic device: in response to detecting the input, change the display position of the first screen and the at least one second screen, regardless of the change in the user's POV, wherein the real space coordinates corresponding to the first screen and the at least one second screen are configured to change according to the content of the input.

9. The electronic device according to claim 1, wherein: The memory is configured to store instructions that, when executed, cause the processor to control the electronic device: receiving a third input for storing information associated with the at least one virtual object in the form of a virtual object mapped to specific real-space coordinates; as well as In response to receiving the third input, information associated with the at least one virtual object is stored in the form of a virtual object mapped to the particular real-space coordinates.

10. A method for operating an electronic device, the method comprising: Displaying at least one virtual object in a first display mode, wherein the at least one virtual object is displayed at a first position of the screen, wherein the at least one virtual object is superimposed on a real object image corresponding to at least one real object, and the first position is fixed and independent of the user's point of view (POV); receiving a first input for changing a display mode of the at least one virtual object; In response to receiving the first input, displaying the at least one virtual object in a second display mode, in which the at least one virtual object is displayed at a second position of the screen, wherein the second position changes according to the POV of the user; receiving a second input for selecting one of the at least one virtual object; In response to receiving the second input, the at least one virtual object is simultaneously displayed using the first display mode and multiple second virtual objects are displayed using the second display mode, wherein the multiple second virtual objects include information corresponding to the identified points of interest in the real space in the field of view FOV and are displayed at the following positions on the screen: the positions correspond to the positions of the corresponding identified points of interest on the screen.

11. The method according to claim 10, wherein: The first input includes at least one of: input for a user interface set on the surface of the electronic device, gesture input for performing a specified action, voice input including a specified keyword, and gaze input based on gazing at a specified area of ​​the screen for at least a specified time.

12. The method according to claim 10, wherein: Displaying the at least one virtual object in the second display mode comprises: determining real-space coordinates corresponding to the at least one virtual object; mapping the determined real-space coordinates to the at least one virtual object; and The at least one virtual object is displayed based on the determined real-space coordinates.

13. The method according to claim 12, wherein: The determining of the real space coordinates corresponding to the at least one virtual object includes: using a first real space coordinate corresponding to the position of the user and a second real space coordinate in the screen corresponding to the position where the at least one virtual object was previously displayed in the first display mode, to determine the real space coordinates corresponding to the at least one virtual object.

14. The method according to claim 13, wherein: Determining the real space coordinates corresponding to the at least one virtual object using the first real space coordinates and the second real space coordinates includes determining the real space coordinates corresponding to the at least one virtual object by dividing the first real space coordinates by the second real space coordinates at a specified ratio.

Images