Method for supporting object selection in virtual environment and electronic device supporting same

Abstract

An embodiment disclosed herein provides a method for supporting object selection in a virtual environment, and an electronic device supporting same. An electronic device according to one embodiment may comprise a display, memory, and a processor. The electronic device may operate to recognize an input device in a virtual environment. The electronic device according to one embodiment may operate to determine a virtual space for determining target virtual objects on the basis of attributes of the input device. The electronic device according to one embodiment may operate to determine the target virtual objects on the basis of the virtual space. The electronic device according to one embodiment may operate to generate affordances corresponding to the respective target virtual objects. The electronic device according to one embodiment may operate to display the affordances according to a designated arrangement mode.

Description

Method for supporting object selection in a virtual environment and electronic device supporting the same

[0001] Embodiments of the present disclosure provide a method for supporting object selection in a virtual environment and an electronic device supporting the same.

[0002] Recently, research and development are being conducted on expanded reality (XR) technologies, such as virtual reality (VR), augmented reality (AR), and / or mixed reality (MR). VR, AR, and / or MR technologies are being utilized in various fields (e.g., entertainment, infotainment, smart homes, and / or smart factories), and the hardware and / or software components of electronic devices for these applications are being continuously researched and developed.

[0003] For example, a wearable electronic device can operate independently (e.g., in a standalone manner) or in conjunction with at least two or more devices (e.g., in a tethered manner), and can provide a single image through the display of the wearable electronic device by overlaying various digital contents (e.g., virtual images) on top of the real world through an application related to the AR service. For example, recently, AR environments have been implemented such as the tethered AR method, in which an electronic device (e.g., a smartphone) is connected to a wearable electronic device and virtual contents generated by the electronic device are provided through the display of the wearable electronic device, and the standalone AR method, in which the wearable electronic device generates virtual contents on its own without being connected to the electronic device and provides them through the display.

[0004] Thus, with recent technological advancements in AR services, the number of users is increasing, and with it, user needs. For example, users of AR services increasingly demand more accurate, intuitive, and convenient selection and display of content within the AR environment.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006] In one embodiment of the present disclosure, a method for supporting object selection in a virtual environment and an electronic device supporting the same are provided.

[0007] In one embodiment of the present disclosure, a method for providing an affordance (or guide handle) for easily selecting an object in a virtual environment based on an input device recognized in the virtual environment and an electronic device supporting the same are provided.

[0008] In one embodiment of the present disclosure, a method for determining a display position and type of affordance that supports object selection in a virtual environment and an electronic device supporting the same are provided.

[0009] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0010] An electronic device according to one embodiment of the present disclosure may include a display, at least one processor including processing circuitry, and a memory storing instructions (or commands). In one embodiment, the memory may store instructions that, when individually and / or collectively executed by the at least one processor, cause the electronic device to perform operations.

[0011] The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to recognize an input device in a virtual environment. The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to determine a virtual space for determining a target virtual object based on a property of the input device. The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to determine the target virtual object based on the virtual space. The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to generate an affordance corresponding to each of the target virtual objects. The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to display the affordance according to a specified arrangement method.

[0012] A method of operating an electronic device according to one embodiment of the present disclosure may include an operation of recognizing an input device in a virtual environment. The method may include an operation of determining a virtual space for determining a target virtual object based on properties of the input device. The method may include an operation of determining the target virtual object based on the virtual space. The method may include an operation of generating an affordance corresponding to each of the target virtual objects. The method may include an operation of displaying the affordance according to a specified arrangement method.

[0013] In order to solve the above-described problem, various embodiments of the present disclosure may include a computer-readable recording medium having recorded thereon a program for executing the method on a processor.

[0014] According to one embodiment, a non-transitory computer-readable recording medium (or storage medium or computer program product) storing one or more programs is described. According to one embodiment, the one or more programs may include commands (or instructions) that, when executed by a processor of an electronic device, perform an operation of recognizing an input device in a virtual environment, an operation of determining a virtual space for determining a target virtual object based on an attribute of the input device, an operation of determining the target virtual object based on the virtual space, an operation of generating an affordance corresponding to each of the target virtual objects, and an operation of displaying the affordance according to a specified arrangement method.

[0015] Further scope of the applicability of the present disclosure will become apparent from the detailed description below. However, since various modifications and variations within the spirit and scope of the present disclosure will readily become apparent to those skilled in the art, it should be understood that the detailed description and specific examples, such as preferred embodiments of the present disclosure, are given by way of example only.

[0016] According to an embodiment of the present disclosure, an electronic device, an operation method thereof, and a recording medium, an affordance (or guide handle) can be provided to easily select an object within a virtual environment based on an input device recognized in the virtual environment. According to an embodiment, in a virtual environment including dense objects, it is possible to support a user to accurately and easily select a desired object among dense objects. According to an embodiment, when selecting an object in a virtual environment, the shape of the object is maintained, and an affordance is provided based on an object included in a selectable range by a user input, so as to maintain a user's sense of immersion in using the virtual environment. According to an embodiment, user needs for using a virtual service (e.g., an AR service and / or a VR service) can be satisfied, and a new user experience (UX) can be provided to the user.

[0017] In addition, various effects may be directly or indirectly realized through this document. The effects obtained through this disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains, based on the description below.

[0018] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0019] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0020] FIG. 2 is a diagram illustrating an example of an electronic device according to one embodiment.

[0021] FIG. 3 is a drawing illustrating an example of the internal structure of an electronic device according to one embodiment.

[0022] FIG. 4A is a diagram illustrating a network environment between an electronic device and another electronic device according to one embodiment.

[0023] FIG. 4b is a diagram illustrating an example of displaying a virtual environment in an electronic device according to one embodiment.

[0024] FIG. 5 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment.

[0025] FIG. 6 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0026] FIGS. 7A, 7B, 7C, and 7D are diagrams illustrating examples of operations of providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0027] FIG. 8 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0028] FIG. 9 is a diagram illustrating examples of indicators for various input devices in an electronic device according to one embodiment of the present disclosure.

[0029] FIG. 10 is a diagram illustrating an example of a virtual space generated by an electronic device according to one embodiment of the present disclosure and an indicator corresponding to an input device.

[0030] FIG. 11 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0031] FIG. 12 is a diagram illustrating an example of providing a virtual space corresponding to an input device in an electronic device according to one embodiment of the present disclosure.

[0032] FIG. 13 is a diagram illustrating an example of resolution for each input device provided by an electronic device according to one embodiment of the present disclosure.

[0033] FIG. 14 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0034] FIG. 15 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0035] FIGS. 16A and 16B are diagrams illustrating examples of determining a target virtual object in an electronic device according to one embodiment of the present disclosure.

[0036] FIG. 17 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0037] FIG. 18 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0038] FIGS. 19A, 19B, and 19C are diagrams illustrating examples of providing affordance in an electronic device according to one embodiment of the present disclosure.

[0039] FIG. 20 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0040] FIGS. 21A, 21B, and 21C are diagrams illustrating examples of determining the size of an affordance in an electronic device according to one embodiment of the present disclosure.

[0041] FIGS. 22A and 22B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0042] FIGS. 23a, 23b, and 23c are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0043] FIGS. 24A, 24B, and 24C are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0044] FIGS. 25A and 25B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the disclosure.

[0045] FIG. 26 is a diagram illustrating an example of an operation of providing affordance in an electronic device according to one embodiment of the present disclosure.

[0046] FIGS. 27a, 27b, and 27c are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0047] FIGS. 28A and 28B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0048] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0049] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0050] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0051] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor (123) (e.g., a graphic processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0052] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0053] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0054] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (OS) (142), middleware (144), or an application (146).

[0055] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0056] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0057] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0058] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0059] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0060] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0061] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0062] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0063] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0064] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0065] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0066] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0067] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB, enhanced mobile broadband), minimizing terminal power and connecting multiple terminals (mMTC, massive machine type communications), or high reliability and low latency communications (URLLC, ultra-reliable and low-latency communications). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0068] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0069] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0070] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0071] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0072] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0073] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0074] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0075] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0076] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0077] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0078] Before describing various embodiments of the present disclosure, an electronic device (101) to which an embodiment of the present disclosure can be applied will be described.

[0079] FIG. 2 is a diagram illustrating an example of an electronic device according to one embodiment.

[0080] According to one embodiment, FIG. 2 illustrates an example in which the electronic device (101) is a wearable electronic device (201) in the form of glasses (e.g., a glasses-type display device or AR (augmented reality) glasses), but is not limited thereto. For example, the wearable electronic device (201) may include various forms of devices that include a display and are worn (or installed) on a part of a user's body (e.g., a face or a head) to provide augmented reality (AR), mixed reality (MR), and / or virtual reality (VR) services. For example, the wearable electronic device (201) may be implemented in at least one form of glasses, goggles, a helmet, or a hat, but is not limited thereto.

[0081] According to one embodiment, the wearable electronic device (201) may include a video see-through (VST) device. For example, the VST device may selectively provide augmented reality and virtual reality services. The VST device may include multiple cameras and may track the gaze of a user based on the multiple cameras (e.g., the gaze tracking cameras 312-1 and 312-2 of FIG. 3). In one embodiment, eye tracking may refer to a technology for tracking the position and / or movement of a user's gaze. According to one embodiment, the VST device may capture or detect the real world in front of the user based on the multiple cameras and display the real world through displays (displays corresponding to each of the user's eyes). The VST device may provide an augmented reality service by displaying virtual objects while displaying the real world.

[0082] The wearable electronic device (201) described below may be a device that includes at least some of the components included in the electronic device (101) described above with reference to FIG. 1. Even if not mentioned in the description below, it may be interpreted that the wearable electronic device (201) according to the present disclosure may include various components as described with reference to FIG. 1.

[0083] According to one embodiment, a wearable electronic device (201) may be worn on a user's face and may provide the user with an image (e.g., an augmented reality image, a mixed reality image, and / or a virtual reality image). According to one embodiment, the wearable electronic device (201) may provide an AR service that superimposes virtual information (or virtual objects) onto at least a portion of an actual real space (or environment). For example, the wearable electronic device (201) may provide the user with virtual information by superimposing it onto an actual real space corresponding to the wearer's field of view (FOV).

[0084] Referring to FIG. 2, a wearable electronic device (201) may include a glass member (or window member) (210) positioned at positions corresponding to the user's two eyes (e.g., left eye and right eye), a main frame (or main body) (240) that fixes the glass member (210), a support frame (or support member) (250) that is connected to both ends of the main frame (240) and is placed on the user's ear, and a camera module (280) (e.g., a camera for taking pictures).

[0085] According to one embodiment, the glass member (210) may include a first glass (220) corresponding to the user's left eye and a second glass (230) corresponding to the user's right eye. According to one embodiment, the glass member (210) may be supported by the main frame (240). For example, the glass member (210) may be fitted into an opening formed in the main frame (240). According to one embodiment, an AR image emitted from a display module (e.g., the display module (160) of FIG. 1) may be projected onto the glass member (210).

[0086] According to one embodiment, the glass member (210) may have a waveguide (e.g., an optical waveguide or a transparent waveguide) formed in at least a portion of the glass member (210). In one embodiment, the waveguide may serve to guide the AR image emitted from the display module to the user's eyes. A detailed description of the waveguide is described with reference to the description related to the first glass (220) and the second glass (230) of FIG. 3.

[0087] According to one embodiment, the glass member (210) is implemented as a form that is separated into a first glass (220) and a second glass (230) to correspond to the left and right eyes of the user, respectively, as illustrated in FIG. 2. However, according to some embodiments, the glass member (210) may be implemented as a single glass form without distinction between the first glass (220) and the second glass (230).

[0088] According to one embodiment, the main frame (240) and the support frame (250) may be implemented in the form of glasses.

[0089] In one embodiment, the main frame (240) may be a structure that is at least partially placed on the user's nose. According to one embodiment, the main frame (240) may support the glass member (210). According to one embodiment, the main frame (240) may be formed of a synthetic resin material. According to one embodiment, the glass member (210) may be fitted into an opening formed in the main frame (240), thereby allowing the main frame (240) to support the glass member (210).

[0090] In one embodiment, the support frame (250) may include a first support frame (260) mounted on a first directional ear (e.g., a left ear) and a second support frame (270) mounted on a second directional ear (e.g., a right ear). For example, the main frame (240) and the support frame (250) (e.g., the first support frame (260) and the second support frame (270)) may be connected through a hinge portion (not shown) and may be combined to enable folding.

[0091] In one embodiment, the support frame (250) may be rotatably connected to the main frame (240). According to one embodiment, the support frame (250) may include a first support frame (260) and a second support frame (270). The first support frame (260) may be connected to the main frame (240) from the left side (e.g., the first direction) with respect to the main frame (240) when viewed from the 'A' direction. The second support frame (270) may be connected to the main frame (240) from the right side (e.g., the second direction) with respect to the main frame (240) when viewed from the 'A' direction.

[0092] In one embodiment, the support frame (250) may be fixedly installed on the main frame (240). For example, a first support frame (260) connected to the left side of the main frame (240) and a second support frame (270) connected to the right side of the main frame (240) may be formed to be connected to each other. According to one embodiment, the support frames (250) connected to both sides of the main frame (240) may be worn in a ring shape and fitted onto the user's head. In addition, the support frame (250) may be modified into various shapes so that the wearable electronic device (201) can be worn on the user's face.

[0093] In one embodiment, the support frame (250) may be configured to be hung over the user's ear. For example, the wearable electronic device (201) may be worn on the user's face in such a way that the support frame (250) connected to the main frame (240) hangs over the user's ear. In one embodiment, the support frame (250) may rotate relative to the main frame (240). In one embodiment, the support frame (250) may be rotated in a direction closer to the main frame (240) to reduce the volume of the wearable electronic device (201).

[0094] According to one embodiment, a display module (e.g., display module (160) of FIG. 1) can output an AR image generated by a processor (120) of an electronic device (201) (e.g., processor (120) of FIG. 1) or a processor of another electronic device (e.g., another electronic device (401) of FIG. 4A) connected to the electronic device (201) (e.g., a tethered electronic device (e.g., a smart phone)). When an AR image is generated in the display module and projected onto a glass member (210), an object included in the AR image can be combined with visible light (L) incident from the front (e.g., in the direction the user is looking) through the glass member (210) to implement AR. The display module may be a very small projector (e.g., a micro projector, a pico projector). For example, the display module may be a laser scanning display (LSD), a digital micro-mirror display (DMD), and / or a liquid crystal on silicon (LCoS). In one embodiment, the display module may be a transparent display. In this case, the light-emitting element included in the display module may be directly placed on the glass member (210). Additionally, the display module may be a variety of display devices for implementing AR.

[0095] In one embodiment, the glass member (210), the support frame (250), and / or the display module may be provided as a pair to correspond to the left and right eyes of the user. For example, the glass member (210) may include a first glass (220) and a second glass (230), and the support frame (250) may include a first support frame (260) and a second support frame (270). According to one embodiment, at least some of the components described above may have different configurations corresponding to the left eye and the right eye.

[0096] In one embodiment, the camera module (280) may represent, for example, a camera for taking pictures (e.g., a front-facing camera). For example, the camera module (280) may be implemented by being placed in the main frame (240) to take pictures of a subject in the front (or the front that the user is looking at) of the wearable electronic device (201). For example, the camera module (280) may be placed in the central portion (or center point) between the first glass (220) and the second glass (230) of the main frame (240) to take pictures of the front of the main frame (240). In one embodiment, the front of the main frame (240) may refer to the direction in which the user is looking when wearing the wearable electronic device (201).

[0097] According to one embodiment, the camera module (280) may include at least one camera, and the wearable electronic device (201) may perform operations such as photographing the real world located in front of the user, recognizing objects, and / or recognizing space including three-dimensional depth information through the camera module (280). According to one embodiment, the wearable electronic device (201) may also obtain relative position information of objects recognized based on the wearable electronic device (201) or position information of objects in the recognized space through the camera module (280).

[0098] In one embodiment, the wearable electronic device (201) may include a camera module (280) and a plurality of other cameras, and a detailed description of the camera module (280) and the plurality of other cameras is described with reference to FIG. 3.

[0099] FIG. 3 is a drawing illustrating an example of the internal structure of an electronic device according to one embodiment.

[0100] According to one embodiment, the wearable electronic device (201) (e.g., the electronic device (101) of FIG. 1) may be a device implemented in a form that is worn on the face or head of the user. According to one embodiment, the wearable electronic device (201) may include a plurality of glasses (e.g., a first glass (220) and a second glass (230)) corresponding to each of the user's eyes (e.g., a left eye and a right eye) and may be implemented in the form of glasses. According to one embodiment, the wearable electronic device (201) may provide the user with an image related to an AR service. According to one embodiment, the wearable electronic device (201) may project or display a virtual object on the first glass (220) and / or the second glass (230), thereby allowing the user to see at least one virtual object superimposed on the real world perceived by the user through the first glass (220) and / or the second glass (230) of the wearable electronic device (201).

[0101] Referring to FIG. 3, a wearable electronic device (201) according to one embodiment may include a main frame (or main body) (240), a support frame (e.g., a first support frame (260), a second support frame (270)), and a hinge portion (e.g., a first hinge portion (340-1), a second hinge portion (340-2)).

[0102] According to one embodiment, the main frame (240) and the support frames (e.g., the first support frame (260) and / or the second support frame (270)) may mount various components of the wearable electronic device (201). According to one embodiment, the main frame (240) and the support frames (260, 270) may be operatively connected via hinge portions (340-1, 340-2).

[0103] In one embodiment, the main frame (240) may include a portion formed to be at least partially resting on the user's nose.

[0104] In one embodiment, the support frame (260, 270) may include a support member that can be worn over the user's ear. In one embodiment, the support frame (260, 270) may include a first support frame (260) that is worn over the user's left ear and a second support frame (270) that is worn over the user's right ear.

[0105] According to one embodiment, the first hinge portion (340-1) may connect the first support frame (260) and the main frame (240) so that the first support frame (260) can rotate relative to the main frame (240). According to one embodiment, the second hinge portion (340-2) may connect the second support frame (270) and the main frame (240) so that the second support frame (270) can rotate relative to the main frame (240). According to another embodiment, the hinge portions (340-1, 340-2) of the wearable electronic device (201) may be omitted. For example, the support frames (260, 270) may be directly connected to and fixedly installed on the main frame (240).

[0106] According to one embodiment, the main frame (240) includes glasses corresponding to the user's two eyes (e.g., left eye and right eye) (e.g., first glass (220) and second glass (230)), display modules (e.g., first display module (314-1) and second display module (314-2)), optical waveguides (e.g., first optical waveguide (320) and second optical waveguide (330)), a camera module (280) (e.g., a camera for shooting or a front shooting camera), a recognition camera (e.g., first recognition camera (311-1), second recognition camera (311-2)), and an eye tracking camera (e.g., first eye tracking camera (312-1), second eye tracking camera (312-2)), at least one microphone (e.g., first microphone (341-1), second microphone (341-2) and / or third microphone (341-3)), and at least one lighting member (e.g., first lighting It may include a first lighting element (342-1) and / or a second lighting element (342-2).

[0107] According to one embodiment, the wearable electronic device (201) can display various information by projecting light generated from the display modules (314-1, 314-2) onto the glasses (220, 230). For example, light generated from the first display module (314-1) can be projected onto the first glass (220), and light generated from the second display module (314-2) can be projected onto the second glass (230). According to one embodiment, at least a portion of the first glass (220) and the second glass (230) can be formed of a transparent material (e.g., a transparent member). According to one embodiment, the first glass (220) (or the first transparent member) corresponding to the user's left eye can be connected to the first display module (314-1), and the second glass (230) (or the second transparent member) corresponding to the user's right eye can be connected to the second display module (314-2). According to one embodiment, the first glass (220) and the second glass (230) may be formed of a glass plate, a plastic plate, and / or a polymer, and may be made transparent or translucent.

[0108] According to one embodiment, the display module (e.g., the first display module (314-1) and the second display module (314-2)) may include a liquid crystal display (LCD), a digital micro-mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0109] According to one embodiment, the first glass (220) and the second glass (230) may include a condensing lens and / or an optical waveguide (or a transparent waveguide) (e.g., the first optical waveguide (320), the second optical waveguide (330)). According to one embodiment, the optical waveguides (320, 330) may be at least partially positioned on a portion of the glass (220, 230). For example, the first optical waveguide (320) may be partially positioned on the first glass (220), and the second optical waveguide (330) may be partially positioned on the second glass (230). According to one embodiment, the optical waveguides (320, 330) may serve to transmit a light source generated by the display module (314-1, 314-2) to the user's eyes. According to one embodiment, light emitted from the display module (314-1, 314-2) may be incident on one side (or one end) of the glass (220, 230). The light incident on one side of the glass (220, 230) may be transmitted to the user through an optical waveguide (320, 330) formed (or positioned) within the glass (220, 230).

[0110] In one embodiment, the optical waveguide (320, 330) may be made of glass, plastic, or polymer, and may include a nano-pattern formed on one surface of the inner or outer surface. The nano-pattern may include a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one surface of the glass (220, 230) may be propagated or reflected by the nano-pattern within the optical waveguide (320, 330) and transmitted to the user. According to one embodiment, the optical waveguide (320, 330) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). According to one embodiment, the optical waveguide (320, 330) can guide light emitted from a display module (314-1, 314-2) (e.g., a light source unit) to a user's eye using at least one diffractive element or reflective element.

[0111] In one embodiment, the diffractive element may include an input optical member / output optical member (not shown). The input optical member may mean an input grating area, and the output optical member (not shown) may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from a display module (314-1, 314-2) (e.g., a micro LED) to transmit it to a glass (e.g., a first glass (220), a second glass (230)). The output grating area may serve as an outlet that diffracts (or reflects) light transmitted to the glass (e.g., a first glass (220), a second glass (230)) of the optical waveguide (320, 330) to a user's eye.

[0112] In one embodiment, the reflective element may include a total internal reflection (TIR) ​​optical element or waveguide for total internal reflection. Total internal reflection is a method of guiding light, which may mean creating an angle of incidence such that light (e.g., a virtual image) input through the input grating region is reflected approximately 100% from one surface (e.g., a designated surface) of the optical waveguide (320, 330), thereby transmitting approximately 100% to the output grating region.

[0113] According to one embodiment, light emitted from a display module (e.g., 314-1, 314-2) may be guided along an optical path to an optical waveguide (320, 330) through an input optical member. Light traveling within the optical waveguide (320, 330) may be guided toward a user's eye through an output optical member.

[0114] According to one embodiment, the display module (314-1, 314-2) may include a plurality of panels (or display areas), and the plurality of panels may be positioned on the glass (220, 230). According to one embodiment, at least a portion of the display module (314-1, 314-2) may be formed of a transparent element, and a user may perceive an actual space behind the display module (314-1, 314-2) by seeing through the display module (314-1, 314-2). According to one embodiment, the display module (314-1, 314-2) may display a virtual object on at least a portion of the transparent element so that the user may see that the virtual object (or virtual information) is added to at least a portion of the actual real space. According to one embodiment, when the display module (314-1, 314-2) is a transparent micro LED, the configuration of the optical waveguide (320, 330) within the glass (220, 230) may be omitted.

[0115] According to one embodiment, the wearable electronic device (201) may include a plurality of cameras (e.g., a first camera, a second camera, and a third camera). For example, the first camera (e.g., the camera module (280) of FIG. 2) may be a camera (280) (e.g., an RGB camera) for capturing an image corresponding to a user's field of view (FoV) and / or for measuring a distance to an object. The second camera may be an eye tracking camera (312-1, 312-2) for identifying a direction of a user's gaze. The third camera may be a recognition camera (gesture camera module) (311-1, 311-2) for recognizing a certain space.

[0116] According to one embodiment, the capturing camera (280) can capture the front direction of the wearable electronic device (201), and the gaze tracking cameras (312-1, 312-2) can capture the direction opposite to the capturing direction of the capturing camera (280). For example, the first gaze tracking camera (312-1) can partially capture the user's left eye, and the second gaze tracking camera (312-2) can partially capture the user's right eye. According to one embodiment, the wearable electronic device (201) can display a virtual object (or virtual information) related to an AR service together with image information related to an actual real space acquired through the capturing camera (280). According to one embodiment, the wearable electronic device (201) can display a virtual object based on display modules arranged to correspond to both eyes of the user (e.g., a first display module (314-1) corresponding to the left eye, and / or a second display module (314-2) corresponding to the right eye). According to one embodiment, the wearable electronic device (201) can display a virtual object based on preset configuration information (e.g., resolution, frame rate, brightness, and / or display area).

[0117] In one embodiment, the camera (280) for recording may include a high-resolution camera, such as a high-resolution (HR) camera and / or a photo video (PV) camera. For example, the camera (280) for recording may be utilized to acquire high-quality images, such as with an autofocus function and optical image stabilization (OIS). In addition to a color camera, the camera (280) for recording may also be implemented as a global shutter (GS) camera and a rolling shutter (RS) camera.

[0118] According to one embodiment, the gaze tracking cameras (312-1, 312-2) can detect the direction of the user's gaze (e.g., eye movement). For example, the gaze tracking cameras (312-1, 312-2) can detect the user's pupils and track the gaze direction. In one embodiment, the tracked gaze direction can be utilized to move the center of a virtual image including a virtual object in response to the gaze direction. For example, the gaze tracking cameras (312-1, 312-2) can utilize a GS (global shutter) camera to detect the pupils and track rapid eye movement without screen tearing, and the performance and specifications of each gaze tracking camera (312-1, 312-2) can be substantially the same.

[0119] According to one embodiment, the recognition camera (311-1, 311-2) can detect a user gesture and / or a certain space within a preset distance (e.g., a certain space). According to one embodiment, the recognition camera (311-1, 311-2) can be used for 3DoF, 6DoF head tracking, hand detection and / or hand tracking. For example, the recognition camera (311-1, 311-2) can be utilized to perform a space recognition function for 6DoF and a simultaneous localization and mapping (SLAM) function through depth shooting. According to some embodiments, the recognition camera (311-1, 311-2) can be utilized for a gesture recognition function for recognizing a user gesture. The recognition camera (311-1, 311-2) can include a camera including a GS. For example, the recognition camera (311-1, 311-2) may include a camera including a GS with less screen dragging (or in which the RS phenomenon can be reduced), such as an RS (rolling shutter) camera, to detect and track rapid hand movements and / or fine movements of fingers.

[0120] According to one embodiment, the wearable electronic device (201) can detect an eye corresponding to the dominant eye and / or the auxiliary eye among the user's left eye and / or right eye using at least one camera (311-1, 311-2, 312-1, 312-2, 280). For example, the wearable electronic device (201) can detect an eye corresponding to the dominant eye and / or the auxiliary eye based on the user's gaze direction toward an external object or a virtual object.

[0121] The number and position of at least one camera (e.g., a camera for shooting (280), a gaze tracking camera (312-1, 312-2), and / or a camera for recognition (311-1, 311-2)) included in the wearable electronic device (201) illustrated in FIG. 3 may not be limited. For example, the number and position of at least one camera (e.g., a camera for shooting (280), a gaze tracking camera (312-1, 312-2), and / or a camera for recognition (311-1, 311-2)) may vary based on the form (e.g., shape or size) of the wearable electronic device (201).

[0122] According to one embodiment, the wearable electronic device (201) may include at least one lighting element (or light-emitting device (illumination LED)) (e.g., a first lighting element (342-1), a second lighting element (342-2)) to increase the accuracy of at least one camera (e.g., a camera for capturing (280), an eye-tracking camera (312-1, 312-2), and / or a recognition camera (311-1, 311-2)). For example, the first lighting element (342-1) may be positioned at a portion corresponding to the user's left eye, and the second lighting element (342-2) may be positioned at a portion corresponding to the user's right eye.

[0123] According to one embodiment, the lighting members (342-1, 342-2) may be utilized for different purposes depending on the location where they are attached to the wearable electronic device (201). In one embodiment, the lighting members (342-1, 342-2) may be used as an auxiliary means to increase accuracy (e.g., ease of eye gaze detection) when capturing the user's pupils with the gaze tracking camera (312-1, 312-2), and may include an IR LED that generates light of an infrared wavelength. In one embodiment, the lighting members (342-1, 342-2) may be used as an auxiliary means to supplement ambient brightness when it is difficult to detect a subject to be captured due to a dark environment or mixing and reflection of multiple light sources when capturing the user's gesture with the recognition camera (311-1, 311-2).

[0124] According to one embodiment, the wearable electronic device (201) may include a microphone (e.g., a first microphone (341-1), a second microphone (341-2), and a third microphone (341-3)) for receiving a user's voice and ambient sounds.

[0125] According to one embodiment, the first support frame (260) (or first housing) and / or the second support frame (270) (second housing) may include a printed circuit board (PCB) (e.g., a first printed circuit board (331-1), a second printed circuit board (331-2)), a speaker for outputting an audio signal (e.g., a first speaker (332-1), a second speaker (332-2)), a battery (e.g., a first battery (333-1), a second battery (333-2)), and / or a hinge (e.g., a first hinge (340-1), a second hinge (340-2)).

[0126] According to one embodiment, the printed circuit board (331-1, 331-2) may include a flexible substrate (e.g., FPCB, flexible PCB). The printed circuit board (331-1, 331-2) may transmit electrical signals to each component (e.g., a camera, a display module, a microphone, and / or a speaker) of the wearable electronic device (201).

[0127] According to one embodiment, the speakers (332-1, 332-2) may include a first speaker (332-1) for transmitting audio signals to the user's left ear and a second speaker (332-2) for transmitting audio signals to the user's right ear.

[0128] According to one embodiment, the batteries (333-1, 333-2) may supply power to the printed circuit boards (331-1, 331-2) through a power management module (e.g., the power management module (188) of FIG. 1). According to one embodiment, the wearable electronic device (201) may include a plurality of batteries (333-1, 333-2) through a first support frame (260) and a second support frame (270), and may supply power to the printed circuit boards (331-1, 331-2) through the power management module. For example, the plurality of batteries (333-1, 333-2) may be electrically connected to the power management module.

[0129] Although not illustrated, a wearable electronic device (201) such as the examples of FIGS. 2 and 3 may be housed by a designated external device (e.g., a case) (not illustrated). In one embodiment, the case may include a function of simply storing and charging the wearable electronic device (201). In some embodiments, in addition to the purpose of charging and storing the wearable electronic device (201), the case may include components such as a camera and / or a processor, and may be used as an auxiliary computing device of the wearable electronic device (201) using these components. For example, when the wearable electronic device (201) is housed in the case, the case may communicate with the case (e.g., wired communication and / or wireless communication), and the case may perform some functions of a host device (e.g., a smartphone).

[0130] According to one embodiment, the wearable electronic device (201) may provide an AR service by itself or in conjunction with at least one other electronic device (e.g., a host device (e.g., electronic device (101)). For example, the wearable electronic device (201) may be connected to a host device (e.g., another electronic device (401) of FIG. 4A) and may provide an AR service by operating in a tethered AR manner by connecting to a network (e.g., a cloud) through the host device. According to some embodiments, the wearable electronic device (201) may also provide an AR service by operating in a standalone manner by connecting to a network (e.g., a cloud) without being connected to a host device (e.g., electronic device (101)).

[0131] FIG. 4A is a diagram illustrating a network environment between an electronic device and another electronic device according to one embodiment.

[0132] Referring to FIG. 4A, an electronic device (101) according to one embodiment (e.g., the wearable electronic device (201) of FIG. 2 or 3) may be connected to another electronic device (401). The electronic device (101) and the other electronic device (401) may be connected by wire or connected (e.g., paired) wirelessly. For example, the electronic device (101) may be connected to the other electronic device (401) via short-range wireless communication (403) such as Bluetooth, low-power Bluetooth, Wi-Fi, Wi-Fi Direct, or ultra-wide band (UWB). According to one embodiment, the other electronic device (401) may include a smart phone, a tablet PC (personal computer), a notebook, and / or a display device (e.g., a monitor or TV). According to one embodiment, the electronic device (101) may include AR glasses, smart glasses, or a head mounted display (HMD).

[0133] According to one embodiment, the electronic device (101) can directly generate (e.g., generate based on stored or processed data) or obtain related data (e.g., AR images) for AR service from an external source (e.g., another electronic device (401) or a server (e.g., server (108) of FIG. 1), and display the data through a display (e.g., display module (160) of FIG. 1). For example, the electronic device (101) can be worn on a user's body (e.g., face) and can display various digital contents (e.g., AR images) overlapping the real world as a single image (e.g., AR screen) through the display module (160). According to one embodiment, the electronic device (101) can receive image data (e.g., AR images) from another electronic device (401) and display the received image data together with various objects of the real world through the display module (160).

[0134] According to one embodiment, when the electronic device (101) is in communication connection with another electronic device (401), the electronic device (101) may periodically transmit image information captured through a camera module (e.g., the camera module (180) of FIG. 1) of the electronic device (101) and user's line of sight information (e.g., FOV and / or AOV) to the other electronic device (401), and / or transmit the image information to the other electronic device (101) when a change in the state of the electronic device (101) occurs (e.g., a change in position or direction). According to one embodiment, when the electronic device (101) is connected to another electronic device (401), the electronic device (101) may provide (e.g., transmit) at least one piece of information, such as image information, line of sight information, device information, sensing information, function information, and / or location information, to the other electronic device (401).

[0135] FIG. 4b is a diagram illustrating an example of displaying a virtual environment in an electronic device according to one embodiment.

[0136] In one embodiment, FIG. 4B may represent a state in which a user wears an electronic device (101) and views an AR screen provided through a display module (160) of the electronic device (101). For example, FIG. 4B may represent a state in which a user views content (410) (or a virtual object or an augmented reality image) superimposed on at least a portion of a real space (400) (or a virtual environment) through the AR screen of the electronic device (101). According to one embodiment, the content (410) may include a virtual object shown on a user's field of view (FOV) (430). In one embodiment, the content (410) (or the augmented reality image) may be graphic data corresponding to at least one application provided as a virtual object in a real real space (400) in which a user wearing the electronic device (101) is located.

[0137] Referring to FIG. 4B, the electronic device (101) may be worn by a user and may display one or more contents (410) (or virtual objects (or information) related to the contents) on at least a portion of a real space (400) through a display module (160) of the electronic device (101). For example, the electronic device (101) may provide the user with at least one virtual piece of information by overlaying it on the real real space (400) corresponding to the user's field of view (430). In one embodiment, a situation in which content (410) is displayed may include, for example, a situation in which a user performs a gesture input and / or an eye gaze on a real object in a real space (400), a situation in which an execution screen of an application (e.g., a browser, a shopping app, and / or an SNS) capable of performing a search function is displayed, a situation in which an execution screen of an application (e.g., a player) for playing media content is displayed (e.g., a video playback screen or an image playback screen), and / or a situation in which a user executes a specified simulation in a virtual space (e.g., a collaboration tool usage situation or a game play situation). For example, the content (410) may include information that can be provided by each corresponding application, such as virtual information (or a virtual object or a graphic image) corresponding to a first application, virtual information corresponding to a second application, and / or virtual information corresponding to a third application.

[0138] According to one embodiment, the electronic device (101) may track the user's gaze (430) while content (410) is provided, and provide virtual information related to at least one real object in a real space (400) corresponding to the user's gaze (430) or an execution screen of an application executed by the user as an AR screen (or virtual environment or execution screen) through the display module (160). For example, the user may use the electronic device (101) to use an AR service by displaying content (410) corresponding to various real objects in a real space (400) according to the movement of the user's gaze (430).

[0139] FIG. 5 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment.

[0140] According to one embodiment, FIG. 5 illustrates an example in which the electronic device (101) is a wearable electronic device in the form of glasses (e.g., the wearable electronic device (201) of FIG. 2 or FIG. 3), but is not limited thereto. According to one embodiment, the electronic device (101) may include components of the electronic device (101) and the wearable electronic device (201) as described in the description with reference to FIGS. 1 to 3. Hereinafter, the term “wearable electronic device (101)” may be used as a term including the electronic device (101) of FIG. 1 and the wearable electronic device (201) of FIG. 2 or FIG. 3.

[0141] Referring to FIG. 5, the wearable electronic device (101) may include a processor (120), a memory (130), a display module (160), an audio module (170), a sensor module (176), a camera module (180), a power management module (188), a battery (189), a communication module (190), an antenna module (197), and / or a charging antenna module (540). According to one embodiment, the wearable electronic device (101) may be connected to an external electronic device (not shown) through a connection terminal (530) (e.g., USB TYPE-C). For example, the power management module (188) of the wearable electronic device (101) may receive power from the external electronic device through the connection terminal (530) and charge the battery (189). For example, the processor (120) of the wearable electronic device (101) can perform power line communication with an external electronic device through the connection terminal (530). For example, the power management module (188) of the wearable electronic device (101) can wirelessly receive power from an external electronic device through the charging antenna module (540) to charge the battery (189). According to one embodiment, the wearable electronic device (101) may be composed of a main frame (or body) (e.g., the main frame (240) of FIG. 2 or 3) and a support frame (or leg member) (e.g., the first support frame (260) and / or the second support frame (270) of FIG. 2 or 3). According to one embodiment, components of the wearable electronic device (101) may be arranged on the main frame (240) and / or the support frames (260, 270).

[0142] According to one embodiment, the processor (120) may execute a program stored in the memory (130) to control at least one other component (e.g., hardware or software component) and perform various data processing or calculations. According to one embodiment, the processor (120) may provide an augmented reality service to a user. The processor (120) may output at least one virtual object so that at least one virtual object is superimposed and displayed in a real space corresponding to the field of view of a user wearing the wearable electronic device (101) through the display module (160).

[0143] According to one embodiment, the memory (130) may correspond to the memory (130) of FIG. 1. According to one embodiment, the memory (130) may store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the wearable electronic device (101). The data may include, for example, software (e.g., the program (140) of FIG. 1) and input data or output data for commands related thereto. The memory (130) may include volatile memory (not shown) or non-volatile memory (not shown).

[0144] According to one embodiment, the memory (130) may store instructions that, when individually and / or collectively executed by the processor (120) (e.g., the processor (120) of FIG. 1), cause the electronic device (101) to perform operations. The instructions may be stored as software (e.g., the program (140) of FIG. 1) on the memory (130) and may be executable by the processor (120).

[0145] According to one embodiment, the instructions, when executed by at least one processor, may cause the wearable electronic device (101) (or the electronic device (101)) to recognize an input device in a virtual environment. According to one embodiment, the instructions, when executed by at least one processor, may cause the wearable electronic device (101) (or the electronic device (101)) to determine a virtual space for determining a target virtual object based on an attribute of the input device. According to one embodiment, the instructions, when executed by at least one processor, may cause the wearable electronic device (101) (or the electronic device (101)) to determine a target virtual object based on the virtual space. According to one embodiment, the instructions, when executed by at least one processor, may cause the wearable electronic device (101) (or the electronic device (101)) to generate an affordance corresponding to each target virtual object. According to one embodiment, the instructions, when executed by at least one processor, may cause the wearable electronic device (101) (or electronic device (101)) to display affordances in a specified arrangement manner.

[0146] The display module (160) can visually provide information to an external device (e.g., a user) of the wearable electronic device (101). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) of the wearable electronic device (101) can include at least one glass (e.g., the first glass (220) of FIG. 2 or 3) and / or the second glass (230)). According to one embodiment, the first glass (220) can include at least a portion of the first display module (551), and the second glass (230) can include at least a portion of the second display module (553). For example, the first display module (551) and / or the second display module (553) can each include a display panel. The display panel may be configured with a transparent element so that a user can perceive an actual space through the display module (160). The display module (160) may display at least one virtual object on at least a portion of the display panel so that a user wearing the wearable electronic device (101) sees the virtual object as being added to an actual space.

[0147] For example, the user's field of view may include an angle and / or range at which the user can recognize objects. According to one embodiment, the display module (160) may include a first display module (551) corresponding to the left eye of the user and / or a second display module (553) corresponding to the right eye. According to one embodiment, the processor (120) may load setting information (e.g., resolution, frame rate, size of the display area, and / or sharpness) related to the performance of the display module (160) from the memory (130), and adjust the performance of the display module (160) based on the setting information.

[0148] According to one embodiment, each display panel included in the display module (160) may have individually determined setting information. For example, a first display panel corresponding to the left eye may be set based on first setting information, and a second display panel corresponding to the right eye may be set based on second setting information. According to one embodiment, the setting information may set at least a portion of one display panel included in the display module (160) differently. For example, the wearable electronic device (101) may set at least one of the resolution, the frame rate, and / or the sharpness of the display module (160) differently. According to one embodiment, the wearable electronic device (101) may reduce power consumption by at least partially changing the setting of the display module (160).

[0149] According to one embodiment, the audio module (170) may convert sound into an electrical signal or, conversely, convert an electrical signal into sound based on the control of the processor (120). For example, the audio module (170) may include a speaker (332-1, 332-2) of FIG. 3 and / or a microphone (e.g., a first microphone (341-1), a second microphone (341-2), and a third microphone (341-3)) of FIG. 3.

[0150] The sensor module (176) can detect the operating status (e.g., power or temperature) of the wearable electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) of the wearable electronic device (101) can include a proximity sensor (521), an illuminance sensor (522), and / or a gyro sensor (523).

[0151] According to one embodiment, the proximity sensor (521) can detect an object adjacent to the wearable electronic device (101).

[0152] According to one embodiment, the light sensor (522) can measure the brightness level around the wearable electronic device (101). According to one embodiment, the processor (120) can use the light sensor (522) to check the brightness level around the wearable electronic device (101) and change the brightness-related setting information of the display module (160) based on the brightness level. For example, if the brightness of the surroundings is brighter than the preset brightness, the processor (120) can set the brightness level of the display module (160) higher so as to increase the user's visibility.

[0153] According to one embodiment, the gyro sensor (523) can detect the posture and position of the wearable electronic device (101). For example, the gyro sensor (523) can detect whether the wearable electronic device (101) is properly worn on the user's head. For example, the gyro sensor (523) can detect the movement of the wearable electronic device (101) or the user wearing the wearable electronic device (101). For example, the gyro sensor (523) can detect the movement of the support frame of the wearable electronic device (101) (e.g., the first support frame (260) and / or the second support frame (270) of FIG. 2 or 3).

[0154] According to one embodiment, the sensor module (176) of the wearable electronic device (101) may include an acceleration sensor and / or a Hall sensor. According to one embodiment, the acceleration sensor may detect a movement of a support frame (e.g., the first support frame (260) and / or the second support frame (270) of FIG. 2 or 3) of the wearable electronic device (101). According to one embodiment, the Hall sensor may detect a change in a magnetic field to detect a movement of the support frame (e.g., the first support frame (260) and / or the second support frame (270) of FIG. 2 or 3) of the wearable electronic device (101).

[0155] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the wearable electronic device (101) and an external electronic device (e.g., another electronic device, a server, and / or a charging device), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the wearable electronic device (101) may perform wireless communication with another electronic device (e.g., a smartphone or a wearable electronic device) through the communication module (190) (e.g., a wireless communication circuit). For example, the wearable electronic device (101) may perform wireless communication with another electronic device and exchange commands and / or data with each other. According to one embodiment, the wearable electronic device (101) can be controlled, at least partially, by another external electronic device through the communication module (190). For example, the wearable electronic device (101) can perform at least one function under the control of another external electronic device. According to one embodiment, the wearable electronic device (101) can transmit object information (e.g., an image) located in an actual real space, distance information to an object, user's gaze information, and / or user's gesture information to an external device (e.g., another electronic device and / or a server) through the communication module (190) through the camera module (180).

[0156] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through at least one selected antenna. According to one embodiment, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0157] According to one embodiment, the wearable electronic device (101) can change at least a part of the settings of the display panel based on the control of another electronic device connected wirelessly and / or wiredly. According to one embodiment, the wearable electronic device (101) can transmit dominant eye / auxiliary eye related information (e.g., distance information to an object located in a real space, user's gaze tracking information, user's gesture information) acquired through a camera (e.g., camera module (180)) of the wearable electronic device (101) to another electronic device.

[0158] According to one embodiment, the other electronic device may transmit setting information of a display panel included in a glass (e.g., the first glass (220) and / or the second glass (230)) corresponding to the detected dominant or auxiliary eye to the wearable electronic device (101) based on the dominant eye / auxiliary eye-related information received from the wearable electronic device (101). According to one embodiment, the wearable electronic device (101) may change at least a part of the setting of the display panel based on the setting information of the display panel received from the other electronic device. For example, the setting of the display panel may be changed to lower the quality of the display panel, and at least a part of the setting may be changed to an extent that is not perceptible to the user. According to one embodiment, the wearable electronic device (101) may reduce the resolution of the display panel, reduce the frame rate, or adjust the size and position of the display area of ​​the display panel.

[0159] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes. According to one embodiment, the camera module (180) of the wearable electronic device (101) may include a gesture camera (511), an eye tracking camera (513), a depth camera (515), and / or an RGB camera (517).

[0160] According to one embodiment, the gesture camera (511) can detect the user's movement. The recognition cameras (311-1, 311-2) of FIG. 3 may include a gesture camera (511). For example, at least one gesture camera (511) may be placed on the wearable electronic device (101) and may detect the user's hand movement within a preset distance. The gesture camera (511) may include a simultaneous localization and mapping camera (SLAM camera) for recognizing information (e.g., location and / or direction) related to the surrounding space of the wearable electronic device (101). The gesture recognition area of ​​the gesture camera (511) may be set based on the shooting range of the gesture camera (511).

[0161] According to one embodiment, the gaze tracking camera (513) (e.g., the first gaze tracking camera (312-1) and the second gaze tracking camera (312-2) of FIG. 3) can track the movements of the user's left and right eyes. According to one embodiment, the processor (120) can use the gaze tracking camera (513) to determine the gaze direction of the left eye and the gaze direction of the right eye. For example, the gaze tracking camera (513) can include a first gaze tracking camera (312-1) for determining the gaze direction of the left eye and a second gaze tracking camera (312-2) for determining the gaze direction of the right eye. According to one embodiment, the processor (120) can determine the dominant eye and the auxiliary eye based on the gaze direction of the left eye and the gaze direction of the right eye.

[0162] According to one embodiment, the distance measuring camera (515) can measure the distance to an object located in front of the wearable electronic device (101). The photographing camera (280) of FIG. 2 or FIG. 3 can include the distance measuring camera (515). The distance measuring camera (515) can include a time of flight (TOF) camera and / or a depth camera. According to one embodiment, the distance measuring camera (515) can capture a front direction of the wearable electronic device (101), and the gaze tracking camera (513) can capture a direction opposite to the capturing direction of the distance measuring camera (515).

[0163] According to one embodiment, the wearable electronic device (101) can measure the distance to an object using a distance measuring camera (515), and change the settings of the display panel when the distance is greater than or equal to a threshold value. For example, the wearable electronic device (101) can maintain the display performance of the display panel when the distance to the object is close to or less than the threshold value. According to one embodiment, the wearable electronic device (101) can recognize one of the objects located in the direction of the user's gaze (e.g., FOV) using the gaze tracking camera (513), and calculate the depth of the distance to the object using a depth camera, or measure the distance to the object using a TOF camera.

[0164] In one embodiment, the RGB (red green blue) camera (517) can detect color-related information of an object and distance information to the object. In one embodiment, the wearable electronic device (101) can include a single type of camera by integrating the distance measurement camera (515) and the RGB camera (517). For example, the photographing camera (280) of FIG. 2 or FIG. 3 can include the distance measurement camera (515) and / or the RGB camera (517).

[0165] According to one embodiment, the gesture camera (511), the gaze tracking camera (513), the distance measurement camera (515), and / or the RGB camera (517) included in the camera module (180) may each be included in the wearable electronic device (101), or some of them may be implemented as integrated cameras. For example, the distance measurement camera (515) and the RGB camera (517) may be implemented as a single integrated camera.

[0166] According to one embodiment, the power management module (188) can manage power supplied to the wearable electronic device (101). The power management module (188) can include a plurality of power management modules (e.g., a first power management module (531), a second power management module (532)). At least some of the first power management module (531) or the second power management module (532) can be directly connected to the processor (120) to supply power. At least some of the first power management module (531) or the second power management module (532) can receive power from an external electronic device through a connection terminal (530) (e.g., TYPE-C) to charge the battery (189) or supply power to other components of the wearable electronic device (101). According to one embodiment, the wearable electronic device (101) can receive power from the external electronic device through a wireless charging method to charge the battery (188). According to one embodiment, the wearable electronic device (101) can wirelessly receive external power from a charging antenna module (540).

[0167] According to one embodiment, the power management module (188) may be electrically connected to components of the wearable electronic device (101) (e.g., memory (130), display module (160), audio module (170), sensor module (176), camera module (180), and / or communication module (190)). For example, the power management module (188) may provide power from the battery (189) to the components of the wearable electronic device (101) based on the control of the processor (120). According to one embodiment, the wearable electronic device (101) may receive power from a first battery (533) through the first power management module (531) and may receive power from a second battery (534) through the second power management module (532). According to one embodiment, the processor (120) can manage power consumption by at least partially changing the settings of the display module (160) based on information obtained using at least one camera (511, 513, 515, 517) included in the camera module (180).

[0168] The battery (189) can supply power to at least one component of the wearable electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. In one embodiment, the battery (189) can be supplied with power and charged or provided with power and discharged under the control of the power management module (188). In one embodiment, the battery (189) can include a plurality of batteries (e.g., a first battery (533), a second battery (534)). For example, the plurality of batteries (e.g., the first battery (533), the second battery (534)) can be arranged in a main frame (e.g., a main frame (240) of FIG. 2 or 3) and a support frame (e.g., a first support frame (260) and / or a second support frame (270) of FIG. 2 or 3). According to one embodiment, the first battery (533) may be placed in the first support frame (260), and the second battery (534) may be placed in the second support frame (270).

[0169] The charging antenna module (540) can receive power from the outside of the wearable electronic device (101). According to one embodiment, the charging antenna module (540) can include at least some of the configurations and / or functions of the antenna module (197). According to one embodiment, the charging antenna module (540) can include a loop-type antenna and / or a coil-type antenna. According to one embodiment, the charging antenna module (540) can include a plurality of antennas (e.g., a first antenna (541), a second antenna (542)). For example, the plurality of antennas (e.g., a first antenna (541), a second antenna (542)) can receive a magnetic field and / or an electromagnetic field from the outside, and can transmit a current generated through magnetic induction and / or magnetic resonance due to the external magnetic field and / or electromagnetic field to the power management module (188) and / or the battery (189). According to one embodiment, a plurality of antennas (e.g., a first antenna (541), a second antenna (542)) may be disposed on a main frame (e.g., a main frame (240) of FIG. 2 or 3) and a support frame (e.g., a first support frame (260) and / or a second support frame (270) of FIG. 2 or 3). According to one embodiment, the first antenna (541) may be disposed on the first support frame (260), and the second antenna (542) may be disposed on the second support frame (270).

[0170] According to one embodiment of the present disclosure, the processor (120) may perform application layer processing functions requested by a user of the wearable electronic device (101). According to one embodiment, the processor (120) may provide control and commands of functions for various blocks of the wearable electronic device (101). According to one embodiment, the processor (120) may perform operations or data processing related to control and / or communication of each component of the wearable electronic device (101). For example, the processor (120) may include at least some of the configurations and / or functions of the processor (120) of FIG. 1. According to one embodiment, the processor (120) may be operatively connected to components of the wearable electronic device (101). According to one embodiment, the processor (120) may load commands or data received from other components of the wearable electronic device (101) into the memory (130), process the commands or data stored in the memory (130), and store result data.

[0171] According to one embodiment, the processor (120) may include processing circuitry and / or executable program elements. According to one embodiment, the processor (120) may control (or process) the overall operations related to supporting the functions of the electronic device (101) based on the processing circuitry and / or executable program elements.

[0172] According to one embodiment, the processor (120) may perform an operation of recognizing an input device in a virtual environment. According to one embodiment, the processor (120) may perform an operation of determining a virtual space for determining a target virtual object based on the properties of the input device. According to one embodiment, the processor (120) may perform an operation of determining a target virtual object based on the virtual space. According to one embodiment, the processor (120) may perform an operation of generating an affordance corresponding to each target virtual object. According to one embodiment, the processor (120) may perform an operation of displaying the affordance according to a specified arrangement method.

[0173] According to one embodiment, the processor (120) may perform an operation of displaying an execution screen for a virtual environment including a plurality of virtual objects through a display. According to one embodiment, the processor (120) may perform an operation of recognizing an input device while displaying the execution screen. According to one embodiment, the processor (120) may perform an operation of recognizing an indicator corresponding to the input device based on recognizing the input device. According to one embodiment, the processor (120) may perform an operation of determining an attribute of the input device based on recognizing the indicator.

[0174] According to one embodiment, the processor (120) may perform an operation of calculating a virtual space corresponding to the properties of the input device based on an indicator. According to one embodiment, the processor (120) may perform an operation of determining a target virtual object included in the virtual space among a plurality of virtual objects of an execution screen based on the virtual space. According to one embodiment, the processor (120) may perform an operation of generating an affordance based on the target virtual object included in the virtual space. According to one embodiment, the processor (120) may perform an operation of displaying the affordance in association with the target virtual object.

[0175] According to one embodiment, the detailed operation of the processor (120) of the electronic device (101) is described with reference to the drawings described below.

[0176] According to one embodiment, the processor (120) may be a system semiconductor that is responsible for various functions of the electronic device (101). According to one embodiment, the processor (120) may be configured in the form of a system-on-chip (SoC), and may include a technology-intensive semiconductor chip that integrates multiple semiconductor technologies into one and implements system blocks into a single chip.

[0177] In one embodiment, the processor (120) may include an application processor (AP). In one embodiment, the processor (120) may include components such as a graphics processing unit (GPU), an image signal processor (ISP), a central processing unit (CPU), a neural processing unit (NPU), a digital signal processor (DSP), a modem, connectivity, and / or security. In one embodiment, the processor (120) may operate individually and / or collectively.

[0178] According to one embodiment, the operations performed by the processor (120) may be implemented by executing instructions stored in a recording medium (or a computer program product or storage medium). For example, the recording medium may include a non-transitory computer-readable recording medium having recorded thereon a program for executing various operations performed by the processor (120).

[0179] The embodiments described in the present disclosure may be implemented in a computer-readable recording medium using software, hardware, or a combination thereof. In a hardware implementation, the operations described in one embodiment may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0180] In one embodiment, a computer-readable recording medium (or computer program product or storage medium) is provided, which records a program for causing an electronic device (101) to perform (or execute) various operations.

[0181] The above operations may include an operation of recognizing an input device in a virtual environment, an operation of determining a virtual space for determining a target virtual object based on properties of the input device, an operation of determining a target virtual object based on the virtual space, an operation of generating an affordance corresponding to each target virtual object, and an operation of displaying the affordance according to a specified arrangement method.

[0182] The above operations may include an operation of displaying an execution screen for a virtual environment including a plurality of virtual objects through a display, an operation of recognizing an input device while displaying the execution screen, an operation of recognizing an indicator corresponding to the input device based on the recognition of the input device, an operation of determining a property of the input device based on the recognition of the indicator, an operation of calculating a virtual space corresponding to the property of the input device based on the indicator, an operation of determining a target virtual object included in the virtual space among a plurality of virtual objects of the execution screen based on the virtual space, an operation of generating an affordance based on the target virtual object included in the virtual space, and an operation of displaying the affordance in association with the target virtual object.

[0183] An electronic device (101, 201) according to one embodiment of the present disclosure may include a display (160), a memory (130), and processing circuitry, and may include at least one processor (120) operatively connected to the display (160) and the memory (130). According to one embodiment, the memory (130) may store instructions that, when individually and / or collectively executed by the at least one processor (120), cause the electronic device (101, 201) to perform operations.

[0184] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to recognize an input device (720) in a virtual environment (400, 700). According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to determine a virtual space (740) for determining a target virtual object (750) based on an attribute of the input device (720). According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to determine the target virtual object (750) based on the virtual space (740). According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to generate an affordance (760) corresponding to each of the target virtual objects (750). According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to display the affordance (760) according to a specified arrangement method.

[0185] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to display an execution screen for the virtual environment including a plurality of virtual objects through the display. According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to recognize the input device while displaying the execution screen. According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to recognize an indicator (900) corresponding to the input device based on recognizing the input device.

[0186] According to one embodiment, the input device may include a device that supports user input for manipulating a virtual object in the virtual environment.

[0187] According to one embodiment, the affordance may include a designated object that supports direct selection of the target virtual object among the plurality of virtual objects.

[0188] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to determine a property of the input device and to calculate a virtual space corresponding to the property of the input device based on the indicator.

[0189] According to one embodiment, the virtual space may include a virtual sphere formed with a diameter having a resolution value based on a property of the input device centered on the indicator.

[0190] According to one embodiment, the resolution value may include a first resolution value predefined for the input device. According to one embodiment, the resolution value may include a second resolution value related to user noise generated by a user using the input device.

[0191] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to calculate the resolution value based on the first resolution value and the second resolution value.

[0192] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to map a center of gravity of the virtual space based on the indicator and form the virtual space having the diameter based on the center of gravity.

[0193] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to determine a size and / or spacing of the affordance based on the resolution value.

[0194] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to determine, based on the virtual space, the target virtual object included in the virtual space among the plurality of virtual objects of the execution screen.

[0195] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to determine whether the target virtual object included in the virtual space satisfies a specified condition. According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to enter a guide generation mode for providing the affordance when the target virtual object satisfies the specified condition.

[0196] According to one embodiment, the specified condition may include a condition in which the target virtual object is a plurality of target virtual objects, at least two in number, and a distance between the centers of gravity of each of the plurality of target virtual objects is less than or equal to a specified proximity distance.

[0197] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to determine whether a plurality of target virtual objects are included in the virtual space. According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to determine a center of gravity corresponding to each of the plurality of target virtual objects, if the plurality of target virtual objects are included. According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to determine a distance between centers of gravity of each of the plurality of target virtual objects. According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to compare the distance between centers of gravity with a specified proximity distance. According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to enter the guide generation mode if at least one distance between the centers of gravity is less than or equal to the specified proximity distance.

[0198] According to one embodiment, the specified arrangement method may include a first method of providing affordances by arranging them in a space in contact with the target virtual object and the surface of the virtual space. According to one embodiment, the specified arrangement method may include a second method of providing affordances by arranging them in a list corresponding to each target virtual object outside the virtual space.

[0199] According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to provide an object-based affordance based on a space in which the target virtual object contacts a surface of the virtual space, if the target virtual object spans the virtual space. According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to provide an object-based affordance based on a space along an extension from the target virtual object to the surface of the virtual space, if the target virtual object is inside the virtual space. According to one embodiment, the instructions, when executed by the at least one processor (120), may cause the electronic device to provide a list-based affordance in an outer space of the virtual space, if a distance between the target virtual objects is less than a radius of the virtual space.

[0200] Hereinafter, the operating method of the electronic device (101) according to various embodiments will be described in detail. The operations performed in the electronic device (101) according to various embodiments may be executed by the processor (120) including various processing circuitry and / or executable program elements of the electronic device (101). According to one embodiment, the operations performed in the electronic device (101) may be stored as instructions in the memory (130) and individually and / or collectively performed (or executed) by the processor (120).

[0201] FIG. 6 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0202] According to one embodiment, FIG. 6 may illustrate an example of an operation in which an electronic device (101) according to one embodiment may support selection of a virtual object in a virtual environment without repositioning or enlarging the virtual object, without disturbing the immersion of the user experience.

[0203] An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 6. The flowchart illustrated in FIG. 6 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 601 to 609 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

[0204] As illustrated in FIG. 6, an operation method performed by an electronic device (101) according to one embodiment may include an operation (601) of recognizing an input device in a virtual environment, an operation (603) of determining a virtual space for determining a target virtual object based on a property of the input device, an operation (605) of determining a target virtual object based on the virtual space, an operation (607) of generating an affordance corresponding to each target virtual object, and an operation (609) of displaying the affordance according to a specified arrangement method.

[0205] Referring to FIG. 6, in operation 601, the processor (120) of the electronic device (101) may perform an operation of recognizing an input device in a virtual environment. According to one embodiment, the processor (120) may display an execution screen (e.g., an AR screen or a VR screen) corresponding to the virtual environment through a user's display (e.g., the display module (160) of FIG. 1 or FIG. 5). According to one embodiment, the processor (120) may provide content overlapping at least a portion of the real world (or the real world) or the virtual world through the execution screen. According to one embodiment, the content may include a plurality of virtual objects visible from the user's field of view. In one embodiment, the content may be graphic data corresponding to an application executed through the display of the electronic device (101) provided as a virtual object.

[0206] According to one embodiment, the processor (120) may recognize an input device while displaying an execution screen corresponding to a virtual environment. In one embodiment, the input device may include various devices (or means) for supporting user input for manipulating content (e.g., virtual objects) in a virtual environment. According to one embodiment, the input device may be connected to the electronic device (101) through a direct (e.g., wired) communication channel or wireless communication, and the connected input device may be recognized by the processor (120). According to one embodiment, the input device may enter an execution screen displayed through a display, and may be recognized (e.g., vision-based recognition) by the processor (120).

[0207] According to one embodiment, the input device may include a hardware device such as a mouse, a smart pen (or an electronic pen), a controller, a wearable device (e.g., a ring (e.g., a smart ring) and / or a watch) that is connected to the electronic device (101) in a specified communication manner. According to one embodiment, the input device may include an object (or object device) such as a user's hand (or finger), a stylus pen, a wearable device (e.g., a smart ring and / or a watch), and / or a glove that is recognized through an execution screen in a virtual environment. According to one embodiment, the input device may include a plurality of cameras (e.g., the eye tracking cameras 312-1 and 312-2 of FIG. 3). For example, the electronic device (101) may track the user's eye gaze using the plurality of cameras, and may recognize the position and / or movement of the user's eye gaze as an indicator corresponding to the input device.

[0208] In operation 603, the processor (120) may perform an operation of determining a virtual space for determining a target virtual object based on the properties of the input device. According to one embodiment, the processor (120) may determine a property (or type) corresponding to the input device based on recognizing the input device. According to one embodiment, the processor (120) may determine whether the recognized input device corresponds to a user's hand, a wearable device, a controller, a glove, a smart pen, a stylus pen, a mouse, and / or a camera for eye tracking. According to one embodiment, the processor (120) may determine the properties of the input device and determine the size (or range) of the virtual space (or virtual sphere or resolution sphere) based on the properties of the input device. According to one embodiment, the virtual space may include a virtual sphere (or resolution sphere) formed with a diameter centered on the input device (e.g., an indicator of the input device) and a resolution value based on the properties of the input device. In one embodiment, the virtual space may be a space virtually (or internally) created to determine a target virtual object for user selection based on the position of an input device among multiple virtual objects on the execution screen. For example, the virtual space may not be displayed on the execution screen.

[0209] In operation 605, the processor (120) may perform an operation of determining a target virtual object based on a virtual space. According to one embodiment, the processor (120) may extract (or infer) a virtual object included in a virtual space formed based on an input device among a plurality of virtual objects on the execution screen as the target virtual object. According to one embodiment, the target virtual object may include a plurality of virtual objects (e.g., at least two virtual objects).

[0210] In operation 607, the processor (120) may perform an operation of generating an affordance corresponding to each target virtual object. According to one embodiment, the processor (120) may generate an affordance corresponding to each target virtual object based on determining the target virtual object based on the virtual space. According to one embodiment, the affordance may include a designated object (e.g., a handle, an icon, an image, or a list) that supports direct selection of a target virtual object among a plurality of virtual objects on an execution screen. According to one embodiment, the affordance may be provided as graphic data.

[0211] In operation 609, the processor (120) may perform an operation of displaying an affordance according to a specified arrangement method. According to one embodiment, the specified arrangement method may include an object-based arrangement method (e.g., a first method) that provides an affordance by arranging it in a space in contact with a surface of a virtual space and a target virtual object, and a list-based arrangement method (e.g., a second method) that provides an affordance by arranging it in a list corresponding to each target virtual object outside of the virtual space. According to one embodiment, an operation example of providing an affordance according to a specified arrangement method will be described with reference to the drawings described below.

[0212] FIGS. 7A, 7B, 7C, and 7D are diagrams illustrating examples of operations of providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0213] According to one embodiment, FIGS. 7A, 7B, 7C, and 7D may illustrate examples of operations for providing an affordance that supports selection of a virtual object in association with a virtual object in a virtual environment in an electronic device (101) according to one embodiment.

[0214] Referring to Figures 7a to 7d, example screens <701> , may indicate a state in which the electronic device (101) displays a virtual environment (700) (or an execution screen (700) of the virtual environment) through a display (e.g., a display module (160) of FIG. 1 or FIG. 5). Example screen <701> As exemplified, the electronic device (101) can display a plurality of virtual objects (710) through a virtual environment (700).

[0215] Example screen <703> and example screen <705> may represent an example of a situation in which an input device (720) is recognized while displaying a virtual environment (700). According to one embodiment, in FIGS. 7B to 7D, the input device (720) is a user's hand (e.g., a finger), and the electronic device (101) may recognize the user's hand (720) entering the virtual environment (700).

[0216] Example screen <705> In the virtual environment (700), the input device (720) recognized may indicate a state in which the input device (720) is stopped at a certain location in the virtual environment (700) for a specified period of time (730). For example, the user may move the input device (720) to a space (or location) where a virtual object to be selected from among a plurality of virtual objects (710) in the virtual environment (700) exists and stop (or hold) the input device (720) for a specified period of time (e.g., about N seconds, where about N seconds is a time for which holding of the input device (720) is recognizable). According to one embodiment, the electronic device (101) may detect a trigger for initiating an operation for selecting a virtual object based on the input device (720) when the input device (720) is stopped for a specified period of time in the virtual environment (700). According to one embodiment, the electronic device (101) may further perform an operation of determining an indicator (e.g., a pointing point) based on the input device (720).

[0217] Example screen <707> An example of forming a virtual space (740) based on an input device (720) may be illustrated. According to one embodiment, the virtual space (740) may include a virtual sphere (or resolution sphere) formed with a diameter having a resolution value based on the properties of the input device (720) as the center (e.g., an indicator of the input device (720)). According to one embodiment, the virtual space (740) may be a space virtually (or internally) created to determine a target virtual object (e.g., a virtual object (750)) to be selected by a user based on the location of the input device (720) among a plurality of virtual objects (710) on a virtual environment (700). For example, the virtual space (740) may not be displayed on a display (or virtual environment (700)). According to one embodiment, the electronic device (101) can search for a virtual object (750) included in a virtual space (740) around an input device (720) (e.g., an indicator of the input device (720)) among a plurality of virtual objects (710).

[0218] Example screen <709> , may represent an example of a searched virtual object (750). For example, the electronic device (101) may determine a target virtual object (e.g., a virtual object (750) included in the virtual space (740)) for generating an affordance (760) (e.g., a handle) based on an input device (720) (e.g., an indicator of the input device (720)) among a plurality of virtual objects (710) based on the virtual space (740). In one embodiment, an example screen <709> In , for the convenience of explanation, an example may be shown in which four virtual objects (750) are extracted (or inferred) as target virtual objects. For example, an example screen <709> In , four virtual objects (750) are shown with emphasis (e.g., highlighting) to distinguish them from other virtual objects, but this may be for convenience of explanation. For example, the target virtual object (750) may be provided with a graphical effect to distinguish it from other virtual objects, taking into account the user's intuitiveness, or may be provided without any change and without a graphical effect.

[0219] Example screen <711> An example of providing an affordance (760) in association with a target virtual object (750) may be shown. According to one embodiment, the electronic device (101) may provide an affordance (760) corresponding to each target virtual object (750) by arranging it around the input device (720) (e.g., virtual space (740)). For example, the electronic device (101) may generate and provide four affordances (760), such as a first affordance (or first handle), a second affordance (or second handle), a third affordance (or third handle), and a fourth affordance (or fourth handle), corresponding to four virtual objects (750), such as a first virtual object, a second virtual object, a third virtual object, and a fourth virtual object, respectively.

[0220] Example screen <713> The input device (720) (e.g., an indicator of the input device (720)) may be moved (e.g., a position change) for example. For example, the user may move the input device (720) outside the provided area of ​​the affordance (760) (e.g., outside the virtual space (740)). For example, the user may move (or change) the position of the input device (720) to select another virtual object among multiple virtual objects (710), perform another action using the input device (720), or release the affordance (760).

[0221] Example screen <715> may represent an example of a state in which the affordance (760) is not displayed by removing it on the virtual environment (700) according to the movement of the input device (720). According to one embodiment, when the electronic device (710) detects the movement of the input device (720), the electronic device (710) may immediately remove the affordance (760) or maintain the state of displaying the affordance (760) for a specified period of time. According to one embodiment, the electronic device (101) may display the affordance (760) on the virtual environment (700) until another target virtual object is determined, and when another target virtual object is determined, the existing affordance (760) may disappear and another affordance may be generated and provided in association with another target virtual object.

[0222] FIG. 8 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0223] FIG. 9 is a diagram illustrating examples of indicators for various input devices in an electronic device according to one embodiment of the present disclosure.

[0224] FIG. 10 is a diagram illustrating an example of a virtual space generated by an electronic device according to one embodiment of the present disclosure and an indicator corresponding to an input device.

[0225] According to one embodiment, FIG. 8 may illustrate an example of an operation of providing an affordance that supports selection of a virtual object in association with a virtual object in a virtual environment in an electronic device (101) according to one embodiment.

[0226] An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 8. The flowchart illustrated in FIG. 8 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 801 to 815 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

[0227] According to one embodiment, the operations described in FIG. 8 may be performed heuristically, for example, in combination with the operations described in FIGS. 6 to 7d, or heuristically performed by replacing at least some of the operations described and combining with at least some other operations, or heuristically performed as detailed operations of at least some of the operations described.

[0228] As illustrated in FIG. 8, an operation method performed by an electronic device (101) according to an embodiment may include an operation (801) of displaying an execution screen for a virtual environment including a plurality of virtual objects through a display, an operation (803) of recognizing an input device while displaying the execution screen, an operation (805) of recognizing an indicator corresponding to the input device, an operation (807) of determining a property of the input device, an operation (809) of calculating a virtual space corresponding to the property of the input device based on the indicator, an operation (811) of determining a target virtual object included in the virtual space among a plurality of virtual objects of the execution screen based on the virtual space, an operation (813) of generating an affordance based on the target virtual object included in the virtual space, and an operation (815) of displaying the affordance in association with the target virtual object.

[0229] Referring to FIG. 8, in operation 801, the processor (120) of the electronic device (101) may perform an operation of displaying an execution screen for a virtual environment including a plurality of virtual objects through a display. According to one embodiment, the processor (120) may display an execution screen (e.g., an AR screen or a VR screen) corresponding to the virtual environment through a user's display (e.g., the display module (160) of FIG. 1 or FIG. 5). According to one embodiment, the processor (120) may provide content (e.g., a plurality of virtual objects) overlapping at least a part of the real world (or the real world) or the virtual world through the execution screen. For example, the processor (120) may display an example screen of FIG. 7A. <701> An execution screen including multiple virtual objects (710) can be displayed through the display.

[0230] In operation 803, the processor (120) may perform an operation of recognizing an input device while displaying an execution screen. In one embodiment, the input device may include various devices (or means) for supporting user input for controlling an operation of content (e.g., a virtual object) in a virtual environment. According to one embodiment, when the processor (120) detects an operation (e.g., a pointer movement) in a virtual environment by an input device connected to the electronic device (101), the processor (120) may recognize the input device connected to the electronic device (101). According to one embodiment, when the processor (120) detects the entry of the input device onto an execution screen displayed through the display, the processor (120) may recognize the input device that has entered the execution screen. An example of this is the example screen of FIG. 7A. <703> and an example screen of Fig. 7b <705> is shown in .

[0231] In operation 805, the processor (120) may perform an operation of recognizing an indicator corresponding to the input device based on recognizing the input device. According to one embodiment, the processor (120) may recognize (or determine) the indicator (900) based on the input shape of the input device (720) and / or the input coordinates of the input device (720). For example, the processor (120) may cognitively infer the indicator (900) based on the input shape for a vision-based input device (e.g., a user's hand, a glove), recognize the coordinates of the input device (720) as the indicator (900) for an input device having sensor-based coordinates (e.g., a controller, a smart pen, a mouse), and recognize the position and / or movement of the user's gaze as the indicator (900) for an eye-tracking-based input device (e.g., the eye-tracking recommendation cameras (312-1, 312-2) of FIG. 3). According to one embodiment, for an input device based on gaze tracking, for example, a function such as a mouse cursor may be performed based on the position and / or movement of the user's gaze, and the position where the user's gaze remains may be recognized as an indicator (900).

[0232] Referring to FIG. 9, examples 901, 903, and 905 illustrate a case where the input device (720) is a user's hand, example 907 illustrates a case where the input device (720) is a controller, example 909 illustrates a case where the input device (720) is a smart pen, example 911 illustrates a case where the input device (720) is a mouse, and example 913 illustrates a case where the input device (720) is a smart ring. In one embodiment, example 913 illustrates and describes a ring-type wearable device (e.g., a smart ring) worn on a user's finger, but is not limited thereto. For example, an exemplary input device (720) of the present disclosure may include a bracelet-type wearable device or a watch-type wearable device.

[0233] In example 901, the processor (120) can vision-recognize the index finger as an input device (720) and recognize the tip (e.g., fingertip) (or the position of the tip) of the index finger as an indicator (900). In example 903, the processor (120) can vision-recognize the thumb and the index finger as input devices (720) and recognize the intermediate position between the first tip of the thumb and the second tip of the index finger as the position of the indicator (900). In example 905, the processor (120) can vision-recognize the entire finger as an input device (720) and recognize the intermediate position between the tips of each of the five fingers as the position of the indicator (900).

[0234] In example 907, the processor (120) can recognize a controller connected to the electronic device (101) as an input device (720), and recognize the coordinates of the pointing point indicated by the controller as the location of the indicator (900). In example 909, the processor (120) can recognize a smart pen connected to the electronic device (101) as an input device (720), and recognize the coordinates of the pointing point indicated by the smart pen (or the tip coordinates of the smart pen) as the location of the indicator (900). In example 911, the processor (120) can recognize a mouse connected to the electronic device (101) as an input device (720), and recognize the pointer coordinates of the mouse as the location of the indicator (900).

[0235] In Example 913, the processor (120) can recognize a smart ring connected to the electronic device (101) as an input device (720), and recognize the location of the smart ring as the location of the indicator (900). For example, the smart ring can include various components such as a motion sensor that detects the movement of the smart ring, a communication circuit for connecting with the electronic device (101), and a biometric sensor (e.g., a PPG sensor) that can detect the user's biometric information. According to one embodiment, when the input device (720) is a smart ring (or a smart bracelet or a smart watch), the electronic device (101) can be connected to the smart ring through a designated wireless communication (e.g., a short-range communication such as UWB communication or Bluetooth communication). According to one embodiment, the processor (120) can recognize (or search) the exact location of the smart ring through the designated wireless communication, and designate the location of the smart ring as the location of the indicator (900).

[0236] According to one embodiment, the processor (120) may determine the location of the smart ring through vision recognition, or may determine the location of the smart ring by using vision recognition together with motion sensing information obtained from the smart ring through wireless communication. For example, the smart ring may include a light-emitting element, such as an LED, to facilitate vision recognition of the smart ring in the electronic device (101). According to one embodiment, the electronic device (101) may determine the initial location of the smart ring by vision recognition of a light-emitting point on the smart ring, and thereafter, may receive motion sensing information from the smart ring to estimate movement from the initial location.

[0237] In one embodiment, the processor (120) may utilize the position or changes thereof of the smart ring as a function of the indicator (900). For example, the processor (120) may perform an interaction to manipulate the smart ring and content (e.g., a virtual object). For example, the processor (120) may determine a target virtual object based on the smart ring and provide affordances corresponding to each target virtual object according to a specified arrangement method.

[0238] According to one embodiment, the processor (120) can monitor the location of the smart ring or the change in the location of the smart ring (e.g., movement) in real time based on designated wireless communication and / or vision recognition, and select a virtual object at the location of the smart ring based on the smart ring being stopped on the virtual object for a certain period of time or receiving a selection signal (e.g., a user input using the smart ring (e.g., a designated touch input, a designated gesture input, or a muscle tension change input)) from the smart ring on the virtual object. According to one embodiment, the processor (120) can detect a change of the smart ring (e.g., movement information based on movement or a designated gesture) while the virtual object is selected, and move the virtual object in response to the change of the smart ring. For example, the processor (120) can operate to recognize the location of the smart ring as an indicator (900) in conjunction with the smart ring and manipulate the virtual object.

[0239] According to one embodiment, the processor (120) may utilize the position or change in the user's gaze as a function of the indicator (900). For example, the processor (120) may perform an interaction to manipulate content (e.g., a virtual object) based on the user's gaze. For example, the processor (120) may determine a target virtual object based on at least one virtual object located in the user's gaze through gaze tracking, and provide affordances corresponding to each target virtual object according to a specified arrangement method.

[0240] According to one embodiment, the processor (120) can monitor in real time the position or position change (e.g., movement) of the user's gaze based on gaze tracking, and select a virtual object at the position of the user's gaze based on whether the user's gaze remains on the virtual object for a certain period of time, or whether a selection signal (e.g., detection of the user's eye blinking or user input (e.g., selection gesture input)) is received (or detected) from gaze tracking on the virtual object located at the user's gaze. According to one embodiment, the processor (120) can detect a change in the user's gaze (e.g., movement information based on movement or a designated gesture) while the virtual object is selected, and move the virtual object in response to the change in the user's gaze. For example, the processor (120) can operate to recognize the position of the user's gaze as an indicator (900) in conjunction with the user's gaze and manipulate the virtual object.

[0241] In operation 807, the processor (120) may perform an operation of determining an attribute of an input device. According to one embodiment, the processor (120) may determine a resolution (e.g., a predefined attribute (or capability) for the input device) according to the size (or type) of the input device. In one embodiment, the resolution may indicate the ability to distinguish and select a specific virtual object using the input device. According to one embodiment, the resolution may vary depending on the input device, and the resolution (e.g., the resolution value) for each input device may be stored in advance in the memory (130) of the electronic device (101). According to one embodiment, the resolution for each input device will be described with reference to the drawings described below.

[0242] In operation 809, the processor (120) may perform an operation of calculating a virtual space corresponding to the properties of the input device based on the indicator. According to one embodiment, the processor (120) may determine the size (or range) of the virtual space (or virtual sphere or resolution sphere) based on the properties (e.g., resolution) of the input device. According to one embodiment, the virtual space may include a virtual sphere (or resolution sphere) formed with the input device (e.g., the indicator of the input device) as the center and the resolution value of the input device as the diameter. An example of this is illustrated in FIG. 10.

[0243] Referring to FIG. 10, the virtual space (740) may be a space virtually (or internally) created to determine a target virtual object to be selected by the user based on the position of the input device (e.g., indicator (900)) among multiple virtual objects on the execution screen. For example, the virtual space (740) may not be displayed on the execution screen. An example of this is the example screen of FIG. 7b. <707> is shown in .

[0244] According to one embodiment, the virtual space (740) may represent a space that needs to be secured in order to accurately select one of a plurality of virtual objects in a virtual environment using the input device (720). According to one embodiment, the virtual space (740) may be expressed in the form of a virtual sphere having a specified distance (or length) (e.g., a length having a radius of about 1 / 2 of the resolution value) from the indicator (900) of the input device (720). According to one embodiment, the virtual space (740) expressed differently depending on the resolution value of each input device will be described with reference to the drawings described below.

[0245] In operation 811, the processor (120) may perform an operation of determining a target virtual object included in a virtual space among a plurality of virtual objects of an execution screen based on a virtual space. According to one embodiment, the processor (120) may extract (or infer) a virtual object included in a virtual space formed based on an input device among a plurality of virtual objects on the execution screen as a target virtual object. According to one embodiment, the target virtual object may include a plurality of virtual objects (e.g., at least two virtual objects). An example of this is the example screen of FIG. 7c. <709> is shown in .

[0246] In operation 813, the processor (120) may perform an operation of generating an affordance based on a target virtual object included in a virtual space. According to one embodiment, the processor (120) may generate an affordance corresponding to each target virtual object based on determining the target virtual object based on the virtual space. According to one embodiment, the affordance may include a designated object (e.g., a handle, an icon, an image, or a list) that supports direct selection of a target virtual object among a plurality of virtual objects on an execution screen. According to one embodiment, the affordance may be provided as graphic data.

[0247] In operation 815, the processor (120) may perform an operation of displaying an affordance in association with a target virtual object. According to one embodiment, the processor (120) may provide an affordance by distinguishing it based on a virtual space corresponding to the secured space, to support easy selection of the target virtual object. An example of this is the example screen of FIG. 7c. <711> In one embodiment, the designated arrangement method may include an object-based arrangement method (e.g., a first method) that provides affordances by arranging them in a space in contact with a surface of a virtual space and a target virtual object, and a list-based arrangement method (e.g., a second method) that provides affordances by arranging them in a list corresponding to each target virtual object outside the virtual space. In one embodiment, an operation example of providing affordances according to the designated arrangement method is described with reference to the drawings described below.

[0248] FIG. 11 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0249] FIG. 12 is a diagram illustrating an example of providing a virtual space corresponding to an input device in an electronic device according to one embodiment of the present disclosure.

[0250] FIG. 13 is a diagram illustrating an example of resolution for each input device provided by an electronic device according to one embodiment of the present disclosure.

[0251] According to one embodiment, FIG. 11 may illustrate an example of an operation of providing a virtual space based on properties of an input device in an electronic device (101) according to one embodiment.

[0252] An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 11. The flowchart illustrated in FIG. 11 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1101 to 1109 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

[0253] According to one embodiment, the operations described in FIG. 11 may be performed heuristically in combination with the operations described in FIGS. 6 to 10, for example, or may be performed heuristically by replacing at least some of the operations described and combining at least some other operations, or may be performed heuristically as detailed operations of at least some of the operations described.

[0254] As illustrated in FIG. 11, an operation method performed by an electronic device (101) according to one embodiment may include an operation (1101) of determining an attribute of an input device, an operation (1103) of obtaining a resolution value corresponding to the attribute of the input device, an operation (1105) of generating a virtual sphere having the resolution value as a diameter, an operation (1107) of mapping a center of gravity of the virtual sphere (e.g., the center point of the sphere) to an indicator, and an operation (1109) of forming a virtual space.

[0255] Referring to FIG. 11, in operation 1101, the processor (120) of the electronic device (101) may perform an operation of determining an attribute of an input device. According to one embodiment, the processor (120) may determine an attribute (e.g., type) of the input device based on recognizing the input device in a state of providing a virtual environment. According to one embodiment, the input device may include a hardware device such as a mouse, a smart pen (or an electronic pen), a controller, a wearable device (e.g., a ring and / or a watch) connected to the electronic device (101) through a specified communication method. According to one embodiment, the input device may include an object (or object device) such as a user's hand (or finger), a stylus pen, a wearable device, and / or a glove recognized through an execution screen in a virtual environment.

[0256] In operation 1103, the processor (120) may perform an operation of acquiring a resolution value corresponding to a property of the input device. According to one embodiment, the processor (120) may determine a resolution (e.g., a predefined property (or capability) for the input device) according to the size (or type) of the input device. In one embodiment, the resolution may indicate the ability to distinguish and select a specific virtual object using the input device. According to one embodiment, the electronic device (101) may store the resolution (e.g., the resolution value) for each input device in advance in the memory (130). Examples thereof are illustrated in FIGS. 12 and 13 .

[0257] As illustrated in FIGS. 12 and 13, the resolution may vary for each input device (720). According to one embodiment, the resolution value for each input device may be stored in advance in the memory (130) of the electronic device (101). According to one embodiment, the resolution value for each input device may be stored externally (e.g., a cloud server) of the electronic device (101), and the electronic device (101) may obtain (e.g., request and receive) the resolution value corresponding to the input device recognized from the external source and use it when necessary.

[0258] Referring to Figure 12, an example <1201> , example <1203> and examples <1205> The input device (720) may represent an example in which the input device is a user's hand (e.g., a thumb and an index finger), a controller, and a mouse. According to one embodiment, the processor (120) may determine the size (or range) of the virtual space (740) based on a predefined resolution value corresponding to the size (or type) of the input device (720). According to one embodiment, the processor (120) may form a virtual space (740) (e.g., a virtual sphere or a resolution sphere) with the resolution value of the input device (720) as the diameter centered on the input device (720) (e.g., an indicator (900) of the input device). For example, the processor (120) may express the virtual space (740) in the form of a virtual sphere having a specified distance (or length) (e.g., a length with the resolution value as the radius) from the indicator (900) of the input device (720).

[0259] In one embodiment, an example <1201> Silver may represent an example in which a first resolution value (e.g., about 40 mm) is set for an input device (720) (e.g., a user's hand), a virtual space (740) having a diameter of the first resolution value is formed by a processor (120), and the center of gravity of the virtual space (740) is mapped to an indicator (900) and provided. According to one embodiment, an example <1203> An example may be provided in which a second resolution value (e.g., about 25 mm) is set for an input device (720) (e.g., a controller), a virtual space (740) having a diameter of the second resolution value is formed by a processor (120), and the center of gravity of the virtual space (740) is mapped to an indicator (900). According to one embodiment, an example <1205> An example may be provided in which a third resolution value (e.g., about 10 mm) is set for an input device (720) (e.g., a mouse), a virtual space (740) having the third resolution value as a diameter is formed by a processor (120), and the center of gravity of the virtual space (740) is mapped to an indicator (900).

[0260] According to one embodiment, the resolution value may be defined and / or learned based on the manipulation precision of object selection using the input device (720). For example, a relatively large (or high) resolution value may be set for an input device (e.g., a user's hand) with low accuracy for object selection, and a relatively small (or low) resolution value may be set for an input device (e.g., a mouse) with high accuracy for object selection. For example, a small resolution value may mean that the virtual space (740) forms a sphere with a small diameter. According to one embodiment, an input device (720) with a small resolution value may mean a device with high accuracy for object selection due to less noise (e.g., hardware noise such as camera noise for recognizing the input device (720) and / or user noise such as hand tremors) resulting from object selection. According to one embodiment, an example of a resolution value that can be set for each input device (720) based on noise that may occur in the input device (720) is illustrated in FIG. 13.

[0261] Referring to FIG. 13, FIG. 13 may illustrate various examples of resolution values ​​predefined for each type of input device (720). For example, a relatively large (or high) resolution value may be set for an input device (e.g., a user's hand) with low accuracy in object selection, and a relatively small (or low) resolution value may be set for an input device (e.g., a mouse) with high accuracy in object selection. For example, a lower resolution value may be assigned as the accuracy of object selection using the input device (720) increases. For example, as illustrated in FIG. 13, for a first input device (1301) (e.g., a user's hand) according to one embodiment, a first resolution value (e.g., approximately 40 mm) based on experimental values ​​may be set. For a second input device (1302) (e.g., a watch) according to one embodiment, a second resolution value (e.g., approximately 35 mm) based on experimental values ​​may be set. For a third input device (1303) according to one embodiment (e.g., a ring), a third resolution value (e.g., about 30 mm) based on experimental values ​​may be set. For a fourth input device (1304) according to one embodiment (e.g., a controller), a fourth resolution value (e.g., about 25 mm) based on experimental values ​​may be set. For a fifth input device (1305) according to one embodiment (e.g., a glove), a fifth resolution value (e.g., about 20 mm) based on experimental values ​​may be set. For a sixth input device (1306) according to one embodiment (e.g., a smart pen), a sixth resolution value (e.g., about 15 mm) based on experimental values ​​may be set. For a seventh input device (1307) according to one embodiment (e.g., a mouse), a seventh resolution value (e.g., about 10 mm) based on experimental values ​​may be set.

[0262] In operation 1105, the processor (120) may perform an operation of generating a virtual sphere having a resolution value as a diameter. In one embodiment, the processor (120) may determine a size (or range) of the virtual space (or virtual sphere or resolution sphere) based on the resolution value of the input device. In one embodiment, the processor (120) may generate a virtual sphere (or resolution sphere) formed with a diameter equal to the resolution value of the input device (or a radius equal to about 1 / 2 of the resolution value from an indicator of the input device).

[0263] In operation 1107, the processor (120) may perform an operation of mapping the center of gravity of a virtual sphere to an indicator. In one embodiment, the processor (120) may map the center of gravity of a virtual space having a diameter corresponding to a resolution value specified for the input device to the indicator. An example of this is illustrated in FIG. 12 .

[0264] In operation 1109, the processor (120) may perform an operation of forming a virtual space. According to one embodiment, the processor (120) may map the center of gravity of the virtual space based on an indicator of an input device, and form a virtual space having a diameter with a resolution value specified based on the center of gravity.

[0265] FIG. 14 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0266] According to one embodiment, FIG. 14 may illustrate an example of an operation for determining a resolution value for an input device in an electronic device (101) according to one embodiment.

[0267] An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 14. The flowchart illustrated in FIG. 14 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1401 to 1405 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

[0268] According to one embodiment, the operations described in FIG. 14 may be heuristically performed in combination with the operations described in FIGS. 6 to 13, for example, or may be heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or may be heuristically performed as a detailed operation of at least some of the operations described.

[0269] As illustrated in FIG. 14, an operation method performed by an electronic device (101) according to one embodiment may include an operation (1401) of obtaining a first resolution value related to an input device, an operation (1403) of obtaining a second resolution value related to user noise for the input device, and an operation (1405) of determining a resolution value for the input device based on the first resolution value and the second resolution value.

[0270] Referring to FIG. 14, in operation 1401, the processor (120) of the electronic device (101) may perform an operation of obtaining a first resolution value related to the input device. In one embodiment, the first resolution value may include a default resolution value predefined for the input device.

[0271] In operation 1403, the processor (120) may perform an operation of acquiring a second resolution value related to user noise for the input device. In one embodiment, the second resolution value may include a correction resolution value related to user noise (user nose) generated by a user using the input device. For example, the second correction value may learn about human errors, such as hand tremors, of the user when the user uses the input device, and may be updated as a correction value for the input device based on the learned result. For example, the second correction value may include a resolution value corrected by considering user noise using the input device for the first resolution value for the input device. According to one embodiment, the resolution value may include a first resolution value pre-stored in the electronic device (101) for each input device, and a second resolution value (e.g., a hand tremors correction value) that corrects user noise that is updated according to user learning.

[0272] In operation 1405, the processor (120) may perform an operation of determining a resolution value for the input device based on the first resolution value and the second resolution value. According to one embodiment, the processor (120) may calculate a final resolution value for the input device by considering the first resolution value of the input device and the second resolution value that compensates for human error (e.g., user noise) of a user using the input device.

[0273] FIG. 15 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0274] FIGS. 16A and 16B are diagrams illustrating examples of determining a target virtual object in an electronic device according to one embodiment of the present disclosure.

[0275] According to one embodiment, FIG. 15 may illustrate an example of an operation for determining entry into a guide generation mode for providing affordance in an electronic device (101) according to one embodiment.

[0276] An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 15. The flowchart illustrated in FIG. 15 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1501 to 1509 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

[0277] According to one embodiment, the operations described in FIG. 15 may be performed heuristically, for example, in combination with the operations described in FIGS. 6 to 14, or may be performed heuristically by replacing at least some of the operations described and combining with at least some other operations, or may be performed heuristically as detailed operations of at least some of the operations described.

[0278] As illustrated in FIG. 15, an operation method performed by an electronic device (101) according to one embodiment may include an operation (1501) of identifying a target virtual object included in a virtual space, an operation (1503) of determining whether the target virtual object satisfies a specified condition, an operation (1505) of determining entry into a guide generation mode based on whether the target virtual object satisfies the specified condition, an operation (1507) of performing an operation related to affordance generation, and an operation (1509) of processing the performance of the corresponding operation based on whether the target virtual object does not satisfy the specified condition.

[0279] Referring to FIG. 15, in operation 1501, the processor (120) of the electronic device (101) may perform an operation of identifying a target virtual object included in a virtual space. According to one embodiment, the processor (120) may extract (or infer) a virtual object included in a virtual space formed based on an input device among a plurality of virtual objects on an execution screen as a target virtual object. According to one embodiment, the processor (120) may identify a virtual object as a target virtual object if part or all of the virtual object is included in a virtual space centered on an indicator of the input device.

[0280] In operation 1503, the processor (120) may perform an operation to determine whether the target virtual object satisfies a specified condition. In one embodiment, the specified condition may include a condition in which the target virtual object is a plurality of virtual objects, at least two in number, and the distance between the centers of gravity of each of the plurality of virtual objects is less than or equal to a specified proximity distance. Examples of this are illustrated in FIGS. 16A and 16B .

[0281] Referring to FIG. 16A, FIG. 16A may illustrate an example in which both a first virtual object (1610) and a second virtual object (1620) are included in a virtual space (740). For example, in a case in which a plurality of virtual objects, such as at least two virtual objects (e.g., the first virtual object (1610) and the second virtual object (1620)), are included in a virtual space (740), it may be difficult for a user to distinguish and select the virtual objects. According to one embodiment, in a case in which at least two virtual objects (1610, 1620) are included in a virtual space (740), the processor (120) may operate to provide affordances for at least two virtual objects (1610, 1620) so that a user can easily select and manipulate the virtual objects.

[0282] Referring to FIG. 16B, FIG. 16B may illustrate an example in which a portion of a third virtual object (1630) and a portion of a fourth virtual object (1640) are included in a virtual space (740). For example, in a case in which portions of a plurality of virtual objects, such as at least two virtual objects (e.g., the third virtual object (1630) and the fourth virtual object (1640)), are included in a virtual space (740), it may be easy or difficult for a user to distinguish and select the virtual objects depending on the distance between the centers of gravity of each of the plurality of virtual objects (1630, 1640). According to one embodiment, as illustrated in FIG. 16B, when the distance (L) between the first center of gravity (1635) of the third virtual object (1630) and the second center of gravity (1645) of the fourth virtual object (1640) is greater than a specified proximity distance, it may be easy for a user to distinguish and select the virtual objects. For example, if the center of gravity of at least one virtual object exists outside the virtual space (740) and there is no overlap between virtual objects (1630, 1640), the user's selection of a distinction for the virtual object may be easy.

[0283] In one embodiment, if the distance (L) between the first center of gravity (1635) of the third virtual object (1630) and the second center of gravity (1645) of the fourth virtual object (1640) in FIG. 16b is less than or equal to a specified proximity distance, the user may have difficulty in distinguishing and selecting the virtual objects. For example, if the centers of gravity (1635, 1645) of multiple virtual objects (1630, 1640) exist within the virtual space (740) or there is overlap between the virtual objects (1630, 1640), the user may have difficulty in distinguishing and selecting the virtual objects. According to one embodiment, the processor (120) may be operable to provide affordances for at least two virtual objects (1630, 1640) so that a user can easily select and manipulate the virtual objects when the distance (L) between the centers of gravity (1635, 1645) of the plurality of virtual objects (1630, 1640) is less than or equal to a specified proximity distance.

[0284] In operation 1503, if the processor (120) determines that the target virtual object satisfies the specified condition (e.g., 'Yes' in operation 1503), in operation 1505, the processor (120) may perform an operation of determining whether to enter a guide generation mode. According to one embodiment, the processor (120) may determine that the specified condition is satisfied if the target virtual object in the virtual space is a plurality of virtual objects, that is, at least two virtual objects, and the distance between the centers of gravity of each of the plurality of virtual objects is less than or equal to a specified proximity distance. According to one embodiment, the processor (120) may determine whether to enter a guide generation mode for generating an affordance based on determining that the specified condition is satisfied. An example of this is illustrated in FIG. 16A.

[0285] In operation 1507, the processor (120) may perform operations related to affordance generation. According to one embodiment, the processor (120) may enter a guide generation mode and perform operations for generating affordances associated with a plurality of virtual objects included in the virtual space. According to one embodiment, an example of operations for generating and providing affordances will be described with reference to the drawings described below.

[0286] In operation 1503, if the processor (120) determines that the target virtual object does not satisfy the specified condition (e.g., 'No' in operation 1503), in operation 1509, the processor (120) may perform an operation for processing the execution of the corresponding operation. According to one embodiment, the processor (120) may recognize a user input using an input device. According to one embodiment, the processor (120) may process an operation for selecting a virtual object based on the user input and executing a related function (e.g., object movement, object editing).

[0287] FIG. 17 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0288] According to one embodiment, FIG. 17 may illustrate an example of an operation for determining entry into a guide generation mode for providing affordance in an electronic device (101) according to one embodiment.

[0289] An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 17. The flowchart illustrated in FIG. 17 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1701 to 1715 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or 5).

[0290] According to one embodiment, the operations described in FIG. 17 may be heuristically performed in combination with the operations described in FIGS. 6 to 16b, for example, or may be heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or may be heuristically performed as a detailed operation of at least some of the operations described.

[0291] As illustrated in FIG. 17, an operation method performed by an electronic device (101) according to an embodiment may include an operation (1701) of identifying a target virtual object included in a virtual space, an operation (1703) of determining whether the virtual space includes a plurality of target virtual objects, an operation (1705) of determining a center of gravity corresponding to each of the plurality of target virtual objects based on the inclusion of the plurality of target virtual objects in the virtual space, an operation (1707) of determining a distance between the centers of gravity of each of the plurality of target virtual objects, an operation (1709) of comparing the distance between the centers of gravity with a specified proximity distance, an operation (1711) of determining whether there is a distance between the centers of gravity that is less than or equal to the specified proximity distance, an operation (1713) of determining entry into a guide generation mode based on the existence of a distance less than or equal to the specified proximity distance, and an operation (1715) of processing the execution of the corresponding operation when the virtual space does not include a plurality of target virtual objects or does not include a distance less than or equal to the specified proximity distance.

[0292] Referring to FIG. 17, in operation 1701, the processor (120) of the electronic device (101) may perform an operation of identifying a target virtual object included in a virtual space. According to one embodiment, the processor (120) may extract (or infer) a virtual object included in a virtual space formed based on an input device among a plurality of virtual objects on an execution screen as a target virtual object. According to one embodiment, the processor (120) may identify a virtual object as a target virtual object if part or all of the virtual object is included in a virtual space centered on an indicator of the input device.

[0293] In operation 1703, the processor (120) may perform an operation of determining whether the virtual space includes a plurality of target virtual objects. According to one embodiment, the processor (120) may determine whether the virtual space includes at least two target virtual objects, as in the examples of FIG. 16A and / or FIG. 16B .

[0294] In operation 1703, if the processor (120) determines that the virtual space does not include multiple target virtual objects (e.g., 'No' in operation 1703), in operation 1715, the processor (120) may process the execution of the corresponding operation. According to one embodiment, the processor (120) may recognize a user input using an input device. According to one embodiment, the processor (120) may process an operation of selecting a virtual object based on the user input and executing a related function (e.g., object movement, object editing).

[0295] In operation 1703, when the processor (120) determines that the virtual space includes multiple target virtual objects (e.g., 'yes' in operation 1703), in operation 1705, the processor may perform an operation of determining a center of gravity corresponding to each of the multiple target virtual objects.

[0296] In operation 1707, the processor (120) may perform an operation of determining a distance between centers of gravity of each of the plurality of target virtual objects. According to one embodiment, when some or all of the plurality of virtual objects, such as at least two virtual objects, are included in a virtual space, the processor (120) may determine the distance between the centers of gravity of each of the plurality of virtual objects. For example, the processor (120) may determine a distance (L) between a first center of gravity (1635) of a third virtual object (1630) and a second center of gravity (1645) of a fourth virtual object (1640), as illustrated in FIG. 16B .

[0297] In operation 1709, the processor (120) may perform an operation of comparing a distance between centers of gravity with a designated proximity distance. According to one embodiment, the processor (120) may compare distances between centers of gravity (e.g., multiple distances such as a first distance and a second distance) corresponding to each of a plurality of target virtual objects in the virtual space with the designated proximity distance. For example, it may be assumed that the virtual space includes a first virtual object, a second virtual object, and a third virtual object. According to one embodiment, the processor (120) may determine a first distance between the centers of gravity of the first virtual object and the second virtual object, a second distance between the centers of gravity of the first virtual object and the third virtual object, and a third distance between the centers of gravity of the second virtual object and the third virtual object. According to one embodiment, the processor (120) may compare the first distance with the designated proximity distance, compare the second distance with the designated proximity distance, and compare the third distance with the designated proximity distance.

[0298] In operation 1711, the processor (120) may perform an operation of determining whether there is a distance between centers of gravity that is less than or equal to a specified proximity distance. In one embodiment, it may be determined whether there is a distance less than or equal to a specified proximity distance among the first distance, the second distance, and / or the third distance.

[0299] In operation 1711, if the processor (120) determines that there is a distance less than or equal to a specified proximity distance (e.g., 'yes' in operation 1711), in operation 1713, the processor (120) may perform an operation of determining whether to enter a guide generation mode. According to one embodiment, if there are at least two target virtual objects in a virtual space and the distance between the centers of gravity of each of the plurality of virtual objects is less than or equal to a specified proximity distance, the processor (120) may determine whether to enter a guide generation mode for generating an affordance.

[0300] In operation 1711, if the processor (120) determines that there is no distance less than the specified proximity distance (e.g., 'No' in operation 1711), in operation 1715, the processor (120) may process the execution of the corresponding operation. According to one embodiment, the processor (120) may recognize a user input using an input device. According to one embodiment, the processor (120) may process an operation of selecting a virtual object based on the user input and executing a related function (e.g., object movement, object editing).

[0301] FIG. 18 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0302] According to one embodiment, FIG. 18 may illustrate an example of an operation of providing affordance in an electronic device (101) according to one embodiment.

[0303] An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 18. The flowchart illustrated in FIG. 18 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1801 to 1803 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or FIG. 5 ).

[0304] According to one embodiment, the operations described in FIG. 18 may be heuristically performed in combination with the operations described in FIGS. 6 to 17, for example, or may be heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or may be heuristically performed as a detailed operation of at least some of the operations described.

[0305] As illustrated in FIG. 18, an operation method performed by an electronic device (101) according to one embodiment may include an operation (1801) of determining an affordance arrangement method, and an operation (1803) of rendering an affordance based on the arrangement method and the resolution value.

[0306] Referring to FIG. 18, in operation 1801, the processor (120) of the electronic device (101) may perform an operation of determining an affordance arrangement method. According to one embodiment, the processor (120) may determine a designated arrangement method (or an affordance type or an affordance display type) for arranging an affordance on the virtual space based on a relationship between the virtual space and the target virtual object. According to one embodiment, the designated arrangement method may include a placement method (e.g., a first placement method) that provides an affordance by arranging it in a space in contact with the surface of the virtual space and the target virtual object. According to one embodiment, the designated arrangement method may include a placement method (e.g., a second placement method) that provides an affordance by arranging it in a list corresponding to each target virtual object outside the virtual space.

[0307] In operation 1803, the processor (120) may perform an operation of rendering an affordance based on a specified arrangement method and resolution value. According to one embodiment, the processor (120) may generate an affordance with two-dimensional or three-dimensional graphic data (e.g., an icon or image of a specified shape (e.g., a cube, a circle, an oval, a triangle, a diamond), an image of an object shape, text based on text recognition included in an object). According to one embodiment, the processor (120) may realistically generate an affordance by considering object information (e.g., information about a location, a color, and / or a light source) related to a target virtual object in a virtual environment. For example, the processor (120) may generate an affordance with a color that contrasts with the color of the target virtual object, and may place the affordance in a space that does not overlap with the location of the target virtual object in the virtual space. For example, the processor (120) may determine the size and / or spacing of the affordance based on the resolution value, and place the affordance based on the determined size and / or spacing. Examples of this are illustrated in FIGS. 19a, 19b, and 19c.

[0308] FIGS. 19A, 19B, and 19C are diagrams illustrating examples of providing affordance in an electronic device according to one embodiment of the present disclosure.

[0309] As illustrated in FIGS. 19A, 19B, and 19C, affordances may be provided in the form of designated objects each corresponding to a target virtual object based on a virtual space (740). According to one embodiment, when the spacing (L) between the arranged affordances is smaller than the radius (r) of the virtual space (e.g., L < r), affordances may be provided in the form of designated lists each corresponding to the target virtual object.

[0310] Referring to FIG. 19A, FIG. 19A may illustrate an example of arranging an affordance when a target virtual object (e.g., a first virtual object (1910), a second virtual object (1920)) spans a virtual space (740). According to one embodiment, when the target virtual objects (1910, 1920) span a virtual space (740), the electronic device (101) may determine an affordance type as an internal arrangement method (or internal type) that places the affordance type inside the target virtual objects (1910, 1920).

[0311] According to one embodiment, the electronic device (101) may provide an affordance (e.g., a first affordance (1915), a second affordance (1925)) (e.g., an internal affordance disposed inside the virtual object) by placing it on a space (or surface) where the target virtual object (1910, 1920) comes into contact with a surface of the virtual space (740). For example, the first affordance (1915) (e.g., a first internal handle) may be placed on a surface where the first virtual object (1910) comes into contact with the virtual space (740), and the second affordance (1925) (e.g., a second internal handle) may be placed on a surface where the second virtual object (1920) comes into contact with the virtual space (740).

[0312] Referring to FIG. 19B, FIG. 19B may illustrate an example of arranging an affordance when a target virtual object (e.g., a third virtual object (1930), a fourth virtual object (1940)) is located inside a virtual space (740). According to one embodiment, when the target virtual objects (1930, 1940) are located inside a virtual space (740), the electronic device (101) may determine an affordance type as an external arrangement method (or external type) that places the affordance type in a space (or plane) on an extension line to the surface of the virtual space (740).

[0313] According to one embodiment, the electronic device (101) may provide an affordance (e.g., a third affordance (1935), a fourth affordance (1945)) (e.g., an external affordance placed on the surface of the virtual space (740)) by placing it in a space on an extension line to a surface of the virtual space (740). For example, the third affordance (1935) (e.g., a first external handle) may be placed in a space on an extension line from a third virtual object (1930) to a close surface (e.g., a surface at a short distance) of the virtual space (740) (or a surface in contact with the extension line), and the fourth affordance (1945) (e.g., a second external handle) may be placed in a space on an extension line from a fourth virtual object (1940) to a close surface (e.g., a surface at a short distance) of the virtual space (740) (or a surface in contact with the extension line).

[0314] Referring to FIG. 19c, FIG. 19c may illustrate an example of arranging affordances when some or all of the target virtual objects (e.g., the fifth virtual object (1950), the sixth virtual object (1960), the seventh virtual object (1970), and the eighth virtual object (1980)) exist within the virtual space (740) or span the virtual space (740), and the spacing (L) between at least some affordances is smaller than the radius of the virtual space (740). According to one embodiment, the electronic device (101) may arrange affordances (e.g., the fifth affordance (1955), the sixth affordance (1965), the seventh affordance (1975), and the eighth affordance (1985)) based on the virtual space (740), and when the interval (L) between each of the affordances (1955, 1965, 1975, 1985) arranged in the virtual space (740) is smaller than the radius of the virtual space (740), the affordance type may be determined in a list arrangement manner (or list type) that arranges the affordance types in a list form on the periphery of the virtual space (740) within the virtual environment.

[0315] According to one embodiment, the electronic device (101) may provide an affordance (1990) (e.g., a list-based affordance) by placing it in an area outside the virtual space (740) within the virtual environment. For example, the electronic device (101) may generate an affordance (1990) in the form of a list and place it in an area within the virtual environment when the distance (L) between at least two of the affordances (e.g., the fifth affordance (1955) and the sixth affordance (1965)) among the affordances (e.g., the fifth affordance (1955), the sixth affordance (1965), the seventh affordance (1975), and the eighth affordance (1985)) placed based on the virtual space (740) is smaller than the radius (e.g., half of the resolution value) of the virtual space (740).

[0316] According to one embodiment, the list-based affordance (1990) may be provided in the form of a list including items (e.g., a fifth item, a sixth item, a seventh item, and an eighth item) each corresponding to a target virtual object (e.g., a fifth virtual object (1950), a sixth virtual object (1960), a seventh virtual object (1970), and an eighth virtual object (1980)).

[0317] FIG. 20 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0318] According to one embodiment, FIG. 20 may illustrate an example of an operation of providing affordance in an electronic device (101) according to one embodiment.

[0319] An operation method supported by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 20. The flowchart illustrated in FIG. 20 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 2001 to 2015 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 or 5).

[0320] According to one embodiment, the operations described in FIG. 20 may be heuristically performed in combination with the operations described in FIGS. 6 to 19c, for example, or may be heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or may be heuristically performed as a detailed operation of at least some of the operations described.

[0321] As illustrated in FIG. 20, an operation method performed by an electronic device (101) according to an embodiment includes an operation (2001) of identifying a state between a target virtual object and a virtual space, an operation (2003) of determining whether the target virtual object is in a state extending over the virtual space, an operation (2005) of providing an affordance based on a space where the target virtual object is in contact with a surface of the virtual space if the target virtual object is in a state extending over the virtual space, an operation (2007) of determining whether the target virtual object is in a state included in the virtual space, an operation (2009) of providing an affordance based on a space in an extension line from the target virtual object to the surface of the virtual space if the target virtual object is in a state included in the virtual space, an operation (2011) of determining whether a gap between target virtual objects is smaller than a radius of the virtual space, an operation (2013) of providing a list-based affordance based on a specified space of the virtual space if the gap between target virtual objects is smaller than a radius of the virtual space, and an operation (2014) of determining whether a state between a target virtual object and a virtual space is the above state (e.g., operation 2003, operation 2007, and the state of the operation 2011) may include operations (2015) that provide affordances based on contact space and / or extension space.

[0322] Referring to FIG. 20, in operation 2001, the processor (120) of the electronic device (101) may perform an operation of identifying a state between a target virtual object and a virtual space. According to one embodiment, the processor (120) may determine whether the state between the target virtual object and the virtual space corresponds to a first state, a second state, or a third state. In one embodiment, the first state may include a state in which the target virtual object spans the virtual space. In one embodiment, the second state may include a state in which the target virtual object is included within the virtual space. In one embodiment, the third state may include a state in which a distance between affordances is smaller than a radius of the virtual space.

[0323] In operation 2003, the processor (120) may perform an operation to determine whether the target virtual object is in a state (e.g., a first state) that spans the virtual space. An example of the first state is illustrated in FIG. 19A.

[0324] In operation 2003, if the target virtual object is in a state (e.g., a first state) extending over the virtual space (e.g., 'yes' in operation 2003), in operation 2005, the processor (120) may perform an operation of providing an affordance based on a space where the target virtual object comes into contact with a surface of the virtual space. According to one embodiment, if the target virtual object extends over the virtual space, the processor (120) may determine an affordance type as an internal placement method (or internal type) that places the affordance inside the target virtual object. According to one embodiment, the processor (120) may place and provide an affordance in a space (or surface) where the target virtual object comes into contact with a surface of the virtual space.

[0325] In operation 2007, the processor (120) may perform an operation to determine whether the target virtual object is in a state (e.g., a second state) contained within the virtual space. An example of the second state is illustrated in FIG. 19b.

[0326] In operation 2007, if the target virtual object is in a state (e.g., a second state) included in the virtual space (e.g., 'yes' in operation 2007), in operation 2009, the processor (120) may perform an operation of providing an affordance based on a space on an extension line from the target virtual object to the surface of the virtual space. According to one embodiment, if the target virtual object exists in the virtual space, the processor (120) may determine the type of affordance as an external placement method (or external type) that places it in a space (or plane) on an extension line to the surface of the virtual space. According to one embodiment, the processor (120) may place and provide an affordance in a space on an extension line to the surface of the virtual space.

[0327] In operation 2011, the processor (120) may perform an operation of determining whether the distance between target virtual objects is smaller than the radius of the virtual space (e.g., a third state). An example of the third state is illustrated in FIG. 19c.

[0328] In operation 2011, if the gap between target virtual objects is smaller than the radius of the virtual space (e.g., the third state), then in operation 2013, the processor (120) may perform an operation of providing a list-based affordance based on a specified space in the virtual space. According to one embodiment, the processor (120) may arrange the affordances based on the virtual space, and if the gap (L) between each of the affordances arranged in the virtual space is smaller than the radius of the virtual space, the affordance type may be determined as a list arrangement method (or list type) that arranges the affordances in a list form on the periphery of the virtual space within the virtual environment. According to one embodiment, the processor (120) may arrange and provide an affordance (e.g., a list-based affordance) in an area outside the virtual space within the virtual environment. For example, the processor (120) may generate a list of affordances and place them in an area within the virtual environment when the distance (L) between at least two affordances arranged based on the virtual space is smaller than the radius of the virtual space (e.g., half of the resolution value).

[0329] In operation 2003, operation 2005, and operation 2011, if the state between the target virtual object and the virtual space is not included in the first state, the second state, and the third state, the processor (120) may perform an operation of providing an affordance based on the contact space and / or the space on the extension line in operation 2015. According to one embodiment, the processor (120) may provide a part or all of the affordances corresponding to each target virtual object by arranging them within the corresponding object in the virtual space, as in the example of FIG. 19A. According to one embodiment, the processor (120) may provide a part or all of the affordances corresponding to each target virtual object by arranging them on a surface on the extension line that contacts the virtual space, as in the example of FIG. 19B. According to one embodiment, the processor (120) may provide a part of the affordances corresponding to each target virtual object by arranging them within the corresponding object in the virtual space, and provide another part of the affordances by arranging them on a surface on the extension line that contacts the virtual space.

[0330] FIGS. 21A, 21B, and 21C are diagrams illustrating examples of determining the size of an affordance in an electronic device according to one embodiment of the present disclosure.

[0331] In one embodiment, the electronic device (101) may determine the size and / or spacing of the affordance based on a resolution value for the virtual space (e.g., a diameter of the virtual space).

[0332] Referring to FIG. 21A, the electronic device (101) may form a virtual space (740) having a resolution value of the input device (720) as a diameter (D) based on an indicator (900) according to the input device (720) (e.g., a user's hand). According to one embodiment, FIG. 21A may show an example in which the input device (720) is a user's hand. According to one embodiment, the electronic device (101) may obtain a resolution value (e.g., a first resolution value set for the user's hand) (e.g., about 40 mm) of the input device (720). According to one embodiment, the electronic device (101) may form a virtual space (740) having a diameter (e.g., D = about 40 mm) equal to the first resolution value (e.g., about 40 mm) of the input device (720). According to one embodiment, the electronic device (101) can determine the size (e.g., width) of the affordance (2110) based on a half (e.g., radius (r), where r = D / 2) of the first resolution value (or diameter (D) of the virtual space (740)) (e.g., about 40 mm). For example, the affordance (2110) can be provided in a designated shape (e.g., cube, circle, oval, triangle, diamond) having a width of half (e.g., about 20 mm) of the first resolution value (or diameter (D) of the virtual space (740)) (e.g., about 40 mm).

[0333] Referring to FIG. 21B, the electronic device (101) may form a virtual space (740) having a resolution value of the input device (720) as a diameter (D) based on an indicator (900) according to the input device (720) (e.g., a controller). According to one embodiment, FIG. 21B may show an example in which the input device (720) is a controller. According to one embodiment, the electronic device (101) may obtain a resolution value of the input device (720) (e.g., a second resolution value set for the controller) (e.g., about 25 mm). According to one embodiment, the electronic device (101) may form a virtual space (740) having a diameter (e.g., D = about 25 mm) equal to the second resolution value of the input device (720) (e.g., about 25 mm). According to one embodiment, the electronic device (101) can determine the size (e.g., width) of the affordance (2120) based on a half (e.g., radius (r), where r = D / 2) of the second resolution value (or diameter (D) of the virtual space (740)) (e.g., about 25 mm). For example, the affordance (2120) can be provided in a designated shape (e.g., cube, circle, oval, triangle, diamond) having a width of half (e.g., about 12.5 mm) of the second resolution value (or diameter (D) of the virtual space (740)) (e.g., about 25 mm).

[0334] Referring to FIG. 21C, the electronic device (101) may form a virtual space (740) having a resolution value of the input device (720) as a diameter (D) based on an indicator (900) according to the input device (720) (e.g., a controller). According to one embodiment, FIG. 21C may show an example in which the input device (720) is a controller and provides an affordance (2130) in a list form. According to one embodiment, the electronic device (101) may obtain a resolution value of the input device (720) (e.g., a second resolution value set for the controller) (e.g., about 25 mm). According to one embodiment, the electronic device (101) may form a virtual space (740) having a diameter (e.g., D = about 25 mm) equal to the second resolution value of the input device (720) (e.g., about 25 mm). According to one embodiment, the electronic device (101) can determine the size of the affordance (2130) (e.g., spacing between items) based on a half (e.g., radius (r), where r = D / 2) of the second resolution value (or diameter (D) of the virtual space (740)) (e.g., about 25 mm). For example, the affordance (2130) can be provided in the form of a list with a spacing of half (e.g., about 12.5 mm) of the second resolution value (or diameter (D) of the virtual space (740)) (e.g., about 25 mm).

[0335] FIGS. 22A and 22B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0336] According to one embodiment, FIGS. 22A and 22B may illustrate an example of an operation of providing an affordance (2250) that can support selection of at least one of a target virtual object in association with a target virtual object in a virtual environment (2200) in an electronic device (101) according to one embodiment.

[0337] Referring to FIGS. 22A and 22B , the electronic device (101) can recognize an input device (720) while displaying a virtual environment (2200) (or an execution screen (2200) of the virtual environment) through a display (e.g., a display module (160) of FIG. 1 or 5 ). According to one embodiment, as an example, in FIGS. 22A and 22B , the input device (720) is a user's hand (e.g., a finger), and the electronic device (101) can recognize the user's hand (720) entering the virtual environment (2200).

[0338] According to one embodiment, the electronic device (101) can generate and display affordances (2250) corresponding to target virtual objects that are the target of user selection based on a virtual space formed based on an input device (720) (e.g., an indicator (900) of the input device (720).

[0339] According to one embodiment, the electronic device (101) may display an affordance (2250) (e.g., a guide handle) related to a target virtual object (e.g., a chess piece) within a virtual space formed around the input device (720) (e.g., an indicator (900) of the input device (720)) when the input device (720) is stationary for a predetermined period of time based on at least one virtual object among a plurality of virtual objects within the virtual environment (2200). According to one embodiment, the user may select and manipulate (e.g., move, delete, copy, and / or edit) at least one target virtual object (e.g., a chess piece) based on the affordance (2250).

[0340] FIGS. 23a, 23b, and 23c are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0341] According to one embodiment, FIGS. 23A through 23C may illustrate an example of an operation of providing an affordance (2350) that can support selection of at least one of a target virtual object in association with a target virtual object in a virtual environment (2300) in an electronic device (101) according to one embodiment.

[0342] Referring to FIGS. 23A to 23B , the electronic device (101) can recognize the input device (720) while displaying a virtual environment (2300) (or an execution screen (2300) of the virtual environment) through a display (e.g., the display module (160) of FIG. 1 or 5 ). According to one embodiment, assuming that the input device (720) is a controller in FIGS. 23A to 23C , the electronic device (101) can recognize the indicator (900) of the input device (720) connected to the electronic device (101).

[0343] According to one embodiment, the electronic device (101) can generate and display affordances (2350) corresponding to target virtual objects that are the target of user selection based on a virtual space formed based on an input device (720) (e.g., an indicator (900) of the input device (720).

[0344] According to one embodiment, the electronic device (101) may display an affordance (2350) (e.g., a guide handle) related to a target virtual object (e.g., a shape) in a virtual space formed around the input device (720) (e.g., the indicator (900) of the input device (720)) when the input device (720) (or the indicator (900) of the input device (720)) is stationary for a predetermined period of time based on at least one virtual object among a plurality of virtual objects in the virtual environment (2300). According to one embodiment, the user may select and manipulate (e.g., move, delete, copy, and / or edit) at least one target virtual object (e.g., a shape) based on the affordance (2350).

[0345] Referring to FIG. 23c, a user may select one of the affordances (2360) among the affordances (2350) corresponding to each target virtual object using the input device (720). According to one embodiment, when the input device (720) (e.g., the indicator (900) of the input device (720)) stops (e.g., holds) one of the affordances (2360) for a predetermined period of time, the electronic device (101) may determine the depth (e.g., depth menu) of the affordance (2360). For example, the electronic device (101) may determine whether menu expansion of the affordance (2360) is possible. According to one embodiment, the electronic device (101) may provide a depth menu (2370) associated with (or adjacent to or around) the selected affordance (2360) when there is a depth menu associated with the affordance (2360) selected by the input device (720). According to one embodiment, the depth menu (2370) may include multiple sub-depths, such as n-depth, and the electronic device (101) may provide up to n-depth.

[0346] FIGS. 24A, 24B, and 24C are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0347] According to one embodiment, FIGS. 24A to 24C may illustrate an example of an operation of providing an affordance (2450) capable of supporting selection of at least one target virtual object in association with a target virtual object of a virtual environment (2400) in an electronic device (101) according to one embodiment. According to one embodiment, FIGS. 24A to 24C may illustrate an example of a case where an input device (720) is a user's hand and an affordance (2450) in the form of a list is provided.

[0348] Referring to FIGS. 24A to 24B , the electronic device (101) can recognize an input device (720) while displaying a virtual environment (2400) (or an execution screen (2400) of the virtual environment) through a display (e.g., a display module (160) of FIG. 1 or 5 ). According to one embodiment, as an example, in FIGS. 24A to 24C , the input device (720) is a user's hand, and the electronic device (101) can recognize the user's hand (720) entering the virtual environment (2400).

[0349] According to one embodiment, the electronic device (101) can generate and display affordances (2450) corresponding to target virtual objects that are the target of user selection based on a virtual space formed based on an input device (720) (e.g., an indicator (900) of the input device (720)).

[0350] According to one embodiment, the electronic device (101) may determine a list arrangement method (or list type) to arrange affordances corresponding to each target virtual object based on the virtual space, and arrange them in a list form on the outskirts of the virtual space within the virtual environment (2400) when the distance (L) between each of the affordances arranged in the virtual space is smaller than the radius of the virtual space. According to one embodiment, the electronic device (101) may arrange and provide an affordance (2450) (e.g., a list-based affordance) in an area outside the virtual space within the virtual environment (2400). For example, the electronic device (101) may generate an affordance (2450) in a list form and arrange it in an area within the virtual environment (2400) when the distance (L) between at least two of the affordances arranged based on the virtual space is smaller than the radius of the virtual space (e.g., half of the resolution value).

[0351] According to one embodiment, the electronic device (101) may display, in a list format, affordances (2450) (e.g., guide handles) related to a target virtual object (e.g., a book) in a virtual space formed around the input device (720) (e.g., the indicator (900) of the input device (720)), when the input device (720) (or the indicator (900) of the input device (720)) stops for a predetermined period of time based on at least one virtual object among a plurality of virtual objects in the virtual environment (2400). According to one embodiment, the user may select and manipulate (e.g., move, delete, copy, and / or edit) at least one target virtual object (e.g., a book) based on the affordances (2450).

[0352] Referring to FIG. 24C, a user may select one of the affordances (2460) corresponding to each target virtual object using an input device (720). According to one embodiment, when the input device (720) (e.g., an indicator (900) of the input device (720)) stops (e.g., holds) for a predetermined period of time for one of the affordances (2460), the electronic device (101) may highlight (2470) (e.g., highlight, select box) the target virtual object corresponding to the affordance (2360) in the virtual environment (2400). For example, when an item (2460) is selected from an affordance (2450) in a list form through an input device (720), the electronic device (101) can distinguish a target virtual object corresponding to the selected item (2460) from other target virtual objects in the virtual environment (2400), thereby increasing the visibility and / or legibility of the target virtual object. According to one embodiment, the electronic device (101) can configure two-dimensional or three-dimensional graphic data (e.g., an icon or image of a specified shape (e.g., a cube, a circle, an oval, a triangle, a diamond) or an image of an object shape or a text based on text recognition included in an object) for each item in the affordance (2450), thereby increasing the visibility and / or legibility of the corresponding target virtual object for each item.

[0353] FIGS. 25A and 25B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the disclosure.

[0354] Referring to FIGS. 25A and 25B , an electronic device (101) may provide an affordance (2500) including a plurality of individual affordances (e.g., items). According to one embodiment, when providing the affordance (2500), the electronic device (101) may determine the depth (e.g., depth menu) of the affordance (2500). For example, the electronic device (101) may determine whether menu expansion of the affordance (2500) is possible. According to one embodiment, if menu expansion is possible, the electronic device (101) may provide a menu expansion button (e.g., “>”) for each affordance for which menu expansion is possible.

[0355] According to one embodiment, a user may select one of the affordances (2510) corresponding to each target virtual object using an input device (720). For example, the user may select a menu expansion button of one of the affordances (2510) using the input device (720) (e.g., an indicator (900) of the input device (720). According to one embodiment, the electronic device (101) may expand the menu of the affordance (2510) based on the user input to provide a depth menu (2550). According to one embodiment, if there is a depth menu related to the affordance (25100) selected by the input device (720), the electronic device (101) may provide the depth menu (2550) in association with (or adjacent to or around) the selected affordance (2510).

[0356] FIG. 26 is a diagram illustrating an example of an operation of providing affordance in an electronic device according to one embodiment of the present disclosure.

[0357] According to one embodiment, FIG. 26 may illustrate an example of a method for arranging list affordances considering a three-dimensional object layout.

[0358] According to one embodiment, the electronic device (101) may provide a list that takes into account the actual 3D layout when selecting multiple overlapping 3D affordance objects in a virtual environment. For example, the electronic device (101) may provide a list based on the 3D object layout (e.g., the 3D affordance object (2610)) to enhance the intuitiveness of selecting multiple overlapping 3D virtual objects.

[0359] In one embodiment, the example <2601> In the case where the gap between multiple affordances in a virtual environment is smaller than the resolution value and it is necessary to generate affordances in the form of a list, the electronic device (101) can project the corresponding 3D affordance object (2610) onto a plane from the user's viewpoint and calculate the center of gravity (2620) of each affordance object (2610).

[0360] In one embodiment, the example <2603> In , the electronic device (101) may place overlapping affordance objects (2610) in a three-dimensionally expanded form by maintaining the size (e.g., size ratio) of the affordance objects (2610) the same and magnifying only the distance between each affordance object (2610) by a specified magnification multiple (e.g., approximately N times) based on the center of gravity (2620). According to one embodiment, the specified magnification multiple may be set to a minimum value (e.g., approximately 1.2 times, approximately 2 times, or approximately 3 times) that satisfies a specified condition (e.g., spacing between centers of gravity, minimum width) according to a resolution value of the input device.

[0361] FIGS. 27a, 27b, and 27c are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0362] According to one embodiment, FIGS. 27A to 27C may illustrate an example of an operation of providing an affordance (2750) capable of supporting selection of at least one target virtual object in association with a target virtual object in a virtual environment (2700) in an electronic device (101) according to one embodiment. According to one embodiment, FIGS. 27A to 27C may illustrate an example of a case where multiple input devices (270) (e.g., a mouse and a user's hand or a user's left hand and a user's right hand) operate in a virtual environment (2700).

[0363] Referring to FIGS. 27A to 27C , the electronic device (101) may recognize a first input device (720) (e.g., a mouse) while displaying a virtual environment (2700) (or an execution screen (2700) of the virtual environment) through a display (e.g., the display module (160) of FIG. 1 or 5 ). According to one embodiment, FIGS. 27A to 27B may represent a state in which a virtual object designated by the first input device (720) in the virtual environment (2700) does not exist. For example, this may represent a state in which an affordance corresponding to the first input device (720) is not provided. According to one embodiment, when a virtual object designated by the first input device (720) exists, an affordance may be provided based on the first input device (720).

[0364] In one embodiment, a user may move a second input device (720) (e.g., a user's hand) into a virtual environment (2700) while manipulating a first input device (720). In one embodiment, the electronic device (101) may recognize the first input device (720) (e.g., a mouse) in the virtual environment (2700) while simultaneously recognizing the second input device (720) (e.g., a user's hand) entering the virtual environment (2700).

[0365] According to one embodiment, the electronic device (101) can generate and display affordances (2750) corresponding to each target virtual object to be selected by the user based on a virtual space formed based on the second input device (720).

[0366] According to one embodiment, the electronic device (101) may display an affordance (2750) (e.g., a guide handle) related to a target virtual object in a virtual space formed around the second input device (720) when the second input device (720) stops for a predetermined period of time based on at least one virtual object among a plurality of virtual objects in the virtual environment (2700). According to one embodiment, the user may select and manipulate (e.g., move, delete, copy, and / or edit) at least one target virtual object based on the affordance (2750).

[0367] According to one embodiment, while the electronic device (101) displays an affordance (2750) centered on the second input device (720), if the first input device (720) stops for a predetermined period of time based on at least one virtual object among a plurality of virtual objects within the virtual environment (2700), the electronic device (101) may simultaneously provide another affordance (not shown) related to another target virtual object within the virtual space formed centered on the first input device (720). For example, the electronic device (101) may simultaneously provide affordances corresponding to each of a plurality of input devices of different types.

[0368] FIGS. 28A and 28B are diagrams illustrating examples of operations for providing affordance in association with a virtual object in an electronic device according to one embodiment of the present disclosure.

[0369] According to one embodiment, FIGS. 28A and 28B may illustrate an example of an operation of providing an affordance (2850) that can support selection of at least one of target virtual objects in association with a target virtual object of a virtual environment (2800) when a plurality of electronic devices (101) (e.g., a first electronic device (2801), a second electronic device (2803)) are connected to each other, share a virtual environment (2800), and collaborate. For example, FIGS. 28A and 28B may illustrate an example of a scenario in which a first electronic device (2801) of a first user and a second electronic device (2803) of a second user are linked to work together in a virtual environment (2800).

[0370] Referring to FIG. 28A, a first electronic device (2801) or a second electronic device (2803) may recognize at least one input device while displaying a virtual environment (2800) (or an execution screen (2800) of the virtual environment) through their respective displays (e.g., the display module (160) of FIG. 1 or FIG. 5). According to one embodiment, among the first electronic device (2801) and the second electronic device (2803), the electronic device (101) executing the virtual environment (2800) may be a master device. According to one embodiment, the electronic device (101) that shares the virtual environment (2800) executed by the master device may be a slave device. According to one embodiment, the recognition of the input device may be recognized by a master device (e.g., the first electronic device (2801) or the second electronic device (2803)) that provides the virtual environment (2800).

[0371] According to one embodiment, a master device (e.g., a first electronic device (2801)) may generate and display affordances (2850) corresponding to target virtual objects that are targets of user selection based on a virtual space formed based on an input device. According to one embodiment, the affordances (2850) generated by the master device may also be shared and displayed on a virtual environment (2800) provided through a display of a slave device (e.g., a second electronic device (2803)).

[0372] According to one embodiment, the master device may display an affordance (2850) (e.g., a guide handle) related to a target virtual object within a virtual space formed around the input device when the input device remains stationary for a predetermined period of time based on at least one virtual object among a plurality of virtual objects within the virtual environment (2800). According to one embodiment, the first user or the second user may select and manipulate (e.g., move, delete, copy, and / or edit) at least one target virtual object based on the affordance (2850).

[0373] Referring to FIG. 28B, a first user or a second user may select one of the affordances (2860) corresponding to a target virtual object using an input device. According to one embodiment, when the input device is stopped (e.g., held) for a predetermined period of time specified for one of the affordances (2860), the master device may determine the depth menu of the affordance (2860). For example, the master device may determine whether menu expansion of the affordance (2860) is possible. According to one embodiment, when there is a depth menu related to the affordance (2860) selected by the input device, the master device may provide a depth menu (2870) associated with (or adjacent to or around) the selected affordance (2860). In one embodiment, a depth menu (2870) generated by a master device may also be shared and displayed on a virtual environment (2800) provided through the display of a slave device.

[0374] An operation method performed in an electronic device (101, 201) according to one embodiment of the present disclosure may include an operation of recognizing an input device (720) in a virtual environment (400, 700). According to one embodiment, the operation method may include an operation of determining a virtual space (740) for determining a target virtual object (750) based on an attribute of the input device (720). According to one embodiment, the operation method may include an operation of determining the target virtual object (750) based on the virtual space (740). According to one embodiment, the operation method may include an operation of generating an affordance (760) corresponding to each of the target virtual objects (750). According to one embodiment, the operation method may include an operation of displaying the affordance (760) according to a specified arrangement method.

[0375] According to one embodiment, the operation of recognizing the input device may include an operation of displaying an execution screen for the virtual environment including a plurality of virtual objects through a display, an operation of recognizing the input device while displaying the execution screen, and an operation of recognizing an indicator (900) corresponding to the input device based on recognizing the input device.

[0376] According to one embodiment, the operation of determining the virtual space may include an operation of determining a property of the input device, an operation of calculating the resolution value based on a first resolution value predefined according to the property of the input device and a second resolution value related to user noise generated by a user using the input device, an operation of calculating the virtual space having the resolution value as a diameter, and an operation of mapping the center of gravity of the virtual space based on the indicator.

[0377] According to one embodiment, the virtual space may include a virtual sphere formed with a diameter having a resolution value based on a property of the input device centered on the indicator.

[0378] According to one embodiment, the operation of determining the virtual object may include an operation of determining, based on the virtual space, the target virtual object included in the virtual space among the plurality of virtual objects of the execution screen, an operation of determining whether the target virtual object included in the virtual space satisfies a specified condition, and an operation of entering a guide generation mode for providing the affordance when the target virtual object satisfies the specified condition.

[0379] According to one embodiment, the specified condition may include a condition in which the target virtual object is a plurality of target virtual objects, at least two in number, and a distance between the centers of gravity of each of the plurality of target virtual objects is less than or equal to a specified proximity distance.

[0380] According to one embodiment, the operation of displaying the affordance may include an operation of providing an object-based affordance based on a space where the target virtual object contacts a surface of the virtual space when the target virtual object spans the virtual space, an operation of providing an object-based affordance based on a space along an extension line from the target virtual object to the surface of the virtual space when the target virtual object is inside the virtual space, and an operation of providing a list-based affordance in an outer space of the virtual space when a distance between the target virtual objects is smaller than a radius of the virtual space.

[0381] According to one embodiment, the affordance may include a virtual object that supports direct selection of the target virtual object among the plurality of virtual objects.

[0382] A non-transitory computer-readable medium storing instructions that, when executed by a processor (120) of an electronic device (101) according to one embodiment of the present disclosure, cause the processor (120) to perform operations, the instructions include a recording medium that, when executed by the processor (120), cause the electronic device (101) to perform an operation of recognizing an input device (720) in a virtual environment (400, 700), an operation of determining a virtual space (740) for determining a target virtual object (750) based on an attribute of the input device (720), an operation of determining the target virtual object (750) based on the virtual space (740), an operation of generating an affordance (760) corresponding to each of the target virtual objects (750), and an operation of displaying the affordance (760) according to a specified arrangement method. Can be.

[0383] It will be appreciated that the above-described embodiments and their technical features may be combined with each other in any and all combinations, as long as there is no potential conflict between the two embodiments or features. For example, any and all combinations of two or more of the above-described embodiments may be envisioned and incorporated within the present disclosure. One or more features from any embodiment may be incorporated into any other embodiment, providing a corresponding advantage or advantages.

[0384] The various embodiments of the present disclosure disclosed in this specification and drawings are merely specific examples to facilitate easy explanation of the technical content of the present disclosure and aid understanding of the present disclosure, and are not intended to limit the scope of the present disclosure. Therefore, the scope of the present disclosure should be interpreted to include all modifications or variations derived based on the technical concepts of the present disclosure, in addition to the embodiments disclosed herein.

Claims

1. In electronic devices, display; At least one processor comprising processing circuitry; and Contains memory that stores instructions, The above instructions, when executed by the at least one processor, cause the electronic device to: Recognize input devices in a virtual environment, Determine a virtual space for determining a target virtual object based on the properties of the above input device, Based on the above virtual space, the target virtual object is judged, Generate affordances corresponding to each of the above target virtual objects, and An electronic device that displays the above affordances according to a specified arrangement method.

2. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, Displaying an execution screen for the virtual environment including a plurality of virtual objects through the above display, Recognize the input device while displaying the above execution screen, Recognize an indicator (900) corresponding to the input device based on recognizing the input device, The input device is an electronic device including a device that supports user input for manipulating a virtual object in the virtual environment.

3. In the second paragraph, the affordance is an electronic device including a designated object that supports direct selection of the target virtual object among the plurality of virtual objects.

4. In the second paragraph, the instructions, when executed by the at least one processor, cause the electronic device to: Determine the properties of the above input device, An electronic device that produces a virtual space corresponding to the properties of the input device based on the above indicator.

5. In the fourth paragraph, the virtual space includes a virtual sphere formed with a diameter having a resolution value based on the properties of the input device centered on the indicator, The above resolution value is a first resolution value predefined for the input device, An electronic device comprising a second resolution value related to user noise generated by a user using the input device.

6. In the fifth paragraph, the instructions, when executed by the at least one processor, cause the electronic device to: An electronic device configured to calculate the resolution value based on the first resolution value and the second resolution value.

7. In the fifth paragraph, the instructions, when executed by the at least one processor, cause the electronic device to: Mapping the center of gravity of the virtual space based on the above indicator, An electronic device that forms a virtual space having the diameter based on the center of gravity.

8. In the fifth paragraph, the instructions, when executed by the at least one processor, cause the electronic device to: An electronic device that determines the size and / or spacing of the affordance based on the resolution value.

9. In the fourth paragraph, the instructions, when executed by the at least one processor, cause the electronic device to: Based on the above virtual space, the target virtual object included in the virtual space is determined among the plurality of virtual objects of the execution screen, Determine whether the target virtual object included in the above virtual space satisfies a specified condition, An electronic device that enters a guide generation mode for providing the affordance when the target virtual object satisfies the specified condition.

10. In paragraph 9, the specified conditions are: An electronic device including a condition that the target virtual object is a plurality of target virtual objects, at least two of which are target virtual objects, and the distance between the centers of gravity of each of the plurality of target virtual objects is less than or equal to a specified proximity distance.

11. In the 9th paragraph, the instructions, when executed by the at least one processor, cause the electronic device to: Determine whether the above virtual space contains multiple target virtual objects, When the above multiple target virtual objects are included, the center of gravity corresponding to each of the above multiple target virtual objects is determined, Determine the distance between the centers of gravity of each of the plurality of target virtual objects, Compare the distance between the centers of gravity with the specified proximity distance, An electronic device that causes the guide generation mode to be entered when at least one of the distances between the centers of gravity is less than or equal to the specified proximity distance.

12. In the second paragraph, the above-mentioned arrangement method is, A first method of providing an affordance by arranging it in a space that comes into contact with the target virtual object and the surface of the virtual space; An electronic device including a second method for providing affordances by arranging them in a list corresponding to each target virtual object outside the virtual space.

13. In the 12th paragraph, the instructions, when executed by the at least one processor, cause the electronic device to: When the target virtual object is over the virtual space, an object-based affordance is provided based on the space where the target virtual object contacts the surface of the virtual space. If the target virtual object is inside the virtual space, an object-based affordance is provided based on the space along the extension line from the target virtual object to the surface of the virtual space. An electronic device that provides a list-based affordance in an outer space of the virtual space when the distance between the target virtual objects is smaller than the radius of the virtual space.

14. In the method of operating an electronic device, Actions to recognize input devices in a virtual environment; An action of determining a virtual space for determining a target virtual object based on the properties of the input device; An action of judging the target virtual object based on the virtual space; An operation of generating an affordance corresponding to each of the above target virtual objects; and A method comprising an action of displaying the above affordance according to a specified arrangement method.

15. A non-transitory computer-readable medium storing instructions that, when executed by a processor of an electronic device, cause the processor to perform operations, The above instructions, when executed by the processor, cause the electronic device to: An action of recognizing an input device (720) in a virtual environment (400, 700); An operation of determining a virtual space (740) for determining a target virtual object (750) based on the properties of the input device (720); An operation of determining the target virtual object (750) based on the virtual space (740); An operation of generating an affordance (760) corresponding to each of the above target virtual objects (750); and A recording medium that performs an operation of displaying the above affordance (760) according to a specified arrangement method.

Images