Wearable electronic device for displaying virtual object, operating method thereof, and recording medium
The wearable electronic device dynamically adjusts the display position of virtual objects based on controller usage and position, addressing the issue of obstruction in conventional devices, thereby enhancing user experience by maintaining visibility during interactions.
Patent Information
- Application Number
- PCT/KR2024/006859
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-12
- Filing Date
- 2024-05-21
- Publication Date
- 2024-11-28
AI Technical Summary
Wearable electronic devices, such as AR glasses and VST devices, face challenges in displaying virtual objects in a manner that is not obstructed by controllers, leading to user inconvenience during interaction, as conventional devices fix the display position of virtual objects in a fixed area, making them hard to view when controlling the device with hands.
The wearable electronic device dynamically adjusts the display position of virtual objects based on the usage and position of controllers, such as moving the display to the controller's location when in use, ensuring the object is not obscured by the user's hand or controller, using sensors and communication circuits to determine controller state and position.
This solution enhances user experience by keeping virtual objects visible and accessible during interactions, preventing obstruction by controllers or hands, thus improving usability and visibility of virtual interfaces in augmented reality scenarios.
Smart Images

Figure KR2024006859_28112024_PF_FP_ABST
Abstract
Description
Wearable electronic device for displaying virtual objects, method of operation thereof, and recording medium
[0001] Embodiments of the present disclosure relate to a wearable electronic device for displaying a virtual object, a method of operating the same, and a recording medium.
[0002] The variety of services and additional features offered through wearable electronic devices, such as augmented reality glasses (AR glasses), video see-through (VST) devices, and head-mounted displays (HMDs), is steadily increasing. To enhance the utility of these devices and satisfy the needs of diverse users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices that offer diverse features and differentiate themselves from competitors. Consequently, the various functions offered through wearable electronic devices are also becoming increasingly sophisticated.
[0003] AR glasses or VST devices, worn on the user's body, can provide a realistic experience by displaying virtual images. AR glasses or VST devices can replace smartphones in various fields, such as gaming, entertainment, education, and social networking services (SNS). Through AR glasses or VST devices, users can receive realistic content and, through interaction, experience the sensation of being immersed in a virtual world.
[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.
[0005] According to one embodiment, a wearable electronic device may include a memory for storing instructions, a camera, communication circuitry, a display, and at least one processor.
[0006] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to display, through the display, a virtual first object representing a user interface in the first area.
[0007] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to determine whether the first controller is in use based on sensing information obtained from the first controller associated with the wearable electronic device via the communication circuit.
[0008] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to determine whether a first condition is satisfied that sets the display location of the virtual first object to be linked with the location of the first controller determined through the camera, if the first controller is determined to be in use.
[0009] In one embodiment, the first condition may be based on a distance between the virtual first object and the first controller.
[0010] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to display the virtual first object in a second area corresponding to a location of the first controller, if the first condition is determined to be satisfied.
[0011] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to change the display position of the virtual first object according to the changed position of the first controller when the position of the first controller is changed.
[0012] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to display the virtual first object in a third area corresponding to a location of the first controller identified through the camera based on a state in which the first controller is placed on the ground, if the first controller is determined to be not in use.
[0013] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to maintain the display location of the first object in the first area if the first condition is determined not to be satisfied while the first controller is in use.
[0014] According to one embodiment, a method of operating a wearable electronic device may include an operation of displaying a virtual first object representing a user interface in a first area through a display of the wearable electronic device.
[0015] According to one embodiment, a method of operating a wearable electronic device may include an operation of determining whether a first controller is in use based on sensing information obtained from a first controller associated with the wearable electronic device through a communication circuit of the wearable electronic device.
[0016] According to one embodiment, a method of operating a wearable electronic device may include an operation of checking whether a first condition is satisfied for setting a display position of the virtual first object to be linked with a position of the first controller confirmed through a camera of the wearable electronic device when the first controller is confirmed to be in use.
[0017] According to one embodiment, the operating method of the wearable electronic device may include an operation of displaying the virtual first object in a second area corresponding to the location of the first controller when the first condition is confirmed to be satisfied.
[0018] According to one embodiment, the method of operating the wearable electronic device is such that when the position of the first controller is changed, the display position of the virtual first object can be changed according to the changed position of the first controller.
[0019] According to one embodiment, a method of operating a wearable electronic device may include, when it is determined that the first controller is not in use, displaying the virtual first object in a third area corresponding to a position of the first controller confirmed through the camera based on a state in which the first controller is placed on the ground.
[0020] According to one embodiment, a method of operating a wearable electronic device may include an operation of maintaining a display position of the virtual first object in the first area when it is determined that the first condition is not satisfied while the first controller is in use.
[0021] According to one embodiment, a non-transitory recording medium may store instructions that can execute an operation of displaying a virtual first object representing a user interface in a first area through a display of a wearable electronic device.
[0022] According to one embodiment, a non-transitory recording medium may store instructions that can perform an operation of determining whether a first controller is in use based on sensing information obtained from a first controller associated with the wearable electronic device through a communication circuit of the wearable electronic device.
[0023] According to one embodiment, the non-transitory recording medium may store instructions that can execute an operation to determine whether a first condition is satisfied for setting the display position of the virtual first object to be linked with the position of the first controller determined through a camera of the wearable electronic device when the first controller is determined to be in use.
[0024] According to one embodiment, the non-transitory recording medium may store instructions that, if it is determined that the first condition is satisfied, can execute an operation of displaying the virtual first object in a second area corresponding to the location of the first controller.
[0025] According to one embodiment, the non-transitory recording medium may be configured such that when the position of the first controller is changed, the display position of the virtual first object may be changed according to the changed position of the first controller.
[0026] According to one embodiment, the non-transitory recording medium may store instructions that, when it is determined that the first controller is not in use, can execute an operation of displaying the virtual first object in a third area corresponding to the position of the first controller confirmed through the camera based on a state in which the first controller is placed on the ground.
[0027] According to one embodiment, the non-transitory recording medium may store instructions that can execute an operation of maintaining the display position of the virtual first object in the first area when the first condition is determined not to be satisfied while the first controller is in use.
[0028] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.
[0029] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to an embodiment of the present disclosure.
[0030] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device according to one embodiment.
[0031] FIG. 4A is a diagram illustrating a system including a wearable electronic device, a first controller, and a second controller according to one embodiment.
[0032] FIG. 4b is a diagram showing a screen displayed by a wearable electronic device according to one embodiment.
[0033] FIG. 4c is a drawing showing a state in which a first controller according to one embodiment is facing a first object.
[0034] FIG. 4d is a drawing showing a state in which the first controller according to one embodiment is placed on the ground.
[0035] FIG. 5 is a schematic block diagram of a system including a wearable electronic device, a first controller, and a second controller according to one embodiment.
[0036] FIG. 6 is a flowchart illustrating an operation of determining a display position of a first object based on whether a wearable electronic device according to one embodiment satisfies a first condition.
[0037] FIG. 7 is a flowchart illustrating an operation of determining a display position of a first object based on whether a wearable electronic device according to one embodiment satisfies a second condition.
[0038] FIG. 8A is a flowchart illustrating an operation of a wearable electronic device according to one embodiment of the present invention to determine a display position of a first object based on a distance between a first controller and a wearable electronic device and a distance between a second controller and a wearable electronic device.
[0039] FIG. 8B is a flowchart illustrating an operation of a wearable electronic device according to one embodiment of the present invention to determine a display position of a first object based on a frequency of use of a first controller and a frequency of use of a second controller.
[0040] FIG. 9 is a flowchart illustrating an operation of a wearable electronic device according to one embodiment of the present invention to adjust the arrangement of a first object and a second object.
[0041] FIG. 10 is a flowchart illustrating an operation of a wearable electronic device according to one embodiment of the present invention to adjust a display position of a first object.
[0042] FIG. 11A is a diagram showing a screen displayed through a display by a wearable electronic device according to one embodiment.
[0043] FIG. 11B is a diagram for explaining an operation of determining a display position of a first object depending on whether a wearable electronic device according to one embodiment satisfies a first condition.
[0044] FIG. 11c is a diagram for explaining an operation of determining a display position of a first object depending on whether a wearable electronic device according to one embodiment satisfies a first condition.
[0045] FIG. 11D is a diagram for explaining an operation of a wearable electronic device according to one embodiment of the present invention to determine a display position of a first object based on satisfaction of a first condition.
[0046] FIG. 12 is a drawing for explaining an operation of not changing the display position of a first object based on a wearable electronic device according to one embodiment not satisfying a first condition.
[0047] FIG. 13A is a diagram for explaining an operation of a wearable electronic device according to one embodiment of the present invention to determine a display position of a first object in an area corresponding to a position of a first controller.
[0048] FIG. 13b is a diagram for explaining an operation of maintaining the display position of the first object in the first area based on the wearable electronic device according to one embodiment not satisfying the second condition.
[0049] FIG. 14A and FIG. 14B are diagrams illustrating an operation of displaying an object that causes a wearable electronic device to display a second object according to one embodiment.
[0050] FIG. 14c and FIG. 14d are drawings for explaining an operation of adjusting the arrangement of a first object and a second object when a wearable electronic device according to one embodiment determines that the first object and the second object overlap.
[0051] FIG. 15 is a drawing showing a screen displayed through a display by a wearable electronic device according to one embodiment when the first controller and the third controller are not in use.
[0052] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components. In addition, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.
[0053] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).
[0054] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.
[0055] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.
[0056] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).
[0057] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).
[0058] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).
[0059] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.
[0060] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.
[0061] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).
[0062] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
[0063] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
[0064] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
[0065] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.
[0066] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.
[0067] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).
[0068] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
[0069] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).
[0070] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.
[0071] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).
[0072] In one embodiment, the antenna module (197) may generate a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.
[0073] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).
[0074] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.
[0075] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to an embodiment of the present disclosure.
[0076] Referring to FIG. 2, a wearable electronic device (200) according to one embodiment of the present disclosure may include at least one of a light output module (211), a display member (201), and a camera module (250).
[0077] According to one embodiment of the present disclosure, the light output module (211) may include a light source capable of outputting an image, and a lens for guiding the image to the display member (201). According to one embodiment of the present disclosure, the light output module (211) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).
[0078] According to one embodiment of the present disclosure, the display member (201) may include an optical waveguide (e.g., a waveguide). According to one embodiment of the present disclosure, an output image of an optical output module (211) incident on one end of the optical waveguide may be propagated inside the optical waveguide and provided to a user. According to one embodiment of the present disclosure, the optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide an output image of the optical output module (211) to a user's eye by using at least one diffractive element or reflective element.
[0079] According to one embodiment of the present disclosure, the camera module (250) can capture still images and / or moving images. According to one embodiment, the camera module (250) is disposed within a lens frame and can be disposed around the display member (201).
[0080] According to one embodiment of the present disclosure, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. According to one embodiment of the present disclosure, the first camera module (251) can periodically or aperiodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).
[0081] According to one embodiment of the present disclosure, the second camera module (253) can capture an external image.
[0082] According to one embodiment of the present disclosure, the third camera module (255) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. According to one embodiment of the present disclosure, the third camera module (255) can be used for 3 degrees of freedom (3DoF), 6DoF head tracking, position (space, environment) recognition, and / or movement recognition. According to one embodiment of the present disclosure, the second camera module (253) can also be used for hand detection and tracking, and user gesture recognition. According to one embodiment of the present disclosure, at least one of the first camera module (251) to the third camera module (255) can be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module can include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.
[0083] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device according to one embodiment.
[0084] Referring to FIGS. 3A and 3B, in one embodiment, camera modules (311, 312, 313, 314, 315, 316) and / or depth sensors (317) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be arranged on the first surface (310) of the housing.
[0085] In one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.
[0086] In one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the camera modules (311, 312) can also be used for hand detection and tracking, and user gesture.
[0087] In one embodiment, a depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (217), the camera modules (313, 314, 315, 316) may determine the distance to an object.
[0088] According to one embodiment, camera modules (325, 326) for facial recognition and / or a display (321) (and / or a lens) may be arranged on the second side (320) of the housing.
[0089] In one embodiment, the face recognition camera modules (325, 326) adjacent to the display may be used to recognize the user's face, or may recognize and / or track the user's two eyes.
[0090] In one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3A and 3B , the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2 .
[0091] As described above, according to one embodiment, the wearable electronic device (300) may have a form factor for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured to a body part of the user. The wearable electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.
[0092] FIG. 4A is a diagram illustrating a system including a wearable electronic device, a first controller, and a second controller according to one embodiment. FIG. 4B is a diagram illustrating a screen displayed by a wearable electronic device according to one embodiment.
[0093] Referring to FIG. 4A, according to one embodiment, a system (500) (e.g., the system (500) of FIG. 5) may include a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5), a first controller (511) (e.g., the first controller (511) of FIG. 5), and a second controller (512) (e.g., the second controller (512)). For example, the first controller (511) may be a pointing device or an input control device of the wearable electronic device (501). For example, the second controller (512) may be a keyboard or a mouse.
[0094] According to one embodiment, the wearable electronic device (501) can be worn by a user. For example, the wearable electronic device (501) can be implemented as either an augmented reality glass (AR glass) or a video see-through (VST) device. According to one embodiment, the wearable electronic device (501) can display a virtual object or a virtual execution screen through a display (560) (e.g., the display (560) of FIG. 5) while being worn by a user. According to one embodiment, the wearable electronic device (501) can adjust the position at which the virtual object or the virtual execution screen is displayed by a first controller (511) and / or a second controller (512).
[0095] Referring to FIG. 4B, a wearable electronic device (501) according to one embodiment may display a virtual first object (1110) (hereinafter, referred to as the first object (1110)) representing a user interface and a virtual execution screen (1121) representing an execution screen of an application through a display (560) while the wearable electronic device (501) is worn by a user. According to one embodiment, the wearable electronic device (501) may display the first object (1110) in a first area. For example, the first object (1110) and the execution screen (1121) may be augmented reality images displayed on a real area. In this case, the first area may be a part of the real area seen through the glasses. Alternatively, the first area may be a part of the real area displayed on the display (560) through a camera included in the wearable electronic device (501). Depending on the implementation, the first object (1110) and the execution screen (1121) may be virtual objects displayed on a virtual area. In this case, the first area may be a portion of the virtual area shown through the display (560).
[0096] According to one embodiment, the wearable electronic device (501) can change the display location of the first object (1110) displayed through the display (560) from the first area to an area corresponding to the location of one of the first controller (511) and the second controller (512). According to one embodiment, the wearable electronic device (501) can cause the display location of the first object (1110) to be linked to the location of the corresponding controller. For example, when a condition based on the distance (e.g., the distance shown through the glasses or the display (560)) between the virtual first object (1110) and the actual first controller (511) is satisfied while the first controller (511) is in use, the wearable electronic device (501) can change the display location of the first object (1110) from the first area to an area corresponding to the location of the first controller (511). For example, if a condition based on a state in which the first controller (511) is placed on the ground while the first controller (511) is not in use is satisfied, the wearable electronic device (501) may change the display location of the first object (1110) from the first area to an area corresponding to the location of the first controller (511). Depending on the implementation, according to one embodiment, the wearable electronic device (501) may not change the display location of the execution screen (1121) displayed through the display (560). For example, the wearable electronic device (501) may determine not to change the display location of a specific object based on a user's setting. For example, the wearable electronic device (501) may not change the display location of the execution screen (1121) while changing the display location of the first object (1110). Alternatively, the wearable electronic device (501) may change both the display position of the first object (1110) and the display position of the execution screen (1121), or may not change both.
[0097] Conventionally, wearable electronic devices can display a first object representing a user interface in a fixed first area. That is, conventional wearable electronic devices can display the first object in the fixed first area when the controller is not in use. At this time, the conventional wearable electronic device can control the functions or operations of the wearable electronic device according to a gesture or interaction using the user's hand. However, the user of the wearable electronic device could see the first object displayed in the fixed first area while it was obscured by the user's hand. Accordingly, the user of the wearable electronic device experienced the inconvenience of not being able to clearly see the first object when controlling the functions or operations of the wearable electronic device according to a gesture or interaction using the user's hand. Furthermore, even when the user of the wearable electronic device controlled the functions or operations of the wearable electronic device using a controller, the first object could be obscured by the controller.
[0098] According to one embodiment, the wearable electronic device (501) can change the display location of the first object (1110) from the first area where it is previously displayed to an area corresponding to the location of one of the first controller (511) and the second controller (512). Through this, the wearable electronic device (501) can display the first object (1110) without being covered by the user's hand when performing an interaction with the user's hand. In addition, the wearable electronic device (501) can display the first object (1110) without being covered by the controller (511 or 512) even when controlling a function or operation of the wearable electronic device (501) using the controller (511 or 512).
[0099] FIG. 4c is a drawing showing a state in which a first controller according to one embodiment is facing a first object.
[0100] Referring to FIG. 4C, according to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may change the display location of the first object (1110) from a first area to an area corresponding to the first controller (511) when a condition based on a distance between the first object (1110) and the first controller (511) is satisfied while the first controller (511) (e.g., the first controller (511) of FIG. 5) is in use.
[0101] According to one embodiment, a condition based on a distance between a first object (1110) and a first controller (511) may mean that one side of the first controller (511) faces the first object (1110) and that the one side of the first controller (511) is located within a specified distance from the first object (1110). For example, the one side of the first controller (511) may mean an upper end (502) of the first controller (511), and the other side of the first controller (511) may mean a lower end (503) of the first controller (511). For example, the lower end (503) of the first controller (511) may mean a part gripped by a hand (or finger) of a user wearing the wearable electronic device (501). However, this is an example, and one side of the first controller (511) is not limited to the upper part (502) of the first controller (511), and the other side of the first controller (511) is not limited to the lower part (503) of the first controller (511).
[0102] According to one embodiment, the shape and type of the first controller (511) illustrated in FIG. 4c are examples for convenience of explanation, and the technical concept of the present invention is not limited thereto. The shape and type of the first controller (511) can be applied in various ways.
[0103] FIG. 4d is a drawing showing a state in which the first controller according to one embodiment is placed on the ground.
[0104] Referring to FIG. 4D, according to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may change the display location of a first object (1110) from a first area to an area corresponding to the first controller (511) when a condition is satisfied based on a state in which the first controller (511) is placed on the ground while the first controller (511) (e.g., the first controller (511) of FIG. 5) is not in use.
[0105] According to one embodiment, a condition based on a state in which the first controller (511) is placed on the ground may mean a state in which one side of the first controller (511) is in contact with the ground and the other side of the first controller (511) is not in contact with the ground. For example, one side of the first controller (511) may mean the upper part (502) of the first controller (511), and the other side of the first controller (511) may mean the lower part (503) of the first controller (511). However, this is an example, and the one side of the first controller (511) is not limited to the upper part (502) of the first controller (511), and the other side of the first controller (511) is not limited to the lower part (503) of the first controller (511).
[0106] According to one embodiment, the shape and type of the first controller (511) illustrated in FIG. 4d are examples for convenience of explanation, and the technical concept of the present invention is not limited thereto. The shape and type of the first controller (511) can be applied in various ways.
[0107] FIG. 5 is a schematic block diagram of a system including a wearable electronic device and a first controller and a second controller according to one embodiment.
[0108] According to one embodiment, the system (500) may include a wearable electronic device (501), a first controller (511), and a second controller (512). However, the number and / or types of the controllers are not limited thereto. For convenience of explanation, the system (500) is hereinafter described as including a first controller (511) and a second controller (512). For example, the wearable electronic device (501) may be implemented as either an augmented reality glass (AR glass) or a video see-through (VST) device. For example, the first controller (511) may be a pointing device or an input control device of the wearable electronic device (501). The first controller (511) may also be collectively referred to as an input device of an augmented reality glass (AR glass), a video see-through (VST) device, or an extended reality (XR) device. For example, the second controller (512) may be a keyboard or a mouse. For example, when a specific condition is satisfied, the wearable electronic device (501) can change the display location of a virtual object displayed through the display (560) of the wearable electronic device (501) from the existing display area to an area corresponding to the location of one of the first controller (511) and the second controller (512).
[0109] According to one embodiment, the wearable electronic device (501) may include a memory (not shown), a processor (520), a display (560), a camera (580), and a communication circuit (590). According to one embodiment, the wearable electronic device (501) may be implemented in the same or similar manner as the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3.
[0110] The processor (520), display (560), camera (580), and communication circuit (590) may be electronically and / or operably coupled with each other by electronic components such as a communication bus (not shown).
[0111] In one embodiment, the hardware of the wearable electronic device (501) being operatively coupled may mean that a direct connection, or an indirect connection, is established between the hardware, either wired or wireless, such that the second hardware is controlled by the first hardware.
[0112] According to one embodiment, the processor (520) can control the overall operation of the wearable electronic device (501). For example, the processor (520) can be implemented identically or similarly to the processor (120) of FIG. 1.
[0113] According to one embodiment, when the wearable electronic device (501) is implemented as AR glasses, the display (560) may be implemented as a light output device (e.g., display member (201) of FIG. 2) for displaying content or objects through the glasses. Alternatively, when the wearable electronic device (501) is implemented as a VST device, the display (560) may be implemented as a device for displaying a screen on a display (e.g., display (321) of FIGS. 3A and 3B).
[0114] According to one embodiment, the processor (520) can communicate with the first controller (511) and the second controller (512) through the communication circuit (590). The processor (520) can communicate with the first controller (511) and the second controller (512) using a short-range communication technology (e.g., BLE (Bluetooth low energy) or WiFi) through the communication circuit (590).
[0115] According to one embodiment, the processor (520) may display a first object (e.g., the first object (1110) of FIG. 4) representing a user interface in the first area through the display (560). The first object (1110) refers to a virtual object. For example, the user interface may include a default menu or a universal menu provided by the wearable electronic device (501). For example, the universal menu may include a menu that may execute a function that can execute an application installed in the wearable electronic device (501), a function that can display devices (e.g., the first controller (511), the second controller (512)) that are communicatively connected to the wearable electronic device (501), and a function that can display the remaining battery level of the wearable electronic device (501). However, this is only an example, and the universal menu may include menus that provide various other functions.
[0116] According to one embodiment, the processor (520) can determine whether the first controller (511) and the second controller (512) are in use. According to one embodiment, if the processor (520) determines that the first controller (511) is not in use when it is determined that there is no movement for a specified period of time, the processor (520) can determine that the first controller (511) is not in use. According to one embodiment, the processor (520) can determine whether the first controller (511) is in use based on sensing information acquired through a sensor included in the first controller (511). For example, the sensing information may include acceleration sensor information and / or gyro sensor information. According to one embodiment, if the processor (520) determines that the first controller (511) is placed on the ground through the camera (580), the processor (520) can determine that the first controller (511) is not in use. In one embodiment, the processor (520) may determine that the second controller (512) is not in use if it determines that there is no input to the second controller (512) for a specified period of time. In one embodiment, the processor (520) may determine that the second controller (512) is not in use if it does not obtain input information for the wearable electronic device (501) from the second controller (512) for a specified period of time through the communication circuit (590).
[0117] According to one embodiment, the processor (520) can determine whether the first controller (511) is held by the user based on sensing information obtained through a grip sensor included in the first controller (511). However, this is an example, and the first controller (511) of the present invention can determine whether the first controller (511) is held by the user based on various sensors that can determine whether the first controller (511) is held by the user. According to one embodiment, the processor (520) can determine whether the first controller (511) is being used based on sensing information obtained through a grip sensor included in the first controller (511) and sensing information (e.g., acceleration sensor information and / or gyro sensor information) obtained through the sensors (e.g., acceleration sensor, gyro sensor) included in the first controller (511). For example, if the processor (520) determines that the first controller (511) is being held by the user and is moving, the processor (520) can determine that the first controller (511) is in use. For example, if the processor (520) determines that the first controller (511) is not being held by the user and is moving, the processor (520) can determine that the first controller (511) is not being used. According to one embodiment, the state in which the first controller (511) is not being held by the user and is moving may mean a state in which the user's body is in contact with the first controller (511) without holding the first controller (511), and the position of the first controller (511) is changed.
[0118] For example, the first controller (511) can obtain a first sensing value through a grip sensor included in the first controller (511). According to one embodiment, the processor (520) can determine that the user's body (e.g., hand or finger) is not close to the first controller (511) if the first sensing value is less than a first threshold value. According to one embodiment, the processor (520) can determine that the user's body (e.g., hand or finger) is close to or in contact with the first controller (511) if the first sensing value is greater than the first threshold value and less than a second threshold value. In this case, the second threshold value may mean a value greater than the first threshold value. According to one embodiment, the processor (520) can determine that the first controller (511) is gripped by the user's body (e.g., hand or finger) if the first sensing value is greater than the second threshold value. For example, the first threshold value may refer to a preset value for determining whether the user's body is in contact with or close to the first controller (511). The second threshold value may refer to a preset value for determining whether the user's body is grasping the first controller (511). The first threshold value and the second threshold value may be automatically set by the processor (520) or may be set by the user.
[0119] According to one embodiment, when the processor (520) determines that the first controller (511) is in use, it can check whether a first condition that sets the display position of the first object (1110) to be linked with the position of the first controller (511) is satisfied. The first condition may be based on a distance between the first object and the first controller (511). For example, setting the display position of the first object (1110) to be linked with the position of the first controller (511) may mean that when the position of the first controller (511) is changed, the display position of the first object is set to change according to the changed position of the first controller (511).
[0120] According to one embodiment, the processor (520) may determine that the first condition is satisfied when it is confirmed that one side of the first controller (511) faces the first object and that the one side is located within a first distance specified from the first object (1110). According to one embodiment, when it is confirmed that the first condition is satisfied, the processor (520) may display the first object (1110) in a second area corresponding to the location of the first controller (511) confirmed through the camera (580). At this time, the processor (520) may apply a visual effect that makes it seem as if the first object (1110) is stuck to the periphery of the first controller (511). For example, the one side of the first controller (511) may mean the upper part of the first controller (511) (e.g., the upper part (502) of the first controller in FIG. 4). However, this is an example, and one side of the first controller (511) is not limited to the upper part (502) of the first controller (511). For example, the specified first distance may mean a reference distance for setting the display position of the first object (1110) to be linked with the position of the first controller (511). For example, the specified first distance may mean a distance (e.g., 2 cm) at which one side of the first controller (511) is determined to be close to the first object (1110). The specified first distance may be automatically set by the processor (520) or may be set by the user.
[0121] According to one embodiment, the processor (520) may determine that the first condition is satisfied when it is confirmed that the display position of the first object (1110) moves within a second distance specified based on the first controller (511). According to one embodiment, the first object (1110) may include a handler (e.g., 1111 of FIG. 11C) that can change the display position of the first object (1110). For example, the handler (1111) may be a virtual object displayed on the first object (1110). According to one embodiment, the processor (520) may move the display position of the first object (1110) based on an input by the handler (e.g., a gesture or interaction of grabbing and moving the handler). According to one embodiment, the processor (520) can check whether the display position of the first object (1110) moves within a specified second distance based on the first controller (511). The input may include a drag input that moves the display position of the first object (1110) to a specific area corresponding to a specific location in the first area using the handler (1111). However, this is an example, and the input may include various types of input. For example, the specified second distance may mean a distance at which the first object (1110) is determined to be close to the first controller (511). The specified second distance may be set by the processor (520). Depending on the implementation, the specified first distance and the specified second distance may be set to be the same or different from each other.
[0122] According to one embodiment, when the processor (520) determines that the first condition is not satisfied while the first controller (511) is in use, the processor (520) may maintain the display location of the first object in the first area. According to one embodiment, when one side of the first controller (511) is not facing the first object (1110) or the one side is not located within a first distance specified from the first object (1110), the processor (520) may determine that the first condition is not satisfied. For example, when the processor (520) determines that the lower part of the first controller (511) (e.g., the lower part (503) of the first controller in FIG. 4) is facing the first object, the processor (520) may determine that the first condition is not satisfied.
[0123] According to one embodiment, if the processor (520) determines that the first controller (511) is not in use, it can determine through the camera (580) whether the state in which the first controller (511) is placed on the ground satisfies a specified second condition. According to one embodiment, if the processor (520) determines that the second condition is satisfied, it can display the first object (1110) in a third area corresponding to the position of the first controller (511). According to one embodiment, if the processor (520) determines that the state in which the first controller (511) is placed on the ground does not satisfy the specified second condition, it can maintain the display position of the first object (1110) in the first area. For example, the second condition may mean a state in which one side of the first controller (511) is in contact with the ground and the other side of the first controller (511) is not in contact with the ground. For example, one side of the first controller (511) may mean the upper part (502) of the first controller (511), and the other side of the first controller (511) may mean the lower part (503) of the first controller (511). However, this is an example, and the one side of the first controller (511) is not limited to the upper part (502) of the first controller (511), and the other side of the first controller (511) is not limited to the lower part (503) of the first controller (511).
[0124] According to one embodiment, the processor (520) can determine that the first controller (511) is not moving when the first controller (511) is not gripped by the user at a first point in time. According to one embodiment, if the processor (520) determines that the second condition is satisfied at the first point in time, the processor (520) can display the first object (1110) in a third area corresponding to the location of the first controller (511). According to one embodiment, the processor (520) can determine that the first controller (511) is moving when the first controller (511) is not gripped by the user at a second point in time subsequent to the first point in time. At this time, the processor (520) can determine whether the first controller (511) is moving while being placed on the ground and satisfying the specified second condition. When the processor (520) determines that the first controller (511) is in a state of moving while satisfying a designated second condition while being placed on the ground, the processor (520) can display the first object (1110) in a third area corresponding to the position of the first controller (511) at the time when the movement of the first controller (511) ends.
[0125] According to one embodiment, when the processor (520) determines that the first controller (511) and the second controller (512) are not in use, the processor (520) may determine the display location of the first object (1110) as a third area corresponding to the location of the first controller (511) or a fourth area corresponding to the location of the second controller (512). According to one embodiment, when the processor (520) determines that the second condition is satisfied, the processor (520) may determine the display location of the first object (1110) by comparing the frequency of use of the first controller (511) and the frequency of use of the second controller (512) or the distance between the first controller (511) and the wearable electronic device (501) and the distance between the second controller (512) and the wearable electronic device (501). According to one embodiment, when the processor (520) is determined to satisfy the second condition, the processor (520) may determine an area corresponding to the position of one of the preset controllers among the first controller (511) and the second controller (512) as the display position of the first object (1110).
[0126] For example, the processor (520) may display the first object in a third area corresponding to the position of the first controller (511) that is frequently used among the first controller (511) and the second controller (512). The frequency of use may refer to the frequency with which the first controller (511) and the second controller (512) are used as input devices for the wearable electronic device (501). For example, the processor (520) may display the first object (1110) in a third area corresponding to the position of the first controller (511) that is close to the wearable electronic device (501) among the first controller (511) and the second controller (512). For example, when the processor (520) presets the area corresponding to the position of the second controller (512) as the display position of the first object (1110), the processor (520) may display the first object (1110) in a fourth area corresponding to the position of the second controller (512). For example, if the processor (520) presets an area corresponding to the location of the first controller (511) as the display location of the first object (1110), the processor (520) may display the first object (1110) in a third area corresponding to the location of the first controller (511). Depending on the implementation, the processor (520) may display the first object in a fourth area corresponding to the location of the second controller (512) that is less frequently used among the first controller (511) and the second controller (512).
[0127] Depending on the implementation, the processor (520) may display the first object (1110) in a fourth area corresponding to a position of the second controller (512) that is far from the wearable electronic device (501) among the first controller (511) and the second controller (512). Depending on the implementation, the processor (520) may display the first object (1110) in a fourth area corresponding to a position of the second controller (512) that is less frequently used among the first controller (511) and the second controller (512).
[0128] According to one embodiment, the processor (520) may maintain the display location of the first object (1110) in the first area when it is determined that the first controller (511) and the second controller (512) are not in use and the state in which the first controller (511) is placed on the ground does not satisfy the specified second condition. Depending on the implementation, the processor (520) may also display the first object (1110) in the fourth area corresponding to the second controller (512) when it is determined that the first controller (511) and the second controller (512) are not in use and the state in which the first controller (511) is placed on the ground does not satisfy the specified second condition.
[0129] According to one embodiment, when a command to change the display location of a first object (1110) displayed in a third area to a fourth area is confirmed, the processor (520) may change the display location of the first object from the third area to the fourth area. For example, the processor (520) may confirm an input to display the first object (1110) in the fourth area. For example, the input may mean a drag input or a pinch input. However, this is an example, and embodiments of the present invention may include various inputs. According to one embodiment, when the input is confirmed, the processor (520) may change the display location of the first object (1110) from the third area to the fourth area.
[0130] According to one embodiment, the processor (520) may display the first object (1110) in the third area based on the display location of the first object (1110) being determined to be the third area, and may display the second object (1420) in the fourth area based on a command to display the second object (e.g., 1420 of FIG. 14B) in the fourth area. For example, the processor (520) may display an object (e.g., 1410 of FIG. 14A) that causes the second object (1420) to be displayed in the fourth area. When an input for the object (1410) that causes the second object (1420) to be displayed is confirmed, the processor (520) may display the second object (1420) in the fourth area. For example, the second object (1420) may include a widget or an execution screen of an application installed in the wearable electronic device (501). However, the second object is not limited to this.
[0131] According to one embodiment, a portion of a first object (1110) and a portion of a second object (1420) may be displayed overlapping each other through the display (560). For example, when a portion of a first object (1110) and a portion of a second object (1420) overlap each other, a portion of the first object (1110) may be displayed over a portion of the second object (1420), or a portion of the second object (1420) may be displayed over a portion of the first object (1110).
[0132] According to one embodiment, the processor (520) can check the gaze of a user wearing the wearable electronic device (501) for a specified period of time through the camera (580) when a part of the first object (1110) and a part of the second object (1420) are displayed overlapping each other. According to one embodiment, the processor (520) can adjust the arrangement of the first object (1110) or the second object (1420) based on the gaze of the user.
[0133] For example, if the processor (520) determines that the user's gaze is directed toward the first object (1110), the processor (520) may display a part of the first object (1110) on a part of the second object (1420). For example, if the processor (520) determines that the user's gaze is directed toward the second object (1420), the processor (520) may display a part of the second object (1420) on a part of the first object (110). For example, the processor (520) may adjust the position at which the first object (1110) or the second object (1420) is displayed so that the first object (1110) and the second object (1420) do not overlap. For example, the processor (520) may move the first object (1110) or the second object (1420) in any one of the up, down, left, and right directions so that the first object (1110) and the second object (1420) do not overlap.
[0134] According to an implementation, according to one embodiment, the processor (520) can determine that the body (e.g., hand) of the user wearing the wearable electronic device (501) is in proximity to the first controller (511) or the second controller (512). According to an implementation, according to one embodiment, the processor (520) can determine that the body (e.g., hand) of the user wearing the wearable electronic device (501) is in proximity to the first object (1110) or the second object (1420). According to one embodiment, when the processor (520) determines that the body of the user is in proximity to the first controller (511) or the first object (1110), the processor (520) can display a part of the first object (1110) on a part of the second object (1420). According to one embodiment, when the processor (520) determines that the user's body is in proximity to the second controller (512) or the second object (1420), the processor (520) may display a portion of the second object (1420) over a portion of the first object (110). For example, the processor (520) may determine that the user's body is in proximity to the first controller (511) or the first object (1110) when the user's body is located within a specified third distance relative to the first controller (511) or the first object (1110). For example, the processor (520) may determine that the user's body is in proximity to the second controller (512) or the second object (1420) when the user's body is located within a specified third distance relative to the second controller (512) or the second object (1420). Depending on the implementation, the specified third distance may be set to be the same as or different from the specified first distance or the specified second distance.
[0135] According to one embodiment, the processor (520) may confirm an input for the first object (1110) to adjust the display position of the first object (1110) from a position corresponding to a third area to a specific position. For example, the input for the first object (1110) may include a drag input for moving the first object (1110) to a specific position in the third area using the handler (1111). According to one embodiment, when the drag input for the first object (1110) is confirmed, the processor (520) may apply a visual effect indicating the movement of the first object (1110) to the first object (1110). For example, the visual effect may be applied until the drag input is released. For example, the visual effect may include an effect for blurring the first object (1110). According to one embodiment, when the input for the first object (1110) is released, the processor (520) may check a third distance between the position of the first object (1110) moved by the drag input and the position of the first controller (511). According to one embodiment, the processor (520) may check whether the third distance is greater than a specified fourth distance. For example, the specified fourth distance may be automatically set by the processor (520) or may be set by the user. According to one embodiment, when the processor (520) determines that the third distance is greater than the specified fourth distance, the processor (520) may maintain the display position of the first object (1110) in the third area. According to one embodiment, when the processor (520) determines that the third distance is not greater than the specified fourth distance, the processor (520) may display the first object (1110) at the position moved by the drag input. Depending on the implementation, the designated fourth distance may be set to be the same as or different from the designated first distance, the designated second distance, or the designated third distance.
[0136] According to one embodiment, operations performed by the first controller (511) may be performed by the second controller (512), and operations performed by the second controller (512) may be performed by the first controller (511).
[0137] The operations of the wearable electronic device (501) described in the drawings below may be performed by the processor (520). However, for convenience of explanation, the operations performed by the processor (520) will be described as being performed by the wearable electronic device (501).
[0138] FIG. 6 is a flowchart illustrating an operation of determining a display position of a first object based on whether a wearable electronic device according to one embodiment satisfies a first condition.
[0139] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.
[0140] According to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) can communicate with a first controller (511) of the wearable electronic device (501) (e.g., the first controller (511) of FIG. 5).
[0141] Referring to FIG. 6, according to one embodiment, in operation 611, the wearable electronic device (501) may display a first object (1110 of FIG. 11A) in a first area through a display (560) (e.g., the display 560 of FIG. 5). According to one embodiment, the first object (1110) may mean a virtual object representing a user interface. For example, the user interface may include a default menu or a universal menu provided by the wearable electronic device (501). For example, the universal menu may include a menu that may execute a function capable of executing an application installed in the wearable electronic device (501), a function capable of displaying devices (e.g., the first controller (511), the second controller (512)) that are communicatively connected to the wearable electronic device (501), and / or a function capable of displaying the remaining battery level of the wearable electronic device (501). However, this is just one example, and the universal menu can include many other functions.
[0142] According to one embodiment, in operation 613, the wearable electronic device (501) can check whether the first controller (511) (e.g., the first controller (511) of FIG. 5) is in use. According to one embodiment, the wearable electronic device (501) can obtain sensing information acquired by a sensor of the first controller (511) through a communication circuit (590) (e.g., the communication circuit (590) of FIG. 5). According to one embodiment, the wearable electronic device (501) can check whether the first controller (511) is in use based on the sensing information. For example, the sensing information can include acceleration sensor information and / or gyro sensor information. According to one embodiment, the wearable electronic device (501) may determine that the first controller (511) is not in use when the first controller (511) is confirmed to be placed on the ground through the camera (580) (e.g., the camera (580) of FIG. 5). According to one embodiment, the wearable electronic device (501) may determine that the first controller (511) is not in use when the first controller (511) is confirmed to be not moving.
[0143] According to one embodiment, the wearable electronic device (501) can determine whether the first controller (511) is held by the user and whether the first controller (511) is moving. For example, the wearable electronic device (501) can determine whether the first controller (511) is held by the user through a grip sensor included in the first controller (511). According to one embodiment, if the first controller (511) is not held by the user and the first controller (511) is moving, the wearable electronic device (501) can determine that the first controller (511) is not in use. According to one embodiment, if the first controller (511) is held by the user and the first controller (511) is moving, the first controller (511) can be determined to be in use.
[0144] According to one embodiment, if the wearable electronic device (501) determines that the first controller (511) is not in use (operation 613-No), in operation 621, the wearable electronic device (501) may display the first object in the third area corresponding to the location of the first controller (511). According to one embodiment, the wearable electronic device (501) may determine the location of the first controller (511) through a camera (580) (e.g., the camera (580) of FIG. 5 ).
[0145] According to one embodiment, when it is confirmed that the first controller (511) is in use (operation 613 - Yes), in operation 615, the wearable electronic device (501) may check whether a first condition that sets the display position of the first object (1110) to be linked with the position of the first controller (511) is satisfied. According to one embodiment, the first condition may be a condition based on the distance between the first object (1110) and the first controller (511). For example, when it is confirmed that the distance between the first object (1110) and the first controller (511) is within a specified distance, the wearable electronic device (501) may check or determine that the first condition is satisfied. According to one embodiment, the operation of setting the display position of the first object (1110) to be linked to the position of the first controller (511) may mean an operation of setting the display position of the first object (1110) to also change according to the changed position of the first controller (511) when the position of the first controller (511) is changed.
[0146] According to one embodiment, the wearable electronic device (501) can determine that the first condition is satisfied when it is confirmed that one side of the first controller (511) faces the first object (1110) and one side of the first controller (511) is located within a specified distance from the first object (1110). For example, the one side of the first controller (511) may refer to the upper part of the first controller (511) (e.g., the upper part (502) of the first controller of FIG. 4). However, this is an example, and the one side of the first controller (511) is not limited to the upper part (502). For example, the specified distance may refer to a reference distance for setting the display position of the first object (1110) to be linked with the position of the first controller (511). For example, the specified distance may refer to a distance (e.g., 2 cm) at which the first controller (511) is determined to be close to the first object. The specified distance may be set automatically by the processor (520) or set by the user.
[0147] According to one embodiment, the wearable electronic device (501) may determine that the first condition is satisfied when it is confirmed that the display position of the first object (1110) moves within a specified distance based on the first controller (511). According to one embodiment, the first object (1110) may include a handler (1111 of FIG. 11C) that can move the display position of the first object (1110). For example, the handler may be a virtual object displayed on the first object. According to one embodiment, the wearable electronic device (501) may move the display position of the first object (1110) based on an input by the handler (1111) (e.g., a gesture or interaction of grabbing and moving the handler).
[0148] According to one embodiment, when the wearable electronic device (501) determines that the first condition is satisfied (operation 615 - Yes), in operation 617, the wearable electronic device (501) may display the first object (1110) in the second area corresponding to the position of the first controller (511). According to one embodiment, when the position of the first controller (511) is changed, the display position of the first object (1110) may be changed according to the changed position of the first controller (511).
[0149] According to one embodiment, if the wearable electronic device (501) determines that the first condition is not satisfied (operation 615-No), in operation 619, the wearable electronic device (501) may maintain the display location of the first object (1110) as the first area. According to one embodiment, if the wearable electronic device (501) determines that one side of the first controller (511) is not facing the first object (1110) or one side of the first controller (511) is not located within a specified distance from the first object (1110), the wearable electronic device (501) may determine that the first condition is not satisfied. For example, if the wearable electronic device (501) determines that the lower portion of the first controller (511) (e.g., the lower portion (503) of the first controller in FIG. 4) is facing the first object (1110), the wearable electronic device (501) may determine that the first condition is not satisfied.
[0150] According to one embodiment, the wearable electronic device (501) may determine that the first condition is not satisfied if it is determined that the display position of the first object (1110) moves beyond a specified distance relative to the first controller (511) based on an input by the handler (1111) (e.g., a gesture or interaction of grabbing and moving the handler).
[0151] FIG. 7 is a flowchart illustrating an operation of determining a display position of a first object based on whether a wearable electronic device according to one embodiment satisfies a second condition.
[0152] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.
[0153] According to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may communicate with a first controller (511) of the wearable electronic device (501) (e.g., the first controller (511) of FIG. 5). According to one embodiment, the wearable electronic device (501) may display a first object (1110 of FIG. 11A) in a first area through a display (560) (e.g., the display (560) of FIG. 5).
[0154] Referring to FIG. 7, according to one embodiment, in operation 711, the wearable electronic device (501) can determine that the first controller (511) is not in use. For example, the wearable electronic device (501) can determine that the first controller (511) is not in use based on sensing information (e.g., acceleration information) obtained from the first controller (511). According to one embodiment, if the wearable electronic device (501) determines that the first controller (511) is placed on the ground through a camera (580) (e.g., the camera (580) of FIG. 5), the wearable electronic device (501) can also determine that the first controller (511) is not in use.
[0155] According to one embodiment, in operation 713, the wearable electronic device (501) can check whether a state in which the first controller (511) is placed on the ground satisfies a designated second condition through a camera (580) (e.g., the camera (580) of FIG. 5). According to one embodiment, the second condition may mean a state in which one side of the first controller (511) is in contact with the ground and the other side of the first controller (511) is not in contact with the ground. For example, the one side of the first controller (511) may mean an upper portion of the first controller (511) (e.g., an upper portion (502) of the first controller of FIG. 4), and the other side of the first controller (511) may mean a lower portion of the first controller (511) (e.g., a lower portion (503) of the first controller of FIG. 4). However, this is an example, and one side and the other side of the first controller (511) are not limited to the upper part (502) and the lower part (503).
[0156] According to one embodiment, if the wearable electronic device (501) determines that the second condition is satisfied (operation 713 - Yes), in operation 715, the wearable electronic device (501) may display the first object (1110) in a third area corresponding to the location of the first controller (511). For example, the wearable electronic device (501) may determine the location of the first controller (511) through the camera (580).
[0157] According to one embodiment, if the wearable electronic device (501) determines that the second condition is not satisfied (713-No), in operation 717, the display location of the first object (1110) can be maintained in the first area.
[0158] FIG. 8A is a flowchart illustrating an operation of a wearable electronic device according to one embodiment of the present invention to determine a display position of a first object by comparing a distance between a first controller and a wearable electronic device and a distance between a second controller and a wearable electronic device.
[0159] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.
[0160] According to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) can communicate with a first controller (511) (e.g., the first controller (511) of FIG. 5) and a second controller (512) (e.g., the second controller (512) of FIG. 5). According to one embodiment, the wearable electronic device (501) can display a first object (1110 of FIG. 11A) in a first area through a display (560) (e.g., the display (560) of FIG. 5). For example, the second controller (512) can be implemented as a keyboard or a mouse.
[0161] Referring to FIG. 8A, according to one embodiment, in operation 811, the wearable electronic device (501) may determine that the first controller (511) and the second controller (512) are not in use. For example, the wearable electronic device (501) may determine that the first controller (511) is not in use if it is determined that the first controller (511) has not moved for a specified period of time. For example, the wearable electronic device (501) may determine that the first controller (511) is not in use based on sensing information (e.g., acceleration sensor information and / or gyro sensor information) acquired by a sensor of the first controller (511). For example, if the wearable electronic device (501) confirms that the first controller (511) is placed on the ground through the camera (580) (e.g., the camera (580) of FIG. 5), the wearable electronic device (501) may confirm that the first controller (511) is not in use. For example, if the wearable electronic device (501) confirms that there is no input to the second controller (512) for a specified period of time, the wearable electronic device (501) may confirm that the second controller (512) is not in use. For example, if the wearable electronic device (501) does not obtain input information from the second controller (512) for a specified period of time through the communication circuit (590), the wearable electronic device (501) may confirm that the second controller (512) is not in use.
[0162] According to one embodiment, in operation 812, the wearable electronic device (501) can determine that the state in which the first controller (511) is placed on the ground satisfies the second condition. According to one embodiment, if the state in which the first controller (511) is placed on the ground does not satisfy the second condition, the wearable electronic device (501) can maintain the display location of the first object (1110) as the first area. Depending on the implementation, if the state in which the first controller (511) is placed on the ground does not satisfy the second condition, the wearable electronic device (501) can also determine the display location of the first object (1110) as the fourth area corresponding to the position of the second controller (512).
[0163] According to one embodiment, in operation 813, the wearable electronic device (501) can check a first distance between the wearable electronic device (501) and a first controller (511) and a second distance between the wearable electronic device (501) and a second controller (512).
[0164] In one embodiment, in operation 815, the wearable electronic device (501) may compare a first distance and a second distance. The wearable electronic device (501) may determine which of the first distance and the second distance is greater. For example, the wearable electronic device (501) may determine whether the first distance is greater than the second distance.
[0165] According to one embodiment, if the wearable electronic device (501) determines that the first distance is greater than the second distance (operation 815 - Yes), in operation 817, the wearable electronic device (501) may determine the display location of the first object (1110) as a fourth area corresponding to the location of the second controller (512).
[0166] According to one embodiment, if the wearable electronic device (501) determines that the first distance is not greater than the second distance (operation 815-No), in operation 819, the wearable electronic device (501) may determine the display location of the first object (1110) as a third area corresponding to the location of the first controller (511).
[0167] Through this, the wearable electronic device (501) can display the first object (1110) in an area corresponding to the position of the controller close to the wearable electronic device (501), thereby allowing the user wearing the wearable electronic device (301) to easily check the position of the object representing the user interface. Through this, the wearable electronic device (501) can provide convenience to the user.
[0168] According to an implementation, according to one embodiment, if the wearable electronic device (501) determines that the first distance is greater than the second distance, the display location of the first object (1110) may be determined as a third area corresponding to the location of the first controller (511). According to an implementation, according to one embodiment, if the wearable electronic device (501) determines that the first distance is not greater than the second distance, the display location of the first object (1110) may be determined as a fourth area corresponding to the location of the second controller (512).
[0169] FIG. 8B is a flowchart illustrating an operation of a wearable electronic device according to one embodiment of the present invention to determine a display position of a first object by comparing the usage frequency of a first controller and the usage frequency of a second controller.
[0170] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.
[0171] According to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may communicate with a first controller (511) (e.g., the first controller (511) of FIG. 5) and a second controller (e.g., the second controller (512) of FIG. 5). According to one embodiment, the wearable electronic device (501) may display a first object (1110 of FIG. 11A) in a first area through a display (560) (e.g., the display (560) of FIG. 5).
[0172] Referring to FIG. 8B, according to one embodiment, at operation 831, the wearable electronic device (501) can determine that the first controller (511) and the second controller (512) are not in use.
[0173] According to one embodiment, in operation 832, the wearable electronic device (501) can determine that the state in which the first controller (511) is placed on the ground satisfies the second condition. According to one embodiment, if the state in which the first controller (511) is placed on the ground does not satisfy the second condition, the wearable electronic device (501) can maintain the display location of the first object (1110) as the first area. Depending on the implementation, if the state in which the first controller (511) is placed on the ground does not satisfy the second condition, the wearable electronic device (501) can also determine the display location of the first object (1110) as an area corresponding to the position of the second controller (512).
[0174] According to one embodiment, in operation 833, the wearable electronic device (501) can check a first usage frequency of the first controller (511) and a second usage frequency of the second controller (512). For example, the first usage frequency and the second usage frequency may refer to the frequency with which the first controller (511) and the second controller (512) are used as input devices of the wearable electronic device (501).
[0175] According to one embodiment, in operation 835, the wearable electronic device (501) may compare a first usage frequency with a second usage frequency. The wearable electronic device (501) may determine which of the first and second usage frequencies is more frequent. For example, the wearable electronic device (501) may determine whether the first usage frequency is more frequent than the second usage frequency.
[0176] According to one embodiment, if the wearable electronic device (501) determines that the first usage frequency is greater than the second usage frequency (operation 835 - Yes), in operation 837, the wearable electronic device (501) may determine the display location of the first object (1110) to be a third area corresponding to the location of the first controller (511). At this time, the wearable electronic device (501) may display the first object in the third area corresponding to the location of the first controller (511).
[0177] According to one embodiment, if the wearable electronic device (501) determines that the first usage frequency is not greater than the second usage frequency (operation 835-No), in operation 839, the wearable electronic device (501) may determine the display location of the first object to be a fourth area corresponding to the location of the second controller (512). At this time, the wearable electronic device (501) may display the first object in the fourth area corresponding to the location of the second controller (512).
[0178] Through this, the wearable electronic device (501) can display the first object (1110) in an area corresponding to the location of a frequently used controller, thereby allowing a user wearing the wearable electronic device (301) to easily check the location of an object in the user interface.
[0179] According to an implementation, if the wearable electronic device (501) determines that the first usage frequency is greater than the second usage frequency, the display location of the first object may be determined as a fourth area corresponding to the location of the second controller (512). At this time, the wearable electronic device (501) may display the first object in the fourth area corresponding to the location of the second controller (512). According to an implementation, according to one embodiment, if the wearable electronic device (501) determines that the first usage frequency is not greater than the second usage frequency, the display location of the first object may be determined as a third area corresponding to the location of the first controller (511). At this time, the wearable electronic device (501) may display the first object in the third area corresponding to the location of the second controller (511).
[0180] FIG. 9 is a flowchart illustrating an operation of a wearable electronic device according to one embodiment of the present invention to adjust the arrangement of a first object and a second object.
[0181] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.
[0182] Referring to FIG. 9, according to one embodiment, in operation 911, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may display a first object (1110 of FIG. 11A) in a third area corresponding to a location of a first controller (511) (e.g., the first controller (511) of FIG. 5). For example, the first object (1110) may mean a user interface (e.g., a control menu).
[0183] According to one embodiment, in operation 913, the wearable electronic device (501) may display a virtual second object in a fourth area corresponding to the location of the second controller (512). The second object (1420 of FIG. 14B) may refer to an object different from the first object (1110). For example, the second object (1420) may include a widget or an application execution screen. However, the second object (1420) is not limited thereto.
[0184] According to one embodiment, in operation 915, the wearable electronic device (501) may display a first object (1110) and a second object (1420) together, such that a portion of the first object (1110) and a portion of the second object (1420) overlap each other. For example, when a portion of the first object (1110) and a portion of the second object (1420) overlap each other, a portion of the first object (1110) may be displayed over a portion of the second object (1420), or a portion of the second object (1420) may be displayed over a portion of the first object (1110).
[0185] According to one embodiment, in operation 917, the wearable electronic device (501) can check the gaze of a user wearing the wearable electronic device (501) for a specified period of time through a camera (580) (e.g., the camera (580) of FIG. 5 ).
[0186] According to one embodiment, in operation 919, the wearable electronic device (501) may adjust the arrangement of the first object (1110) or the second object (1420) based on the user's gaze. For example, if the wearable electronic device (501) determines that the user's gaze is directed toward the first object (1110), the wearable electronic device (501) may display a part of the first object (1110) over a part of the second object (1420). For example, if the wearable electronic device (501) determines that the user's gaze is directed toward the second object (1420), the wearable electronic device (501) may display a part of the second object (1420) over a part of the first object (110). For example, the wearable electronic device (501) may adjust the position at which the first object (1110) or the second object (1420) is displayed so that the first object (1110) and the second object (1420) do not overlap. For example, the wearable electronic device (501) can move the first object (1110) or the second object (1420) in any one of the up, down, left, or right directions so that the first object (1110) and the second object (1420) do not overlap.
[0187] According to an implementation, according to one embodiment, the wearable electronic device (501) can determine that the body (e.g., hand) of the user wearing the wearable electronic device (501) is in proximity to the first controller (511) or the first object (1110). According to one embodiment, when the wearable electronic device (501) determines that the body of the user is in proximity to the first controller (511) or the first object (1110), the wearable electronic device (501) can display a part of the first object (1110) on a part of the second object (1420). According to one embodiment, the wearable electronic device (501) can determine that the body (e.g., hand) of the user wearing the wearable electronic device (501) is in proximity to the second controller (512) or the second object (1420). According to one embodiment, when the wearable electronic device (501) determines that the user's body is in proximity to the second controller (512) or the second object (1420), the wearable electronic device (501) may display a portion of the second object (1420) on a portion of the first object (1110). For example, the wearable electronic device (501) may determine that the user's body is in proximity to the first controller (511) or the first object (1110) when the user's body is located within a specified distance relative to the first controller (511) or the first object (1110). The wearable electronic device (501) may determine that the user's body is in proximity to the second controller (512) or the second object (1420) when the user's body is located within a specified distance relative to the second controller (512) or the second object (1420).
[0188] FIG. 10 is a flowchart illustrating an operation of a wearable electronic device according to one embodiment of the present invention to adjust a display position of a first object.
[0189] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.
[0190] Referring to FIG. 10, according to one embodiment, in operation 1011, the wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may display a first object (1110 of FIG. 11A) in a third area corresponding to the position of the first controller (511) based on the second condition being satisfied that the first controller (511) (e.g., the first controller (511) of FIG. 5) is not in use and the first controller (511) is placed on the ground.
[0191] According to one embodiment, in operation 1013, the wearable electronic device (501) may confirm an input for moving the first object (1110) to adjust the display position of the first object (1110). According to one embodiment, the first object (1110) may include a handler (e.g., 1111 of FIG. 11C) that may change the display position of the first object (1110). For example, the input for the first object (1110) may include a drag input for moving the first object (1110) to a specific position in a third region using the handler (1111). According to one embodiment, when the drag input for the first object (1110) is confirmed, the wearable electronic device (501) may apply a visual effect indicating the movement of the first object (1110) to the first object (1110). For example, the visual effect may be applied until the drag input is released. For example, the visual effect may include an effect that blurs the first object (1110).
[0192] According to one embodiment, in operation 1015, when the drag input for the first object (1110) is released, the wearable electronic device (501) can check a third distance between the position of the first object (1110) moved by the drag input and the position of the first controller (511).
[0193] In one embodiment, at operation 1017, the wearable electronic device (501) may determine whether the third distance is greater than a specified distance. For example, the specified distance may be set by the processor (520).
[0194] According to one embodiment, if the wearable electronic device (501) determines that the third distance is greater than the specified distance (operation 1017 - Yes), in operation 1019, the display location of the first object (1110) can be maintained in the existing third area.
[0195] According to one embodiment, if the wearable electronic device (501) determines that the third distance is not greater than the specified distance (operation 1017-No), in operation 1021, the wearable electronic device (501) may display the first object (1110) at the location moved by the drag input.
[0196] FIG. 11A is a diagram showing a screen displayed through a display by a wearable electronic device according to one embodiment.
[0197] Referring to FIG. 11A, according to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may display a first object (1110) in a first area through a display (560) (e.g., the display (560) of FIG. 5). The first object (1110) may refer to a virtual object representing a user interface. For example, the user interface may include a default menu or a universal menu provided by the wearable electronic device (501). For example, the universal menu may include a menu that can execute a function that can execute an application installed on the wearable electronic device (501), a function that can display devices (e.g., the first controller (511) and the second controller (512) of FIG. 5) that are connected to the wearable electronic device (501) by communication, and a function that can display the remaining battery level of the wearable electronic device (501). However, this is only an example, and the universal menu may include menus that provide various other functions.
[0198] According to one embodiment, the wearable electronic device (501) may display a running screen (1121). For example, the running screen (1121) may mean a virtual running screen of an application installed on the wearable electronic device (501).
[0199] For example, the first object (1110) and the execution screen (1121) may be augmented reality images displayed on a real area. In this case, the first area may be a portion of the real area seen through the glasses. Alternatively, the first area may be a portion of the real area displayed on the display (560) through a camera included in the wearable electronic device (501) (e.g., the camera (580) of FIG. 5). Depending on the implementation, the first object (1110) and the execution screen (1121) may be virtual objects displayed on a virtual area. In this case, the first area may be a portion of the virtual area seen through the display (560).
[0200] FIG. 11B is a diagram for explaining an operation of determining a display position of a first object depending on whether a wearable electronic device according to one embodiment satisfies a first condition.
[0201] Referring to FIG. 11B, according to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may determine that the first controller (511) is in use based on sensing information (e.g., acceleration sensor information and / or gyro sensor information) obtained from the first controller (511) (e.g., the first controller (511) of FIG. 5). According to one embodiment, the wearable electronic device (501) may also determine that the first controller (511) is in use through a camera (580) (e.g., the camera (580) of FIG. 5).
[0202] According to one embodiment, the wearable electronic device (501) can determine whether a first condition is satisfied. According to one embodiment, if it is determined that the first condition is satisfied, the wearable electronic device (501) can link the display position of the first object (1110) with the position of the first controller (511). For example, the operation of linking the display position of the first object (1110) with the position of the first controller (511) may mean an operation of moving the display position of the first object (1110) from an area corresponding to the first position to an area corresponding to the second position when the first controller (511) is moved from the first position to the second position. The first condition may be a condition based on a distance between the first object (1110) and the first controller (511). For example, the wearable electronic device (501) can be determined to satisfy the first condition if it is confirmed that the upper part (502) of the first controller (511) is directed toward the first object (1110) and the upper part (502) of the first controller (511) is located within a specified distance from the first object (1110).
[0203] The wearable electronic device (501) may determine not to change the display position of a specific object according to the user's settings. According to one embodiment, the wearable electronic device (501) may preset so that the display position of the execution screen (1121) does not change while changing the display position of the first object (1110). According to one embodiment, the wearable electronic device (501) may preset so that the display position of the execution screen (1121) does not change even if it is confirmed that the upper portion (502) of the first controller (511) faces the execution screen (1121) and the upper portion (502) of the first controller (511) is located within a specified distance from the execution screen (1121).
[0204] FIG. 11c is a diagram for explaining an operation of determining a display position of a first object depending on whether a wearable electronic device according to one embodiment satisfies a first condition.
[0205] Referring to FIG. 11C, according to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may determine that the first controller (511) is in use based on sensing information (e.g., acceleration sensor information and / or gyro sensor information) obtained from the first controller (511) (e.g., the first controller (511) of FIG. 5). According to one embodiment, the wearable electronic device (501) may also determine that the first controller (511) is in use through a camera (580) (e.g., the camera (580) of FIG. 5).
[0206] According to one embodiment, the wearable electronic device (501) can determine whether the first controller (511) satisfies a first condition. The first condition may be a condition based on a distance between the first object (1110) and the first controller (511). According to one embodiment, the wearable electronic device (501) can determine that the first condition is satisfied when it is confirmed that the display position of the first object (1110) moves within a specified distance based on the first controller (511). According to one embodiment, the first object (1110) may include a handler (1111) that can change the display position of the first object (1110). For example, the handler (1111) may be a virtual object displayed on the first object (1110). According to one embodiment, the wearable electronic device (501) may move the display position of the first object (1110) based on an input by the handler (1111) (e.g., a gesture or interaction of grabbing and moving the handler). According to one embodiment, the processor (e.g., the processor (520) of FIG. 5) may determine whether the display position of the first object (1110) moves within a second distance specified based on the first controller (511). The input may include a drag input that moves the display position of the first object (1110) to a specific position in the first area using the handler (1111).
[0207] FIG. 11D is a diagram for explaining an operation of a wearable electronic device according to one embodiment of the present invention to determine a display location of a first object as an area corresponding to a location of a first controller.
[0208] Referring to FIG. 11D, according to one embodiment, when the wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) determines that the first controller (511) (e.g., the first controller (511) of FIG. 5) satisfies the first condition, the wearable electronic device (501) may change the display location of the first object (1110) from the first area to an area corresponding to the location of the first controller (511). At this time, the wearable electronic device (501) may apply a visual effect that makes it appear as if the first object (1110) is stuck around the first controller (511).
[0209] According to one embodiment, the wearable electronic device (501) may link the display position of the first object (1110) with the position of the first controller (511). According to one embodiment, the wearable electronic device (501) may move the display position of the first object (1110) from an area corresponding to the first position to an area corresponding to the second position based on the first controller (511) moving from the first position to the second position.
[0210] FIG. 12 is a drawing for explaining an operation of not changing the display position of a first object based on a wearable electronic device according to one embodiment not satisfying a first condition.
[0211] Referring to FIG. 12, according to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may determine that the first controller (511) is in use based on sensing information (e.g., acceleration information) obtained from the first controller (511) (e.g., the first controller (511) of FIG. 5). According to one embodiment, the wearable electronic device (501) may also determine that the first controller (511) is in use through a camera (580) (e.g., the camera (580) of FIG. 5).
[0212] According to one embodiment, the wearable electronic device (501) can determine whether a first condition is satisfied.
[0213] In one embodiment, the wearable electronic device (501) can determine that the lower portion (503) of the first controller (511) is facing the first object. In one embodiment, the wearable electronic device (501) can determine that the first condition is not satisfied based on the determination that the upper portion (502) of the first controller (511) is not facing the first object (1110).
[0214] According to one embodiment, the wearable electronic device (501) may maintain the display location of the first object (1110) in the first area based on the determination that the first condition is not satisfied.
[0215] FIG. 13A is a diagram illustrating an operation of a wearable electronic device according to one embodiment to determine a display location of a first object in an area corresponding to the location of a first controller. FIG. 13B is a diagram illustrating an operation of a wearable electronic device according to one embodiment to maintain the display location of a first object in the first area based on a failure to satisfy a second condition.
[0216] Referring to FIG. 13A, according to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) can determine that a first controller (511) (e.g., the first controller (511) of FIG. 5) and a second controller (512) (e.g., the second controller (512) of FIG. 5) are not in use. According to one embodiment, the wearable electronic device (501) can determine that the first controller (511) is not in use by sensing information (e.g., acceleration sensor information and / or gyro sensor information) acquired by a sensor of the first controller (511). According to one embodiment, the wearable electronic device (501) may determine that the first controller (511) is not in use if the first controller (511) is confirmed to be placed on the ground through a camera (580) (e.g., the camera (580) of FIG. 5). According to one embodiment, the wearable electronic device (501) may determine that the second controller (512) is not in use if it does not obtain input information from the second controller (512) for a specified period of time through a communication circuit (590) (e.g., the communication circuit (590) of FIG. 5).
[0217] According to one embodiment, the wearable electronic device (501) can determine, through the camera (580), whether the state in which the first controller (511) is placed on the ground satisfies the second condition. For example, the second condition may mean that the upper part (502) of the first controller (511) is in contact with the ground and the lower part (503) of the first controller (511) is not in contact with the ground.
[0218] According to one embodiment, the wearable electronic device (501) may display the first object (1110) in a third area corresponding to the location of the first controller (511) based on the determination that the second condition is satisfied.
[0219] Alternatively, according to one embodiment, the wearable electronic device (501) may determine the usage frequency of the first controller (511) and the usage frequency of the second controller (512) based on the determination that the second condition is satisfied. According to one embodiment, the wearable electronic device (501) may display the first object (1110) in the third area corresponding to the location of the first controller (511) based on the determination that the usage frequency of the first controller (511) is greater than the usage frequency of the second controller (512).
[0220] Alternatively, according to one embodiment, the wearable electronic device (501) may determine the distance between the first controller (511) and the wearable electronic device (501) and the distance between the second controller (512) and the wearable electronic device (501) based on the determination that the second condition is satisfied. According to one embodiment, the wearable electronic device (501) may display the first object (1110) in the third area corresponding to the position of the first controller (511) based on the determination that the distance between the first controller (511) and the wearable electronic device (501) is smaller than the distance between the second controller (512) and the wearable electronic device (501).
[0221] Referring to FIG. 13B, according to one embodiment, the wearable electronic device (501) can confirm that the state in which the first controller (511) is placed on the ground does not satisfy the second condition through the camera (580). According to one embodiment, the wearable electronic device (501) can confirm that the second condition is not satisfied based on the confirmation that the upper portion (502) and the lower portion (503) of the first controller (511) are in contact with the ground.
[0222] According to one embodiment, the wearable electronic device (501) may change the display location of the first object (1110) from the third area to the first area based on determining that the second condition is not satisfied.
[0223] FIG. 14A and FIG. 14B are diagrams for explaining an operation of displaying an object that causes a wearable electronic device to display a second object according to one embodiment.
[0224] Referring to FIG. 14A, according to one embodiment, the system (500) (e.g., the system (500) of FIG. 5 ) may further include a third controller (513). The third controller (513) may be a pointing device or an input control device of a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5 ). The third controller (513) may also be collectively referred to as an input device of an augmented reality glass (AR glass), a video see-through (VST) device, or an extended reality (XR) device. According to one embodiment, operations performed by the first controller (511) and the second controller (512) may be performed by the third controller (513).
[0225] According to one embodiment, the wearable electronic device (501) can communicate with a third controller (513) via a communication circuit (590) (e.g., the communication circuit (590) of FIG. 5).
[0226] According to one embodiment, the wearable electronic device (501) may display a first object (1110) in an area corresponding to a position of the first controller (511). According to one embodiment, the wearable electronic device (501) may display an object (1410) that causes a second object (e.g., the second object (1420) in FIG. 14B) to be displayed in an area corresponding to a position of the second controller (512) (e.g., the second controller (512) in FIG. 5). For example, the second object (1420) may include an execution screen or widget of an application.
[0227] According to one embodiment, the wearable electronic device (501) can determine that the state in which the third controller (513) is placed on the ground satisfies the second condition. According to one embodiment, the wearable electronic device (501) can determine that the second condition is satisfied when it is determined that one side (e.g., the upper side) of the third controller (513) is placed on the ground and the other side (e.g., the lower side) of the third controller (513) is not placed on the ground.
[0228] According to one embodiment, the wearable electronic device (501) may display an object (1430) that causes a second object (1420) to be displayed in an area corresponding to the location of the third controller (513).
[0229] Referring to FIG. 14B, according to one embodiment, when an input for an object (1430) is confirmed, the wearable electronic device (501) may display a second object (1420) in an area corresponding to the location of the third controller (513). According to one embodiment, a portion of the first object (1110) and a portion of the second object (1420) may be displayed overlapping each other.
[0230] FIG. 14c and FIG. 14d are drawings for explaining an operation of adjusting the arrangement of a first object and a second object when a wearable electronic device according to one embodiment determines that the first object and the second object overlap.
[0231] According to one embodiment, the wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) can detect the user's gaze for a specified period of time through the camera (580) (e.g., the camera (580) of FIG. 5) when it is determined that a portion of the first object (1110) and a portion of the second object (1420) overlap. According to one embodiment, the wearable electronic device (501) can adjust the arrangement of the first object (1110) or the second object (1420) based on the user's gaze.
[0232] Referring to FIG. 14C, according to one embodiment, the wearable electronic device (501) may display a part of the second object (1420) instead of displaying a part of the first object (1110) in an area where a part of the first object (1110) and a part of the second object (1420) overlap, based on determining that the user's gaze is on the second object (1420). According to one embodiment, the wearable electronic device (501) may also apply a visual effect to a part of the first object (1110) and / or a part of the second object (1420) in an area where a part of the first object (1110) and a part of the second object (1420) overlap. For example, a part of the first object (1110) may be displayed transparently in a background area of a part of the second object (1420). However, this is an example, and the technical idea of the present invention is not limited thereto. According to one embodiment, when a user's body (e.g., a hand) is positioned within a specified distance relative to a second object (1420) or a third controller (513), a portion of the first object (1110) may be displayed over a portion of the second object (1420) when a portion of the first object (1110) overlaps a portion of the second object (1420).
[0233] Referring to FIG. 14d, according to one embodiment, the wearable electronic device (501) can adjust the positions at which the first object (1110) and / or the second object (1420) are displayed so that the first object (1110) and the second object (1420) do not overlap.
[0234] FIG. 15 is a drawing showing a screen displayed through a display by a wearable electronic device according to one embodiment when the first controller and the third controller are not in use.
[0235] Referring to FIG. 15, according to one embodiment, a wearable electronic device (501) (e.g., the wearable electronic device (501) of FIG. 5) may be connected to a first controller (511) (e.g., the first controller (511) of FIG. 5) and a third controller (513) (e.g., the third controller (513) of FIG. 14a) through a communication circuit (590) (e.g., the communication circuit (590) of FIG. 5).
[0236] In one embodiment, the wearable electronic device (501) can confirm that the first controller (511) and the third controller (513) are not in use. In one embodiment, the wearable electronic device (501) can confirm that the first controller (511) and the third controller (513) are not in use based on the confirmation that the first controller (511) and the third controller (513) are placed on the ground through the camera (580) (e.g., the camera (580) of FIG. 5). According to one embodiment, the wearable electronic device (501) may determine that the first controller (511) and the third controller (513) are not in use based on sensing information (e.g., acceleration information) obtained from the first controller (511) and sensing information (e.g., acceleration sensor information and / or gyro sensor information) obtained from the third controller (513) through the communication circuit (590).
[0237] According to one embodiment, the wearable electronic device (501) can display a screen (1510) through the display (560) (e.g., the display (560) of FIG. 5) when the first controller (511) and the third controller (513) are not in use for a specified period of time. For example, the screen (1510) may be a screen preset by the processor (520) (e.g., the processor (520) of FIG. 5).
[0238] According to one embodiment, the wearable electronic device (501) can determine that the state in which the first controller (511) is placed on the ground satisfies the second condition. For example, the wearable electronic device (501) can determine that the state in which the first controller (511) is placed on the ground satisfies the second condition based on the determination that the upper portion (502) of the first controller (511) is in contact with the ground and the lower portion (503) is not in contact with the ground.
[0239] In one embodiment, the wearable electronic device (501) can determine that the state in which the third controller (513) is placed on the ground does not satisfy the second condition. For example, the wearable electronic device (501) can determine that the state in which the third controller (513) is placed on the ground does not satisfy the second condition based on the determination that the upper and lower portions of the third controller (513) are in contact with the ground.
[0240] According to one embodiment, the wearable electronic device (501) may apply a visual effect to the first controller (511) based on determining that the state in which the first controller (511) is placed on the ground satisfies the second condition. For example, the wearable electronic device (501) may display the first controller (511) in a blurred state. Depending on the implementation, according to one embodiment, the wearable electronic device (501) may also apply a visual effect to the third controller (513) based on determining that the state in which the third controller (513) is placed on the ground does not satisfy the second condition.
[0241] According to one embodiment, the wearable electronic device (501) can check whether the distance between the first controller (511) and the wearable electronic device (501) is within a specified distance. According to one embodiment, if the wearable electronic device (501) determines that the distance between the first controller (511) and the wearable electronic device (501) is within the specified distance, the wearable electronic device (501) can disable the visual effect applied to the first controller (511). According to one embodiment, if the wearable electronic device (501) determines that the distance between the first controller (511) and the wearable electronic device (501) is not within the specified distance, the wearable electronic device (501) can not disable the visual effect applied to the first controller (511).
[0242] According to one embodiment, a wearable electronic device (501) may include a memory for storing instructions, a camera (580), a communication circuit (590), a display (560), and at least one processor (520).
[0243] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display, through the display (560), a virtual first object representing a user interface in the first area.
[0244] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to determine whether the first controller (511) is in use based on sensing information obtained from the first controller (511) associated with the wearable electronic device (501) via the communication circuit (590).
[0245] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to check whether a first condition is satisfied that sets the display position of the virtual first object to be linked with the position of the first controller (511) confirmed through the camera (580) when the first controller (511) is confirmed to be in use.
[0246] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display the virtual first object in a second area corresponding to the location of the first controller (511) if the first condition is confirmed to be satisfied.
[0247] According to one embodiment, when the position of the first controller is changed, the display position of the virtual first object may be changed according to the changed position of the first controller (511).
[0248] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display the virtual first object in a third area corresponding to the position of the first controller (511) confirmed through the camera (580) based on a state in which the first controller is placed on the ground, if the first controller (511) is determined to be not in use.
[0249] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to maintain the display location of the virtual first object in the first area if the first condition is determined not to be satisfied while the first controller (511) is in use.
[0250] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to determine that the first condition is satisfied when it is determined that one side of the first controller (511) is directed toward the virtual first object and the one side of the first controller (511) is located within a specified distance from the virtual first object.
[0251] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to determine that the first condition is satisfied when it is determined that the display position of the virtual first object moves within a specified distance relative to the first controller (511).
[0252] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to determine, via the camera (580), whether the first controller (511) is placed on the ground if the first controller (511) is determined to be not in use.
[0253] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display the virtual first object in the third area if it is determined that the first controller is placed on the ground and that the first controller satisfies a specified second condition.
[0254] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display the virtual first object in the first area if it is determined that the first controller is placed on the ground and the first controller does not satisfy the second condition.
[0255] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to identify a second controller (512) associated with the wearable electronic device (501) that is communicatively connected via the communication circuit (590).
[0256] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display the virtual first object in the third area or in a fourth area corresponding to the location of the second controller if it is determined that the first controller (511) and the second controller (512) are not in use.
[0257] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to determine a first usage frequency of the first controller (511) and a second usage frequency of the second controller (512).
[0258] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display the virtual first object in the third area if the first usage frequency is determined to be greater than the second usage frequency.
[0259] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to determine a first distance between the wearable electronic device (501) and the first controller (511) and a second distance between the wearable electronic device (501) and the second controller (512).
[0260] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display the virtual first object in the third area if the first distance is determined to be shorter than the second distance.
[0261] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to change the display location of the virtual first object from the third area to the fourth area when a command to change the display location of the virtual first object to the fourth area is confirmed.
[0262] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display the virtual first object in the third area based on the display location of the virtual first object being determined to be the third area.
[0263] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to display the virtual second object in the fourth area based on a command to display the virtual second object in the fourth area.
[0264] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to adjust the arrangement between the first object and the virtual second object based on the user's gaze identified through the camera (580) when it is determined that at least a portion of the first object and at least a portion of the virtual second object overlap.
[0265] According to one embodiment, the instructions, when executed by the at least one processor (520), may cause the wearable electronic device (501) to adjust a position at which the first object or the second object is displayed so that the virtual first object and the virtual second object do not overlap.
[0266] According to one embodiment, a method of operating a wearable electronic device (501) may include an operation of displaying a virtual first object representing a user interface in a first area through a display of the wearable electronic device.
[0267] According to one embodiment, a method of operating a wearable electronic device (501) may include an operation of determining whether the first controller (511) is in use based on sensing information obtained from a first controller (511) related to the wearable electronic device (501) through a communication circuit (590) of the wearable electronic device (501).
[0268] According to one embodiment, the operating method of the wearable electronic device (501) may include an operation of checking whether a first condition is satisfied for setting the display position of the first object to be linked with the position of the first controller confirmed through a camera of the wearable electronic device when the first controller is confirmed to be in use.
[0269] According to one embodiment, the operating method of the wearable electronic device (501) may include an operation of displaying the virtual first object in a second area corresponding to the location of the first controller (511) when the first condition is confirmed to be satisfied.
[0270] According to one embodiment, the method of operating the wearable electronic device (501) is such that when the position of the first controller (511) is changed, the display position of the virtual first object can be changed according to the changed position of the first controller (511).
[0271] According to one embodiment, the method of operating the wearable electronic device (501) may include an operation of displaying the first object in a third area corresponding to the position of the first controller (511) confirmed through the camera based on a state in which the first controller (511) is placed on the ground, when it is determined that the first controller (511) is not in use.
[0272] According to one embodiment, the operating method of the wearable electronic device (501) may include an operation of maintaining the display position of the virtual first object in the first area when it is determined that the first condition is not satisfied while the first controller (511) is in use.
[0273] According to one embodiment, the method of operating the wearable electronic device (501) may include an operation of determining that the first condition is satisfied when it is confirmed that one side of the first controller (511) is directed toward the virtual first object and the one side of the first controller (511) is located within a specified distance from the virtual first object.
[0274] According to one embodiment, the operating method of the wearable electronic device (501) may include an operation of confirming that the first condition is satisfied when it is confirmed that the display position of the virtual first object moves within a specified distance based on the first controller (511).
[0275] According to one embodiment, the method of operating the wearable electronic device (501) may include an operation of checking whether the first controller (511) is placed on the ground through the camera when it is determined that the first controller (511) is not in use.
[0276] According to one embodiment, the method of operating the wearable electronic device (501) may include an operation of displaying the virtual first object in the third area when the first controller is placed on the ground and it is confirmed that the first controller satisfies a specified second condition.
[0277] According to one embodiment, the method of operating the wearable electronic device (501) may include an operation of displaying the virtual first object in the first area when the first controller is placed on the ground and it is determined that the first controller does not satisfy the second condition.
[0278] According to one embodiment, a method of operating a wearable electronic device (501) may include an operation of identifying a second controller (512) associated with the wearable electronic device (501) that is communicatively connected through the communication circuit (590).
[0279] According to one embodiment, the method of operating the wearable electronic device (501) may include an operation of displaying the virtual first object in the third area or a fourth area corresponding to the position of the second controller when it is determined that the first controller (511) and the second controller (512) are not in use.
[0280] According to one embodiment, the method of operating the wearable electronic device (501) may include an operation of checking a first usage frequency of the first controller (511) and a second usage frequency of the second controller (512).
[0281] According to one embodiment, the operating method of the wearable electronic device (501) may include an operation of displaying the virtual first object in the third area when the first usage frequency is determined to be greater than the second usage frequency.
[0282] According to one embodiment, a method of operating a wearable electronic device (501) may include an operation of checking a first distance between the wearable electronic device (501) and the first controller (511) and a second distance between the wearable electronic device (501) and the second controller (512).
[0283] According to one embodiment, the operating method of the wearable electronic device (501) may include an operation of displaying the virtual first object in the third area when the first distance is determined to be shorter than the second distance.
[0284] According to one embodiment, the operating method of the wearable electronic device (501) may include, when a command to change the display position of the virtual first object to the fourth area is confirmed, an operation of changing the display position of the virtual first object from the third area to the fourth area through the display (560).
[0285] According to one embodiment, the method of operating the wearable electronic device (501) may include an operation of displaying the first object in the third area based on determining the display location of the virtual first object as the third area.
[0286] According to one embodiment, a method of operating a wearable electronic device (501) may include an operation of displaying a virtual second object in the fourth area based on a command to display the virtual second object in the fourth area.
[0287] According to one embodiment, the operating method of the wearable electronic device (501) may include an operation of adjusting the arrangement between the virtual first object and the virtual second object based on the user's gaze confirmed through the camera (580) when it is confirmed that at least a portion of the virtual first object and at least a portion of the virtual second object overlap.
[0288] According to one embodiment, the method of operating the wearable electronic device (501) may include an operation of adjusting at least one of a display position of the first object or a position of the second object so that the virtual first object and the virtual second object do not overlap.
[0289] According to one embodiment, the non-transitory recording medium may store instructions that can execute an operation of displaying a virtual first object representing a user interface in a first area through a display (560) of a wearable electronic device (501).
[0290] According to one embodiment, a non-transitory recording medium may store instructions that can execute an operation of determining whether the first controller (511) is in use based on sensing information obtained from the first controller (511) associated with the wearable electronic device (501) through the communication circuit (590) of the wearable electronic device (501).
[0291] According to one embodiment, the non-transitory recording medium may store instructions that can execute an operation to check whether a first condition is satisfied for setting the display position of the virtual first object to be linked with the position of the first controller (511) confirmed through the camera (580) of the wearable electronic device (501) when the first controller (511) is confirmed to be in use.
[0292] According to one embodiment, the non-transitory recording medium may store instructions that, if it is determined that the first condition is satisfied, can execute an operation of displaying the virtual first object in a second area corresponding to the location of the first controller (511).
[0293] According to one embodiment, when the position of the first controller (511) is changed, the display position of the virtual first object can be changed according to the changed position of the first controller.
[0294] According to one embodiment, the non-transitory recording medium may store instructions that can execute an operation of displaying the virtual first object in a third area corresponding to the position of the first controller (511) confirmed through the camera (580) based on a state in which the first controller (511) is placed on the ground when the first controller (511) is determined to be not in use.
[0295] According to one embodiment, the non-transitory recording medium may store instructions that can execute an operation of maintaining the display position of the virtual first object in the first area when the first condition is determined not to be satisfied while the first controller (511) is in use.
[0296] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.
[0297] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used merely to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first) is referred to as “coupled” or “connected” to another (e.g., a second) component, with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.
[0298] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).
[0299] Various embodiments of the present document may be implemented as software (e.g., program (140)) including one or more commands stored in a storage medium (e.g., built-in memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101, 200, 300, 501)). For example, a processor (e.g., processor (120, 520)) of a machine (e.g., electronic device (101, 200, 300, 501)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the called at least one command. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'Non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.
[0300] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.
[0301] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.
Claims
1. In a wearable electronic device (101 of FIG. 1; 200 of FIG. 2; 300 of FIGS. 3a and 3b; 501 of FIG. 5), Memory for storing instructions (130 in Figure 1); Camera (180 in FIG. 1; 250 in FIG. 2; 311; 312; 313; 314; 315; 316 in FIG. 3; 350 in FIG. 5); Communication circuit (190 in Fig. 1; 590 in Fig. 5); display (160 in FIG. 1; 201 in FIG. 2; 321 in FIG. 3; 360 in FIG. 5); and comprising at least one processor (120 in FIG. 1; 520 in FIG. 5), The above instructions, when executed by the at least one processor, cause the wearable electronic device to: Through the above display, a virtual first object representing a user interface is displayed in the first area, Based on sensing information obtained from the first controller (511 of FIG. 5) related to the wearable electronic device through the above communication circuit, it is determined whether the first controller is in use, Based on the confirmation that the first controller is in use, it is confirmed whether a first condition is satisfied for setting the display position of the virtual first object to be linked with the position of the first controller confirmed through the camera, wherein the first condition is based on the distance between the first object and the first controller, Based on the confirmation that the first condition is satisfied, the virtual first object is displayed in a second area corresponding to the position of the first controller, and when the position of the first controller is changed, the display position of the virtual first object is changed according to the changed position of the first controller. Based on the determination that the first controller is not in use, the virtual first object is displayed in a third area corresponding to the position of the first controller confirmed through the camera based on the state in which the first controller is placed on the ground. A wearable electronic device set to maintain the display position of the virtual first object in the first area based on determining that the first condition is not satisfied while the first controller is in use.
2. In paragraph 1, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device configured to determine that the first condition is satisfied based on determining that one side of the first controller faces the virtual first object and that the one side of the first controller is located within a specified distance from the virtual first object.
3. In any one of paragraphs 1 and 2, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device set to determine that the first condition is satisfied based on confirming that the display position of the virtual first object moves within a specified distance relative to the first controller.
4. In any one of paragraphs 1 to 3, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: Based on the determination that the first controller is not in use, the camera determines whether the first controller is placed on the ground, A wearable electronic device configured to display the virtual first object in the third area based on the first controller being placed on the ground and the first controller satisfying a specified second condition.
5. In any one of paragraphs 1 to 4, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device configured to display the virtual first object in the first area based on the first controller being placed on the ground and determining that the first controller does not satisfy the second condition.
6. In any one of paragraphs 1 to 5, The above instructions, when executed by the one processor, cause the wearable electronic device to: Check the second controller (512 in Fig. 5) connected to the communication circuit above, A wearable electronic device configured to display the virtual first object in the third area or in a fourth area corresponding to the location of the second controller based on determining that the first controller and the second controller are not in use.
7. In any one of paragraphs 1 to 6, The above instructions, when executed by the one processor, cause the wearable electronic device to: Check the first usage frequency of the first controller and the second usage frequency of the second controller, A wearable electronic device set to display the virtual first object in the third area based on determining that the first usage frequency is greater than the second usage frequency.
8. In any one of paragraphs 1 to 7, The above instructions, when executed by the one processor, cause the wearable electronic device to: Checking a first distance between the wearable electronic device and the first controller and a second distance between the wearable electronic device and the second controller, A wearable electronic device set to display the virtual first object in the third area based on determining that the first distance is shorter than the second distance.
9. In any one of paragraphs 1 to 8, The above instructions, when executed by the one processor, cause the wearable electronic device to: A wearable electronic device configured to change the display location of the virtual first object from the third area to the fourth area based on confirmation of a command to change the display location of the virtual first object from the third area to the fourth area.
10. In any one of paragraphs 1 to 9, The above instructions, when executed by the one processor, cause the wearable electronic device to: A wearable electronic device configured to adjust the arrangement between the first object and the second object based on the user's gaze identified through the camera, based on determining that a portion of the first object and a portion of the second object overlap.
11. In any one of paragraphs 1 to 10, The above instructions, when executed by the one processor, cause the wearable electronic device to: A wearable electronic device configured to adjust at least one of the display position of the virtual first object or the display position of the virtual second object so that the virtual first object and the virtual second object do not overlap.
12. In a method of operating a wearable electronic device (101 of FIG. 1; 200 of FIG. 2; 300 of FIGS. 3a and 3b; 501 of FIG. 5), An operation of displaying a virtual first object representing a user interface in a first area through a display (160 of FIG. 1; 201 of FIG. 2; 321 of FIG. 3; 360 of FIG. 5) of the wearable electronic device; An operation of determining whether the first controller is in use based on sensing information obtained from a first controller (511 in FIG. 5) related to the wearable electronic device through a communication circuit (190 in FIG. 1; 590 in FIG. 5) of the wearable electronic device; An operation of checking whether a first condition is satisfied for setting the display position of the virtual first object to be linked with the position of the first controller confirmed through a camera (180 of FIG. 1; 250 of FIG. 2; 311; 312; 313; 314; 315; 316 of FIG. 3; 350 of FIG. 5) of the wearable electronic device based on the confirmation that the first controller is in use; wherein the first condition is based on the distance between the virtual first object and the first controller, An operation of displaying the virtual first object in a second area corresponding to the position of the first controller based on verifying that the first condition is satisfied, wherein when the position of the first controller is changed, the display position of the virtual first object is changed according to the changed position of the first controller; An operation of displaying the virtual first object in a third area corresponding to the position of the first controller confirmed through the camera based on the state in which the first controller is placed on the ground, based on the determination that the first controller is not in use; and A method of operating a wearable electronic device, comprising: an operation of maintaining the display position of the virtual first object in the first area based on determining that the first condition is not satisfied while the first controller is in use.
13. In paragraph 12, The action to check whether the first condition is satisfied is: A method of operating a wearable electronic device, comprising an operation of determining that the first condition is satisfied based on determining that one side of the first controller is directed toward the virtual first object and that the one side of the first controller is located within a specified distance from the virtual first object.
14. In any one of paragraphs 12 to 13, The action to check whether the first condition is satisfied is: A method of operating a wearable electronic device, comprising an operation of determining that the first condition is satisfied based on confirming that the display position of the virtual first object moves within a specified distance relative to the first controller.
15. In non-transitory recording media, An operation of displaying a virtual first object representing a user interface in a first area through a display (160 of FIG. 1; 201 of FIG. 2; 321 of FIG. 3; 360 of FIG. 5) of a wearable electronic device (101 of FIG. 1; 200 of FIG. 2; 300 of FIGS. 3A and 3B; 501 of FIG. 5); An operation of determining whether the first controller is in use based on sensing information obtained from a first controller (511 in FIG. 5) related to the wearable electronic device through a communication circuit (190 in FIG. 1; 590 in FIG. 5) of the wearable electronic device; An operation of determining whether a first condition is satisfied for setting the display position of the virtual first object to be linked with the position of the first controller confirmed through the camera of the wearable electronic device based on the determination that the first controller is in use; wherein the first condition is based on the distance between the virtual first object and the first controller, An operation of displaying the virtual first object in a second area corresponding to the position of the first controller based on verifying that the first condition is satisfied, wherein when the position of the first controller is changed, the display position of the virtual first object is changed according to the changed position of the first controller; An operation of displaying the virtual first object in a third area corresponding to the position of the first controller confirmed through the camera based on the state in which the first controller is placed on the ground, based on the determination that the first controller is not in use; and A recording medium storing at least one instruction capable of executing an operation of maintaining the display position of the virtual first object in the first area based on determining that the first condition is not satisfied while the first controller is in use.
Citation Information
Patent Citations
Video display device, video display method, and program
JP2016186727A
Display control device and display control method
JP2019125397A
Mobile terminal and operation control method thereof
KR1020120067445A
Head mounted display and method for controlling the same
KR1020180113301A
Drain-Trap
KR102341397B1
