Wearable device and method for displaying extended reality, and non-transitory computer-readable recording medium

Abstract

A wearable device is disclosed. The wearable device may identify the gaze of a user on the basis of an image of the eyes acquired through a sensor. The wearable device may identify a gesture of the user on the basis of an image acquired through a camera. The wearable device may display, through a display, a first three-dimensional image having binocular disparity. The wearable device may identify a specified gesture of the user through the camera. The wearable device, in response to the specified gesture being identified while the gaze of the user identified through the sensor is located in a non-execution area of the first three-dimensional image, may display a second three-dimensional image having binocular disparity, so as to overlap a partial area in the first three-dimensional image in which the gaze is located. The first three-dimensional image and the second three-dimensional image may be three-dimensional images for mutually different realities.

Description

Wearable device, method, and non-transitory computer-readable recording medium for displaying extended reality

[0001] The following descriptions relate to a wearable device, method, and non-transitory computer-readable recording medium for displaying extended reality.

[0002] Electronic devices are being developed that provide extended reality (XR) services that display computer-generated images. Furthermore, the XR services may include services for augmented reality (AR), virtual reality (VR), or mixed reality (MR), which combines AR and VR. The electronic devices may be wearable devices worn by a user. For example, the electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] A wearable device is disclosed. The wearable device may include a display arranged to face a user's eyes when worn. The wearable device may include a camera arranged to face the front of the user when worn. The wearable device may include a sensor arranged to face the user's eyes when worn. The wearable device may include at least one processor including a processing circuit. The wearable device may include a memory storing instructions and including one or more storage media. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify the user's gaze based on an image of the eye acquired through the sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a gesture of the user based on an image acquired through the camera. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a first three-dimensional image having binocular parallax through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a specified gesture of the user through the camera.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to, in response to the identification of the designated gesture while the user's gaze identified through the sensor is located in a non-executable area of ​​the first three-dimensional image, display a second three-dimensional image having binocular disparity by overlaying it on a portion of the first three-dimensional image where the gaze is located. The first three-dimensional image and the second three-dimensional image may be three-dimensional images for different realities.

[0004] A method is disclosed. The method may be performed by a wearable device including a display arranged to face a user's eyes when worn, a camera arranged to face the front of the user when worn, and a sensor arranged to face the user's eyes when worn. The method may include an operation of identifying a gaze of the user based on an image of the eye acquired through the sensor. The method may include an operation of identifying a gesture of the user based on an image acquired through the camera. The method may include an operation of displaying a first 3D image having binocular parallax through the display. The method may include an operation of identifying a designated gesture of the user through the camera. The method may include an operation of superimposing a second 3D image having binocular parallax on a portion of the first 3D image where the designated gesture is identified while the user's gaze identified through the sensor is located in a non-executable region of the first 3D image. The first three-dimensional image and the second three-dimensional image may be three-dimensional images for different realities.

[0005] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium can store a program including instructions. The instructions, when executed individually or collectively by at least one processor of a wearable device including a display arranged to face a user's eyes when worn, a camera arranged to face the front of the user when worn, and a sensor arranged to face the user's eyes when worn, can cause the wearable device to identify a gaze of the user based on an image of the eyes acquired through the sensor. The instructions, when executed individually or collectively by the at least one processor, can cause the wearable device to identify a gesture of the user based on an image acquired through the camera. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a first three-dimensional image having binocular parallax through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a designated gesture of the user through the camera. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to superimpose a second three-dimensional image having binocular parallax on a portion of the first three-dimensional image where the gaze of the user, identified through the sensor, is located in a non-execution area of ​​the first three-dimensional image, in response to identifying the designated gesture.The first three-dimensional image and the second three-dimensional image may be three-dimensional images for different realities.

[0006] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0007] FIG. 2A illustrates an example of an appearance of a wearable device according to one embodiment.

[0008] FIG. 2b illustrates an example of an appearance of a wearable device according to one embodiment.

[0009] FIG. 3 illustrates an example block diagram of a wearable device according to one embodiment.

[0010] Figure 4a is a diagram illustrating the field of view (FOV) of a user wearing a wearable device.

[0011] Figure 4b is a diagram illustrating a gesture of a user wearing a wearable device.

[0012] FIG. 5A illustrates an example of a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0013] FIG. 5b illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0014] FIG. 5c illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0015] FIG. 5d illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0016] FIG. 5e illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0017] FIG. 5f illustrates an example of a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0018] FIG. 5g illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0019] FIG. 5h illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0020] FIG. 6A illustrates an example of a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0021] FIG. 6b illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0022] FIG. 6c illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0023] FIG. 6d illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0024] FIG. 6e illustrates an example of a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0025] FIG. 6f illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0026] FIG. 7A illustrates an example of a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0027] FIG. 7b illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0028] FIG. 8A illustrates an example of transitions between three-dimensional images of a second mode displayed by a wearable device in one embodiment.

[0029] FIG. 8b illustrates an example of transitions between three-dimensional images of a second mode displayed by a wearable device in one embodiment.

[0030] Figure 9 is a flowchart illustrating the operation of a wearable device in one embodiment.

[0031] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0032] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0033] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0034] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0035] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0036] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0037] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0038] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0039] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0040] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0041] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0043] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0044] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0045] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0047] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0048] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0049] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, a loss coverage (e.g., 664 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip) for realizing URLLC.

[0050] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0051] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0052] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0053] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0054] FIGS. 2A and 2B illustrate an example of an exterior appearance of a wearable device (200) according to one embodiment. The wearable device (200) of FIGS. 2A and 2B may be included in the electronic device (101) of FIG. 1. An example of an exterior appearance of a first side (210) of a housing of the wearable device (200) according to one embodiment is illustrated in FIG. 2A, and an example of an exterior appearance of a second side (220) opposite to the first side (210) may be illustrated in FIG. 2B.

[0055] According to one embodiment, the wearable device (200) can be worn on a part of a user's body. The wearable device (200) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (200). For example, the wearable device (200) can output a virtual reality image to the user through at least one display (250) in response to a designated gesture of the user acquired through the motion recognition camera (240-2) of FIG. 2B.

[0056] FIGS. 2A and 2B illustrate an example of an exterior appearance of a wearable device (200) according to one embodiment. The wearable device (200) of FIGS. 2A and 2B may be included in the electronic device (101) of FIG. 1. An example of an exterior appearance of a first side (210) of a housing of the wearable device (200) according to one embodiment is illustrated in FIG. 2A, and an example of an exterior appearance of a second side (220) opposite to the first side (210) may be illustrated in FIG. 2B.

[0057] Referring to FIG. 2A, according to one embodiment, a first surface (210) of a wearable device (200) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (200) may further include a strap for fixing to a body part of the user, and / or one or more. A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (210). The wearable device (200) may further include a rubber or silicone packing formed on the first surface (210) to prevent interference by light (e.g., external light) different from light radiated from the first display (250-1) and the second display (250-2).

[0058] According to one embodiment, a wearable device (200) may include cameras (240-1, 240-2) for photographing and / or tracking both eyes of a user adjacent to the first display (250-1) and the second display (250-2), respectively. The cameras (240-1, 240-2) may be referred to as eye tracking (ET) cameras.

[0059] The gaze tracking camera (240-1, 240-2) can output data indicating the gaze of a user wearing the wearable device (200). For example, the wearable device (200) can detect the gaze from an image including the user's pupils obtained through the gaze tracking camera (240-1, 240-2). The gaze tracking camera (240-1) can be positioned toward the user's right eye, and the gaze tracking camera (240-2) can be positioned toward the user's left eye.

[0060] According to one embodiment, a wearable device (200) may include cameras (240-3, 240-4) for capturing and / or recognizing a user's face. The cameras (240-3, 240-4) may be referred to as FT (face tracking) cameras.

[0061] Referring to FIG. 2B, a camera (e.g., cameras (240-5, 240-6, 240-7, 240-8, 240-9, 240-10)) and / or a sensor (e.g., depth sensor (230)) for obtaining information related to the external environment of the wearable device (200) may be disposed on a second surface (220) opposite to the first surface (210) of FIG. 2A. For example, the cameras (240-5, 240-6, 240-7, 240-8, 240-9, 240-10) may be disposed on the second surface (220) to recognize an external object different from the wearable device (200).

[0062] For example, the cameras (240-5, 240-6, 240-7, 240-8) can capture images of the user's entire body or a part of the user's body, such as the user's torso, hands, or face. The cameras (240-5, 240-6, 240-7, 240-8) may be referred to as motion recognition cameras. For example, the processor (120) may provide a specific event to a screen provided on at least one display (250-1, 250-2) based on images captured of the user's entire body or a part of the user's body. The processor (120) may recognize the user's motion based on images acquired through the cameras (240-5, 240-6, 240-7, 240-8). The processor (120) can perform a function assigned (or mapped) to the user's action based on the user's action identified through the cameras (240-5, 240-6, 240-7, 240-8).

[0063] For example, using cameras (240-9, 240-10), the wearable device (200) can acquire images and / or videos to be transmitted to each of the user's eyes. For example, the cameras (240-9, 240-10) can capture real images or backgrounds to be aligned with virtual images to implement augmented reality or mixed reality content. For example, the camera (240-9) can be positioned on the second face (220) of the wearable device (200) to acquire an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. For example, the camera (240-10) can be positioned on the second face (220) of the wearable device (200) to acquire an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes.

[0064] According to one embodiment, the wearable device (200) may include a depth sensor (230) disposed on the second surface (220) to identify a distance between the wearable device (200) and an external object. Using the depth sensor (230), the wearable device (200) may obtain spatial information (e.g., a depth map) for at least a portion of a field of view (FoV) of a user wearing the wearable device (200).

[0065] Although not shown, a microphone may be placed on the second side (220) of the wearable device (200) to acquire sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0066] Although not shown, the wearable device (200) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (200) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (200) may identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable device (200) based on the IMU.

[0067] As described above, according to one embodiment, the wearable device (200) may have a form factor for being worn on a user's head. The wearable device (200), while worn on the head, may provide a user experience based on augmented reality, virtual reality, and / or mixed reality. Using cameras (240-5, 240-6, 240-7, 240-8, 240-9, 240-10) for recording video of an external space, the wearable device (200) and a server connected to the wearable device (200) (e.g., server (108) of FIG. 1) may provide an on-demand service and / or a metaverse service that provides video of a location and / or place selected by the user.

[0068] According to one embodiment, the wearable device (200) may display frames acquired through cameras (240-9, 240-10) on the first display (250-1) and the second display (250-2), respectively. The wearable device (200) may combine virtual objects within frames displayed through the first display (250-1) and the second display (250-2), which include real objects, to provide the user with a mixed user experience of real objects and virtual objects (e.g., video see-through (VST)). The wearable device (200) may change the virtual objects based on information acquired by the cameras (240-1, 240-2, 240-3, 240-4, 240-5, 240-6, 240-7, 240-8) and / or the depth sensor (230). For example, if a visual object corresponding to a real object and a virtual object at least partially overlap within the frame, the wearable device (200) may stop displaying the virtual object based on detecting a motion for interacting with the real object. By stopping displaying the virtual object, the wearable device (200) may prevent the visibility of the real object from being reduced as the visual object corresponding to the real object is obscured by the virtual object.

[0069] FIG. 3 illustrates an example of a block diagram of a wearable device (200) according to one embodiment. The wearable device (200) of FIG. 3 may correspond to the electronic device (101) of FIG. 1. The wearable device (200) of FIG. 3 may correspond to the wearable devices (200) of FIGS. 2A and 2B.

[0070] In one embodiment, the wearable device (200) may include a head-mounted display (HMD) that can be worn on a user's head.

[0071] Referring to FIG. 3, a wearable device (200) according to one embodiment may include at least one of a processor (310), a memory (315), a display (320), a camera (325), a sensor (330), or a communication circuit (335). The processor (310) of FIG. 3 may correspond to the processor (120) of FIG. 1. The memory (315) of FIG. 3 may correspond to the memory (130) of FIG. 1. The display (320) of FIG. 3 may correspond to the display module (160) of FIG. 1. The camera (325) of FIG. 3 may correspond to the camera module (180) of FIG. 1. The sensor (330) of FIG. 3 may correspond to the sensor module (176) of FIG. 1. The communication circuit (335) of FIG. 3 may correspond to the communication module (190) of FIG. 1.

[0072] In one embodiment, the processor (310), the memory (315), the display (320), the camera (325), the sensor (330), and the communication circuit (335) may be electrically and / or operatively connected to each other by electronic devices such as a communication bus (302). The type and / or number of hardware components included in the wearable device (200) is not limited to those illustrated in FIG. 3. For example, the wearable device (200) may include only some of the hardware components illustrated in FIG. 3. Elements (e.g., layers and / or modules) within the memory described below may be logically separated, but are not limited thereto.

[0073] According to one embodiment, the processor (310) of the wearable device (200) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (310) may be one or more. For example, the processor (310) may have a multi-core processor structure such as a dual core, quad core, or hexa core.

[0074] The memory (315) of the wearable device (200) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (310). The memory (315) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi media card (eMMC).

[0075] In one embodiment, the display (320) of the wearable device (200) can output visualized information to a user of the wearable device (200). For example, the display (320) can be controlled by a processor (310) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (320) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0076] In one embodiment, the camera (325) of the wearable device (200) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (325) may be arranged in the form of a two-dimensional array. The camera (325) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (325) may mean one (a) two-dimensional frame data acquired from the camera (325). For example, video data captured using the camera (325) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (325) according to a frame rate. The camera (325) may further include a flash light that is positioned toward the direction in which the camera (325) receives light and outputs light toward the direction.

[0077] According to one embodiment, the wearable device (200) may include a plurality of cameras, for example, cameras (325), arranged facing different directions. A first camera among the plurality of cameras may be referred to as a motion recognition camera (e.g., motion recognition cameras 240-5 to 240-8 of FIG. 2B ), and a second camera may be referred to as a gaze tracking camera (e.g., gaze tracking cameras 240-1 and 240-2 of FIG. 2A ). The wearable device (200) may identify a position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (200) may identify a gaze direction of a user wearing the wearable device (200) using an image acquired using the second camera. For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces. For example, the first camera may be arranged to face the front of the user when worn. For example, the second camera may be arranged to face the user's eyes when worn.

[0078] According to one embodiment, a sensor (330) of a wearable device (200) may generate electrical information that may be processed by a processor (310) and / or a memory (315) of the wearable device (200) from non-electrical information related to the wearable device (200). The information may be referred to as sensor data. The sensor (330) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (200), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (200).

[0079] In one embodiment, the communication circuit (335) of the wearable device (200) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (200) and an external electronic device. The communication circuit (335) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (335) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0080] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by the processor (310) of the wearable device (200) may be stored in the memory (315) of the wearable device (200). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (200) and / or the processor (310) may perform at least one of the operations of FIG. 9 when a set of multiple instructions distributed in the form of an operating system, firmware, driver, and / or application is executed. Hereinafter, the fact that an application is installed in a wearable device (200) may mean that one or more instructions provided in the form of an application are stored in a memory (315), and that the one or more applications are stored in a format executable by the processor (310) (e.g., a file having an extension specified by the operating system of the wearable device (200). As an example, an application may include a program and / or a library related to a service provided to a user.

[0081] Referring to FIG. 3, programs installed in the wearable device (200) may be classified into any one of different layers, including an application layer (340), a framework layer (350), and / or a hardware abstraction layer (HAL) (380), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (200) (e.g., the display (320), the camera (325), and / or the sensor (330)) may be classified within the hardware abstraction layer (380). The framework layer (350) may be referred to as an XR framework layer in that it includes one or more programs for providing an XR (extended reality) service. For example, FIG. 3 illustrates layers being divided within the memory (315), the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (315).

[0082] For example, within the framework layer (350), programs designed to target at least one of the hardware abstraction layer (380) and / or the application layer (340) (e.g., a position tracker (371), a space recognizer (372), a gesture tracker (373), and / or an eye tracker (374), a face tracker (375)) may be classified. Programs classified within the framework layer (350) may provide an executable API (application programming interface) based on other programs.

[0083] For example, within the application layer (340), programs designed to target users controlling wearable devices (200) may be classified. Examples of programs classified into the application layer (340) include, but are not limited to, an XR (extended reality) system user interface (UI) and / or an XR application (342). For example, programs (e.g., software applications) classified into the application layer (340) may call an API to cause execution of functions supported by programs classified into the framework layer (350).

[0084] For example, the wearable device (200) may display one or more visual objects on the display (320) for performing interaction with a user to use a virtual space based on the execution of the XR system UI (341). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (200) may provide a service for controlling functions available within a virtual space to the user based on the execution of the XR system UI (341).

[0085] Referring to FIG. 3, a lightweight renderer (343) and / or an XR plug-in (344) are illustrated as being included within the XR system UI (341), but are not limited thereto. For example, the XR system UI (341) may cause execution of functions supported by the lightweight renderer (343) and / or the XR plug-in (344) included within the framework layer (350).

[0086] For example, the wearable device (200) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (342). The lightweight renderer (343) may be referred to as a lightweight render pipeline from the perspective of defining a rendering pipeline that allows partial changes. The lightweight renderer (343) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (200) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (344). The XR plug-in (344) may be referred to as an open XR native client from the perspective of defining (or configuring) an entire rendering pipeline.

[0087] For example, the wearable device (200) may display a screen representing at least a portion of a virtual space on the display (320) based on the execution of the XR application (342). The XR plug-in (344-1) included in the XR application (342) may be referenced by the XR plug-in (344) of the XR system UI (341). Descriptions of the XR plug-in (341-1) that overlap with the description of the XR plug-in (344) may be omitted. The wearable device (200) may cause the execution of the virtual space manager (351) based on the execution of the XR application (342).

[0088] According to one embodiment, the wearable device (200) may provide a virtual space service based on the execution of the virtual space manager (351). For example, the virtual space manager (351) may include a platform (e.g., an Android platform) for supporting the virtual space service. The wearable device (200) may display the posture of a virtual object representing the user's posture rendered using data acquired through the sensor (330) on the display based on the execution of the virtual space manager (351). The virtual space manager (351) may be referred to as a composition presentation manager (CPM).

[0089] For example, the virtual space manager (351) may include a runtime service (352). As an example, the runtime service (352) may be referred to as an OpenXR runtime module. The wearable device (200) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (200) based on the execution of the runtime service (352). As an example, the wearable device (200) may be used to perform rendering for a virtual space service to a user based on the execution of the runtime service (352). For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the execution of the runtime service (352).

[0090] For example, the virtual space manager (351) may include a pass-through manager (353). Based on the execution of the pass-through manager (353), the wearable device (200) may display a screen representing a virtual space on the display (320), while another screen representing an actual space acquired through the camera (325) may be superimposed on at least a portion of the screen.

[0091] For example, the virtual space manager (351) may include an input manager (354). Based on the execution of the input manager (354), the wearable device (200) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (370). The wearable device (200) may initiate execution of at least one of the functions of the wearable device (200) using the acquired data.

[0092] For example, the perception abstraction layer (360) can be used for data exchange between the virtual space manager (351) and the perception service layer (370). From the perspective of being used for data exchange between the virtual space manager (351) and the perception service layer (370), the perception abstraction layer (360) can be referred to as an interface. For example, the perception abstraction layer (360) can be referred to as OpenPX. The perception abstraction layer (360) can be used for a perception client and a perception service.

[0093] According to one embodiment, the recognition service layer (370) may include one or more programs for processing data acquired from a sensor (330) (or a camera (325)). The one or more programs may include at least one of a position tracker (371), a space recognizer (372), a gesture tracker (373), an eye tracker (374), and / or a face tracker (373). The type and / or number of the one or more programs included in the recognition service layer (370) are not limited to those illustrated in FIG. 3.

[0094] For example, the wearable device (200) can identify the pose of the wearable device (200) using the sensor (330) based on the execution of the position tracker (371). The wearable device (200) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (200) using data acquired using the camera (325) and the IMU based on the execution of the position tracker (371). The position tracker (371) can be referred to as a head tracking (HeT) module.

[0095] For example, the wearable device (200) may be used to configure the surrounding environment of the wearable device (200) (or the user of the wearable device (200)) into a three-dimensional virtual space based on the execution of the space recognizer (372). The wearable device (200) may reconstruct the surrounding environment of the wearable device (200) in three dimensions using data acquired using the camera (325) based on the execution of the space recognizer (372). The wearable device (200) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (200) reconstructed in three dimensions based on the execution of the space recognizer (372). The space recognizer (372) may be referred to as a scene understanding (SU) module.

[0096] For example, the wearable device (200) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (200) based on the execution of the gesture tracker (373). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (330) based on the execution of the gesture tracker (373). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using a first camera (e.g., motion recognition cameras (240-5 to 240-8) of FIG. 2B) based on the execution of the gesture tracker (373). The gesture tracker (373) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0097] For example, the wearable device (200) may identify (or track) eye movements of a user of the wearable device (200) based on the execution of the gaze tracker (374). As an example, the wearable device (200) may identify eye movements of the user using data acquired from at least one sensor based on the execution of the gaze tracker (374). As an example, the wearable device (200) may identify eye movements of the user based on data acquired using a second camera (e.g., gaze tracking cameras 240-1 and 240-2 of FIG. 2A) and / or an infrared light emitting diode (IR LED) based on the execution of the gaze tracker (374). The gaze tracker (374) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0098] For example, the recognition service layer (370) of the wearable device (200) may further include a face tracker (375) for tracking the user's face. For example, the wearable device (103) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (375). The wearable device (200) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (375). As an example, the wearable device (200) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (375).

[0099] The wearable device (200) of FIGS. 4A and 4B may correspond to the electronic device (101) of FIG. 1. The wearable device (200) of FIGS. 4A and 4B may correspond to the wearable device (200) of FIGS. 2A and 2B. The wearable device (200) of FIGS. 4A and 4B may correspond to the wearable device (200) of FIG. 3. FIGS. 4A and 4B may be described with reference to FIGS. 1, 2A, 2B, and 3.

[0100] Referring to FIGS. 4A and 4B, a user (401) may wear a wearable device (200). The user (401) may view at least one content within a field of view (FOV, 410) through the wearable device (200). Here, the FOV (410) may refer to an area viewable according to the user's (401) gaze. The FOV (410) may refer to a display area of ​​the wearable device (200) that the user (401) may view.

[0101] In one embodiment, the processor (310) of the wearable device (200) may display an image for a designated mode through the display (320). Here, the designated mode may be a mode for virtual reality (VR) or a mode for augmented reality (AR) (or, video see through (VST)). In one embodiment, the mode for virtual reality may be a mode in which the wearable device (200) provides the user with an image generated through an application (or an image representing a virtual space reconstructed based on the image). In one embodiment, the mode for augmented reality may be a mode in which the wearable device (200) provides the user with an image acquired through the camera (325) (or, cameras (240-9, 240-10)) (or an image representing a real space reconstructed based on the image). In one embodiment, the image for the designated mode may be a stereoscopic image. A stereoscopic image may be an image that takes into account the binocular disparity of a user (401). The stereoscopic image may be an image that provides a three-dimensional (3D) sense of space to the user (401). Hereinafter, an image for a specified mode may be referred to as a three-dimensional image. Hereinafter, a mode for augmented reality may be referred to as an AR mode. Hereinafter, a mode for virtual reality may be referred to as a VR mode.

[0102] In one embodiment, the 3D image for a given mode may include a non-executable area (or non-interaction area) (or background area). For example, the non-executable area (or non-interaction area) (or background area) may be an area where interaction with the user (401) is restricted. For example, the non-executable area (or non-interaction area) (or background area) may be an area where a function to be executed based on an input (or gesture) of the user (401) is not specified.

[0103] For example, a 3D image for a given mode may include an executable area (or interaction area). For example, the executable area (or interaction area) may include a user interface (UI) (or a visual object) (or a virtual object). For example, the UI (or a visual object) (or a virtual object) may represent an object that can be positioned within the FOV (410) for providing information and / or interacting, such as an icon (or a widget) and / or a frame (or a window). The UI (or a visual object) (or a virtual object) may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. For example, the executable area (or interaction area) may be an area in which a function to be executed is specified based on an input (or a gesture) of a user (401). For example, the executable area (or interaction area) may be an area for triggering (or starting) the execution of a function of an application related to a UI (or a visual object) (or a virtual object) based on an input (or a gesture) of a user (401). For example, the executable area (or interaction area) may be an area for activating an application related to a UI (or a visual object) (or a virtual object) based on an input (or a gesture) of a user (401) (or displaying a three-dimensional image of the application).

[0104] In one embodiment, the processor (310) can identify an interaction (or user input) of a user (401). In one embodiment, the interaction (or user input) of the user (401) can include a gaze (420) of the user identified through the user's eyes (or pupils). In one embodiment, the interaction (or user input) of the user (401) can include a gesture (430) of the user identified through a part of the user's body (e.g., a hand, a finger, a head, a face). In one embodiment, the gesture (430) can include a movement and / or a posture of a part of the user's body.

[0105] For example, the processor (310) can identify an interaction between a user's (401) gaze (420) and a three-dimensional image displayed through the display (320). In one embodiment, the processor (310) can identify (or track) the gaze (420). For example, the processor (310) can identify (or track) the movement of the eye (or pupil) (or gaze (420)) of the user's (401) eye of the wearable device (200) based on the execution of the gaze tracker (374). Here, the gaze (420) of the user's (401) can be identified based on a sensor (330) and / or a camera (325) for ET (eye tracking) (e.g., cameras (240-1, 240-2)). For example, the gaze (420) of the user (401) can be identified based on a glint image. For example, the processor (310) can identify the gaze (420) of the user (401) based on the position of the eye (or pupil) (or iris) included in the glint images acquired through the cameras (240-1, 240-2).

[0106] For example, the processor (310) can identify an interaction based on a gesture (430) of the user (401). For example, the wearable device (200) can identify (or recognize) a pose and / or gesture (430) of a hand of the user (401) of the wearable device (200) based on the execution of the gesture tracker (373). For example, the wearable device (200) can identify a pose and / or gesture (430) of a hand of the user (401) based on data (or images) acquired using a sensor (330) and / or a camera (325) (or cameras (240-5 to 240-8)) for HaT (hand tracking) based on the execution of the gesture tracker (373). For example, the processor (310) can identify (or recognize) a pose and / or gesture (430) of a user's (401) hand corresponding to a change in a part of the user's (401) body included in images continuously acquired through cameras (240-5 to 240-8).

[0107] In one embodiment, the processor (310) may determine whether the interaction (or user input) of the user (401) is an interaction (or user input) for overlaying (or displaying) a display area for a designated other mode on a three-dimensional image for the designated mode. Hereinafter, the interaction (or user input) for overlaying (or displaying) a display area for a designated other mode on a three-dimensional image for the designated mode may be referred to as a designated interaction (or designated user input). Herein, the designated mode and the designated other mode may be different modes. For example, the designated mode may be an AR mode, and the designated other mode may be a VR mode. For example, the designated mode may be a VR mode, and the designated other mode may be an AR mode. However, the present invention is not limited thereto. For example, the designated mode and the designated other mode may be the same mode. For example, the designated mode and the designated other mode may be a VR mode. For example, a given mode and another given mode may be a VR mode that displays 3D images generated by different applications.

[0108] For example, the processor (310) can determine whether an interaction (or user input) indicated by the gaze (420) and gesture (430) of the user (401) is a designated interaction (or a designated user input). For example, the processor (310) can determine that the interaction (or user input) is a designated interaction (or a designated user input) when the gaze (420) of the user (401) is located in a designated area (e.g., a non-execution area (or a non-interaction area) (or a background area)) and the gesture (430) is a designated gesture (e.g., a pinch gesture, a squeeze gesture).

[0109] For example, the processor (310) may superimpose (or display) a display area for another mode specified in an area where the user's (401) gaze (420) is located on a three-dimensional image for the specified mode based on determining that the interaction (or user input) is a specified interaction (or user input).

[0110] For example, the processor (310) may overlay (or display) a display area for the AR mode on an area where the user's (401) gaze (420) is located while a three-dimensional image for the VR mode is displayed. Hereinafter, an operation of overlaying (or displaying) a display area for the AR mode on a three-dimensional image for the VR mode may be described with reference to FIGS. 5A to 5E.

[0111] For example, the processor (310) may overlay (or display) a display area for the VR mode on an area where the user's (401) gaze (420) is located while a three-dimensional image for the AR mode is displayed. Hereinafter, an operation of overlaying (or displaying) a display area for the VR mode on a three-dimensional image for the AR mode may be described with reference to FIGS. 6A to 6C.

[0112] For example, the processor (310) may overlay (or display) a display area for the VR mode on an area where the user's (401) gaze (420) is located while a three-dimensional image for the VR mode is displayed. Hereinafter, an operation of overlaying (or displaying) a display area for the VR mode on a three-dimensional image for the VR mode may be described with reference to FIGS. 7A, 7B, 8A, and 8B.

[0113] For example, the processor (310) may not overlap (or display) a display area for a designated other mode based on determining that the interaction (or user input) indicated by the user's (401) gaze (420) and gesture (430) is not a designated interaction (or a designated user input). For example, the processor (310) may determine that the interaction (or user input) is not a designated interaction (or a designated user input) if a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe) is input while the user's (401) gaze (420) is positioned in the designated other area (e.g., the executable area (or interaction area)). For example, the processor (310) may determine that an interaction (or user input) is not a designated interaction (or designated user input) if a gesture (e.g., a finger flick) other than a designated gesture (e.g., a pinch, a squeeze, a zoom out, a drag, or a swipe) is input while the user's (401) gaze (420) is positioned in a designated area (e.g., a non-execution area (or a non-interaction area) (or a background area)).

[0114] For example, the processor (310) may not overlap (or display) a display area for another mode specified in an area where the user's (401) gaze (420) is located based on determining that the interaction (or user input) is not a designated interaction (or designated user input). For example, the processor (310) may execute a function mapped to an area where the user's (401) gaze (420) is located and an input gesture (430) based on determining that the interaction (or user input) is not a designated interaction (or designated user input). Hereinafter, the operation of not overlapping (or displaying) a display area for another mode specified in an area may be described with reference to FIGS. 5F and 6E, but is not limited thereto. For example, the processor (310) may determine that an interaction (or user input) is a designated interaction (or a designated user input) when a designated other gesture is input regardless of the area where the user's (401) gaze (420) is located. For example, the designated other gesture may be a combination of two or more gestures. For example, the designated other gesture may be a gesture assigned for overlapping (or displaying) a display area for another mode designated in the application layer (340) and the framework layer (350). Hereinafter, an operation of overlapping (or displaying) a display area for another mode designated while the gaze (420) is located in the designated other area (e.g., an executable area (or interaction area)) may be described with reference to FIGS. 5g, 6f, 7a, and 7b.

[0115] FIG. 5A illustrates an example of a three-dimensional image displayed by a wearable device (200) in one embodiment. FIG. 5B illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment. FIG. 5C illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment. FIG. 5D illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment. FIG. 5E illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0116] FIGS. 5A to 5E can be described with reference to FIGS. 1, 2A, 2B, 3, 4A, and 4B.

[0117] Referring to FIG. 5A, the wearable device (200) can display a three-dimensional image of the first mode on the display (320). For example, the wearable device (200) can display a three-dimensional image (or an image having binocular disparity) (or a stereoscopic image) of the first mode in an area of ​​the FOV (410) of the user (401). For example, the first mode may be a mode for virtual reality (VR). However, the present invention is not limited thereto. For example, the first mode may be a mode for augmented reality (AR). Hereinafter, the mode for VR may be referred to as the first mode.

[0118] In one embodiment, the three-dimensional image of the first mode may include a non-executable area (501) and executable areas (511, 513, 515). In one embodiment, the three-dimensional image of the first mode may include a partial image (or layer) (or view object) (or window) (hereinafter, non-executable area image) for displaying the non-executable area (501) and a partial image (or layer) (or view object) (or window) (hereinafter, executable area image) for displaying each of the executable areas (511, 513, 515). In one embodiment, the three-dimensional image of the first mode may be an image having binocular parallax that is a composite of the non-executable area image and the executable area image.

[0119] In one embodiment, the non-executable area (501) may be an area to which a function for processing an interaction (or input) of a user (401) is not assigned. In one embodiment, the non-executable area (501) may be an area to which a function for inputting the interaction (or input) of the user (401) as a parameter is not assigned. In one embodiment, the non-executable area (501) may be an area to which the interaction (or input) of the user (401) is not assigned a destination (or application) as a parameter. Hereinafter, the interaction (or input) of the user (401) may be referred to as a gesture (430). However, the interaction (or input) of the user (401) is not limited to a gesture. Depending on the embodiment, the interaction (or input) of the user (401) may also include input (e.g., touch input) obtained through the input module (150) and / or the display module (160). Depending on the embodiment, the interaction (or input) of the user (401) may also include voice input (e.g., input based on a sound signal) obtained through the audio module (170).

[0120] In one embodiment, the executable area (511, 513, 515) may be an area to which a function for processing an interaction (or input) of a user (401) is assigned. In one embodiment, the executable area (511, 513, 515) may be an area to which a function in which a gesture (430) is input as a parameter is assigned. In one embodiment, the executable area (511, 513, 515) may be an area to which a destination (or application) in which the gesture (430) is input as a parameter is assigned. In one embodiment, the function may include, but is not limited to, executing an application related to the executable area (511, 513, 515) and / or executing a service provided through the application.

[0121] Referring to FIG. 5A, the wearable device (200) can identify a gesture (430) while the gaze (420) identified through a sensor (330) and / or a camera (325) for eye tracking (ET) (e.g., cameras (240-1, 240-2)) is located in a non-execution area (501). For example, the wearable device (200) can identify a gesture (430) through a sensor (330) and / or a camera (325) for hand tracking (HaT) (or cameras (240-5 to 240-8)) while the gaze (420) is located in a non-execution area (501).

[0122] In one embodiment, the wearable device (200) can determine whether the gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe).

[0123] In one embodiment, the wearable device (200) may display (or superimpose) a display area of ​​a second mode on a three-dimensional image of a first mode based on determining that a gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe). For example, the second mode may be a mode for AR, but is not limited thereto. For example, the second mode may be a mode for virtual reality (VR). Hereinafter, a mode that provides a service for AR may be referred to as the second mode.

[0124] In one embodiment, displaying a display area of ​​the second mode on a three-dimensional image of the first mode may include synthesizing a portion (or a portion corresponding to the display area) of an image for displaying a three-dimensional image of the second mode onto a portion of an image for displaying a three-dimensional image of the first mode. In one embodiment, displaying a display area of ​​the second mode on a three-dimensional image of the first mode may include superimposing a portion (or a portion corresponding to the display area) of an image for displaying a three-dimensional image of the second mode onto a portion of an image for displaying a three-dimensional image of the first mode.

[0125] For example, referring to FIG. 5B, the wearable device (200) may display a display area (520) of a second mode on a three-dimensional image of a first mode based on determining that a gesture (430) is a designated gesture. In one embodiment, the display area (520) of the second mode may include at least a portion of an image representing a real space acquired through a camera (325) for video see through (VST) (e.g., cameras (240-9, 240-10)).

[0126] For example, the wearable device (200) may change the display of the display area (520) based on a user interaction (e.g., a gesture (430) and / or a gesture different from the gesture (430)) while displaying the display area (520) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may enlarge the display area (520) to correspond to the state (e.g., the holding time and / or the intensity) of the gesture (430) while displaying the display area (520) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may enlarge the display area (520) in a specified direction (521) based on the state (e.g., the holding time and / or the intensity) of the gesture (430). For example, the wearable device (200) may enlarge the display area (520) in a designated direction (521) within a maximum size (530) based on the state (e.g., holding time and / or intensity) of the gesture (430). For example, the designated direction (521) may be a direction away from the line of sight (420), but is not limited thereto. For example, the wearable device (200) may change the display of the display area (520) based on another gesture (e.g., a gesture input subsequent to or substantially simultaneously with the gesture (430) while displaying the display area (520) in a second mode on a three-dimensional image in a first mode. For example, the gesture subsequent to the gesture (430) may be a gesture identified through the same body part as the gesture (430). For example, a gesture input substantially simultaneously with the gesture (430) may be a gesture identified through a body part different from the gesture (430).

[0127] Referring to FIG. 5B, as the display area (520) is displayed (or enlarged), the display area (520) may be displayed over executable areas (511, 513) other than the non-executable area (501). For example, as the display area (520) is displayed (or enlarged), the display area (520) may be composited with executable areas (511, 513, 515) on the three-dimensional image of the first mode. For example, as the display area (520) is displayed (or enlarged), the display area (520) may be overlapped with executable areas (511, 513, 515) on the three-dimensional image of the first mode.

[0128] For example, referring to FIG. 5C, the wearable device (200) can maintain the display of the display area (525) based on the display area (525) being expanded to the maximum size. For example, the wearable device (200) can maintain the display of the display area (525) regardless of a change (or release) of a user interaction (e.g., gesture (430)). For example, referring to FIG. 5C, even after the display area (525) is expanded to the maximum size and the gesture (430) is released, the wearable device (200) can maintain the display of the display area (525).

[0129] For example, the wearable device (200) may change the display of the display area (520) based on the state of the gaze (420) (e.g., movement (or motion) of the gaze (420)) while displaying the display area (520) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may move the display area (520) based on the movement (or motion) of the gaze (420). For example, the wearable device (200) may move the display area (520) based on the movement (or motion) of the gaze (420) that occurs after the display area (520) is enlarged to the maximum size (530). However, the present invention is not limited thereto. The wearable device (200) can move the display area (520) based on the movement (or motion) of the gaze (420) that occurs while the display area (520) is within the maximum size (530).

[0130] For example, referring to FIG. 5c, the wearable device (200) can move the maximum size display area (525) in the direction (550, 555) in which the gaze (420) moves based on the movement (or motion) of the gaze (420). For example, while the wearable device (200) moves the display area (525) in the direction (550, 555) in which the gaze (420) moves, the wearable device (200) can change (or move) a part of the image for displaying the 3D image of the second mode shown in the display area (525). For example, referring to FIG. 5d, based on the direction (550) in which the gaze (420) moves, the wearable device (200) can change (or move) a part of the image for displaying the 3D image of the second mode shown through the display area (525). For example, referring to FIG. 5e, based on the direction (555) in which the gaze (420) moves, the wearable device (200) can change (or move) a portion of the image for displaying the three-dimensional image of the second mode shown through the display area (525).

[0131] For example, the wearable device (200) may stop displaying the display area (520) of the second mode based on a user interaction (e.g., a gesture (430) and / or a gesture different from the gesture (430)) while displaying the display area (520) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may identify that the gesture (430) has been released while displaying the display area (520) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may stop displaying the display area (520) based on identifying that the gesture (430) has been released. For example, the wearable device (200) may stop displaying the display area (520) based on identifying that the gesture (430) is released while the display area (520) is displayed to be less than the maximum size (530). For example, the wearable device (200) may identify a gesture different from the gesture (430) while displaying the display area (520) of the second mode (or the maximum size display area (525)) on the three-dimensional image of the first mode. For example, the wearable device (200) may stop displaying the display area (520) based on identifying a gesture different from the gesture (430). Here, the gesture different from the gesture (430) may be a gesture for selecting a button (540) for stopping displaying the display area (520) as exemplified through FIGS. 5C to 5D, but is not limited thereto. A gesture other than the gesture (430) may be a gesture assigned to stop display of the display area (520).

[0132] As described above, the wearable device (200) can overlap (or display) a display area for the AR mode while a 3D image for the VR mode is displayed. Accordingly, the wearable device (200) can improve convenience by allowing the user to check the external mode through a portion of the area for the VR mode without having to switch from the VR mode to the AR mode to check the external mode.

[0133] FIG. 5f illustrates an example of a three-dimensional image of a first mode displayed by a wearable device in one embodiment. FIG. 5g illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0134] FIGS. 5F and 5G can be described with reference to FIGS. 1, 2A, 2B, 3, 4A, 4B, and 5A.

[0135] Referring to FIG. 5F, the wearable device (200) can display a three-dimensional image of the first mode on the display (320). For example, the wearable device (200) can display a three-dimensional image (or an image having binocular disparity) (or a stereoscopic image) of the first mode in an area of ​​the FOV (410) of the user (401). For example, the first mode may be a VR mode. However, the present invention is not limited thereto. For example, the first mode may be an AR mode. Hereinafter, the VR mode may be referred to as the first mode.

[0136] In one embodiment, the three-dimensional image of the first mode may include a non-executable area (501) and an executable area (511, 513, 515). In one embodiment, the non-executable area (501) may be an area to which no function for processing user (401) interaction (or input) is assigned. In one embodiment, the executable area (511, 513, 515) may be an area to which a function for processing user (401) interaction (or input) is assigned.

[0137] Referring to FIG. 5F, the wearable device (200) can identify a gesture (430) while the gaze (421, 423, 425) identified through the sensor (330) and / or the camera (325) for ET (e.g., cameras (240-1, 240-2)) is located in the non-executable area (501). For example, the wearable device (200) can identify a gesture (430) through the sensor (330) and / or the camera (325) for HaT (or the cameras (240-5 to 240-8)) while the gaze (421, 423, 425) is located in the executable area (511, 513, 515).

[0138] In one embodiment, the wearable device (200) can determine whether the gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe). The designated gesture in FIG. 5f and / or FIG. 5g may be an interaction (or designated user input) that requests overlapping (or displaying) the display area (520) for another mode, regardless of the area where the gaze (420) is located. For example, the designated gesture in FIG. 5f and / or FIG. 5g may be a combination of two or more gestures. For example, the designated gesture in FIG. 5f and / or FIG. 5g may be a gesture assigned for overlapping (or displaying) the display area for another mode, as specified in the application layer (340) and the framework layer (350).

[0139] In one embodiment, the wearable device (200) may display (or superimpose) a display area of ​​a second mode on a three-dimensional image of a first mode based on determining that the gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe). For example, the second mode may be an AR mode, but is not limited thereto. For example, the second mode may be a VR mode. Hereinafter, the AR mode may be referred to as the second mode.

[0140] For example, referring to FIG. 5G, the wearable device (200) may display a display area (520) of a second mode on a three-dimensional image of a first mode based on determining that a gesture (430) is a designated gesture while the gaze (421, 423, 425) is positioned in an executable area (511, 513, 515). In one embodiment, the display area (520) of the second mode may include at least a portion of an image representing a real space acquired through a camera (325) for pass-through (or VST) (e.g., cameras (240-9, 240-10)).

[0141] For example, the wearable device (200) may enlarge the display area (520) in a specified direction (521) based on the state (e.g., holding time and / or intensity) of the gesture (430). For example, the wearable device (200) may move the display area (520) based on the movement (or motion) of the gaze (421, 423, 425).

[0142] For example, the wearable device (200) may identify that the gesture (430) has been released while displaying the display area (520) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may stop displaying the display area (520) based on identifying that the gesture (430) has been released.

[0143] As described above, the wearable device (200) can overlap (or display) the display area for the AR mode on the executable area (511, 513, 515) while a three-dimensional image for the VR mode is displayed. Accordingly, the user's inconvenience of having to move the gaze (420) outside the executable area (511, 513, 515) to display the display area for the AR mode can be reduced.

[0144] FIG. 5h illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0145] FIG. 5h can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, FIG. 3, FIG. 4a, FIG. 4b, FIG. 5a, and FIG. 5b.

[0146] In FIG. 5B, the display area (520) is enlarged to correspond to the duration and / or intensity of the gesture (430), but this is only an example. Referring to FIG. 5H, the wearable device (200) can enlarge the display area (520) to correspond to the movement of the gesture (430). For example, the wearable device (200) can enlarge the display area (520) based on the gesture (430) moving two fingers (561, 565) in opposite directions (or increasing the distance between the two fingers (561, 565)).

[0147] In one embodiment, the wearable device (200) may reduce the display area (520) to correspond to the movement of the gesture (430). For example, the wearable device (200) may reduce the display area (520) based on the gesture (430) moving two fingers (561, 565) toward each other (or reducing the distance between the two fingers (561, 565)).

[0148] FIG. 6A illustrates an example of a three-dimensional image of a first mode displayed by a wearable device in one embodiment. FIG. 6B illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment. FIG. 6C illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0149] FIGS. 6A to 6C can be described with reference to FIGS. 1, 2A, 2B, 3, 4A, and 4B.

[0150] Referring to FIG. 6A, the wearable device (200) can display a three-dimensional image of the first mode on the display (320). For example, the wearable device (200) can display a three-dimensional image (or an image having binocular disparity) (or a stereoscopic image) of the first mode in an area of ​​the FOV (410) of the user (401). For example, the first mode may be a mode for augmented reality (AR). However, the present invention is not limited thereto. For example, the first mode may be a mode for virtual reality (VR). Hereinafter, the mode for AR may be referred to as the first mode.

[0151] In one embodiment, the three-dimensional image of the first mode may include at least a portion of an image representing a real space acquired through a camera (325) (e.g., camera (240-9, 240-10)) for pass-through (or video see through (VST)). For example, the wearable device (200) may display an area corresponding to the FOV (410) of the user (401) in the image representing the real space on the display (320).

[0152] In one embodiment, the three-dimensional image of the first mode may include a non-executable area (601) and executable areas (611, 613, 615). In one embodiment, the three-dimensional image of the first mode may include a partial image (or layer) (or view object) (or window) (hereinafter, non-executable area image) for displaying the non-executable area (601) and a partial image (or layer) (or view object) (or window) (hereinafter, executable area image) for displaying each of the executable areas (611, 613, 615). In one embodiment, the three-dimensional image of the first mode may be an image having binocular parallax that is a composite of the non-executable area image and the executable area image.

[0153] In one embodiment, the non-executing area (601) may be an area to which no function for processing an interaction (or input) of a user (401) is assigned. Hereinafter, the interaction (or input) of the user (401) may be referred to as a gesture (430). However, the interaction (or input) of the user (401) is not limited to a gesture. Depending on the embodiment, the interaction (or input) of the user (401) may also include an input (e.g., a touch input) obtained through the input module (150) and / or the display module (160). Depending on the embodiment, the interaction (or input) of the user (401) may also include a voice input (e.g., an input based on a sound signal) obtained through the audio module (170).

[0154] In one embodiment, the executable area (611, 613, 615) may be an area assigned with a function for processing user (401) interaction (or input). In one embodiment, the function may include, but is not limited to, executing an application associated with the executable area (611, 613, 615) and / or executing a service provided through the application.

[0155] Referring to FIG. 6A, the wearable device (200) can identify a gesture (430) while the gaze (420) identified through the sensor (330) and / or the camera (325) for ET (e.g., cameras (240-1, 240-2)) is located in the non-execution area (601). For example, the wearable device (200) can identify a gesture (430) through the sensor (330) and / or the camera (325) for HaT (or cameras (240-5 to 240-8)) while the gaze (420) is located in the non-execution area (601).

[0156] In one embodiment, the wearable device (200) can determine whether the gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe).

[0157] In one embodiment, the wearable device (200) may display (or superimpose) a display area of ​​a second mode on a three-dimensional image of a first mode based on determining that the gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe). For example, the second mode may be a VR mode, but is not limited thereto. For example, the second mode may be an AR mode. Hereinafter, the VR mode may be referred to as the second mode.

[0158] In one embodiment, displaying a display area of ​​the second mode on a three-dimensional image of the first mode may include synthesizing a portion (or a portion corresponding to the display area) of an image for displaying a three-dimensional image of the second mode onto a portion of an image for displaying a three-dimensional image of the first mode. In one embodiment, displaying a display area of ​​the second mode on a three-dimensional image of the first mode may include superimposing a portion (or a portion corresponding to the display area) of an image for displaying a three-dimensional image of the second mode onto a portion of an image for displaying a three-dimensional image of the first mode.

[0159] For example, referring to FIG. 6B, the wearable device (200) may display a display area (620) of a second mode on a three-dimensional image of a first mode based on determining that a gesture (430) is a designated gesture. In one embodiment, the display area (620) of the second mode may include at least a portion of an image representing an execution screen of an application (or a virtual space according to execution of the application).

[0160] For example, the wearable device (200) may change the display of the display area (620) based on a user interaction (e.g., gesture (430) and / or a gesture different from gesture (430)) while displaying the display area (620) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may enlarge the display area (620) to correspond to the state (e.g., holding time and / or intensity) of the gesture (430) while displaying the display area (620) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may enlarge the display area (620) in a specified direction (621) based on the state (e.g., holding time and / or intensity) of the gesture (430). For example, the wearable device (200) may enlarge the display area (620) in a designated direction (621) within a maximum size (630) based on the state (e.g., holding time and / or intensity) of the gesture (430). For example, the designated direction (621) may be a direction away from the line of sight (420), but is not limited thereto. For example, the wearable device (200) may change the display of the display area (620) based on another gesture (e.g., a gesture input subsequent to or substantially simultaneously with the gesture (430) while displaying the display area (620) in a second mode on a three-dimensional image in a first mode. For example, the gesture subsequent to the gesture (430) may be a gesture identified through the same body part as the gesture (430). For example, a gesture input substantially simultaneously with the gesture (430) may be a gesture identified through a body part different from the gesture (430).

[0161] Referring to FIG. 6B, as the display area (620) is displayed (or enlarged), the display area (620) may be displayed over executable areas (611, 613) other than the non-executable area (601). For example, as the display area (620) is displayed (or enlarged), the display area (620) may be composited with executable areas (611, 613, 615) on the three-dimensional image of the first mode. For example, as the display area (620) is displayed (or enlarged), the display area (620) may be overlapped with executable areas (611, 613, 615) on the three-dimensional image of the first mode.

[0162] For example, referring to FIG. 6C, the wearable device (200) can maintain the display of the display area (625) based on the display area (625) being expanded to the maximum size. For example, the wearable device (200) can maintain the display of the display area (625) regardless of a change (or release) of a user interaction (e.g., gesture (430)). For example, referring to FIG. 6C, even after the gesture (430) is released after the display area (625) is expanded to the maximum size, the wearable device (200) can maintain the display of the display area (625).

[0163] For example, the wearable device (200) may change the display of the display area (620) based on the state of the gaze (420) (e.g., movement (or motion) of the gaze (420)) while displaying the display area (620) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may move the display area (620) based on the movement (or motion) of the gaze (420). For example, the wearable device (200) may move the display area (620) based on the movement (or motion) of the gaze (420) that occurs after the display area (620) is enlarged to the maximum size (630). However, the present invention is not limited thereto. The wearable device (200) can move the display area (620) based on the movement (or motion) of the gaze (420) that occurs while the display area (620) is within the maximum size (630).

[0164] For example, the wearable device (200) may move the maximum size display area (625) in the direction in which the gaze (420) moves based on the movement (or motion) of the gaze (420). For example, while the wearable device (200) moves the display area (625) in the direction in which the gaze (420) moves, the wearable device (200) may change (or move) a part of an image for displaying a three-dimensional image of the second mode shown in the display area (625).

[0165] For example, the wearable device (200) may stop displaying the display area (620) of the second mode based on a user interaction (e.g., gesture (430) and / or a gesture different from gesture (430)) while displaying the display area (620) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may identify that the gesture (430) has been released while displaying the display area (620) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may stop displaying the display area (620) based on identifying that the gesture (430) has been released. For example, the wearable device (200) may stop displaying the display area (620) based on identifying that the gesture (430) is released while the display area (620) is displayed to be less than the maximum size (630). For example, the wearable device (200) may identify a gesture different from the gesture (430) while displaying the display area (620) of the second mode (or the maximum size display area (625)) on the three-dimensional image of the first mode. For example, the wearable device (200) may stop displaying the display area (620) based on identifying a gesture different from the gesture (430). Here, the gesture different from the gesture (430) may be a gesture for selecting a button (640) for stopping displaying the display area (620) as exemplified through FIGS. 6C to 6D, but is not limited thereto. A gesture other than the gesture (430) may be a gesture assigned to stop display of the display area (620).

[0166] As described above, the wearable device (200) can overlap (or display) a display area for the VR mode while a 3D image for the AR mode is displayed. Accordingly, the wearable device (200) can improve convenience by allowing the user to check the external mode through a portion of the area for the AR mode without having to switch from the AR mode to the VR mode to check the external mode.

[0167] FIG. 6d illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0168] FIG. 6d can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, FIG. 3, FIG. 4a, FIG. 4b, FIG. 6a, and FIG. 6c.

[0169] FIG. 6d may illustrate a situation in which the real space in which the user (401) is located is different from that in FIG. 6c. For example, the real space in which the user (401) is located in FIG. 6d may be a bedroom, and the real space in which the user (401) is located in FIG. 6c may be a workroom.

[0170] In one embodiment, the wearable device (200) may determine an application for the display area (620) of the second mode based on determining that the identified gesture (430) is a designated gesture while the gaze (420) is positioned in the non-execution area (601). For example, the application for the display area (620) of the second mode may be an application that generates an image for the display area of ​​the second mode.

[0171] In one embodiment, the wearable device (200) may determine an application for the display area (620) of the second mode based on the behavioral pattern of the user (401) within the space where the user (401) is located. For example, the wearable device (200) may determine an application frequently used by the user (401) within the space where the user (401) is located as the application for the display area (620) of the second mode. For example, the wearable device (200) may determine an application most recently used by the user (401) within the space where the user (401) is located as the application for the display area (620) of the second mode. However, the present invention is not limited thereto. For example, the wearable device (200) may determine the application most recently used by the user (401) (or the application frequently used by the user (401)) as the application for the display area (620) of the second mode, regardless of the space where the user (401) is located.

[0172] Referring to FIG. 6d, the wearable device (200) can display a display area (650) having an image different from the image displayed in the display area (625) of FIG. 6c on the three-dimensional image for the first mode. For example, the wearable device (200) can display a display area (650) including a portion of an image determined based on a behavioral pattern of the user (401) within a space where the user (401) is located on the three-dimensional image for the first mode.

[0173] As described above, the wearable device (200) can display (or overlay) a display area for VR mode generated by an application based on the user's behavioral patterns while displaying a 3D image for AR mode. Accordingly, the user can check an optimized display area for VR mode based on their behavioral patterns without inputting an application designation for display.

[0174] FIG. 6e illustrates an example of a three-dimensional image of a first mode displayed by a wearable device in one embodiment. FIG. 6f illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0175] FIGS. 6E and 6F can be described with reference to FIGS. 1, 2A, 2B, 3, 4A, 4B, and 6A.

[0176] Referring to FIG. 6E, the wearable device (200) can display a three-dimensional image of the first mode on the display (320). For example, the wearable device (200) can display a three-dimensional image (or an image having binocular disparity) (or a stereoscopic image) of the first mode in an area of ​​the FOV (421, 423, 425) of the user (401). For example, the first mode can be an AR mode. However, the present invention is not limited thereto. For example, the first mode can be a VR mode. Hereinafter, the AR mode may be referred to as the first mode.

[0177] In one embodiment, the three-dimensional image of the first mode may include a non-executable area (601) and an executable area (611, 613, 615). In one embodiment, the non-executable area (601) may be an area to which no function for processing user (401) interaction (or input) is assigned. In one embodiment, the executable area (611, 613, 615) may be an area to which a function for processing user (401) interaction (or input) is assigned.

[0178] Referring to FIG. 6E, the wearable device (200) can identify a gesture (430) while the gaze (421, 423, 425) identified through the sensor (330) and / or the camera (325) for ET (e.g., cameras (240-1, 240-2)) is located in the non-executable area (601). For example, the wearable device (200) can identify a gesture (430) through the sensor (330) and / or the camera (325) for HaT (or the cameras (240-5 to 240-8)) while the gaze (421, 423, 425) is located in the executable area (611, 613, 615).

[0179] In one embodiment, the wearable device (200) can determine whether the gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe). The designated gesture in FIGS. 6E and / or 6F may be an interaction (or designated user input) requesting overlapping (or displaying) display areas for another mode, regardless of the area where the gaze (421, 423, 425) is located. For example, the designated gesture in FIGS. 6E and / or 6F may be a combination of two or more gestures. For example, the designated gesture in FIGS. 6E and / or 6F may be a gesture assigned for overlapping (or displaying) display areas for another mode, as specified in the application layer (340) and the framework layer (350).

[0180] In one embodiment, the wearable device (200) may display (or superimpose) a display area (620) of a second mode on a three-dimensional image of a first mode based on determining that a gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe). For example, the second mode may be a VR mode, but is not limited thereto. For example, the second mode may be an AR mode. Hereinafter, the VR mode may be referred to as the second mode.

[0181] In one embodiment, the application for the display area (620) of the second mode may be distinguished from the applications associated with the executable areas (611, 613, 615). In one embodiment, the application for the display area (620) of the second mode may be an application determined based on a behavioral pattern of the user (401) within a space where the user (401) is located. For example, the application for the display area (620) of the second mode may be an application that generates an image for the display area of ​​the second mode.

[0182] For example, referring to FIG. 6F, the wearable device (200) may display a display area (620) of a second mode on a three-dimensional image of a first mode based on determining that a gesture (430) is a designated gesture while the gaze (421, 423, 425) is positioned in an executable area (611, 613, 615). In one embodiment, the display area (620) of the second mode may include at least a portion of an image representing an execution screen of an application (or a virtual space according to execution of the application).

[0183] For example, the wearable device (200) may enlarge the display area (620) in a specified direction (621) based on the state (e.g., holding time and / or intensity) of the gesture (430). For example, the wearable device (200) may move the display area (620) based on the movement (or motion) of the gaze (421, 423, 425).

[0184] For example, the wearable device (200) may identify that the gesture (430) has been released while displaying the display area (620) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may stop displaying the display area (620) based on identifying that the gesture (430) has been released.

[0185] As described above, the wearable device (200) can overlap (or display) the display area for the VR mode on the executable area (611, 613, 615) while displaying a three-dimensional image for the AR mode. Accordingly, the user's inconvenience of having to move their gaze (421, 423, 425) outside the executable area (611, 613, 615) to display the display area for the VR mode can be reduced.

[0186] FIG. 7A illustrates an example of a three-dimensional image of a first mode displayed by a wearable device in one embodiment. FIG. 7B illustrates an example of a display area of ​​a second mode on a three-dimensional image of a first mode displayed by a wearable device in one embodiment.

[0187] FIGS. 7a and 7b can be described with reference to FIGS. 1, 2a, 2b, 3, 4a, and 4b.

[0188] In comparison with FIGS. 6e and 6f, FIGS. 7a and 7b illustrate that the application for the display area (725) of the second mode may be an application related to the executable area.

[0189] Referring to FIG. 7A, the wearable device (200) can display a three-dimensional image of a first mode on the display (320). For example, the first mode may be a mode that provides services for an AR mode. Hereinafter, the AR mode may be referred to as the first mode.

[0190] In one embodiment, the three-dimensional image of the first mode may include a non-executable area (701) and an executable area (611, 613, 615, 711). In one embodiment, the non-executable area (701) may be an area to which no function for processing user (401) interaction (or input) is assigned. In one embodiment, the executable area (611, 613, 615, 711) may be an area to which a function for processing user (401) interaction (or input) is assigned.

[0191] Referring to FIG. 7A, the wearable device (200) can identify a gesture (430) while the gaze (420) identified through the sensor (330) and / or camera (325) for ET (e.g., cameras (240-1, 240-2)) is located in the non-execution area (701).

[0192] In one embodiment, the wearable device (200) can determine whether the gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe). The designated gesture in FIG. 7A and / or FIG. 7B may be an interaction (or designated user input) requesting overlapping (or displaying) a display area for another mode, regardless of the area where the gaze (420) is located.

[0193] In one embodiment, the wearable device (200) may display (or superimpose) a display area (725) of a second mode on a three-dimensional image of a first mode based on determining that the gesture (430) is a designated gesture (e.g., pinch, squeeze, zoom out, drag, or swipe). For example, the second mode may be a VR mode. Hereinafter, the VR mode may be referred to as the second mode.

[0194] In one embodiment, the application for the display area (725) of the second mode may be an application associated with an executable area (711) where the gaze (420) is located. In one embodiment, the application being associated with the executable area may include that an image displayed in the executable area is generated by the application. In one embodiment, the application being associated with the executable area may include that an input for the executable area is input to the application as a destination.

[0195] For example, referring to FIG. 7B, the wearable device (200) may display a display area (725) of a second mode on a three-dimensional image of a first mode based on determining that a gesture (430) is a designated gesture while the gaze (420) is positioned in the executable area (711). In one embodiment, the display area (725) of the second mode may include at least a portion of an image representing an execution screen of an application (or a virtual space according to execution of the application) related to the executable area (711).

[0196] For example, the wearable device (200) may enlarge the display area (725) in a specified direction (621) based on the state (e.g., holding time and / or intensity) of the gesture (430). For example, the wearable device (200) may move the display area (725) based on the movement (or motion) of the gaze (420).

[0197] For example, the wearable device (200) may identify that the gesture (430) is released while displaying the display area (725) of the second mode on the three-dimensional image of the first mode. For example, the wearable device (200) may stop displaying the display area (725) based on identifying that the gesture (430) is released. For example, the wearable device (200) may stop displaying the display area (725) based on identifying a gesture different from the gesture (430). Here, the gesture different from the gesture (430) may be a gesture for selecting a button (740) for stopping displaying the display area (725).

[0198] As described above, the wearable device (200) can display (or overlap) a display area for a VR mode related to an executable area (611, 613, 615, 711) on the executable area (611, 613, 615, 711) while a three-dimensional image for an AR mode is displayed. Accordingly, the number of inputs required for a user to display a display area for a VR mode related to an executable area (611, 613, 615, 711) can be reduced.

[0199] FIG. 8A illustrates an example of transitions between three-dimensional images of a second mode displayed by a wearable device in one embodiment.

[0200] FIG. 8a can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, FIG. 3, FIG. 4a, and FIG. 4b.

[0201] FIG. 8a may illustrate operations for overlapping (or switching) between three-dimensional images for augmented reality (AR) or virtual reality (VR).

[0202] Referring to FIG. 8A, in state (801), a three-dimensional image (810) for a first virtual reality may be displayed on a display (320). In one embodiment, at least one executable area (820) (or a visual guide, a virtual object, a visual element, a UI element, a view object, or a view element) may be placed on the three-dimensional image (810) for the first virtual reality.

[0203] In one embodiment, the wearable device (200) can identify an input (or gesture) for the executable area (820) while the user's (401) gaze (830) is positioned in the executable area (820).

[0204] In one embodiment, the wearable device (200) may execute an assigned function for the executable area (820) based on an input (or gesture) for the executable area (820). Referring to FIG. 8A, in a state (803), an enlarged menu (840) may be displayed on a three-dimensional image (810) for the first virtual reality according to the execution of the assigned function. In one embodiment, the enlarged menu (840) may be referred to as an executable area (or visual guide, virtual object, visual element, UI element, view object, or view element).

[0205] In one embodiment, the wearable device (200) can identify an input (or gesture) for a specific UI element while the user's (401) gaze (830) is positioned on the specific UI element within an enlarged menu (840).

[0206] In one embodiment, the wearable device (200) may execute a function assigned to a specific UI element based on an input (or gesture) for the specific UI element. Referring to FIG. 8A, in state (805), upon execution of the assigned function, a three-dimensional image (815) for the second virtual reality may be displayed on the display (320) instead of the three-dimensional image (810).

[0207] In one embodiment, the wearable device (200) can identify an input (or gesture) for overlapping a 3D image while the user's (401) gaze (830) is positioned in a non-executable area within the 3D image (815).

[0208] In one embodiment, the wearable device (200) may determine an application for overlapping 3D images based on a behavioral pattern of the user (401) based on an input (or gesture) for overlapping 3D images. For example, the wearable device (200) may identify an application most recently used by the user (401) (or an application frequently used by the user (401)).

[0209] Referring to FIG. 8A, in state (807), a portion of an image for the first virtual reality can be displayed (or superimposed) on the 3D image (815) in a display area (850) within the 3D image (815).

[0210] According to an embodiment, in state (807), the user (401) may input an input for displaying a 3D image (810) instead of a 3D image (815). For example, the input for displaying a 3D image (810) instead of a 3D image (815) may be a 3D image switching input. For example, the input for displaying a 3D image (810) instead of a 3D image (815) may be a continuous input to an input (or gesture) for overlapping 3D images. However, the present invention is not limited thereto. For example, the input for displaying a 3D image (810) instead of a 3D image (815) may be a different input distinct from the input (or gesture) for overlapping 3D images. Referring to FIG. 8a, in state (809), a three-dimensional image (810) instead of a three-dimensional image (815) may be displayed on the display (320).

[0211] FIG. 8b illustrates an example of transitions between three-dimensional images of a second mode displayed by a wearable device in one embodiment.

[0212] FIG. 8b can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, FIG. 3, FIG. 4a, and FIG. 4b.

[0213] FIG. 8b may illustrate operations for overlapping (or switching) between three-dimensional images for augmented reality (AR) or virtual reality (VR).

[0214] FIG. 8B illustrates, compared to FIG. 8A, that an input (or gesture) for overlapping three-dimensional images can be input into an executable area (e.g., a UI element on a menu (840)). The input (or gesture) for overlapping three-dimensional images in FIG. 8B may be an interaction (or a designated user input) requesting overlapping (or displaying) display areas for another mode, regardless of the area where the gaze (830) is located. For example, the input (or gesture) for overlapping three-dimensional images in FIG. 8B may be a combination of two or more gestures. For example, the input (or gesture) for overlapping three-dimensional images in FIG. 8B may be a gesture assigned for overlapping (or displaying) display areas for another mode specified in the application layer (340) and the framework layer (350).

[0215] Referring to FIG. 8B, in state (801), a three-dimensional image (810) for the first virtual reality may be displayed on the display (320). Referring to FIG. 8B, in state (803), an enlarged menu (840) may be displayed on the three-dimensional image (810) for the first virtual reality according to the execution of the assigned function.

[0216] In one embodiment, the wearable device (200) can identify an input (or gesture) for a specific UI element while the user's (401) gaze (830) is positioned on the specific UI element within the expanded menu (840). Referring to FIG. 8B, in a state (804), the wearable device (200) can display, in a display area (850), at least a portion of an image representing an execution screen of an application (or a virtual space according to the execution of the application) related to the specific UI element on which the gaze (830) is positioned while the user's (401) gaze (830) is positioned on the specific UI element within the expanded menu (840).

[0217] Referring to FIG. 8b, in state (805), a portion of an image for the second virtual reality can be displayed (or superimposed) on the 3D image (810) in a display area (850) within the 3D image (810).

[0218] According to an embodiment, in state (805), the user (401) may input an input for displaying a 3D image (815) instead of a 3D image (810). For example, the input for displaying a 3D image (815) instead of a 3D image (810) may be a 3D image switching input. For example, the input for displaying a 3D image (815) instead of a 3D image (810) may be a continuous input to an input (or gesture) for overlapping 3D images. However, the present invention is not limited thereto. For example, the input for displaying a 3D image (815) instead of a 3D image (810) may be a different input distinct from the input (or gesture) for overlapping 3D images. Referring to FIG. 8a, in state (805), a three-dimensional image (815) instead of a three-dimensional image (810) may be displayed on the display (320).

[0219] Figure 9 is a flowchart illustrating the operation of a wearable device in one embodiment.

[0220] Figure 9 can be explained with reference to Figures 1 to 8b.

[0221] Referring to FIG. 9, in operation 910, the wearable device (200) may display a three-dimensional image of the first mode. For example, the wearable device (200) may display a three-dimensional image (or an image having binocular disparity) (or a stereoscopic image) of the first mode in an area of ​​the FOV (410) of the user (401). For example, the first mode may be a mode for virtual reality (VR). However, the present invention is not limited thereto. For example, the first mode may be a mode for augmented reality (AR). Hereinafter, the mode for VR may be referred to as the first mode.

[0222] In one embodiment, the three-dimensional image of the first mode may include a non-executable area (501) and an executable area (511, 513, 515).

[0223] In operation 920, the wearable device (200) can identify an input. The wearable device (200) can identify the input while the gaze (420) identified through the sensor (330) for eye tracking (ET) and / or the camera (325) (e.g., the cameras (240-1, 240-2)) is located in the non-execution area (501). For example, the wearable device (200) can identify the input through the sensor (330) for hand tracking (HaT) and / or the camera (325) (or the cameras (240-5 to 240-8)) while the gaze (420) is located in the non-execution area (501). In one embodiment, the input can include, but is not limited to, a gesture (430). For example, the input may also include input (e.g., touch input) obtained through the input module (150) and / or the display module (160).

[0224] In operation 930, the wearable device (200) can determine whether the input is for superimposing a three-dimensional image display of the second mode. In one embodiment, the wearable device (200) can determine whether the input is a designated input (e.g., pinch, hold, zoom out, drag, or swipe). In one embodiment, the wearable device (200) can determine whether the input is a designated input (e.g., pinch, hold, zoom out, drag, or swipe) that is input while the gaze (420) is positioned in the non-execution area (501).

[0225] At operation 930, based on the input being an input for superimposing a three-dimensional image display of the second mode, the wearable device (200) can perform operation 940. At operation 930, based on the input not being an input for superimposing a three-dimensional image display of the second mode, the wearable device (200) can perform operation 950.

[0226] In operation 940, the wearable device (200) may display a three-dimensional image of the second mode by overlaying it on the three-dimensional image of the first mode. For example, the second mode may be an AR mode. However, the present invention is not limited thereto. For example, the second mode may be a VR mode. Hereinafter, the AR mode may be referred to as the second mode.

[0227] In one embodiment, displaying a three-dimensional image of the second mode by overlaying it on the three-dimensional image of the first mode may include synthesizing a portion (or a portion corresponding to the display area) of an image for displaying the three-dimensional image of the second mode onto a portion of an image for displaying the three-dimensional image of the first mode. In one embodiment, displaying a three-dimensional image of the second mode by overlaying it on the three-dimensional image of the first mode may include overlaying a portion (or a portion corresponding to the display area) of an image for displaying the three-dimensional image of the second mode onto a portion of an image for displaying the three-dimensional image of the first mode.

[0228] In operation 950, the wearable device (200) can execute a function mapped to an area and an input. In one embodiment, the wearable device (200) can trigger (or start) the execution of a function of an application related to an area (or an executable area) (or an interaction area) where the gaze (420) is located. The wearable device (200) can transmit an input as a parameter to the application related to the area (or an executable area) (or an interaction area) where the gaze (420) is located.

[0229] As described above, the wearable device (200) may include a display (320) arranged to face the eyes of a user (401) when worn. The wearable device (200) may include a camera (325) arranged to face the front of the user (401) when worn. The wearable device (200) may include a sensor (330) arranged to face the eyes of the user (401) when worn. The wearable device (200) may include at least one processor (310) including a processing circuit. The wearable device (200) may include a memory (315) that stores instructions and includes one or more storage media. The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify a gaze (420) of the user (401) based on an image of the eye acquired through the sensor (330). The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify a gesture (430) of the user (401) based on an image acquired through the camera (325). The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to display a first three-dimensional image having binocular parallax through the display (320). The above instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify a designated gesture (430) of the user (401) through the camera (325).The above instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to display a second three-dimensional image having binocular parallax by overlapping a portion of the first three-dimensional image where the gaze (420) of the user (401) identified through the sensor (330) is located in a non-executable area (501) of the first three-dimensional image in response to the identification of the designated gesture (430). The first three-dimensional image and the second three-dimensional image may be three-dimensional images for different modes.

[0230] The above instructions, when executed individually or collectively by the at least one processor (310), may cause the wearable device (200) to perform a function assigned to the designated gesture (430) in response to the designated gesture (430) being identified while the gaze (420) is positioned in an executable area (511, 513, 515) of the first three-dimensional image.

[0231] The above different modes may include a mode for virtual reality and a mode for augmented reality.

[0232] The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify that the gesture (430) has been released. The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to stop displaying the second three-dimensional image based on that the gesture (430) has been released.

[0233] The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify that the gesture (430) has been released. The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to maintain display of the second three-dimensional image based on the gesture (430) being released after the portion of the second three-dimensional image displayed has been enlarged to a designated size. The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to stop displaying of the second three-dimensional image based on the gesture (430) being released before the portion of the second three-dimensional image displayed has been enlarged to the designated size.

[0234] The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify that the gaze (420) has moved. The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to move the display position of the second three-dimensional image in response to the movement of the gaze (420).

[0235] The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify a location of the wearable device (200). The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to, in response to the designated gesture (430) being identified while the gaze (420) is positioned in the non-execution area (501), display the second three-dimensional image of the application corresponding to the identified location, overlapping the partial area on the first three-dimensional image.

[0236] The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify another gesture (430) of the user (401) through the second camera (325) while the second three-dimensional image is displayed in the partial region. The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to control the display (320) to switch the three-dimensional image displayed through the display (320) from the first three-dimensional image to the second three-dimensional image in response to the other gesture (430) being identified while the gaze (420) is positioned in the partial region.

[0237] The above instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to enlarge the size of the portion of the area where the second three-dimensional image is displayed while the gesture (430) is maintained.

[0238] The above instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to enlarge the size of the portion of the area where the second three-dimensional image is displayed based on the distance by which the gesture (430) is moved.

[0239] As described above, the method may be performed by a wearable device (200) including a display (320) arranged to face the eyes of a user (401) when worn, a camera (325) arranged to face the front of the user (401) when worn, and a sensor (330) arranged to face the eyes of the user (401) when worn. The method may include an operation of identifying a gaze (420) of the user (401) based on an image of the eyes acquired through the sensor (330). The method may include an operation of identifying a gesture (430) of the user (401) based on an image acquired through the camera (325). The method may include an operation of displaying a first three-dimensional image having binocular disparity through the display (320). The method may include an operation of identifying a designated gesture (430) of the user (401) through the camera (325). The method may include an operation of displaying a second 3D image having binocular parallax by overlapping a portion of the first 3D image where the gaze (420) of the user (401) identified through the sensor (330) is located in a non-executable area of ​​the first 3D image in response to the identification of the designated gesture (430). The first 3D image and the second 3D image may be 3D images for different modes.

[0240] The method may include an action of performing a function assigned to the designated gesture (430) in response to the designated gesture (430) being identified while the gaze (420) is positioned in an executable area of ​​the first three-dimensional image.

[0241] The above different modes may include a mode for virtual reality and a mode for augmented reality.

[0242] The method may include an action of identifying that the gesture (430) has been released. The method may include an action of stopping display of the second three-dimensional image based on that the gesture (430) has been released.

[0243] The method may include an action of identifying that the gesture (430) is released. The method may include an action of maintaining the display of the second three-dimensional image based on the gesture (430) being released after the partial area where the second three-dimensional image is displayed has been enlarged to a designated size. The method may include an action of stopping the display of the second three-dimensional image based on the gesture (430) being released before the partial area where the second three-dimensional image is displayed has been enlarged to the designated size.

[0244] The method may include an operation of identifying that the gaze (420) has moved. The method may include an operation of moving the display position of the second three-dimensional image in response to the movement of the gaze (420).

[0245] The method may include an operation of identifying a location of the wearable device (200). The method may include an operation of displaying, in response to the identification of the designated gesture (430) while the gaze (420) is positioned in the non-execution area (501), the second three-dimensional image of the application corresponding to the identified location, overlapping the partial area on the first three-dimensional image.

[0246] The method may include an operation of identifying another gesture (430) of the user (401) through the second camera (325) while the second three-dimensional image is displayed in the partial area. The method may include an operation of controlling the display (320) to switch the three-dimensional image displayed through the display (320) from the first three-dimensional image to the second three-dimensional image in response to the other gesture (430) being identified while the gaze (420) is positioned in the partial area.

[0247] The above method may include an operation of enlarging the size of the partial area where the second three-dimensional image is displayed while the gesture (430) is maintained.

[0248] The method may include an operation of enlarging the size of the partial area where the second three-dimensional image is displayed based on the distance by which the gesture (430) is moved.

[0249] As described above, a non-transitory computer readable storage medium can store a program including instructions. The instructions, when individually or collectively executed by at least one processor (310) of a wearable device (200) including a display (320) arranged to face the eyes of a user (401) when worn, a camera (325) arranged to face the front of the user (401) when worn, and a sensor (330) arranged to face the eyes of the user (401) when worn, can cause the wearable device (200) to identify the gaze (420) of the user (401) based on an image of the eyes acquired through the sensor (330). The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify a gesture (430) of the user (401) based on an image acquired through the camera (325). The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to display a first three-dimensional image having binocular parallax through the display (320). The instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to identify a specified gesture (430) of the user (401) through the camera (325).The above instructions, when individually or collectively executed by the at least one processor (310), may cause the wearable device (200) to display a second three-dimensional image having binocular parallax by overlapping a portion of the first three-dimensional image where the gaze (420) of the user (401) identified through the sensor (330) is located in a non-executable area (501) of the first three-dimensional image in response to the identification of the designated gesture (430). The first three-dimensional image and the second three-dimensional image may be three-dimensional images for different modes.

[0250] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0251] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0252] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0253] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0254] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., by download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0255] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (200), A display (320) arranged to face the eyes of a user (401) when worn; A camera (325) arranged to face the front of the user (401) when worn; A sensor (330) arranged to face the eyes of the user (401) when worn; At least one processor (310) comprising a processing circuit, and A wearable device (200) comprises a memory (315) storing instructions and including one or more storage media, wherein the instructions, when individually or collectively executed by the at least one processor (310), Based on the image of the eye acquired through the sensor (330), the gaze (420) of the user (401) is identified, Based on the image acquired through the above camera (325), the gesture (430) of the user (401) is identified, A first three-dimensional image having binocular parallax is displayed through the above display (320), Identifying the designated gesture (430) of the user (401) through the camera (325), While the gaze (420) of the user (401) identified through the sensor (330) is located in a non-executable area (501) of the first three-dimensional image, in response to the identification of the designated gesture (430), a second three-dimensional image having binocular parallax is superimposed and displayed on a part of the first three-dimensional image where the gaze (420) is located. The above first 3D image and the above second 3D image are 3D images for different modes. Wearable devices.

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (310), cause the wearable device (200) to: While the above gaze (420) is positioned in the executable area (511, 513, 515) of the first three-dimensional image, in response to the identification of the designated gesture (430), causing the designated gesture (430) to perform the assigned function. Wearable devices.

3. In claim 1 or 2, The above different modes include a mode for virtual reality and a mode for augmented reality. Wearable devices.

4. In any one of claims 1 to 3, The above instructions, when individually or collectively executed by the at least one processor (310), cause the wearable device (200) to: Identify that the above gesture (430) is released, Based on the release of the above gesture (430), causing the display of the second three-dimensional image to be stopped, Wearable devices.

5. In claim 4, The above instructions, when individually or collectively executed by the at least one processor (310), cause the wearable device (200) to: Identify that the above gesture (430) is released, The gesture (430) is released after the partial area where the second three-dimensional image is displayed is enlarged to a specified size, thereby maintaining the display of the second three-dimensional image. The above gesture (430) causes the display of the second three-dimensional image to be stopped based on the release of the partial area where the second three-dimensional image is displayed before it is enlarged to the specified size. Wearable devices.

6. In any one of claims 1 to 5, The above instructions, when individually or collectively executed by the at least one processor (310), cause the wearable device (200) to: Identify that the above gaze (420) has moved, In response to the movement of the above gaze (420), causing the display position of the second three-dimensional image to move in response to the movement of the above gaze (420). Wearable devices.

7. In any one of claims 1 to 6, The above instructions, when individually or collectively executed by the at least one processor (310), cause the wearable device (200) to: Identifying the location of the wearable device (200), While the above gaze (420) is located in the non-execution area (501), in response to the identification of the designated gesture (430), causing the second 3D image of the application corresponding to the identified position to be displayed over the partial area on the first 3D image. Wearable devices.

8. In any one of claims 1 to 7, The above instructions, when individually or collectively executed by the at least one processor (310), cause the wearable device (200) to: While the second 3D image is displayed in the above-mentioned part of the area, another gesture (430) of the user (401) is identified through the second camera (325), The above other gesture (430) causes the display (320) to be controlled to switch the three-dimensional image displayed through the display (320) from the first three-dimensional image to the second three-dimensional image in response to being identified while the gaze (420) is positioned in the above partial area. Wearable devices.

9. In any one of claims 1 to 8, The above instructions, when individually or collectively executed by the at least one processor (310), cause the wearable device (200) to: While the above gesture (430) is maintained, causing the size of the part of the area where the second three-dimensional image is displayed to be enlarged. Wearable devices.

10. In any one of claims 1 to 9, The above instructions, when individually or collectively executed by the at least one processor (310), cause the wearable device (200) to: Based on the distance the above gesture (430) moves, causing the size of the part of the area where the second 3D image is displayed to be enlarged. Wearable devices.

11. A method of a wearable device (200) including a display (320) arranged to face the eyes of a user (401) when worn, a camera (325) arranged to face the front of the user (401) when worn, and a sensor (330) arranged to face the eyes of the user (401) when worn, An operation of identifying the gaze (420) of the user (401) based on the image of the eye acquired through the sensor (330). An action of identifying a gesture (430) of the user (401) based on an image acquired through the camera (325). An operation of displaying a first three-dimensional image having binocular parallax through the above display (320); An action of identifying a designated gesture (430) of the user (401) through the camera (325), and An operation of superimposing and displaying a second 3D image having binocular parallax on a portion of the first 3D image where the gaze (420) of the user (401) identified through the sensor (330) is located in a non-executable area of the first 3D image in response to the identification of the designated gesture (430). The above first 3D image and the above second 3D image are 3D images for different modes. method.

12. In claim 11, While the above gaze (420) is located in the executable area of the first three-dimensional image, in response to the identification of the designated gesture (430), an action is performed to perform the function assigned to the designated gesture (430). method.

13. In claim 11 or 12, The above different modes include a mode for virtual reality and a mode for augmented reality. method.

14. In any one of claims 11 to 13, An action that identifies that the above gesture (430) has been released, and Based on the release of the above gesture (430), an action is included to stop displaying the second three-dimensional image. method.

15. In claim 14, An action that identifies that the above gesture (430) has been released; An action of maintaining the display of the second three-dimensional image based on the gesture (430) being released after the partial area where the second three-dimensional image is displayed is enlarged to a specified size, and The gesture (430) includes an action of stopping display of the second three-dimensional image based on the release of the partial area where the second three-dimensional image is displayed before it is enlarged to the specified size. method.

Images