Electronic device and method for identifying user input, and non-transitory computer-readable storage medium

Abstract

A wearable device may comprise: at least one display; at least one camera; at least one processor including processing circuitry; and a memory storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, may cause the wearable device to: identify, through the at least one camera, initiation of a user input based on pinching of a user's fingers; while the user input based on movement of a user's hand object is being provided, determine whether pinched portions of the fingers for making the pinch are visible in a field of view of the at least one camera; according to a determination that the pinched portions of the fingers are not visible in the field of view of the at least one camera, apply the user input based on the movement of the user's hand object while the pinching of the fingers is applied for an expected duration; and on the basis of identifying that the expected duration has elapsed while the pinched portions of the fingers are not visible in the field of view of the at least one camera, execute a function corresponding to release of the pinch.

Description

Electronic device, method, and non-transitory computer-readable storage medium for identifying user input

[0001] The present disclosure relates to electronic devices, methods, and non-transitory computer-readable media for identifying user input.

[0002] A wearable device may include a display and communication circuitry. The wearable device may be utilized as a tool for implementing virtual reality, augmented reality, and mixed reality. The wearable device may display a three-dimensional (3D) space on the display.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] In embodiments of the present disclosure, a wearable device is provided. The wearable device may include at least one display; at least one camera; at least one processor including a processing circuit; and a memory storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify an initiation of a user input based on a pinch of a user's fingers through the at least one camera, determine whether a pinch portion of the fingers for the pinch is visible in the field of view of the at least one camera while the user input is provided based on a movement of a hand object of the user, and, upon determining that the pinch portion of the fingers is not visible in the field of view of the at least one camera, apply the user input based on the movement of the hand object of the user while the pinch portion of the fingers is applied for an expected duration, and, based on identifying that the expected duration has elapsed while the pinch portion of the fingers is not visible in the field of view of the at least one camera, execute a function corresponding to the release of the pinch.

[0005] In embodiments of the present disclosure, a method performed by a wearable device is provided. The method may include: identifying an initiation of a user input according to a pinch of a user's fingers through at least one camera of the wearable device; determining, while the user input is provided according to a movement of a hand object of the user, whether a pinch portion of the user's fingers for the pinch is visible in the field of view of the at least one camera; applying the user input according to the movement of the hand object of the user while the pinch portion of the fingers is applied for an expected duration based on a determination that the pinch portion of the fingers is not visible in the field of view of the at least one camera; and executing a function corresponding to the release of the pinch based on identifying that the expected duration has elapsed while the pinch portion of the fingers is not visible in the field of view of the at least one camera.

[0006] In embodiments of the present disclosure, a wearable device is provided. The wearable device may include at least one display; at least one camera; at least one processor including a processing circuit; and a memory storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify the initiation of a gesture input using a hand object of a user, determine whether at least a portion of the hand object of the user is visible in the field of view of the at least one camera while the gesture input is being performed, determine an expected duration for the duration of the gesture input based on at least one of a posture of the hand object of the user, information about the mobility of the hand object of the user, information about the rotation of a joint of the hand object of the user, and information about the state of an application layer, display a visual object to indicate the expected duration while the at least a portion of the hand object of the user is not visible, and execute a function corresponding to the completion of the gesture input based on identifying that the expected duration has elapsed while the at least a portion of the hand object of the user is not visible.

[0007] In embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store instructions. The instructions, when executed by at least one processor, may cause an electronic device to perform operations including identifying the initiation of a gesture input using a hand object of a user, determining whether at least a portion of the hand object of the user is visible in the field of view of the at least one camera while the gesture input is performed, determining an expected duration for the duration of the gesture input based on at least one of a posture of the hand object of the user, information about the mobility of the hand object of the user, information about the rotation of a joint of the hand object of the user, and information about the state of an application layer, displaying a visual object to indicate the expected duration while the at least a portion of the hand object of the user is not visible, and executing a function corresponding to the completion of the gesture input based on identifying that the expected duration has elapsed while the at least a portion of the hand object of the user is not visible.

[0008] Figure 1 is a block diagram of an electronic device within a network environment.

[0009] Figure 2a shows an example of a perspective view of a wearable device.

[0010] FIG. 2b illustrates an example of one or more hardware devices arranged within a wearable device.

[0011] Figures 3a and 3b show an example of the appearance of a wearable device.

[0012] Figure 4 shows an example of a block diagram of a wearable device.

[0013] Fig. 5 shows an example of a block diagram of an electronic device for displaying an image in a virtual space.

[0014] Figures 6a and 6b illustrate examples of structures of multiple layers.

[0015] Figure 7 shows an example of user input according to pinching of the user's fingers.

[0016] Figure 8 illustrates the operation flow of a wearable device for identifying user input using a pinch.

[0017] Figure 9a illustrates an example of prediction of user input using the mobility of at least one hand object of the user.

[0018] Figures 9b and 9c illustrate examples of prediction of user input using the rotation of a joint of at least one hand object of the user.

[0019] Figure 9d shows an example of prediction of user input using the pose of at least one hand object of the user.

[0020] Fig. 10 illustrates an operation flow of a wearable device for identifying a gesture input using at least one hand object of a user.

[0021] Figure 11a shows an example of a visual object corresponding to a visibility state.

[0022] Figures 11b, 11c, and 11d illustrate examples of visual objects for indicating expected durations.

[0023] Figure 12 shows an example of a visual object for representing user input.

[0024] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0025] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0026] In the following description, terms referring to images (e.g., image, frame, camera frame, captured image, camera image), terms referring to the user's hand (e.g., hand object, candidate object, hand candidate object, bounding box, candidate hand object), terms referring to signals (e.g., signaling, control signal, data, control data, request signal, information), terms referring to locations (e.g., location information, area information, object information, object location, object coordinates, reference object, coordinate information, location, coordinates, relative coordinates, absolute coordinates, coordinate system), terms referring to values ​​(e.g., threshold, reference value, reference area, reference range, level, threshold level, threshold, range, value, area), terms for operation states (e.g., step, operation, procedure), terms referring to network entities, terms referring to components of a device, etc., are examples for convenience of description. Therefore, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings may be used. In addition, the terms '...bu', '...gi', '...mul', '...che', etc. used below may mean at least one shape structure or a unit that processes a function.

[0027] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled, but this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." A condition described as "more than" may be replaced with "more than," a condition described as "less than" may be replaced with "less than," and a condition described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of elements from A (including A) to B (including B). hereinafter, "C" and / or "D" mean at least one of "C" or "D," that is, including {"C", "D", "C" and "D"}.

[0028] Figure 1 is a block diagram of an electronic device within a network environment.

[0029] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0030] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0031] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0032] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0033] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0034] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0035] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0036] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0037] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0038] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0039] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0040] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0041] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0042] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0043] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0044] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0045] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0046] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0047] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0048] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0049] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0050] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0051] In embodiments of the present disclosure, an electronic device (e.g., electronic device (101) of FIG. 1) for displaying an image in a virtual space may be a wearable device. The wearable device (101) may include a head-mounted display (HMD) that is wearable on a user's head. The wearable device (101) may be referred to as a head-mounted device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of a hardware configuration included in the wearable device (101) is exemplarily described with reference to FIG. 4. An example of the structure of a wearable device (101) that can be worn on the head of a user (110) is described with reference to FIGS. 2A, 2B, 3A, and / or 3B. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) for attaching to the head of a user to form an HMD.

[0052] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) may see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0053] In one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include a housing that covers the eyes of the user (110). The wearable device (101), in this state, may include a display disposed on a first side of the housing facing the eyes. The wearable device (101) may include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (101) may acquire images and / or videos representing ambient light. The wearable device (101) can output the image and / or video within the display disposed on the first surface, thereby allowing the user (110) to recognize the ambient light through the display. The displaying area (or displaying region) (or active area or active region) of the display disposed on the first surface can be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object into the image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with a real object recognized by the ambient light.

[0054] According to one embodiment, the wearable device (101) can identify or recognize a position (or location) and / or direction (or orientation) of the wearable device (101) based on an image (and / or video) obtained or acquired using a camera. The wearable device (101) can obtain information about the external space using one or more cameras and / or one or more sensors. The information can include a geographic location (e.g., global positioning system (GPS) coordinates) of the external space identified from one or more sensors. The information can include images and / or videos of the external space identified from one or more cameras. The wearable device (101) can perform object recognition on the images and / or videos to identify external objects included in the external space from the images and / or videos.

[0055] Below, an example of a hardware configuration of a wearable device (101) is described with reference to FIGS. 2a, 2b, 3a, 3b, and 4.

[0056] FIG. 2A illustrates an example of a perspective view of a wearable device. FIG. 2B illustrates an example of one or more hardware components arranged within the wearable device. According to one embodiment, the wearable device (101) may have a form of glasses that can be worn on a body part (e.g., head) of a user. The wearable device (101) of FIGS. 2A and 2B may be an example of the wearable device (101) of FIG. 1. The wearable device (101) may include a head-mounted display (HMD). For example, the housing of the wearable device (101) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (101) may include one or more straps capable of being twined around the user's head, and / or one or more temples attachable to the ears of the head.

[0057] Referring to FIG. 2A, according to one embodiment, a wearable device (101) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0058] According to one embodiment, a wearable device (101) can be worn on a part of a user's body. The wearable device (101) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (101). For example, the wearable device (101) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0059] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0060] Referring to FIG. 2B, at least one display (250) can provide the user with visual information transmitted from external light and other visual information distinct from the visual information through a lens included in the at least one display (250). The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When the user wears the wearable device (101), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0061] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (101) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0062] The wearable device (101) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (101) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (101) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (101) can view an image displayed on at least one display (250).

[0063] According to one embodiment, the frame (200) may be formed as a physical structure that allows the wearable device (101) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that, when the user wears the wearable device (101), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0064] Referring to FIG. 2A, the frame (200) may include a region (220) that is in contact with at least a portion of a user's body when the user wears the wearable device (101). For example, the region (220) of the frame (200) that is in contact with a portion of the user's body may include a region that is in contact with a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (101) makes contact with. According to one embodiment, the frame (200) may include a nose pad (210) that is in contact with a portion of the user's body. When the wearable device (101) is worn by the user, the nose pad (210) may be in contact with a portion of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another portion of the user's body that is distinct from the portion of the user's body.

[0065] For example, the frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (101) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200).

[0066] According to one embodiment, the wearable device (101) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (200).

[0067] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (101) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B, but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B. When the number of microphones (265) included in the wearable device (101) is two or more, the wearable device (101) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (200).

[0068] According to one embodiment, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or may be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (101) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0069] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (101). For example, the wearable device (101) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (101) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (101) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other. The wearable device (101) can obtain an image having visual quality of a first area matching the user's gaze and visual quality of a second area using foveated rendering.For example, if the wearable device (101) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1). An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0070] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (101) can compensate for depth information (e.g., the distance between the wearable device (101) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (260-4). The wearable device (101) can perform object recognition through an image acquired using the capturing camera (260-4). The wearable device (101) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capturing camera (260-4). The wearable device (101) can perform a pass-through function to display an image acquired through the capturing camera (260-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the capturing camera (260-4) can be disposed on a bridge (203) disposed between the first rim (201) and the second rim (202).

[0071] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing a wearable device (101), thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (101) looks straight ahead, the wearable device (101) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (101) is positioned.

[0072] The motion recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The motion recognition camera (260-2, 260-3) can recognize the user's motion (gesture recognition), obtain a signal corresponding to the motion, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the motion, and perform a designated function based on the identification. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor can perform a gesture recognition function and / or an object tracking function using the motion recognition camera (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) may be positioned on the first rim (201) and / or the second rim (202).

[0073] The camera (260) included in the wearable device (101) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (101) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (101) can identify an external object based on a sensor for identifying the distance between the wearable device (101) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (101) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (101).

[0074] Although not shown, in one embodiment, the wearable device (101) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame (200) and the hinge units (206, 207).

[0075] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (101). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0076] The antenna module (275) can transmit signals or power to the outside of the wearable device (101), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0077] The speaker (255) can output an acoustic signal to the outside of the wearable device (101). The acoustic output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (101). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus positioned adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus positioned adjacent to the user's right ear.

[0078] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (101) to the user. For example, when the wearable device (101) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0079] Referring to FIG. 2B, according to one embodiment, a wearable device (101) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (101) (e.g., hardwares illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (101) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0080] According to one embodiment, a wearable device (101) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (101) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (101). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (101) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (101) based on the IMU.

[0081] FIGS. 3A and 3B illustrate an example of an exterior appearance of a wearable device (e.g., a wearable device (101)). The wearable device (101) of FIGS. 3A and 3B may be an example of the wearable device (101) of FIG. 1 . According to one embodiment, an example of an exterior appearance of a first side (310) of a housing of the wearable device (101) is illustrated in FIG. 3A , and an example of an exterior appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B .

[0082] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (101) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (101) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (101) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0083] According to one embodiment, the wearable device (101) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (101) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (101) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (101) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar expressing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (101).

[0084] Referring to FIG. 3B, a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor (330)) for obtaining information related to the external environment of the wearable device (101) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0085] Using cameras (260-11, 260-12), the wearable device (101) can acquire images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be positioned on the second face (320) of the wearable device (101) to acquire an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be positioned on the second face (320) of the wearable device (101) to acquire an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0086] According to one embodiment, the wearable device (101) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (101) and an external object. Using the depth sensor (330), the wearable device (101) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (101). Although not illustrated, a microphone may be disposed on the second face (320) of the wearable device (101) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0087] Hereinafter, with reference to FIG. 4, the hardware or software configuration of the wearable device (101) is described.

[0088] Fig. 4 illustrates an example of a block diagram of a wearable device (e.g., a wearable device (101)). The wearable device (101) of Fig. 4 may be an example of the electronic device (101) of Fig. 1 or the wearable devices (101) of Figs. 2A to 3B.

[0089] Referring to FIG. 4, a wearable device (101) according to one embodiment may include a processor (410) (e.g., processor (120)), a memory (415), a display (250) (e.g., the first display (250-1) and / or the second display (250-2) of FIGS. 2A, 2B, 3A, and 3B), a sensor (420) (e.g., an image sensor (421) and / or a motion sensor (422)), and / or a communication circuit (430) (e.g., including at least a portion of the communication module (190) of FIG. 1). The processor (410), the memory (415), the display (250), the sensor (420), and / or the communication circuit (430) may be electrically and / or operatively connected to each other by electronic components such as a communication bus (402). In the present disclosure, the operational connection of electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components, such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (101) is not limited to those illustrated in FIG. 4. For example, the wearable device (101) may include only some of the electronic components illustrated in FIG. 4.

[0090] A processor (410) of a wearable device (101) according to one embodiment may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. The processor (410) may have a multi-core processor structure such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (410) may include a structure (e.g., a big-little structure) based on a plurality of core circuits that are distinguished by power consumption, clock, and / or calculation amount per unit time. In one embodiment comprising a processor (410) having a multi-core processor architecture, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (410).

[0091] The memory (415) of the wearable device (101) according to one embodiment may include electronic components for storing data and / or instructions input to and / or output from the processor (410). The memory (415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC). In one embodiment, the memory (415) may be referred to as storage.

[0092] In one embodiment, the display (250) of the wearable device (101) can output visualized information to the user of the wearable device (101). The display (250), which is arranged in front of the eyes of the user wearing the wearable device (101), can be arranged on at least a portion of the housing of the wearable device (101) (e.g., the first display (250-1) and / or the second display (250-2) of FIGS. 2A, 2B, 3A, and 3B). For example, the display (250) can be included in a display assembly. For example, the display (250) can be controlled by a processor (410) including circuits such as a CPU (411), a GPU (graphics processing unit) (412), and / or a DPU (display processing unit) (413), to output visualized information to the user. The display (250) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (250) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). The embodiment is not limited thereto, and for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (250) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (250) may be referred to as a display panel and / or a display module.The pixels included in the display (250) may be arranged to face either of the user's two eyes when the wearable device (101) is worn by the user. For example, the display (250) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0093] In one embodiment, the sensor (420) of the wearable device (101) may generate electrical information that may be processed by the processor (410) and / or the memory (415) from non-electronic information related to the wearable device (101). For example, the sensor (420) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (101). In addition to the GPS method, the sensor (420) may generate information indicating the geographic location of the wearable device (101) based on a global navigation satellite system (GNSS) such as, for example, Galileo or Beidou (compass). The information may be stored in the memory (415), processed by the processor (410), and / or transmitted to another electronic device distinct from the wearable device (101) via a communication circuit.

[0094] Referring to FIG. 4, an image sensor (421) and / or a motion sensor (422) are illustrated as examples of a sensor (420) included in a wearable device (101). The sensor (420) may include one or more optical sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate electrical signals representing the color and / or brightness of light. The image sensor (421) may be referred to as a camera. A plurality of optical sensors included in the image sensor (421) may be arranged in the form of a two-dimensional grid (4-dimensional array). The image sensor (421) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional grid. For example, photographic data captured using the image sensor (421) may mean one (a) two-dimensional frame data acquired from the image sensor (421). For example, video data captured using the image sensor (421) may mean a sequence of a plurality of two-dimensional frame data acquired from the image sensor (421) according to a frame rate. The image sensor (421) may be arranged toward the direction in which the image sensor (421) receives light and may further include a flash light for outputting light toward the direction.

[0095] According to one embodiment, the wearable device (101) may include a plurality of image sensors, as an example of an image sensor (421), arranged in different directions. As described above with reference to FIGS. 2A, 2B, 3A, and 3B, the plurality of image sensors may include gaze tracking cameras (e.g., gaze tracking camera 260-1 of FIGS. 2B and 3A) configured to be arranged toward the eyes of a user wearing the wearable device (101). The plurality of image sensors may include outward cameras. The processor (410) may identify the direction of the user's gaze using images and / or videos acquired from the gaze tracking cameras. The gaze tracking cameras may include infrared (IR) sensors. The gaze tracking cameras may be referred to as eye sensors and / or eye trackers.

[0096] The external camera may be positioned facing the front of the user wearing the wearable device (101) (e.g., in the direction that both eyes may face). The wearable device (101) may include multiple external cameras. The embodiment is not limited thereto, and the external camera may be positioned facing the external space. Using images and / or videos acquired from the external cameras, the processor (410) may identify external objects. For example, the processor (410) may identify the position, shape, and / or gesture (e.g., hand gesture) of the hand of the user wearing the wearable device (101) based on the images and / or videos acquired from the external cameras. Using images and / or videos of the external environment acquired from the external cameras, the processor (410) may recognize or track one or more objects within the external environment.

[0097] In one embodiment, the motion sensor (422) may output electrical signals representing gravitational accelerations, accelerations, and / or angular velocities of a plurality of axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other and based on a designated origin within the wearable device (101) and / or the motion sensor (422). For example, the processor (410) may repeatedly receive or acquire sensor data including accelerations, angular velocities, and / or magnitudes of magnetic fields of the plurality of axes from the motion sensor (422) based on a designated period (e.g., 1 millisecond). In one embodiment, the motion sensor (422) may be referred to as an inertial measurement unit (IMU). The sensor (420) included in the wearable device (101) is not limited to those described above, and may include a grip sensor, a proximity sensor, a heart rate sensor, a fingerprint sensor, an ambient light sensor, and / or a ToF sensor. Using the motion sensor (422), the processor (410) can detect motion of the wearable device (101) (e.g., motion of the wearable device (101) caused by a user wearing the wearable device (101).

[0098] In one embodiment, the communication circuit (430) of the wearable device (101) may include hardware components for supporting transmission and / or reception of signals between the wearable device (101) and an external electronic device (e.g., electronic device (102), electronic device (104)). The communication circuit (430) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (430) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), zigbee, long term evolution (LTE), and 5G new radio (NR).

[0099] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (410) of the wearable device (101) may be stored in the memory (415) of the wearable device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (101) and / or the processor (410) may perform at least one of the operations described below when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, a program, and / or a software application is executed. Hereinafter, the fact that a software application is installed in a wearable device (101) may mean that one or more instructions provided in the form of a software application (or package) are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (101)). For example, the application may include a program and / or a library related to a service provided to a user.

[0100] Referring to FIG. 4, programs installed in the wearable device (101) may be included in any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (250), and / or the sensor (420)) of the wearable device (101) may be included in the hardware abstraction layer (480). The framework layer (450) may be referred to as an XR framework layer from the perspective of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 4 may be logically (or for convenience of explanation) separated, and may not mean that the address space of the memory (415) is separated by the layers.

[0101] Within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a position tracker (471), a space recognizer (472), a gesture tracker (473), and / or an eye-gaze tracker (474)) may be included. The programs included in the framework layer (450) may provide an API (application programming interface) that is executable (or callable) based on other programs.

[0102] The application layer (440) may include programs designed to target users of the wearable device (101). Examples of programs included in the application layer (440) include an extended reality (XR) system user interface (UI) (441) and / or an XR application (442), but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (440) may call APIs to cause execution of functions supported by programs included in the framework layer (450).

[0103] The wearable device (101) may display one or more visual objects on the display (250) for performing interaction with the user based on the execution of the XR system UI (441). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user with functions available within a virtual space based on the execution of the XR system UI (441).

[0104] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plug-in (444) are illustrated to be included within the XR system UI (441), but are not limited thereto. For example, based on the XR system UI (441), the processor (410) may execute a lightweight renderer (443) and / or an XR plug-in (444) within the framework layer (450).

[0105] The wearable device (101) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline from the perspective of defining a rendering pipeline that allows partial changes. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (444). The XR plug-in (444) may be referred to as an open XR native client from the perspective of defining (or configuring) an entire rendering pipeline.

[0106] The wearable device (101) may display a screen representing at least a portion of a virtual space on the display (250) based on the execution of the XR application (442). The XR plug-in (444-1) included in the XR application (442) may include instructions that support functions similar to those of the XR plug-in (444) of the XR system UI (441). Descriptions of the XR plug-in (444-1) that overlap with those of the XR plug-in (444) may be omitted. The wearable device (101) may cause the execution of the virtual space manager (451) based on the execution of the XR application (442).

[0107] The wearable device (101) can display an image on the display (250) in a virtual space based on the execution of the application (445). The application (445) can be configured to output image information for displaying a two-dimensional image. The wearable device (101) can cause the execution of the virtual space manager (451) based on the execution of the application (445). The wearable device (101) can generate dual image information to display the two-dimensional image in a three-dimensional virtual space based on the execution of the application (445). Here, the dual image information can include first image information for the left eye and second image information for the right eye, taking into account binocular disparity. In order to display the two-dimensional image in the three-dimensional virtual space, the wearable device (101) can generate the dual image information based on the image information for displaying the two-dimensional image.

[0108] According to one embodiment, the wearable device (101) can provide a virtual space service based on the execution of the virtual space manager (451). For example, the virtual space manager (451) can include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (451), the wearable device (101) can identify a virtual space formed based on the user's location indicated by data acquired through the sensor (420), and display at least a portion of the virtual space on the display (250). The virtual space manager (451) can be referred to as a composition presentation manager (CPM).

[0109] The virtual space manager (451) may include a runtime service (452). For example, the runtime service (452) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (101) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (452). For example, the wearable device (101) may perform rendering for a virtual space service for the user based on the execution of the runtime service (452). For example, a function related to a virtual space, executable by the application layer (440), may be supported based on the execution of the runtime service (452).

[0110] The virtual space manager (451) may include a pass-through manager (453). Based on the execution of the pass-through manager (453), the wearable device (101) may display an image and / or video representing an actual space acquired through an external camera on at least a portion of the screen while displaying a screen representing a virtual space on the display (250).

[0111] The virtual space manager (451) may include an input manager (454). The wearable device (101) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (470) based on the execution of the input manager (454). The wearable device (101) may use the acquired data to identify user input related to the wearable device (101). The user input may be related to a motion (e.g., a hand gesture), gaze, and / or speech of the user identified by a sensor (420) (e.g., an image sensor (421) such as an external camera). The user input may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0112] The perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). From the perspective of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referenced as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0113] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data acquired from the sensor (420). The one or more programs may include at least one of a position tracker (471), a space recognizer (472), a gesture tracker (473), and / or an eye tracker (474). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those illustrated in FIG. 4.

[0114] The wearable device (101) can identify the posture of the wearable device (101) using the sensor (420) based on the execution of the position tracker (471). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using an external camera (e.g., an image sensor (421)) and / or an IMU (e.g., a motion sensor (422) including a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (471). The position tracker (471) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0115] The wearable device (101) can obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of the space recognizer (472). The wearable device (101) can reproduce the surrounding environment of the wearable device (101) in three dimensions using data obtained using an external camera (e.g., an image sensor (421)) based on the execution of the space recognizer (472). The wearable device (101) can identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (101) reproduced in three dimensions based on the execution of the space recognizer (472). The space recognizer (472) can be referred to as a scene understanding (SU) module (or a scene recognition program).

[0116] The wearable device (101) can identify (or recognize) the pose and / or gesture of the user's hand based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user's hand using data acquired from an external camera (e.g., an image sensor (421)) based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user's hand based on data (or images) acquired using the external camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0117] The wearable device (101) can identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (474). For example, the wearable device (101) can identify eye movements of the user using data acquired from a gaze tracking camera (e.g., an image sensor (421)) based on the execution of the gaze tracker (474). The gaze tracker (474) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0118] The recognition service layer (470) of the wearable device (101) may further include a face tracker (475) for tracking the user's face. For example, the wearable device (101) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (475). The wearable device (101) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (475). As an example, the wearable device (101) may identify the movement of the user's face and / or the user's expression based on data (e.g., images and / or videos) acquired using a camera (425) (e.g., a camera directed at at least a portion of the user's face) based on the execution of the face tracker (475).

[0119] Referring to FIG. 4, examples of a processor (410) include a CPU (411), a GPU (graphics processing unit) (412), and / or a DPU (display processing unit) (413). A renderer (490) may include instructions for rendering images in a three-dimensional virtual space. A processor (410) (e.g., DPU (413)) executing the renderer (490) may obtain at least one image to be at least partially displayed in a display area of ​​a display (250) from a software application (e.g., a software application executed by the CPU (411) and / or GPU (412)). For example, a processor (410) executing the renderer (490) may determine a location of an area in which an application (e.g., an XR application (442), an application (445)) is to be rendered. The processor (410) executing the renderer (490) can generate an image of the application to be displayed on the display (250). The renderer (490) can synthesize images to generate a composite image to be displayed on the display (250).

[0120] The processor (410) executing the renderer (490) can divide the display area of ​​the display (250) into a foveated portion (or may be referred to as the foveated area) and a peripheral portion (or may be referred to as the residual area) using the gaze position calculated using the position tracker (471) and / or the gaze tracker (474). For example, the processor (410) detecting the coordinate values ​​of the gaze position can determine the portion of the display area including the coordinate values ​​as the foveated area. The DPU (413) executing the renderer (490) can obtain at least one image corresponding to each of the foveated area and the residual area, and having a size smaller than the size of the entire display area of ​​the display (250) or a resolution smaller than the resolution of the display area.

[0121] The processor (410) executing the renderer (490) may obtain or generate a composite image to be displayed on the display (250) by synthesizing an image corresponding to the foveated area and an image corresponding to the peripheral area. For example, the processor (410) may perform upscaling to enlarge the image corresponding to the peripheral area to the size of the entire display area of ​​the display (250). On the enlarged image, the processor (410) may combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (250). Along the boundary line of the image corresponding to the foveated area, the processor (410) may apply a visual effect, such as blur, to blend the enlarged image and the image corresponding to the foveated area.

[0122] Fig. 5 shows an example of a block diagram of an electronic device (e.g., electronic device (101), wearable device (101)) for displaying an image in a virtual space. In Fig. 5, an example of executing a plurality of programs / instructions for displaying an image in a virtual space is described. The plurality of programs / instructions may all be executed in one processor (e.g., AP) or may be executed by a plurality of processors (e.g., AP, GPU (graphics processing unit), NPU (neural processing unit)). The meaning of being executed by the plurality of processors means that some programs / instructions may be executed by a first processor and other some programs / instructions may be executed by a second processor different from the first processor.

[0123] Referring to FIG. 5, the electronic device (101) may execute a virtual space manager (550) (e.g., the virtual space manager (451) of FIG. 4, CPM) to render an image in a virtual space. For the virtual space manager (550), at least some of the descriptions of the virtual space manager (451) of FIG. 4 may be referred to. The virtual space manager (550) may include a platform for supporting a virtual space service. The virtual space manager (550) may include a runtime service (551) (e.g., OpenXR Runtime), a panel rendering (552) (e.g., 2D Panel Render), and an XR composition unit (553) (XR Compositor). The electronic device (101) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (551). For the runtime service (551), at least some of the descriptions of the runtime service (452) of FIG. 4 may be referred to. The electronic device (101) can display at least one image (video) on a panel (e.g., a 2D panel) to implement a virtual space through the display based on the execution of the panel rendering (552). For example, the electronic device (101) can display a rendering image corresponding to RGB (red green blue) information (566) for the panel from the spatialization manager (540) described below through the display (e.g., the display (250)). The electronic device (101) can synthesize an image of an actual area captured by a camera in a virtual space (hereinafter, a pass-through image) with a virtual area image based on the execution of the XR synthesis unit (553). For example, the electronic device (101) can generate a composite image by merging the pass-through image and the virtual area image based on the execution of the XR synthesis unit (553).The electronic device (101) can transmit the generated composite image to the display buffer so that the composite image is displayed. The electronic device (101) can identify a virtual space through a virtual space manager (550) and display at least a portion of the virtual space on the display (250). The virtual space manager (550) may be referred to as a CPM. The electronic device (101) can execute the virtual space manager (550) to render an image corresponding to at least a portion of the virtual space.

[0124] According to one embodiment, the electronic device (101) may execute a spatialization manager (540). The spatialization manager (540) may perform processes for displaying an image in a three-dimensional virtual space. The electronic device (101) may perform preprocessing based on the execution of the spatialization manager (540) so that the image can be rendered in a three-dimensional virtual space through the virtual space manager (550). For example, the electronic device (101) may perform at least some of the functions of the renderer (490) of FIG. 4 based on the execution of the spatialization manager (540). The electronic device (101) may process image information provided by an application (e.g., an XR application (510), an application (520) that provides a general 2D screen other than XR, and an application that provides a system UI (530)) based on the execution of the spatialization manager (540). A spatialization manager (540) (e.g., Space Flinger) may include a system scene manager (541) (e.g., System scene), an input manager (542) (e.g., Input Routing), and a lightweight rendering engine (543) (e.g., Impress Engine). The system scene manager (541) may be executed to display a system UI (530). System UI-related information (564) may be transmitted to the system scene manager (541) from a program (e.g., API) that provides the system UI (530). The system UI-related information (564) may be obtained through a spatializer API and / or a same-process private API. The spatialization manager (540) may determine the layout (e.g., location, display order) of the screen of the system UI (530) in a three-dimensional space through pre-allocated resources. The system screen manager (541) can transmit image information (567) for rendering the screen of the system UI (530) to the virtual space manager (550) according to the above layout.The input manager (542) may be configured to process user input (e.g., user input on a system screen or an app screen). The lightweight rendering engine (543) may be a renderer for generating images (e.g., the lightweight renderer (443)). For example, the lightweight rendering engine (543) may be used to display the system UI (530). According to one embodiment, the spatialization manager (540) may include a lightweight rendering engine (543) for rendering the system UI. According to one embodiment, when the lightweight rendering engine (543) does not have sufficient resources to render an avatar used in the HMD, at least one external rendering engine may be used. In this case, in order to resolve compatibility issues with external rendering (e.g., a 3rd party engine), an external rendering engine support module may be added within the spatialization manager (540).

[0125] According to one embodiment, the electronic device can execute an application. For example, in response to the execution of an XR application (510) (e.g., an XR application (442), a 3D game, an XR map, or other immersive application), the electronic device can execute a virtual space manager (550). The electronic device (101) can provide dual image information (561) provided from the XR application (510) to the virtual space manager (550). In order to display an image in a three-dimensional space, the dual image information (561) can include two pieces of image information that take binocular parallax into account. For example, the dual image information (561) can include first image information for the user's left eye and second image information for the user's right eye for rendering in a three-dimensional virtual space. Hereinafter, in the present disclosure, the term dual image information is used to refer to image information for displaying images for both eyes in a three-dimensional space. In addition to dual image information, the above dual image information may also include binocular image information, dual image information, dual image data, dual images, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimensional conversion image data, binocular parallax image data, and / or equivalent technical terms. The electronic device (101) can generate a composite image by merging image layers through a virtual space manager (550). The electronic device (101) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (250) of the electronic device (101).

[0126] According to one embodiment, the electronic device can execute at least one application among an XR application (510) and other applications (520) (e.g., a first application (520-1), a second application (520-2), ..., an Nth application (520-N)). According to one embodiment, the application (520) can be configured to output image information for displaying a two-dimensional image. In other words, the application (520) can provide a two-dimensional image. For example, the application (520) can be a video application, a schedule application, or an Internet browser application. Let us assume that, in response to the execution of the application (520), image information (562) provided from the application (520) is provided to the virtual space manager (550). Since the image information (562) has only x-coordinates and y-coordinates within a two-dimensional plane, it may be difficult to consider the precedence relationship (i.e., the distance from the user) between other applications with respect to the user. The electronic device (101) may execute the spatialization manager (540) to provide dual image information to the virtual space manager (550) even when displaying an application (520) that provides a general 2D screen. For example, based on the execution of the spatialization manager (540), the electronic device (101) may receive application-related information (563) from the first application (520-1). For example, the application-related information (563) may include image information representing a 2D image of the first application (520-1) (e.g., information including RGB for each pixel) and / or content information in the first application (520-1) (e.g., characteristics of content executed in the first application, type of content). The application-related information (563) may be acquired through a spatialization API.Based on the execution of the spatialization manager (540), the electronic device (101) can identify information (hereinafter, location information) about the location of the area to be rendered and the size of the area to be rendered by the first application (520-1). Based on the execution of the spatialization manager (540), the electronic device (101) can generate dual image information (565, e.g., RGBx2) that takes into account the user's binocular disparity through the image information and the location information. Based on the execution of the spatialization manager (540), the electronic device (101) can provide the dual image information (565) to the virtual space manager (550). By converting a simple two-dimensional image into the dual image information (565), a problem that occurs when the image information (562) is directly transmitted to the virtual space manager (550) can be resolved. In addition, since at least some of the functions for displaying images in a virtual space are performed by the spatialization manager (540) instead of the virtual space manager (550), the burden on the virtual space manager (550) may be reduced. However, since the image information from the application (520) is not directly transmitted to the virtual space manager (550) but is transmitted through the spatialization manager (540), the quality of the image ultimately output to the user may be reduced. For example, in the first application (520-1), an image is rendered at a resolution of about 2756 x 1846, but the image may be downsampled in the process of being transmitted to the virtual space manager (550) through the spatialization manager (540) (e.g., downsampled from a resolution of about 2756 x 1846 to a resolution of about 1160 x 680). Thereafter, the virtual space manager (550) can upsample the downsampled image (e.g., upsample from a resolution of about 1160 x 680 to a resolution of about 1625 x 1070) and transfer the upsampled image to the display buffer.In this way, when an image is transferred from an application (520) to a spatialization manager (540) and then transferred from the spatialization manager (540) to a virtual space manager (550), a resolution mismatch may occur, or an aliasing problem or a deterioration in image quality may occur during the upsampling process. To solve the above-described problems, the present disclosure describes techniques for controlling the resolution of an area to be displayed in an application and performing foveation rendering based on the system structure illustrated in FIG. 5.

[0127] Figure 6a shows an example of a structure of multiple layers.

[0128] Referring to FIG. 6A, programs installed in the wearable device (101) can be classified into one of the platform layer (610), the recognition service layer (620) (e.g., the recognition service layer (470) of FIG. 4), and the sensor service layer (630). For example, the wearable device (101) can operate based on the platform layer (610), the recognition service layer (620), and the sensor service layer (630).

[0129] According to one embodiment, the platform layer (610) may be configured for an XR service. For example, the platform layer (610) may include a platform (e.g., an Android platform) for supporting an XR service. For example, the platform layer (610) may include a virtual space manager (550) of FIG. 5. The platform layer (610) may include a runtime service (611). For the runtime service (611), reference may be made to the descriptions of the runtime service (551) of FIG. 5 and the descriptions of the runtime service (452) of FIG. 4. As an example, the runtime service (611) may be referred to as an OpenXR runtime module. The runtime service (611) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (101). As an example, the runtime service (611) may be used to perform rendering for an XR service to a user. For example, an application (e.g., unity or OpenXR native application) can be implemented based on the runtime service (611).

[0130] The perception abstraction layer (612) can be used for data exchange between the platform layer (610) and the perception service layer (620). For the perception abstraction layer (612), the descriptions of the perception abstraction layer (460) of FIG. 4 can be referenced. For example, the perception abstraction layer (612) can be referenced as OpenPX. The perception abstraction layer (612) can be used for a perception client and a perception service.

[0131] According to one embodiment, the recognition service layer (620) may include a service module (621), a recognition plug-in layer (622), a sensor management module (623), a playback module (624), and / or an external data management module (625). For example, the recognition service layer (620) may include at least one of the service module (621), the recognition plug-in layer (622), the sensor management module (623), the playback module (624), and / or the external data management module (625). For example, at least some of the service module (621), the recognition plug-in layer (622), the sensor management module (623), the playback module (624), and the external data management module (625) may be omitted.

[0132] The service module (621) can manage input data of the wearable device (101). The service module (621) can be used to manage data (e.g., gesture information) acquired from a plurality of recognition modules included in the recognition plug-in layer (622). For example, the service module (621) can be referenced as SxrDataService.

[0133] The service module (621) can interface with an upper layer (e.g., the platform layer (610) or the runtime service (611)). The service module (621) can exchange data with the upper layer (e.g., the platform layer (610) or the runtime service (611)) through the recognition abstraction layer (612). For example, the recognition abstraction layer (612) can be referred to as OpenPX. According to an embodiment, the service module (621) can support not only OpenPX but also OpenXR Extension. The service module (621) can be used to exchange data (e.g., gesture information) between a plurality of recognition modules. The service module (621) can be configured to manage data processed in the recognition service layer (620). The service module (621) can select data to be recognized as input of the wearable device (101) from among the above data. The above data may include data acquired from multiple recognition modules and data acquired through an external data management module (625). The service module (621) may manage data to be used in the recognition abstraction layer (612). The service module (621) may select data to be recognized as input from the wearable device (101) among the above data and provide the data to the recognition abstraction layer (612).

[0134] The recognition plug-in layer (622) may include multiple recognition modules. The multiple recognition modules may be referred to as a plurality of perception solutions.

[0135] For example, the plurality of recognition modules may include at least one of a head tracking (HeT) module (622-1), a scene understanding (SU) module (622-2), a hand tracking (HaT) module (622-3), an eye tracking (ET) module (622-4), and a face tracking (FT) module (622-5). Each of the plurality of recognition modules included in the recognition plug-in layer (622) may include a common interface for connection (or interworking) with the sensor management module (623). Each of the plurality of recognition modules may include a common interface for connection (or interworking) with the sensor management module (623).

[0136] According to one embodiment, the head tracking module (622-1) can identify the pose of the wearable device (101) using at least one sensor of the wearable device (101). For example, the head tracking module (622-1) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) based on data acquired using a camera (e.g., image sensor (421) of FIG. 4) and an IMU.

[0137] According to one embodiment, the environment recognition module (622-2) may be used to construct a surrounding environment of the wearable device (101) (or a user of the wearable device (101)) into a three-dimensional virtual space. The environment recognition module (622-2) may be used to reconstruct the surrounding environment of the wearable device (101) in three dimensions based on data acquired using a camera (e.g., the image sensor (421) of FIG. 4). The environment recognition module (622-2) may identify at least one of a plane, a slope, and stairs based on the three-dimensionally reconstructed surrounding environment of the wearable device (101).

[0138] In one embodiment, the hand tracking module (622-3) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (101). For example, the hand tracking module (622-3) may identify a pose and / or gesture of a hand of a user based on data acquired from at least one sensor. For example, the hand tracking module (622-3) may identify a pose and / or gesture of a hand of a user based on data (e.g., an image) acquired using a camera.

[0139] According to one embodiment, the eye tracking module (622-4) may be used to identify (or track) eye movements of a user of the wearable device (101). For example, the eye tracking module (622-4) may identify eye movements of the user based on data acquired from at least one sensor. For example, the eye tracking module (622-4) may identify eye movements of the user based on data acquired using a camera (e.g., the gaze tracking camera (260-1) of FIGS. 2B and 3A) and / or an infrared light emitting diode (IR LED).

[0140] In one embodiment, the face tracking module (622-5) may be used to identify (or track) the user's facial movements and / or the user's facial expressions. The face tracking module (622-5) may estimate the user's facial expressions based on the user's facial movements. For example, the face tracking module (622-5) may identify the user's facial movements and / or the user's facial expressions based on data (e.g., images) acquired using a camera (e.g., camera (260) of FIGS. 2A and 2B ).

[0141] For example, a plurality of recognition modules included in the recognition plug-in layer (622) may be configured in a plug-in structure. For example, some of the plurality of recognition modules may be replaced with other modules regardless of the sensor service layer (630) and platform layer (610), which are lower layers of the recognition service layer (620).

[0142] In one embodiment, the sensor management module (623) may be used to provide (or transmit) data to each of a plurality of recognition modules through a common interface. For example, the sensor management module (623) may be used to separate (or remove) the dependency between the sensor service layer (630), which is a lower layer, and the recognition plug-in layer (622), which is an upper layer. For example, the sensor management module (623) may be referred to as SxrSensorSeviceManger.

[0143] The sensor management module (623) can support various modules (or sensor services) of the sensor service layer (630). The plurality of recognition modules may not directly interface with the sensor service layer (630). The plurality of recognition modules may receive data (e.g., sensor data) through the sensor management module (623). Therefore, even if a module of the sensor service layer (630) is changed, the plurality of recognition modules may not be affected.

[0144] The sensor management module (623) may further include a load balancing module. The load balancing module may identify data provided from the sensor service layer (630). The load balancing module may identify at least some of the plurality of recognition modules based on the data provided from the sensor service layer (630). The load balancing module may provide data to at least some of the identified recognition modules. For example, the load balancing module may distribute data to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable device (101). For example, the load balancing module may filter the data provided to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable device (101). Depending on the embodiment, the load balancing module may be configured independently from the sensor management module (623). The load balancing module may be referred to as SxrPerceptionLoadBalancer.

[0145] The playback module (624) can be used to provide a stored dataset to at least one of a plurality of recognition modules in real time through playback. For example, the dataset can be stored through the playback module (624) based on a specified standard. The dataset can include first data acquired from the sensor service layer (630) as well as second data (e.g., virtual object data or synthetic data) acquired based on the first data acquired from the sensor service layer (630). For example, the first data can be referred to as sensor data. The second data can be referred to as virtual data.

[0146] According to an embodiment, the wearable device (101) may receive data from an external electronic device. For example, the data received from the external electronic device may include first data obtained from a service layer included in the external electronic device and / or second data obtained based on the first data. The wearable device (101) may perform playback (or a playback function) using the data received from the external electronic device. The wearable device (101) may transmit the result of performing the playback (or the playback function) to the external electronic device. For example, the wearable device (101) may be used to process the data obtained from the external electronic device on its behalf. The wearable device (101) may receive data obtained from at least one sensor of the external electronic device. Based on the received data, the wearable device (101) may obtain information (e.g., information on a 6-degrees-of-freedom posture) through the playback module (624) (or multiple recognition modules). The wearable device (101) can transmit the acquired information to an external electronic device. The external electronic device can provide an XR service based on the acquired information.

[0147] The playback module (624) can perform playback (or a playback function) based on at least one of the first data and the second data. In some embodiments, the playback module (624) can perform playback by combining (or mixing) real-time data (e.g., runtime data) and pre-stored data.

[0148] For example, playback may refer to a function that utilizes data (or gesture information) stored based on the operation of a wearable device (101). For example, playback may refer to a function that identifies a value for the performance of an XR service by comparing gesture information acquired based on a specified operation related to the XR service with reference gesture information based on the specified operation.

[0149] For example, playback may refer to a function for obtaining performance information of an XR service provided to a user of a wearable device (101). The playback module (624) may identify information (e.g., gesture information) about a user who performed a specified action (e.g., a mission) regarding the XR service. The playback module (624) may identify reference information about the specified action. The reference information may refer to information for determining the completion of performance of the specified action. The playback module (624) may identify the similarity between the information about the user who performed the specified action and the reference information. Based on the similarity, the playback module (624) may identify whether the performance of the action specified by the user has been completed.

[0150] In some embodiments, the playback module (624) may be included in the sensor management module (623). For example, the playback module (624) may perform playback through the sensor management module (623) without changing multiple recognition modules.

[0151] The external data management module (625) may be used to manage data acquired through an external electronic device (e.g., a smartwatch, a smartphone, or a tablet PC) (or at least one sensor of the external electronic device) connected to the wearable device (101). For example, the external data management module (625) may use data acquired from the external electronic device to improve the accuracy of multiple recognition modules. For example, the external data management module (625) may use data acquired from the external electronic device to correct data (or gesture information) acquired from multiple recognition modules. Depending on the embodiment, the external data management module (625) may not be included in the recognition service layer (620).

[0152] The sensor service layer (630) may be used to control at least one sensor (e.g., a camera, an IMU, a time of flight (TOF) sensor). For example, the sensor service layer (630) may be used to provide a service for accessing at least one sensor. For example, the sensor service layer (630) may include at least one of a module for a VR service (e.g., QVRservice), a module for an XR service (e.g., SxrSensorService), a sensor API (e.g., android sensor API), and a sensor hardware abstraction layer (sensor HAL).

[0153] According to one embodiment, the sensor management module (623) may provide sensor data to the recognition plug-in layer (622) through a common interface. For example, the sensor management module (623) may provide sensor data to each of a plurality of recognition modules through the same interface. For example, the sensor management module (623) may provide sensor data according to the operation of the recognition module to the recognition module without changing the configuration information of the recognition plug-in layer (622) based on changing (or modifying) the configuration information (e.g., configuration file) regarding the sensor management module (623).

[0154] The sensor management module (623) can identify sensor data for at least one recognition module based on the operation of at least one recognition module among a plurality of recognition modules. The sensor management module (623) can provide the identified sensor data to the at least one recognition module.

[0155] According to one embodiment, when the head tracking module (622-1) is driven, the sensor management module (623) may obtain camera data and IMU data through at least one of a module for VR service, a module for XR service, a sensor API, and a sensor hardware abstraction layer in the sensor service layer (630). The sensor management module (623) may provide the camera data and IMU data to the head tracking module (622-1). Depending on the embodiment, the camera data and the IMU data may be obtained through different modules.

[0156] In one embodiment, when the environment recognition module (622-2) is operated in playback mode, the sensor management module (623) can identify stored camera data and stored posture data. The sensor management module (623) can provide the camera data and posture data to the environment recognition module (622-2).

[0157] In one embodiment, the service module (621) may be configured to eliminate dependency on a higher layer of the recognition plugin layer (622). For example, the higher layer of the recognition plugin layer (622) may include a platform layer (610) (e.g., Android XR) and / or an application layer (e.g., the application layer (440) of FIG. 4).

[0158] The service module (621) can manage input data of the wearable device (101). The service module (621) can be configured to integrate and manage information (e.g., gesture information or tracking data) obtained from multiple recognition modules. The service module (621) can convert information (e.g., gesture information or tracking data) according to the requirements of a higher layer without changing the multiple recognition modules, and then provide the converted information to the higher layer.

[0159] For example, the service module (621) can obtain information on a 6-degree-of-freedom posture from the head tracking module (622-1). The information on the 6-degree-of-freedom posture obtained from the head tracking module (622-1) can be configured in a quaternion format. On the other hand, a higher layer (e.g., the platform layer (610)) can request information on the 6-degree-of-freedom posture configured in an axis-angle representation format. The service module (621) can change (or convert) the information on the 6-degree-of-freedom posture configured in a quaternion format into information on the 6-degree-of-freedom posture configured in an axis-angle representation format. The service module (621) can provide the information on the 6-degree-of-freedom posture configured in an axis-angle representation format to the higher layer (e.g., the platform layer (610)). However, the present invention is not limited thereto. For example, the service module (621) can change (or convert) information about a 6-degree-of-freedom pose configured in an axis-angle representation format into information about a 6-degree-of-freedom pose configured in a quaternion format and provide it to an upper layer.

[0160] For example, the service module (621) can obtain information about hand movements from the hand tracking module (622-3). The information about hand movements can be obtained based on the movements of a first number of joints. On the other hand, a higher layer (e.g., the platform layer (610)) can request information about hand movements obtained based on the movements of a second number of joints. The service module (621) can perform either a joint interpolation procedure or a simplification procedure. The service module (621) can support the structure of the joints required by the higher layer based on performing either a joint interpolation procedure or a simplification procedure.

[0161] The wearable device (101) according to embodiments of the present disclosure can detect a hand object and recognize the posture of the hand object based on data acquired from the sensor management module (623) (e.g., vision processing module) by the hand tracking module (622-3). The wearable device (101) can determine an expected gesture (e.g., pinch, pinch release, move, drag) based on the posture of the hand object, the movement pattern of the hand object, the current state of the wearable device (101), and the rotation range of the joint connected to the hand object. According to one embodiment, a pinch means performing a motion as if grabbing something between fingers, and may mean a finger gesture motion that performs a specific function (e.g., zooming in or out of an image or a function) by spreading or bringing fingers together on a touch screen. The wearable device (101) can provide information about the gesture to a system UI (e.g., system UI (530)). The present disclosure is not intended to improve the accuracy of data obtained from the sensor management module (623), but rather to determine a gesture that matches the user's intention based on the data.

[0162] According to one embodiment, to determine a gesture that matches a user's intention, the wearable device (101) may include an input gesture inference module. If detection of a gesture is uncertain based solely on data acquired from the sensor management module (623), the wearable device (101) may identify a gesture that is most likely to match the user's intention based on the continuity of movement of the hand object, the continuity of joint rotation, a weight for the current posture of the hand object, and / or the system status. The wearable device (101) may provide information on the status of the gesture and / or the time for which the gesture is maintained. The wearable device (101) may store information on the tendency of the gesture (e.g., location, movement pattern, speed, rotation, relationship with the UI) to recognize a gesture that matches the user's intention. The wearable device (101) may include a buffer (e.g., an input tendency buffer) for storing the information.

[0163] In one embodiment, to notify the user of a gesture that matches the user's intention, the wearable device (101) may include a gesture inference UI module. Using the gesture inference UI module, the wearable device (101) may notify that the gesture is currently being provided or may notify how much of the gesture will be provided in the future. Through the notification, the wearable device (101) may prevent unintended input from being provided by the user or guide the user to make more accurate gestures.

[0164] Figure 6b shows an example of a structure of multiple layers.

[0165] Referring to FIG. 6B, the wearable device (101) may include an input / output unit (650). The wearable device (101) may include, as the input / output unit (650), a system UI (651), an input processing module (653), a hand tracking processing module (655), and a gesture inference UI (656). The hand tracking processing module (655) may be used to detect a hand object and track the movement of the hand object. The input processing module (653) may be configured to recognize the shape of a skeleton. The input processing module (653) may be configured to provide the system UI (651) with a result on what kind of hand gesture the user makes through the shape of the skeleton. The gesture inference UI (656) may be used to inform the user of an input corresponding to the user's intention predicted through a series of operations described below. The gesture inference UI (656) can be used to guide the user to input corresponding to the user's intention. For example, the gesture inference UI (656) can display a UI to prevent unintended input from being provided to the user and guide the user to make more accurate gestures.

[0166] The wearable device (101) may include an input gesture inference module (661) and an input tendency storage buffer (663). When the accuracy of gesture detection is low (i.e., when it is determined to be uncertain), the input gesture inference module (655) may determine a gesture that is determined to have a high probability of matching the actual user's intention as an input gesture based on the continuity of hand tracking position movement, the movement speed of hand tracking, the continuity of joint rotation, and the weights of the shape of the skeleton of the hand object. The determined input gesture may have a certain tendency as an input (e.g., speed, rotation, relationship with the UI). Data related to the tendency may be stored in the input tendency storage buffer (663). The data stored in the input tendency storage buffer (663) may be used to infer a gesture intended by the user. For example, when the user's hand object is occluded by an external object, the data may be used to predict the user's intention.

[0167] The wearable device (101) may include a vision processing module (670). The vision processing module (670) may acquire data through processing of a camera (671) (e.g., an RGB camera, a ToF camera), at least one sensor (e.g., an IMU sensor (673)), and / or a processor (675). The vision processing module (670) may be used to recognize a hand object based on the data. The vision processing module (670) may provide the data to a hand tracking processing module (655).

[0168] Figure 7 shows an example of user input according to pinching of the user's fingers.

[0169] Referring to FIG. 7, a user (701) can wear a wearable device (101). The user (701) can provide user input to control the wearable device (101). The wearable device (101) can receive the user input. As a means for providing the user input, at least one hand object (e.g., hand object (710)) of the user (701) can be used. The at least one hand object can include the user's left hand, right hand, or both the left and right hands. The wearable device (101) can identify the movement of the at least one hand object (e.g., hand object (710)) through a hand tracking module (e.g., gesture tracker (473), hand tracking module (622-3)). For example, the wearable device (101) can acquire an image of a front area of ​​a user (701) through at least one camera, and detect at least one hand object (e.g., hand object (710)) within the image. The wearable device (101) can identify a movement of at least one hand object (e.g., hand object (710)) in consecutive images, or identify a movement of at least one hand object (e.g., hand object (710)) through at least one sensor (e.g., IMU sensor, gyro sensor, acceleration sensor).

[0170] Various input methods may be used to control the wearable device (101). Since the camera in the wearable device (101) is constantly running, the position and gesture of the hand object tracked by the camera may be used as user input. According to one embodiment, a pinch input may be used as a user input for the wearable device (101). For example, if a user (701) of the wearable device (101) wishes to move a visual object displayed in a virtual environment, the user (701) may drag the visual object by pinching his / her fingers on the visual object and moving the hand object while the fingers are pinched. The drag may be referred to as a 'pinch-drag'. For example, if a user of the wearable device (101) wishes to draw a picture in the virtual environment, the user (701) may pinch his / her fingers and move the hand object while the fingers are pinched. A pinch may represent a pen input in a virtual environment, and movement of a hand object may represent movement of the pen input. The movement of the pen input may be referred to as 'pinch-drawing'. For example, when a user (701) of a wearable device (101) wishes to select a visual object displayed in a virtual environment, the user (701) may pinch his or her fingers on the visual object. The pinch may be referred to as a 'pinch-click'.

[0171] A user (701) of a wearable device (101) can pinch fingers of at least one hand object (e.g., a hand object (710)). For example, the hand object (710) can include a thumb (711), an index finger (712), a middle finger (713), a ring finger (714), and a little finger (715). The user (701) of the wearable device (101) can pinch two fingers among the fingers. For example, the user (701) of the wearable device (101) can pinch the thumb (711) and the index finger (712). Hereinafter, in the present disclosure, a pinch of the thumb (711) and the index finger (712) is described as an example, but a pinch of other two fingers (e.g., a thumb (711) and a middle finger (713), a thumb (711) and a ring finger (714)) may also be understood as a user input of the present disclosure. The wearable device (101) may detect the pinch through at least one camera. The wearable device (101) may perform image analysis on a pinch portion corresponding to the tip portion of the fingers of at least one hand object (e.g., a hand object (710)) of the user (701). The wearable device (101) may detect the pinch of the user (701) through the image analysis. For example, the wearable device (101) may acquire a skeleton of at least one hand object (e.g., a hand object (710)) of the user (701) through hand tracking. The wearable device (101) can detect a pinch by detecting whether the portion corresponding to the tips of the fingers in the skeleton is in contact.

[0172] In one embodiment, when dragging a panel (e.g., pinch-drag) or drawing a line (e.g., pinch-drawing) in a virtual environment, at least one hand object (e.g., hand object (710)) may be required to move while the pinch is maintained. However, for example, depending on the posture of the hand object (710), the pinched portion of the hand object (710) may not be visible in the field of view of the user (701) (i.e., the field of view of the camera of the wearable device (101)). Referring to example (731), depending on the posture of the hand object (710), the tips of the fingers (e.g., thumb (711), index finger (712)) of the hand object (710) may be obscured by the back of the hand of the hand object (710). Referring to example (732), the tip of at least one finger (e.g., thumb (711), index finger (712)) of the hand object (710) may be obscured by an external object (799). Although not illustrated in FIG. 7, the tip of at least one finger (e.g., thumb (711), index finger (712)) of the hand object (710) may be outside the field of view of the user (701) (i.e., field of view, FoV (FoV)) of the camera of the wearable device (101).

[0173] In one embodiment, the user's (701) intention is to maintain user input by pinching, but if the user's pinch is not recognized, as in example (732), the wearable device (101) may determine that the user input has ended. This may result in a drawing being interrupted or a drag being abruptly terminated. In this disclosure, a technique for predicting the user's intention to maintain user input, such as a pinch, is described.

[0174] The wearable device (101) according to embodiments of the present disclosure can predict the duration of a user input based on the posture and movement of the hand object (710) of the user (701). The wearable device (101) can provide better usability to the user (701) by determining that the user input is applied when a specific condition is met. Unlike estimating the visible gesture recognition more accurately, the wearable device (101) according to embodiments of the present disclosure can provide a user input that matches the intention of the user (701) by predicting features of an invisible part of the hand object (710) through a visible part of the hand object (710) and / or a previous movement of the hand object (710).

[0175] Figure 8 shows the operation flow of a wearable device (wearable device (101)) for identifying user input using a pinch.

[0176] Referring to FIG. 8, in operation (801), the wearable device (101) can identify the start of a user input based on a pinch of the fingers of a user (e.g., the user (701)). For example, the user input may be a pinch-drag or a pinch-draw. The wearable device (101) can detect a pinch of the fingers of the user (701) (e.g., the thumb (711) and the index finger (712)). For example, the wearable device (101) can acquire an image through at least one camera (e.g., the camera (260)) and detect that the tips of the fingers (hereinafter, referred to as pinch portions) are in contact with each other in the image. As an example, the wearable device (101) can detect that the tips of the fingers (hereinafter, referred to as pinch portions) are in contact with each other through a skeleton-shaped guide. According to one embodiment, the wearable device (101) may detect a pinch state when it recognizes that the distance between the fingers has become somewhat close, and when it determines that the distance is within a distance that can be recognized as a pinch (e.g., a predefined threshold distance, within about 2 mm). The wearable device (101) may identify the initiation of the user input through the pinch.

[0177] In operation (803), the wearable device (101) can determine whether a pinch portion is visible in the field of view. A user input according to the pinch may be provided. The user (701) may be pinching his / her fingers. The wearable device (101) can track the movement of at least one hand object (e.g., hand object (710)) of the user (701) while the pinch is applied. The wearable device (101) can identify the movement of the at least one hand object (e.g., hand object (710)) through a hand tracking module (e.g., gesture tracker (473), hand tracking module (622-3)). The wearable device (101) can track the pinch portion through at least one camera (e.g., camera (260)). The wearable device (101) can determine whether the pinch portion is included in an image acquired through at least one camera. The wearable device (101) may determine whether the pinch portion is visible in the field of view of the at least one camera. In one embodiment, to determine whether the pinch portion is visible in the field of view of the at least one camera, the wearable device (101) may determine whether a part of the user's body (e.g., at least one finger) is recognizable enough to determine a pinch. As a non-limiting example, if it is difficult to recognize a part of the user's body (e.g., at least one finger) enough to determine a pinch, the wearable device (101) may determine that the pinch portion is not visible in the field of view of the at least one camera. If the pinch portion is not visible in the field of view of the at least one camera, the wearable device (101) may perform operation (805). If the pinch portion is visible in the field of view of the at least one camera, the wearable device (101) may perform operation (809).

[0178] In operation (805), according to one embodiment, the wearable device (101) may apply a user input in a pinched state for an expected duration. A state in which the pinched portion is not visible may be recognized as a type of gesture. The wearable device (101) may determine the expected duration through a gesture inference algorithm and apply a user input corresponding to the gesture for the expected duration. The wearable device (101) may determine the expected duration. The expected duration may indicate a time for maintaining the pinched state. The expected duration may be referred to as a predicted duration, a buffering time, a buffer duration, an input buffer time, a buffer holding time, an extended duration, an extended duration, an additional duration, an extended time, an extended time, an inference time, a guess time, a delay time, an end delay time, a completion delay time, an input holding time, a selection holding time, a selection duration, a pinch holding time, and / or equivalent technical terms thereof in addition to the expected duration.

[0179] A wearable device (101) according to embodiments of the present disclosure can infer the intention of a user (701). According to one embodiment, the wearable device (101) can determine an expected duration for which a pinch of fingers of at least one hand object (e.g., the hand object (710)) of the user (701) is maintained based on information about the mobility of at least one hand object (e.g., the hand object (710)) of the user (701). For example, the information about the mobility of the hand object (710) can indicate a movement path or a movement pattern of the hand object (710). Since the hand object (710) has a continuous movement in a user input such as a pinch-drag or pinch-drawing, the intention of the user (701) can be predicted. The wearable device (101) can determine a weight for the mobility of the hand object (710) based on data cached for a certain period of time (e.g., 1 second) before determining the expected duration. Even if the pinch portion of the hand object (710) is covered by the back of the hand or an external object (e.g., an external object (799)), the weight may be set higher as the hand object (710) is expected to move continuously. The higher the weight, the longer the expected duration may be set. For example, information about the mobility of the hand object (710) may indicate the speed of the hand object (710). The wearable device (101) may cache information about the speed for a certain period of time (e.g., 1 second) before determining the expected duration. The wearable device (101) may determine the weight for the mobility of the hand object (710) based on the speed. For example, as the speed of the hand object (710) is constant (e.g., the deviation is lower than a threshold value), the user's (701) intention may be to maintain the pinch. The wearable device (101) may set the weight higher as the deviation in the speed of the hand object (710) is small. The higher the weight, the longer the expected duration can be set.For example, as the speed of the hand object (710) suddenly slows down, the user's (701) intention may be to release the pinch. The wearable device (101) may set the weight low when the acceleration of the hand object (710) is negative and the magnitude of the acceleration is greater than or equal to a threshold value.

[0180] According to one embodiment, the wearable device (101) can determine an expected duration for which a pinch of fingers of at least one hand object (e.g., hand object (710)) of the user (701) is maintained based on information about the rotation of joints of at least one hand object (e.g., hand object (710)) of the user (701). Each joint (e.g., each joint of an individual finger) of the at least one hand object has a limited range of rotation (hereinafter, referred to as a rotation range). The wearable device (101) can predict a movement of the at least one hand object (e.g., hand object (710)) based on the rotation range of the joint. The movement of each joint of the at least one hand object is within a predictable range. The wearable device (101) can determine a weight for the continuity of the joint rotation of the at least one hand object (e.g., hand object (710)) based on data cached for a predetermined period of time (e.g., 1 second) before determining the expected duration. For example, the closer the pinch portion of the hand object (710) is to the boundary of the predictable range, the lower the weight for the continuity of the joint rotation can be set by the wearable device (101). This is because it is more difficult to perform the joint rotation. The lower the weight, the shorter the expected duration can be determined. For example, the closer the pinch portion of the hand object (710) is to the interrupted region of the predictable range (e.g., a certain distance or more from the boundary), the higher the weight for the expected duration can be set by the wearable device (101).

[0181] According to one embodiment, the wearable device (101) may determine an expected duration for which a pinch of fingers of at least one hand object (e.g., hand object (710)) of the user (701) is maintained based on a posture of at least one hand object (e.g., hand object (710)) of the user (701). The wearable device (101) may detect the posture of at least one hand object (e.g., hand object (710)) from an image acquired through at least one camera. The at least one hand object (e.g., hand object (710)) may include an invisible pinch portion and a visible portion excluding the pinch portion. The wearable device (101) may determine a shape weight based on the visible portion. The shape weight may be used to determine the expected duration of the pinch. The wearable device (101) may determine the expected duration based on the remaining portion excluding the hidden portion. For example, the wearable device (101) may set the weight low when the distance between the thumb (e.g., the thumb (711)) and the index finger (e.g., the index finger (712)) in the visible portion is greater than or equal to a threshold value. The low weight may cause the expected duration to be short or set to a specific time (e.g., '0'). Here, a weight of '0' may indicate that it is difficult to see that the pinch input is maintained. As another example, the wearable device (101) may set the weight high when the probability that the direction of the joint of the thumb (e.g., the thumb (711)) and the direction of the joint of the index finger (e.g., the index finger (712)) meet each other in the visible portion is high (e.g., the closer the angle formed by the two directions is to a specified angle (e.g., about 30 degrees, 60 degrees)).

[0182] In one embodiment, the wearable device (101) may determine an expected duration for which a pinch of the fingers of the user's (701) hand object (710) is maintained based on information about the state of the application layer of the user's (701) hand object (710). For example, the wearable device (101) may determine a weight based on a state in which the user (701) is operating on the system user interface. The wearable device (101) may determine the expected duration based on the weight. For example, the wearable device (101) may be drawing a line in a drawing app. If the current line is directed toward the starting point of the drawing, the wearable device (101) may determine that the user's intention is to draw a closed curve. Even if the pinch portion is obscured during the drawing, the wearable device (101) may determine that the user's intention is to maintain the pinch input. The wearable device (101) can determine a weight value higher than a reference value.

[0183] According to one embodiment, the wearable device (101) may determine an expected duration for which a pinch of the fingers of the hand object (710) of the user (701) is maintained based on at least one of information about the mobility of the hand object (710) of the user (701), information about the rotation of the joints of the hand object (710), the posture of the hand object (710) of the user (701), or information about the state of the application layer of the hand object (710) of the user (701). The wearable device (101) may output a more accurate prediction result by considering multiple parameters (e.g., information about the mobility of the hand object (710) of the user (701), information about the rotation of the joints of the hand object (710), the posture of the hand object (710) of the user (701), or information about the state of the application layer of the hand object (710) of the user (701)) together with weights. By weighting each parameter, the wearable device (101) can determine the probability that an invisible part of the hand object (710) is pinched and predict the duration of the user input corresponding to the pinch. For example, the user's (701) intention can be determined based on the mathematical formula below.

[0184] <Mathematical Formula 1>

[0185] e = f(a, b, c, d)

[0186] Here, a represents information on the mobility of the hand object (710). For example, the information on the mobility may include weight information on the continuity of the position movement speed. b represents information on the rotation of the joint of the hand object (710). For example, the information on the rotation of the joint may include continuity information indicating the degree to which the joint rotation is a continuous motion. c represents the posture (shape) of the hand object (710) of the user (701). d represents the state of the application layer of the hand object (710) of the user (701). e represents the intention of the user (701). According to one embodiment, the intention of the user (701) may correspond to the expected duration of the user input. For example, when the expected duration is set to a specific time (e.g., '0'), the intention of the user (701) to no longer maintain the pinch input may be indicated. f() represents a function according to the input variable.

[0187] According to one embodiment, the wearable device (101) can determine an expected duration for which a pinch input is maintained according to the above-described method. At least some of the parameters required to determine the expected duration can be cached in the wearable device (101). The wearable device (101) can store at least some of the parameters for a certain period of time. For example, at least one of information about the mobility of the hand object (710) of the user (701), information about the rotation of the joints of the hand object (710), or information about the state of the application layer of the hand object (710) of the user (701) can be cached for a certain period of time (e.g., about 1 second). Thereafter, if it is determined that the pinch portion is covered, the wearable device (101) can determine the tendency of the user input, i.e., the duration for which the pinch is maintained, i.e., the state for which the selection is maintained.

[0188] As a non-limiting example, the wearable device (101) may additionally utilize information about the movement of the hand object (710). The wearable device (101) may determine that the pinch of the user's (701) fingers is maintained for an expected duration and may apply a drag or drawing input according to the movement of the hand object (710). For example, even if the pinch portion is not visible, the wearable device (101) may apply a user input according to the movement, such as a drag or drawing, through hand tracking while the selection is maintained within the expected duration.

[0189] In operation (807), the wearable device (101) may determine whether the expected duration has elapsed. The wearable device (101) may determine whether the expected duration has elapsed while the pinched portion of the user's fingers is not visible from the view of the at least one camera. If the expected duration has elapsed, the wearable device (101) may perform operation (809). If the expected duration has not elapsed, the wearable device (101) may perform operation (803) again. In other words, the wearable device (101) may apply the user input with the pinch applied until the expected duration has elapsed.

[0190] Although not shown in FIG. 8, even before the expected duration elapses, if the pinched portion of the user's fingers becomes invisible from the view of the at least one camera, the wearable device (101) may stop maintaining the selection input according to the expected duration. The wearable device (101) may determine whether to apply the user input based on the pinched portion shown through the at least one camera. If the tips of the fingers shown through the at least one camera are separated, that is, if the pinching of the pinched portion is released, the wearable device (101) may determine the end of the user input.

[0191] In operation (809), the wearable device (101) may execute a function corresponding to the release of the pinch (or may be referred to as a pinch release). The release of the pinch may correspond to the completion of a user input. For example, the user input may be a pinch drag. The user input may include a pinch, a drag, and a pinch release. The wearable device (101) may move a visual object in a virtual environment in response to the user input. Upon completion of the user input, the movement of the visual object may be completed. The wearable device (101) may display the virtual environment in which the visual object has moved. The wearable device (101) may determine the release of the pinch based on identifying that an expected duration has elapsed. In response to determining the release of the pinch, the wearable device (101) may determine an expected location. The above predicted position can be determined based on the movement of the hand object (710) at the time when the release of the pinch is determined (e.g., the time when the expected duration has elapsed). The wearable device (101) can display the visual object at the predicted position according to the release of the pinch. In addition, for example, the user input can be a pinch drawing. The user input can include pinching, dragging, and releasing the pinch. The wearable device (101) can draw a line in the virtual environment according to the user input. Upon completion of the user input, the wearable device (101) can display a line extending from the point where the pinch is performed to the point where the pinch is released. The shape of the line can be determined based on hand tracking of the hand object (710). The wearable device (101) can determine the release of the pinch based on identifying that the expected duration has elapsed. The wearable device (101) can determine the expected end position upon release of the pinch.The wearable device (101) can display a line drawn from the starting position of the pinch (i.e., the starting point) to the expected ending position (i.e., the end point) upon release of the pinch.

[0192] Although FIG. 8 illustrates an example in which various types of parameters and weights for each parameter are determined based on the situation, the embodiments of the present disclosure are not limited thereto. In another embodiment, the weights may be manually set by input from a user (701).

[0193] FIG. 9a illustrates an example of prediction of user input using the mobility of at least one hand object (e.g., hand object (710)) of a user (e.g., user (701)).

[0194] Referring to FIG. 9A, the wearable device (101) can obtain information on the mobility of at least one hand object (e.g., hand object (710)). The information on the mobility can include a change in position according to the movement of the corresponding hand object, a speed according to the change in position, and / or a change in shape. For example, the hand object (710) can move (910) in three-dimensional space while the fingers are pinched. The wearable device (101) can track the movement (910) of the hand object (710). In the field of view of the wearable device (101), the hand object (710) can be observed in the following order: a first shape (911), a second shape (912), a third shape (913), a fourth shape (914), and a fifth shape (915). Since the pinched portion of the hand object (710) is not visible from the fifth form (915), the wearable device (101) can determine whether to maintain the pinched state. According to one embodiment, the wearable device (101) can determine the continuity of the pinched state based on the velocity of at least one hand object (e.g., the hand object (710)). For example, information about the mobility of the at least one hand object can represent the velocity of the hand object (710). Acceleration can be obtained by differentiating the velocity. For example, when the velocity suddenly decreases, the acceleration can rapidly increase. Since the velocity suddenly decreases, the wearable device (101) can predict that the user input will end. That is, when the magnitude of the slope of the velocity is greater than or equal to a threshold value (e.g., the slope is less than a threshold angle with respect to the y-axis), the wearable device (101) can predict that the user input will end. As the change in velocity becomes more rapid, the possibility of differentiating the velocity decreases. Therefore, the wearable device (101) can determine the continuity of the pinched state based on the possibility of differentiating the velocity. As the velocity decreases rapidly, the wearable device (101) can predict that the user input will end. The wearable device (101) can store information indicating a trend, such as velocity, in a buffer.For example, the wearable device (101) may cache information about the velocity for a certain period of time (e.g., 1 second) before determining the expected duration. The wearable device (101) may determine a weight for the mobility of the hand object (710) based on the velocity. For example, the more constant the velocity of the hand object (710) is (e.g., the lower the deviation is than a threshold), the more likely the user's (701's) intention may be to maintain the pinch. The wearable device (101) may set a higher weight as the deviation in the velocity of the hand object (710) is smaller. The higher the weight, the longer the expected duration may be set. For example, the more likely the user's (701's) intention is to release the pinch as the velocity of the hand object (710) suddenly slows down.

[0195] FIGS. 9b and 9c illustrate examples of prediction of user input using the rotation of a joint of at least one hand object (e.g., hand object (710)) of a user (e.g., user (701)).

[0196] Referring to FIG. 9b, the wearable device (101) can predict user input based on a range of rotation (hereinafter, rotation range) of each joint of at least one hand object (e.g., hand object (710)). The rotation range may be limited for each joint. Examples (951a) and (951b) represent a range of rotation of the elbow joint of the user (701). The range of rotation may be from about 0 degrees to about 145 degrees. Referring to example (951a), a rotation angle of a forearm portion connected to the elbow joint of the user (701) may be about 145 degrees based on an upper arm portion connected to the elbow joint of the user (701). Referring to example (951b), a rotation angle of a forearm portion connected to the elbow joint of the user (701) may be about 0 degrees based on an upper arm portion connected to the elbow joint of the user (701). Examples (952a) and (952b) represent the rotation range of the forearm joint of the user (701). The rotation range may be approximately 90 degrees to 90 degrees. Referring to example (952a), the rotation angle of the hand portion connected to the forearm joint of the user (701) may be approximately 80 degrees based on the arm connected to the forearm joint. Referring to example (952b), the rotation angle of the hand portion connected to the forearm joint of the user (701) may be approximately 90 degrees based on the arm connected to the forearm joint of the user (701). Examples (953a) and (953b) represent the rotation range of the wrist joint of the user (701). Example (953a) corresponds to flexion, and as the wrist joint bends, the hand and arm move closer together. Example (953b) corresponds to extension, and as the wrist joint straightens, the hand and arm move away from each other. Examples (954a) and (954b) illustrate the range of rotation of the wrist joint of the user (701). Example (954a) illustrates ulnar deviation in which the wrist moves toward the little finger, and example (954b) illustrates radial deviation in which the wrist moves toward the thumb.

[0197] According to one embodiment, the wearable device (101) can determine an expected duration for a user input according to a pinch based on information about the rotation of a joint of at least one hand object (e.g., a hand object (710)). Even if the pinch portion is obscured, the intention of the user (701) can be inferred through information about the rotation of the joint of at least one hand object (e.g., a hand object (710)). For example, assume a situation where the user (701) drags the hand object (710). A certain pattern can be identified in the rotation of the joint of the hand object (710). The speed can vary depending on the rotation range and state of the joint. For example, when dragging from left to right, the user (701) may have difficulty moving the shoulder any further. In this state, the user (701) may try to maintain the drag to a desired point by turning the wrist. The rotation speed and angle of the wrist in this user input may be within a certain pattern.

[0198] Referring to FIG. 9C, the wearable device (101) can predict user input through the movement of the joints of the fingers of at least one hand object (e.g., the hand object (710)). Example (970) shows a skeleton that takes into account the fingers of the hand object (710) and each finger joint. The skeleton represents a part connected to the joint, and when the joint rotates, the skeleton can move as a whole. The wearable device (101) can predict user input based on the joints for each finger and the skeleton connected to the joint. For example, the wearable device (101) can confirm a curved circular shape through the skeleton of the index finger (712), and can confirm that the tip of the skeleton of the index finger (712) touches the skeleton of the thumb (711). Example (975) shows a 3D visualization model (976) based on the skeletons of each finger of the hand object (710). The wearable device (101) can predict user input based on a three-dimensional visualization model (976) corresponding to skeletons that take into account the movement of the finger joints of the hand object (710).

[0199] FIG. 9d illustrates an example of prediction of user input using the pose of at least one hand object (e.g., hand object (710)) of a user (e.g., user (701)).

[0200] Referring to FIG. 9D, the wearable device (101) can acquire an image (980) including at least one hand object (e.g., hand object (710)) through at least one camera. For example, the image (980) can include the hand object (710). The hand object (710) can include a thumb (711), an index finger (712), a middle finger (713), a ring finger (714), and a little finger (715). The tips (i.e., pinch portions) of the fingers (e.g., the thumb (711), the index finger (712)) of the hand object (710) can be covered by an external object (799). Since the pinch portions of the fingers are not visible, the wearable device (101) can predict whether the fingers are in a pinched state or not. According to one embodiment, the wearable device (101) can determine a pinch state for an invisible portion through a visible portion of the hand object (710). For example, the wearable device (101) can predict a pinch state through a first direction (971) of the thumb (711) and a second direction (972) of the index finger (712) in the visible portion, in a posture determined through the visible portion of the hand object (710). If the first direction (971) and the second direction (972) move further away from the fingertip, the wearable device (101) can predict that the pinch is released. If the first direction (971) and the second direction (972) move closer to the fingertip, the wearable device (101) can predict that the pinch is maintained. As the probability of a pinch being sustained increases, the wearable device (101) may determine a higher shape weight for at least one hand object (e.g., hand object (710)). The shape weight may be used to determine the expected duration of the pinch.

[0201] As a non-limiting example, the wearable device (101) may determine whether the fingers are pinched or not based on the shape of the muscles on the hand object (710) identified through the image (980). As a non-limiting example, the wearable device (101) may also determine the shape weight based on a movement pattern (e.g., a pattern of lifting the fingers) in addition to the posture of the hand object (710).

[0202] Figure 9d shows an example of prediction of user input using the rotation of the finger joints.

[0203]

[0204] FIG. 10 illustrates an operation flow of a wearable device (e.g., a wearable device (101)) for identifying a gesture input using at least one hand object of a user (e.g., a hand object (710)). In FIGS. 7 to 9d, an example of predicting a user input using a pinch is described. However, embodiments of the present disclosure are not limited thereto. FIG. 10 illustrates operations of the wearable device (101) for predicting a user input corresponding to not only a pinch but also other gesture inputs.

[0205] Referring to FIG. 10, in operation (1001), the wearable device (101) can identify the initiation of a gesture input using at least one hand object (e.g., hand object (710)). The wearable device (101) can track the movement of at least one hand object (e.g., hand object (710)). The wearable device (101) can identify the movement of at least one hand object (e.g., hand object (710)) through a hand tracking module (e.g., gesture tracker (473), hand tracking module (622-3)). The wearable device (101) can identify the initiation of a gesture input through the movement of at least one hand object (e.g., hand object (710)). For example, a pinch of the fingers of the hand object (710) can correspond to the initiation of a gesture input. For example, a specific posture of a hand object (710) (e.g., a posture with all fingers extended, a posture with a fist clenched) may correspond to the initiation of a gesture input. For example, movement of a hand object (710) along a specific path (e.g., an action of drawing a character with a finger) may correspond to the initiation of a gesture input.

[0206] In operation (1003), the wearable device (101) may determine that at least a portion of at least one hand object (e.g., hand object (710)) is not visible in a camera field of view (a field of view of at least one camera of the wearable device (101). The wearable device (101) may track the movement of the at least one hand object (e.g., hand object (710)) via the at least one camera. The gesture input of operation (1001) may be ongoing. The wearable device (101) may determine whether at least a portion of the at least one hand object (e.g., hand object (710)) is included in an image acquired via the at least one camera. In other words, while the gesture input is being performed, the wearable device (101) may determine whether at least a portion of the at least one hand object (e.g., hand object (710)) is visible in the field of view of at least one camera of the wearable device (101). If at least a portion of the at least one hand object (e.g., hand object (710)) has moved out of the image or is obscured by an external object or another portion (e.g., back of the hand, wrist) of the at least one hand object (e.g., hand object (710)), the wearable device (101) may determine that at least a portion of the at least one hand object (e.g., hand object (710)) is not visible.

[0207] In operation (1005), the wearable device (101) may determine an expected duration for the duration of a gesture input. A state in which at least a portion of at least one hand object (e.g., hand object (710)) is not visible may be recognized as a type of gesture. The gesture may be referred to as an ambiguous pinch gesture in that it is ambiguous what kind of input it is because at least a portion of the at least one hand object is not visible. The wearable device (101) may determine the expected duration through a gesture inference algorithm. During the expected duration, the wearable device (101) may determine that the gesture input is ongoing. The wearable device (101) may apply that the gesture input is ongoing. The wearable device (101) may perform operations assuming that the gesture input is ongoing. The above expected duration may be referred to as expected duration, predicted duration, buffering time, buffer duration, input buffer time, buffer hold time, extended duration, extended duration, additional duration, extended time, extended time, inference time, guess time, delay time, termination delay time, completion delay time, input hold time, selection hold time, selection duration, gesture hold time, and / or equivalent technical terms.

[0208] A wearable device (101) according to embodiments of the present disclosure can infer an intention of a user (701) related to the duration of a gesture input. According to one embodiment, the wearable device (101) can determine an expected duration for a state in which the gesture input continues based on information about the mobility of a hand object (710) of the user (701). According to one embodiment, the wearable device (101) can determine an expected duration for a state in which the gesture input continues based on information about the rotation of a joint of a hand object (710) of the user (701). According to one embodiment, the wearable device (101) can determine an expected duration for a state in which the gesture input continues based on a posture of a hand object (710) of the user (701). According to one embodiment, the wearable device (101) may determine an expected duration for a state in which the gesture input continues based on information about the state of the application layer of the hand object (710) of the user (701). According to one embodiment, the wearable device (101) may determine an expected duration for a state in which the gesture input continues based on at least one of information about the mobility of the hand object (710) of the user (701), information about the rotation of the joints of the hand object (710), the posture of the hand object (710) of the user (701), or information about the state of the application layer of the hand object (710) of the user (701). To determine the expected duration, the descriptions of FIG. 8 may be referred to.

[0209] In action (1007), the wearable device (101) may display a visual object to indicate the expected duration.

[0210] The above visual object may be used by the wearable device (101) to indicate to the user (701) that a gesture input using the hand object (710) is maintained while at least a portion of the hand object (710) is not visible, and to indicate that the state may end soon. The wearable device (101) may display a visual object to inform the user (701) of the expected duration corresponding to the time for which the state is maintained through at least one display (e.g., at least one display (250)). For example, the wearable device (101) may display the visual object in an area adjacent to the hand object (710). As an example, the visual object may indicate the expected duration and / or remaining time (i.e., the remaining time for a state in which the gesture input is maintained even when the hand object (710) is not visible) in the form of a status bar. As another example, the visual object may indicate the expected duration and / or remaining time in the form of a circle. For example, the wearable device (101) may display a visual object in a specific area of ​​the field of view (e.g., the upper right area) in the virtual environment. As an example, the wearable device (101) may display the visual object together with the system user interface (UI). For example, the wearable device (101) may display an emphasis effect (e.g., blinking, a dark outline, a specific colored outline) on a shape corresponding to a hand object (710) in the virtual environment. As an example, the degree of blinking or the thickness of the outline may indicate the remaining time of the expected duration.

[0211] In operation (1009), the wearable device (101) may execute a function corresponding to the completion of the gesture input based on identifying that an expected duration has elapsed. The wearable device (101) may identify that the expected duration has elapsed while at least a portion of the user's hand object is not visible. As a non-limiting example, the wearable device (101) may identify that the expected duration has elapsed while the visual object is displayed. When the expected duration has elapsed, the wearable device (101) may determine a location corresponding to the completion of the gesture input. Since at least a portion of the hand object (710) is not visible from the field of view, the exact location corresponding to the completion of the gesture input cannot be determined. The wearable device (101) may predict a location corresponding to the completion of the gesture input based on a gesture inference algorithm and / or movement of the hand object (710). The location may be referred to as the predicted location. The wearable device (101) may execute a function corresponding to the completion of the gesture input based on the predicted position. For example, releasing the pinch of the fingers of the hand object (710) may correspond to the completion of the gesture input. For example, a specific posture of the hand object (710) (e.g., a posture of extending all fingers, a posture of making a fist) may correspond to the completion of the gesture input. For example, a movement of the hand object (710) to a specific area (e.g., an action of moving a finger to a specific area in the virtual environment (e.g., a trash can, a desktop area)) may correspond to the initiation of the gesture input.

[0212] Although an example of displaying a visual object to indicate an expected duration to a user (701) is described in FIG. 10 , embodiments of the present disclosure are not limited thereto. According to one embodiment, the wearable device (101) may indicate the expected duration to the user (701) in a manner other than a visual object (e.g., an audio signal, vibration). According to another embodiment, the wearable device (101) may not notify the user (701) of the expected duration. The wearable device (101) may apply a state in which the gesture input is maintained for the expected duration without the operation (1007) of FIG. 10 . According to another embodiment, the wearable device (101) may display a visual object that only indicates that the current hand object (710) is not visible from the camera view and the gesture input is maintained, instead of indicating the expected duration. The wearable device (101) may not display separate information about the time.

[0213] Although FIG. 10 illustrates an example in which various types of parameters and weights for each parameter are determined based on the situation, the embodiments of the present disclosure are not limited thereto. In another embodiment, the weights may be manually set by input from a user (701).

[0214] Although FIG. 10 illustrates an example of caching at least one of the parameters for a certain period of time (e.g., 1 second) and utilizing the cached data, embodiments of the present disclosure are not limited thereto. For example, the wearable device (101) may determine an expected duration for holding a pinch in a user input based on user tendency data stored in memory. As a non-limiting example, the user tendency data may be determined based on data acquired through a hand object as well as data acquired through a controller connected to the wearable device (101). As another example, the wearable device (101) may learn a typical human input pattern and determine an expected duration based on the learned data.

[0215] In addition to the methods described above, the wearable device (101) may determine whether the pinch of the fingers is maintained in an invisible area using at least one of various methods. In one embodiment, the wearable device (101) may predict the shape of the pinch area through a shape reflected in a mirror or shadow in an image acquired through at least one camera. As another example, in addition to a mirror or shadow, a shape reflected on a metal or liquid surface may be used to predict the shape of the tip of the fingers. In one embodiment, the wearable device (101) may predict the shape of the pinch area based on another type of input method. For example, the wearable device (101) may predict the shape of the pinch area based on the position of the user's head acquired through head tracking and / or the position of the user's gaze acquired through eye tracking. For example, when the user's hand object moves from left to right, the user may turn his or her head from left to right. The wearable device (101) can determine that a drag input is in progress while the pinch portion is maintained based on identifying the movement of the user's head through head tracking.

[0216] Fig. 11a illustrates an example of a visual object corresponding to a visibility state. The visibility state may indicate whether a pinch portion of at least one hand object (e.g., hand object (710)) is visible or invisible in the field of view of at least one camera (e.g., camera (260)) of the wearable device (101).

[0217] Referring to FIG. 11A, in example (1110), the wearable device (101) may display a visual object (1111) through at least one display (e.g., at least one display (250)). The visual object (1111) may indicate a state in which at least a portion (e.g., a pinch portion (1115)) of a hand object (710) is visible in the field of view of at least one camera (e.g., a camera (260)) of the wearable device (101). The visual object (1111) may indicate that the thumb and index finger of the hand form a pinch.

[0218] In one embodiment, as the hand object (710) of the user (701) moves, at least a portion (e.g., a pinch portion) of the hand object (710) may become invisible to the view of at least one camera (e.g., camera (260)) of the wearable device (101). The wearable device (101) may detect a change in the visibility state of at least a portion (e.g., a pinch portion) of the hand object (710). As the visibility state changes, the wearable device (101) may display a visual object different from the visual object (1111). In an example (1120), the wearable device (101) may display the visual object (1121) through at least one display (e.g., at least one display (250)). The visual object (1121) may indicate a state in which at least a portion (e.g., a pinch portion) of the hand object (710) is not visible from the view of at least one camera (e.g., camera (260)) of the wearable device (101) and a pinch input is continuously being applied. By displaying the visual object (1221), the user (701) of the wearable device (101) may confirm that the pinch input is being applied within the wearable device (101). As a non-limiting example, the wearable device (101) may further display the hand object (710) of the user (701) through pass-through. Referring to example (1130), the wearable device (101) may display the visual object (1121) and the hand object (710) together through at least one display (e.g., at least one display (250)). A visual object (1121) can be displayed overlappingly on a hand object (710).

[0219] Figures 11b, 11c, and 11d illustrate examples of visual objects for indicating expected durations.

[0220] Referring to FIG. 11B, in example (1150), the wearable device (101) may display a first visual object (1151) and a second visual object (1152). The first visual object (1151) may indicate a visibility state of a pinch portion of a hand object (710). The visibility state may indicate a state in which the pinch portion is not visible from the field of view of at least one camera (e.g., camera (260)) of the wearable device (101). The second visual object (1152) may indicate an expected duration for a state in which the pinch portion of the hand object (710) is not visible through the at least one camera and a pinch is input. According to one embodiment, the wearable device (101) may indicate that the currently inferred input is provided through a UI or indicate a delay time when making a pinch inference. For example, a visual object such as a graph, diagram, icon, shader, texture, progress bar, etc., may be used to indicate that the inferred pinch input is provided, and / or to indicate a delay time, such as some tendency such as speed, rotation, etc. for the pinch. The UI may be displayed on the at least one hand object (e.g., hand object (710)) itself, or may be displayed near an area where the at least one hand object (e.g., hand object (710)) is rendered. Alternatively, the UI may be displayed on a user-perceived area, such as a heads-up display (HUD) layer. For example, the second visual object (1152) may indicate an expected duration and a remaining time in the form of a status bar. The expected duration may represent the time from a point in time when the pinch is started to be maintained while the pinch portion is not visible through the at least one camera (hereinafter, referred to as a start time) to a point in time when the pinch is ended (hereinafter, referred to as a completion time). The remaining time may represent the time from a current point in time to the completion time.In the second visual object (1152), the size of the entire bar may represent the expected duration, the occupied portion of the bar may represent the time from the start time to the present time, and the empty portion of the bar may represent the remaining time from the present time to the completion time. The wearable device (101) may display the first visual object (1151) and the second visual object (1152) together with the hand object (710) of the user (701). According to one embodiment, the hand object (710) may be displayed through pass-through, which displays an image corresponding to an actual hand object captured by at least one camera of the wearable device (101) together with virtual visual objects (e.g., the first visual object (1151) and the second visual object (1152)). The wearable device (101) may display the first visual object (1151) overlapping the hand object (710). The wearable device (101) may display a second visual object (1152) in an area adjacent to the hand object (710). As a non-limiting example, the instructions and / or delay time may be provided to the user without any UI.

[0221] Referring to FIG. 11C, in example (1160), the wearable device (101) may display a first visual object (1161) and a second visual object (1162). The first visual object (1161) may indicate a visibility state of a pinch portion of a hand object (710). The visibility state may indicate a state in which the pinch portion is not visible from the view of at least one camera (e.g., camera (260)) of the wearable device (101). The second visual object (1162) may indicate an expected duration for a state in which the pinch portion of the hand object (710) is not visible through the at least one camera and a pinch is input. For example, the second visual object (1162) may indicate an expected duration and a remaining time in the form of a circle graph. The expected duration may indicate the time from the start time of holding the pinch to the completion time. The remaining time may indicate the time from the current time to the completion time. In the second visual object (1162), a circle may represent an expected duration, an occupied portion of the circle may represent the time from the start time to the present time, and an empty portion of the circle may represent the remaining time from the present time to the completion time. The wearable device (101) may display the first visual object (1161) and the second visual object (1162) together with the hand object (710) of the user (701). As an example, the hand object (710) may be displayed through pass-through. The wearable device (101) may display the first visual object (1161) overlapping the hand object (710). The wearable device (101) may display the second visual object (1162) in an area adjacent to the hand object (710).

[0222] Referring to FIG. 11D, in example (1170), the wearable device (101) can display a visual object (1171). The visual object (1171) can indicate the visibility state of the pinch portion of the hand object (710) as well as the expected duration. For example, the wearable device (101) can indicate the expected duration through blinking of the visual object (1171). For example, the visual object (1171) can repeat a first state (1171a), a second state (1171b), a third state (1171c), and / or a fourth state (1171d). The speed of the blinking (e.g., the speed of transition from the first state (1171a) to the second state (1171b) to the third state (1171c)) can indicate the expected duration. The shorter the remaining time, the faster the blinking can be.

[0223] In addition to the visual objects exemplified in FIGS. 11A, 11B, 11C, and 11D, various types of visual objects may be utilized. For example, trends in indicators such as speed or rotation may be provided in the form of graphs, charts, icons, shaders, and / or textures. As a non-limiting example, information regarding the visibility status or expected duration may be provided in text form without a separate user interface.

[0224] Figure 12 illustrates an example of a visual object for indicating user input. In addition to displaying a visibility status or expected duration in the system UI, the wearable device (101) may also display a status and expected duration indicating whether an input is being applied in a manner specific to the application in use.

[0225] Referring to FIG. 12, a wearable device (101) can display a virtual environment (1200). The wearable device (101) can execute an application for drawing a boundary area. While the application is executing, the wearable device (101) can receive a user input (e.g., pinch-drawing) using a pinch on the virtual environment (1200). Assume a situation where a user (e.g., user (701)) draws a line (e.g., a boundary line for a safety zone) from a starting point (1215). The wearable device (101) can identify the starting point (1215) based on a hand ray (1211) of a hand object (710). The wearable device (101) can receive a selection input in response to a pinch of the fingers of the hand object (710). The wearable device (101) can display a line (1230) according to the movement of the hand object (710) on at least one display (e.g., at least one display (250)) while the above selection input is continuously received.

[0226] When at least a portion of the hand object (710) (e.g., a pinch portion of the fingers) is obscured from the user's (701's) field of view (e.g., a field of view of at least one camera), the wearable device (101) may determine the user's (701's) intention. The user's (701's) intention may relate to whether to maintain the user input while holding the pinch and / or to the time for which the pinch is to be maintained. The wearable device (101) may determine an expected duration for which the pinch is to be maintained even if the pinch portion is not visible from the field of view. The wearable device (101) may maintain the user input assuming that the pinch is applied for the expected duration even if the pinch portion is not visible from the field of view. As an example, the wearable device (101) may continuously perform an action of drawing a boundary area.

[0227] The wearable device (101) may display a visual object (1220) to indicate that the pinch is being continuously applied. The visual object (1220) may move according to the movement of the hand object (710). The visual object (1220) may move along a line (1230). While moving along the line (1230), the state of the visual object (1220) may change from a first state (1221) to a second state (1222), and from the second state (1222) to a third state (1223). The visual object (1220) may indicate the state of the pinch portion (e.g., a state in which the pinch portion is visible and user input is provided, a state in which the pinch portion is invisible and user input according to the pinch is maintained) as well as the expected duration of the selection input. For example, if the pinched portion is not visible from the field of view and the pinch is continuously applied, the wearable device (101) may display a visual object (1220) having a shape including a circle around the point. The size of the circle may represent an expected duration. For example, when the remaining time of the expected duration is in a first range, the state of the visual object (1220) may change from a first state (1221) to a second state (1222). For example, when the remaining time of the expected duration enters a second range that is smaller than the first range, the state of the visual object (1220) may change from a second state (1222) to a second state (1223).

[0228] A wearable device (101) according to embodiments of the present disclosure can execute a function corresponding to a user input while maintaining a user input corresponding to the intention of a user (e.g., a user (701)) for a certain period of time. Even if at least a portion (e.g., a pinch portion) of an object (e.g., a hand object (710)) corresponding to a hand of the user (701) is obscured, the implementation of the present disclosure can be confirmed as the user input is continuously applied. In addition, the implementation of the present disclosure can be confirmed as the time for which the user input (e.g., a pinch) is maintained changes while changing the moving speed of the hand object (710), the posture of the hand object (710), or the degree of rotation of a joint of the hand object (710). In addition, even if at least a portion (e.g., a pinch portion) of the hand object (710) is temporarily out of view of the wearable device (101), the implementation of the disclosure can be confirmed as the user input using the hand object (710) is maintained.

[0229] In embodiments of the present disclosure, a wearable device (101) is provided. The wearable device (101) may include at least one display (250); at least one camera (260); at least one processor (120) including a processing circuit; and a memory (130) for storing instructions. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify the initiation of a user input according to a pinch of the user's fingers through the at least one camera (260), determine whether a pinch portion of the fingers for the pinch is visible in the field of view of the at least one camera (260) while the user input according to the movement of the user's hand object is provided, and, upon determining that the pinch portion of the fingers is not visible in the field of view of the at least one camera (260), apply the user input according to the movement of the user's hand object while the pinch of the fingers is applied for an expected duration, and, based on identifying that the expected duration has elapsed while the pinch portion of the fingers is not visible in the field of view of the at least one camera (260), execute a function corresponding to the release of the pinch.

[0230] For example, the instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to determine the expected duration based on at least one of information about the posture of the user's hand object, information about the mobility of the user's hand object, information about the rotation of a joint of the user's hand object, and information about the state of an application layer, upon determining that the pinch portion of the fingers is not visible in the field of view of the at least one camera (260).

[0231] For example, information about the mobility of the user's hand object, information about the rotation of the joints of the user's hand object, and information about the state of the application layer may be cached in a buffer of the wearable device (101) for a certain period of time.

[0232] For example, information about the mobility of the hand object may be used to determine a continuity weight for the pinched state. Information about the mobility of the hand object may include information about the velocity of the hand object. The expected duration may be determined according to a first value of the continuity weight when a deviation in the velocity of the hand object is greater than or equal to a threshold range. The expected duration may be determined according to a second value of the continuity weight that is lower than the first value when a deviation in the velocity of the hand object is less than the threshold range.

[0233] For example, the instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to display a visual object through the at least one display (250) in an area adjacent to the user's hand object to indicate that the pinch of the fingers has been applied for the expected duration. The visual object may be displayed to move in accordance with movement of the user's hand object.

[0234] For example, the instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to display a visual object representing the expected duration through the at least one display (250).

[0235] For example, the instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to apply the user input according to the pose of the pinch portion and the movement of the user's hand object when the pinch portion of the fingers changes from a state where it is not visible from the view of the at least one camera (260) to a state where it is visible from the view of the at least one camera (260).

[0236] In embodiments of the present disclosure, a method performed by a wearable device (101) is provided. The method may include: identifying an initiation of a user input according to a pinch of a user's fingers through at least one camera (260) of the wearable device (101); determining, while the user input according to movement of a hand object of the user is provided, whether a pinch portion of the user's fingers for the pinch is visible in the field of view of the at least one camera (260); applying the user input according to movement of the hand object of the user while the pinch portion of the fingers is applied for an expected duration of time based on a determination that the pinch portion of the fingers is not visible in the field of view of the at least one camera (260); and executing a function corresponding to the release of the pinch based on identifying that the expected duration of time has elapsed while the pinch portion of the fingers is not visible in the field of view of the at least one camera (260).

[0237] For example, the action of applying the user input may include an action of determining the expected duration based on at least one of information about the posture of the user's hand object, information about the mobility of the user's hand object, information about the rotation of a joint of the user's hand object, and information about the state of an application layer, based on a determination that the pinch portion of the fingers is not visible in the field of view of the at least one camera (260).

[0238] For example, information about the mobility of the user's hand object, information about the rotation of the joints of the user's hand object, and information about the state of the application layer may be cached in a buffer of the wearable device (101) for a certain period of time.

[0239] For example, information about the mobility of the hand object may be used to determine a continuity weight for the pinched state. Information about the mobility of the hand object may include information about the velocity of the hand object. The expected duration may be determined according to a first value of the continuity weight when a deviation in the velocity of the hand object is greater than or equal to a threshold range. The expected duration may be determined according to a second value of the continuity weight that is lower than the first value when a deviation in the velocity of the hand object is less than the threshold range.

[0240] For example, the method may include displaying a visual object via at least one display (250) of the wearable device (101) in an area adjacent to the user's hand object to indicate that the pinch of the fingers is applied for the expected duration. The visual object may be displayed to move according to the movement of the user's hand object.

[0241] For example, the method may include displaying a visual object representing the expected duration via at least one display (250).

[0242] For example, the method may include an operation of applying the user input according to the posture of the pinch portion and the movement of the user's hand object when the pinch portion of the fingers changes from a state where it is not visible in the view of the at least one camera (260) to a state where it is visible in the view of the at least one camera (260).

[0243] In embodiments of the present disclosure, a wearable device (101) is provided. The wearable device (101) may include at least one display (250); at least one camera (260); at least one processor (120) including a processing circuit; and a memory (130) for storing instructions. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify the initiation of a gesture input using a hand object of a user, determine whether at least a portion of the hand object of the user is visible in the field of view of the at least one camera (260) while the gesture input is being performed, determine an expected duration for the duration of the gesture input based on at least one of a posture of the hand object of the user, information about the mobility of the hand object of the user, information about the rotation of a joint of the hand object of the user, and information about the state of an application layer, display a visual object to indicate the expected duration while at least a portion of the hand object of the user is not visible, and execute a function corresponding to the completion of the gesture input based on identifying that the expected duration has elapsed while at least a portion of the hand object of the user is not visible.

[0244] For example, information about the mobility of the user's hand object, information about the rotation of the joints of the user's hand object, and information about the state of the application layer may be cached in a buffer of the wearable device (101) for a certain period of time.

[0245] For example, information about the mobility of the hand object may be used to determine a continuity weight for a state in which the gesture input is maintained. Information about the mobility of the hand object may include information about the velocity of the hand object. The expected duration may be determined according to a first value of the continuity weight when a deviation in the velocity of the hand object is greater than or equal to a threshold range. The expected duration may be determined according to a second value of the continuity weight that is lower than the first value when a deviation in the velocity of the hand object is less than the threshold range.

[0246] For example, the instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to display a guide visual object through the at least one display (250) of the wearable device (101) in an area adjacent to the user's hand object to indicate that the gesture input is maintained for the expected duration. The guide visual object may be displayed to move according to movement of the user's hand object.

[0247] For example, the instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to continuously apply the gesture input according to the posture of the user's hand object and the movement of the user's hand object when at least a portion of the user's hand object changes from being invisible to the view of the at least one camera (260) to being visible to the view of the at least one camera (260).

[0248] For example, the gesture input may include a pinch-draw or pinch-drag input.

[0249] In embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store instructions. The instructions, when executed by at least one processor, may cause an electronic device to perform operations including identifying the initiation of a gesture input using a hand object of a user, determining whether at least a portion of the hand object of the user is visible in the field of view of the at least one camera while the gesture input is performed, determining an expected duration for the duration of the gesture input based on at least one of a posture of the hand object of the user, information about the mobility of the hand object of the user, information about the rotation of a joint of the hand object of the user, and information about the state of an application layer, displaying a visual object to indicate the expected duration while the at least a portion of the hand object of the user is not visible, and executing a function corresponding to the completion of the gesture input based on identifying that the expected duration has elapsed while the at least a portion of the hand object of the user is not visible.

[0250] For one or more embodiments, at least one of the components described in one or more of the preceding drawings may be configured to perform one or more operations, techniques, processes, and / or methods as described herein. For example, a processor (e.g., a baseband processor) described herein with respect to one or more of the preceding drawings may be configured to operate according to one or more examples described herein. For another example, circuitry associated with a user equipment (UE), a base station, a network element, and the like, as described above with respect to one or more of the preceding drawings, may be configured to operate according to one or more examples described herein.

[0251] Any of the embodiments described above may be combined with any other embodiment (or combination of embodiments) unless explicitly stated otherwise. The foregoing description of one or more implementations provides examples and descriptions, but is not intended to be exhaustive or limit the scope of the embodiments to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be learned from practicing various embodiments.

[0252] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0253] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0254] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0255] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0256] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0257] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

In wearable devices, At least one display; At least one camera; At least one processor comprising a processing circuit; and A wearable device comprising a memory storing instructions, wherein the instructions, when individually or collectively executed by the at least one processor: Identifying the initiation of user input by pinching the user's fingers through at least one camera, While the user input is provided according to the movement of the user's hand object, determining whether the pinch portion of the fingers for the pinch is visible in the field of view of at least one camera; Applying the user input according to the movement of the user's hand object while the pinch of the fingers remains applied for an expected duration, based on a determination that the pinched portion of the fingers is not visible in the view of at least one camera, Causing a function corresponding to the release of the pinch to be executed based on identifying that the expected duration has elapsed while the pinched portion of the fingers is not visible from the view of the at least one camera. Wearable devices. In claim 1, When the above instructions are individually or collectively executed by the at least one processor, the wearable device: Causing the expected duration to be determined based on at least one of information about the posture of the user's hand object, information about the mobility of the user's hand object, information about the rotation of the joints of the user's hand object, and information about the state of the application layer, based on a determination that the pinch portion of the fingers is not visible in the view of the at least one camera. Wearable devices. In claim 2, Information about the mobility of the user's hand object, information about the rotation of the joints of the user's hand object, and information about the state of the application layer are cached in a buffer of the wearable device for a certain period of time. Wearable devices. In claim 2, Information about the mobility of the hand object is used to determine continuity weights for the pinched state, Information about the mobility of the hand object includes information about the velocity of the hand object, The above expected duration is determined according to the first value of the continuity weight when the deviation of the velocity of the hand object is greater than a critical range, The above expected duration is determined according to a second value lower than the first value of the continuity weight when the deviation of the velocity of the hand object is less than the threshold range. Wearable devices. In claim 1, When the above instructions are individually or collectively executed by the at least one processor, the wearable device: Causing a visual object to be displayed through at least one display in an area adjacent to the user's hand object to indicate that the pinch of the fingers has been applied for the expected duration; The above visual object is displayed to move according to the movement of the user's hand object. Wearable devices. In claim 5, When the above instructions are individually or collectively executed by the at least one processor, the wearable device: Causing a visual object representing the expected duration to be displayed through at least one display; Wearable devices. In claim 1, When the above instructions are individually or collectively executed by the at least one processor, the wearable device: When the pinch portion of the fingers changes from being invisible to being visible in the view of at least one camera, causing the user input to be applied according to the posture of the pinch portion and the movement of the user's hand object. Wearable devices. In a method performed by a wearable device, An action of identifying the initiation of user input based on a pinch of the user's fingers through at least one camera of the wearable device; An operation for determining whether a pinch portion of the user's fingers for the pinch is visible in the field of view of at least one camera while the user input is provided according to the movement of the user's hand object; An action of applying the user input according to the movement of the user's hand object while the pinch of the fingers is applied for an expected duration, based on a determination that the pinched portion of the fingers is not visible in the view of at least one camera; An action comprising: executing a function corresponding to the release of the pinch, based on identifying that the expected duration has elapsed while the pinched portion of the fingers is not visible from the view of the at least one camera; method. In claim 8, the operation of applying the user input comprises: Including an operation of determining the expected duration based on at least one of information about the posture of the user's hand object, information about the mobility of the user's hand object, information about the rotation of the joint of the user's hand object, and information about the state of the application layer, based on a determination that the pinched portion of the fingers is not visible in the view of the at least one camera. method. In claim 9, Information about the mobility of the user's hand object, information about the rotation of the joints of the user's hand object, and information about the state of the application layer are cached in a buffer of the wearable device for a certain period of time. method. In claim 9, Information about the mobility of the hand object is used to determine continuity weights for the pinched state, Information about the mobility of the hand object includes information about the velocity of the hand object, The above expected duration is determined according to the first value of the continuity weight when the deviation of the velocity of the hand object is greater than a critical range, The above expected duration is determined according to a second value lower than the first value of the continuity weight when the deviation of the velocity of the hand object is less than the threshold range. method. In claim 8, Further comprising an action of displaying a visual object through at least one display of the wearable device in an area adjacent to the user's hand object to indicate that the pinch of the fingers is applied for the expected duration; The above visual object is displayed to move according to the movement of the user's hand object. method. In claim 12, Further comprising an action of displaying a visual object representing the expected duration through the at least one display, method. In claim 8, Further comprising an action of applying the user input according to the posture of the pinch portion and the movement of the user's hand object when the pinch portion of the fingers changes from being invisible to being visible in the view of the at least one camera. method. In a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores instructions, The above instructions, when executed by at least one processor, cause the electronic device to: Identify the initiation of a gesture input using the user's hand object, While the gesture input is being performed, determining whether at least a portion of the user's hand object is visible in the field of view of the at least one camera; Upon determining that at least a portion of the user's hand object is not visible, an expected duration for the continuation of the gesture input is determined based on at least one of information about the posture of the user's hand object, information about the mobility of the user's hand object, information about the rotation of a joint of the user's hand object, and information about the state of the application layer. Displaying a visual object to indicate the expected duration while at least part of the user's hand object is not visible; Causing the user to perform actions including executing a function corresponding to completion of the gesture input based on identifying that the expected duration has elapsed while at least a portion of the user's hand object is not visible. Non-transitory computer-readable storage medium.

Images