Wearable device, method, and non-transitory computer-readable storage medium for displaying object in virtual environment

Abstract

A wearable device is provided. The wearable device may comprise at least one processor, a display, and at least one processor. The at least one processor is configured to: identify, using at least one camera, an object included in a virtual screen displayed through the display; determine whether the identified object corresponds to a type for limiting a representation of a hand object of a user; display, through the display, a virtual screen including the hand object on the basis of hand tracking according to a determination that the identified object does not correspond to the type; and display, through the display, a virtual screen that does not display a movement of the hand object according to the hand tracking, on the basis of identifying that the hand object is located within a threshold distance from the object according to a determination that the identified object corresponds to the type.

Description

Wearable device, method, and non-transient computer-readable storage medium for displaying objects in a virtual environment

[0001] The present disclosure relates to a wearable device, a method, and a non-transient computer-readable storage medium for displaying an object in a virtual environment.

[0002] Wearable devices that provide extended reality can perform hand tracking and object tracking.

[0003] The information described above is merely background information to aid in understanding the present disclosure. No assertion or determination is made as to whether any of the foregoing can be applied as prior art in relation to the present disclosure.

[0004] Aspects of the disclosure address the problems and / or disadvantages described above and provide at least the advantages described below. Accordingly, aspects of the disclosure provide a wearable device, a method, and a non-transient computer-readable storage medium for displaying an object in a virtual environment.

[0005] Additional aspects are some described in the following description, some become apparent from the description, or may be known by practicing the presented embodiments.

[0006] According to an aspect of the present disclosure, a wearable device is provided. The wearable device may include at least one camera. The wearable device may include a display. The wearable device may include a memory that stores instructions and includes one or more storage media. The wearable device may include at least one processor that includes a processing circuit and is communicatively connected to the at least one camera, the display, and the memory. When the above instructions are executed individually or collectively by the at least one processor, the wearable device may be caused to identify an object included in a virtual screen displayed through the display using the at least one camera, determine whether the identified object corresponds to a type for restricting the representation of a user's hand object, and, based on the determination that the identified object does not correspond to the type, display a virtual screen containing the hand object through the display based on hand tracking, and based on the determination that the identified object corresponds to the type, display a virtual screen through the display that does not display the movement of the hand object according to the hand tracking based on identifying that the hand object is located within a threshold distance from the object.

[0007] According to another aspect of the present disclosure, a method is provided by a wearable device comprising a display and at least one camera. The method may include: identifying an object included in a virtual screen displayed through the display using the at least one camera; determining whether the identified object corresponds to a type for restricting the representation of a user's hand object; displaying a virtual screen containing the hand object through the display based on hand tracking, based on the determination that the identified object does not correspond to the type; and displaying a virtual screen through the display that does not display the movement of the hand object according to the hand tracking, based on identifying that the hand object is located within a threshold distance from the object, based on the determination that the identified object corresponds to the type.

[0008] According to another aspect of the present disclosure, one or more non-transient computer-readable storage media are provided. The one or more non-transient computer-readable storage media may store one or more programs. The one or more programs may include instructions that, when executed individually or collectively by at least one processor of a wearable device comprising a display and at least one camera, use the at least one camera to identify an object included in a virtual screen displayed through the display, determine whether the identified object corresponds to a type for restricting the representation of a user's hand object, and, based on the determination that the identified object does not correspond to the type, display a virtual screen containing the hand object through the display based on hand tracking, and based on the determination that the identified object corresponds to the type, cause the wearable device to display a virtual screen that does not display the movement of the hand object according to the hand tracking through the display based on identifying that the hand object is located within a threshold distance from the object.

[0009] According to another aspect of the present disclosure, a wearable device is provided. The wearable device may include at least one display. The wearable device may include at least one sensor. The wearable device may include a memory that stores instructions and includes one or more storage media. The wearable device may include at least one processor that includes a processing circuit and is communically connected to the at least one display, the at least one sensor, and the memory. When the instructions are executed individually or collectively by the at least one processor, the wearable device may cause the wearable device to acquire object information data for an object identified by the at least one sensor, acquire tracking data by tracking a part of the body identified by the at least one sensor, identify whether a specified condition is satisfied, and, based on the identification, determine the target or scope of the tracking, or determine the target or scope of the tracking data processed by the at least one processor.

[0010] According to another aspect of the present disclosure, a method performed by a wearable device is provided. The method may include the operation of acquiring object information data for an object identified by the at least one sensor, the operation of acquiring tracking data by tracking a part of a body identified by the at least one sensor, the operation of identifying whether a specified condition is satisfied, and the operation of determining a target or range of tracking based on the identification, or determining a target or range of the tracking data processed by the at least one processor.

[0011] According to another aspect of the present disclosure, one or more non-transient computer-readable storage media are provided. The one or more non-transient computer-readable storage media may store one or more programs. The one or more programs may include instructions that, when executed individually or collectively by at least one processor of a wearable device, cause the wearable device to acquire object information data for an object identified by the at least one sensor, acquire tracking data by tracking a part of the body identified by the at least one sensor, identify whether a specified condition is satisfied, and, based on the identification, determine the target or range of the tracking, or determine the target or range of the tracking data processed by the at least one processor.

[0012] Other aspects, advantages, and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description disclosing various embodiments of the present disclosure together with the accompanying drawings.

[0013] The above-described and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description together with the accompanying drawings.

[0014] FIG. 1 is a block diagram of an electronic device in a network environment according to an embodiment of the present disclosure.

[0015] FIG. 2a shows a perspective view of a wearable device according to an embodiment of the present disclosure.

[0016] FIG. 2b illustrates one or more hardware components disposed within a wearable device according to an embodiment of the present disclosure.

[0017] FIGS. 3a and 3b illustrate the appearance of a wearable device according to various embodiments of the present disclosure.

[0018] FIG. 4 illustrates a block diagram of a wearable device according to an embodiment of the present disclosure.

[0019] FIG. 5 shows a block diagram of a wearable device for displaying an image in a virtual space according to an embodiment of the present disclosure.

[0020] FIG. 6 is a block diagram illustrating the operation of a wearable device that performs object tracking and hand tracking according to an embodiment of the present disclosure.

[0021] FIG. 7 is a flowchart illustrating the operation of a wearable device for restricting the display of a user's hand object within a virtual screen according to an embodiment of the present disclosure.

[0022] FIGS. 8a, 8b, and 8c illustrate the operation of a wearable device for limiting the display of a user's hand object within a virtual screen according to various embodiments of the present disclosure.

[0023] FIG. 9 is a flowchart illustrating the operation of a wearable device for restricting the interaction of a user's hand object within a virtual screen according to an embodiment of the present disclosure.

[0024] FIG. 10 illustrates the limitation of interaction by a user's hand object within a virtual screen according to an embodiment of the present disclosure.

[0025] FIG. 11 is a flowchart illustrating the operation of a wearable device for processing a hand object within a virtual screen according to the type of object within the virtual screen according to an embodiment of the present disclosure.

[0026] FIG. 12 is a flowchart illustrating the operation of a wearable device for processing a hand object within a virtual screen according to the type of object within the virtual screen according to an embodiment of the present disclosure.

[0027] Throughout the drawing, the same reference number may be used to describe identical or similar elements, features, and structures.

[0028] With reference to the accompanying drawings, the following description is provided to facilitate a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and equivalents. This description includes various specific details to aid such understanding, but these are merely illustrative. Accordingly, those skilled in the art will be able to make various changes and modifications to the various embodiments described in the present disclosure without departing from the scope of the present disclosure. Additionally, for clarity and conciseness, descriptions of well-known functions and constructions may be omitted.

[0029] The terms and words used in the following description and claims are not limited to their bibliographical meaning but are used by the inventor to enable a clear and consistent understanding of the present disclosure. Accordingly, it will be apparent to those skilled in the art that the following description of the various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure as defined by the appended claims and their equivalents.

[0030] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of various embodiments. A singular expression may include a plural expression unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art described in this disclosure. Terms used in this disclosure that are defined in a general dictionary may be interpreted as having the same or similar meaning as they have in the context of the relevant technology, and are not to be interpreted in an ideal or overly formal sense unless explicitly defined in this disclosure. In some cases, even terms defined in this disclosure are not to be interpreted to exclude the embodiments of this disclosure.

[0031] In the various embodiments of the present disclosure described below, a hardware-based approach is described as an example. However, since the various embodiments of the present disclosure include techniques using both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0032] Terms used in the following description to refer to images (e.g., image, frame, camera frame, captured image, camera image), terms referring to a user's hand (e.g., hand object, candidate object, hand candidate object, bounding box, candidate hand object), terms referring to signals (e.g., signaling, control signal, data, control data, request signal, information), terms referring to locations (e.g., location information, area information, object information, object location, object coordinates, reference object, coordinate information, location, coordinate, relative coordinate, absolute coordinate, coordinate system), terms referring to values ​​(e.g., threshold value, reference value, reference area, reference range, level, threshold level, threshold, range, value, area), terms for operation states (e.g., step, operation, procedure), or terms referring to components of a device are examples provided for the convenience of explanation. Accordingly, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings may be used. Additionally, terms such as '...part', '...device', '...piece', '...body' used below may refer to at least one shape structure or a unit that processes a function.

[0033] Additionally, in this disclosure, expressions of "greater than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled; however, this is merely for the purpose of expressing an example and does not exclude descriptions of "greater than" or "less than." Conditions described as "greater than" may be replaced with "greater than," conditions described as "less than" may be replaced with "less than," and conditions described as "greater than and less than" may be replaced with "greater than and less than." Furthermore, "A" to "B" below refer to at least one of elements from A (including A) to B (including B). Below, "C" and / or "D" refers to including at least one of "C" or "D," i.e., {"C", "D", "C" and "D"}.

[0034] It should be understood that blocks in each flowchart and combinations of flowcharts can be executed by one or more programs containing computer-executable instructions. One or more programs as a whole may be stored in a single memory device, or one or more computer programs may be divided into multiple parts and stored in multiple memory devices.

[0035] The functions or operations described in this disclosure may be processed by a single processor or a combination of processors. The single processor or combination of processors performs processing and may include circuits such as an application processor (e.g., CPU), a communication processor (e.g., modem), a graphical processing unit (GPU), a neutral processing unit (NPU) (e.g., an artificial intelligence chip), a wireless-fidelity chip, a Bluetooth chip, a global positioning chip, a near-field communication (NFC) chip, a connectivity chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver IC (integrated circuit), an audio codec chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on chip (SoC), and an IC.

[0036] FIG. 1 is a block diagram of an electronic device in a network environment according to an embodiment of the present disclosure.

[0037] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an external electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an external electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to an embodiment of the present disclosure, the electronic device (101) may communicate with an external electronic device (104) through a server (108). According to an embodiment of the present disclosure, an electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments of the present disclosure, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments of the present disclosure, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0038] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to an embodiment of the present disclosure, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to an embodiment of the present disclosure, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0039] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to embodiments of the present disclosure, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to embodiments of the present disclosure, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0040] Memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. Memory (130) may include volatile memory (132) or non-volatile memory (134). Non-volatile memory (134) may include internal memory (136) and / or external memory (138).

[0041] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0042] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0043] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to an embodiment of the present disclosure, the receiver may be implemented separately from the speaker or as part thereof.

[0044] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to an embodiment of the present disclosure, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0045] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to an embodiment of the present disclosure, the audio module (170) can acquire sound through an input module (150) or output sound through an audio output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0046] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to an embodiment of the present disclosure, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0047] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., external electronic device (102)). According to an embodiment of the present disclosure, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0048] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., external electronic device (102)). According to an embodiment of the present disclosure, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0049] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to an embodiment of the present disclosure, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0050] The camera module (180) can capture still images and video. According to an embodiment of the present disclosure, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0051] The power management module (188) can manage power supplied to the electronic device (101). According to an embodiment of the present disclosure, the power management module (188) may be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0052] The battery (189) can supply power to at least one component of the electronic device (101). According to an embodiment of the present disclosure, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0053] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., external electronic device (102), external electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to an embodiment of the present disclosure, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G (5th generation) network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0054] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G (4th generation) networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., external electronic device (104)), or network system (e.g., second network (199)). According to an embodiment of the present disclosure, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0055] An antenna module (197) can transmit a signal or power to an external source (e.g., an external electronic device) or receive it from an external source. According to an embodiment of the present disclosure, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to an embodiment of the present disclosure, the antenna module (197) may include a plurality of antennas (e.g., array antennas). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiment of the present disclosure, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0056] According to various embodiments of the present disclosure, the antenna module (197) may form a mmWave antenna module. According to embodiments of the present disclosure, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0057] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0058] According to an embodiment of the present disclosure, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be a device of the same or different type as the electronic device (101). According to an embodiment of the present disclosure, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In an embodiment of the present disclosure, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks.According to an embodiment of the present disclosure, an external electronic device (104) or server (108) may be included within the second network (199). The electronic device (101) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0059]

[0060] In embodiments of the present disclosure, an electronic device for displaying an image in a virtual space (e.g., the electronic device (101) of FIG. 1) may be a wearable device. The wearable device (101) may include a head-mounted display (HMD) that is wearable on a user's head. The wearable device (101) may be referred to as a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of the wearable device (101) having the form of glasses is illustrated, embodiments of the present disclosure are not limited thereto. An example of a hardware configuration included within the wearable device (101) is described exemplarily with reference to FIG. 4. An example of the structure of a wearable device (101) that can be worn on the head of a user (110) is described with reference to FIG. 2a, FIG. 2b, FIG. 3a, and / or FIG. 3b. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) for attaching to the user's head to form an HMD.

[0061] In an embodiment of the present disclosure, the wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, while a user (e.g., user (110) of FIG. 8a) is wearing the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eye. The wearable device (101) may combine light emitted from the display of the wearable device (101) with ambient light passing through the lens. The display area of ​​the display may be formed within the lens through which the ambient light passes. Since the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) can see an image in which a real object (or physical object) recognized by the ambient light and a virtual object formed by the light emitted from the display are mixed. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0062] In an embodiment of the present disclosure, the wearable device (101) may perform functions related to VST (video see-through or visible see-through) and / or virtual reality (VR). For example, while a user (110) is wearing the wearable device (101), the wearable device (101) may include a housing that covers the user's (110) eyes. The wearable device (101) may include a display disposed on a first surface of the housing facing the eyes while in the state. The wearable device (101) may include a camera disposed on a second surface opposite to the first surface. Using the camera, the wearable device (101) may acquire an image and / or video representing ambient light. A wearable device (101) may output the image and / or video within a display placed on the first surface, thereby allowing a user (110) to perceive the ambient light through the display. A displaying area (or displaying region) (or active area or active region) of the display placed on the first surface may be formed by one or more pixels included in the display. The wearable device (101) may composite a virtual object with the image and / or video output through the display, thereby allowing the user (110) to perceive the virtual object together with a real object perceived by the ambient light.

[0063] In an embodiment of the present disclosure, the wearable device (101) can identify or recognize the position or location and / or direction or orientation of the wearable device (101) based on images and / or videos obtained or acquired using a camera. The wearable device (101) can obtain information about the external space using one or more cameras and / or one or more sensors. The information may include the geographic location of the external space (e.g., Global Positioning System (GPS) coordinates) identified by one or more sensors. The information may include images and / or videos of the external space identified by one or more cameras. The wearable device (101) can identify external objects contained in the external space from the images and / or videos by performing object recognition on the images and / or videos.

[0064] Hereinafter, with reference to FIGS. 2a, FIGS. 2b, FIGS. 3a, FIGS. 3b, and FIGS. 4, an example of a hardware configuration of a wearable device (101) is described.

[0065] FIG. 2a shows a perspective view of a wearable device according to an embodiment of the present disclosure.

[0066] FIG. 2b illustrates one or more hardware components disposed within a wearable device according to an embodiment of the present disclosure.

[0067] According to one embodiment, the wearable device (101) may have the form of glasses that are wearable on a body part (e.g., head) of a user (110). The wearable device (101) of FIGS. 2a and 2b may be an example of the electronic device (101) of FIG. 1. The wearable device (101) may include a head-mounted display (HMD). For example, the housing of the wearable device (101) may include a flexible material such as rubber and / or silicone that has a shape that adheres to a part of the user's (110) head (e.g., a part of the face covering both eyes). For example, the housing of the wearable device (101) may include one or more straps that can be twined around the user's (110) head and / or one or more temples that are attachable to the ears of the head.

[0068] Referring to FIG. 2a, a wearable device (101) according to an embodiment of the present disclosure may include at least one display (250) and a frame (200) supporting at least one display (250).

[0069] According to an embodiment of the present disclosure, a wearable device (101) may be worn on a part of the body of a user (110). The wearable device (101) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user (110) wearing the wearable device (101). For example, the wearable device (101) may display a virtual reality image provided by at least one optical device (282, 284) of FIG. 2b on at least one display (250) in response to a designated gesture of the user (110) obtained through the motion recognition camera (260-2, 260-3) of FIG. 2b.

[0070] According to an embodiment of the present disclosure, at least one display (250) can provide visual information to a user (110). For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at locations corresponding to the left and right eyes of the user (110), respectively.

[0071] Referring to FIG. 2b, at least one display (250) may provide visual information transmitted from external light to the user (110) through a lens included in at least one display (250), and other visual information distinct from said visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) may include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area may be formed on the second surface (232) of at least one display (250). When the user (110) wears the wearable device (101), external light may be transmitted to the user (110) by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided by at least one optical device (282, 284) on a real image transmitted through external light in a display area formed on the second surface (232).

[0072] In an embodiment of the present disclosure, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits it to a user (110). At least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed on the exterior or at least a portion of the interior of at least one waveguide (233, 234). The nano pattern may be formed based on a polygonal and / or curved grating structure. Light incident on one end of at least one waveguide (233, 234) may be propagated to the other end of at least one waveguide (233, 234) by the nano pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be placed within a wearable device (101) to guide a screen displayed by at least one display (250) to the eyes of a user (110). For example, the screen may be transmitted to the eyes of the user (110) based on total internal reflection (TIR) ​​occurring within at least one waveguide (233, 234).

[0073] A wearable device (101) can analyze an object included in a real-world image collected through a camera (260-4), combine a virtual object corresponding to an object among the analyzed objects that is the target of augmented reality provision, and display it on at least one display (250). The virtual object may include at least one of text and an image regarding various information related to the object included in the real-world image. The wearable device (101) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (101) can perform spatial recognition (e.g., SLAM (simultaneous localization and mapping)) using a multi-camera and / or time-of-flight (ToF). A user (110) wearing the wearable device (101) can view the image displayed on at least one display (250).

[0074] According to an embodiment of the present disclosure, the frame (200) may be formed as a physical structure that allows the wearable device (101) to be worn on the body of a user (110). According to an embodiment of the present disclosure, the frame (200) may be configured such that when the user (110) wears the wearable device (101), the first display (250-1) and the second display (250-2) can be positioned corresponding to the left and right eyes of the user (110). The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) so that they are positioned corresponding to the left and right eyes of the user (110).

[0075] Referring to FIG. 2a, the frame (200) may include a region (220) in which at least a portion of the frame contacts a part of the user's (110) body when the user (110) wears the wearable device (101). For example, the region (220) of the frame (200) in contact with a part of the user's (110) body may include a region in contact with a part of the user's (110) nose, a part of the user's (110) ear, and a part of the side of the user's (110) face that the wearable device (101) contacts. According to an embodiment of the present disclosure, the frame (200) may include a nose pad (210) that contacts a part of the user's (110) body. When the wearable device (101) is worn by the user (110), the nose pad (210) may contact a part of the user's (110) nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another part of the user's (110) body that is distinct from the part of the user's (110) body.

[0076] For example, the frame (200) may include a first rim (201) covering at least a portion of a first display (250-1), a second rim (202) covering at least a portion of a second display (250-2), a bridge (203) positioned between the first rim (201) and the second rim (202), a first pad (211) positioned along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) positioned along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) extending from the first rim (201) and fixed to a portion of the wearer's ear, and a second temple (205) extending from the second rim (202) and fixed to a portion of the ear opposite to the first. The first pad (211) and the second pad (212) may come into contact with a portion of the user's (110) nose, and the first temple (204) and the second temple (205) may come into contact with a portion of the user's (110) face and a portion of the user's (110) ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through a first hinge unit (206) positioned between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through a second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to an embodiment of the present disclosure, the wearable device (101) may identify an external object touching the frame (200) (e.g., the fingertip of the user (110)) and / or a gesture performed by said external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame (200).

[0077] According to an embodiment of the present disclosure, the wearable device (101) may include hardware that performs various functions (e.g., hardware to be described later based on the block diagram of FIG. 4). For example, the hardware may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers (255-1, 255-2)), a microphone (e.g., microphones (265-1, 265-2, 265-3)), a light-emitting module (not shown), and / or a PCB (printed circuit board) (290) (e.g., a printed circuit board). The various hardware may be placed within a frame (200).

[0078] According to an embodiment of the present disclosure, a microphone (e.g., microphones (265-1, 265-2, 265-3)) of a wearable device (101) may be placed on at least a portion of a frame (200) to acquire a sound signal. A first microphone (265-1) placed on a bridge (203), a second microphone (265-2) placed on a second rim (202), and a third microphone (265-3) placed on a first rim (201) are shown in FIG. 2b, but the number and placement of the microphones (265) are not limited to the embodiment of FIG. 2b. If there are two or more microphones (265) included in the wearable device (101), the wearable device (101) may identify the direction of the sound signal using a plurality of microphones placed on different portions of the frame (200).

[0079] According to an embodiment of the present disclosure, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user (110). For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or included within at least one display (250) as part of at least one display (250). According to an embodiment of the present disclosure, a wearable device (101) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) positioned at the edge of a first display (250-1) and a second optical device (284) positioned at the edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) positioned on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) positioned on the second display (250-2).

[0080] In an embodiment of the present disclosure, the camera (260) may include a shooting camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 206-3). The shooting camera (260-4), the eye tracking camera (260-1), and the motion recognition camera (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position of the eyes or the gaze of a user (110) wearing the wearable device (101). For example, the wearable device (101) may detect the gaze from an image containing the pupils of the user (110) obtained through the eye tracking camera (260-1). A wearable device (101) can identify an object (e.g., a real object, and / or a virtual object) focused by the user (110) by using the user's (110) gaze obtained through an eye-tracking camera (260-1). The wearable device (101), having identified the focused object, can perform a function (e.g., gaze interaction) for interaction between the user (110) and the focused object. The wearable device (101) can represent a part corresponding to the eyes of an avatar representing the user (110) in a virtual space by using the user's (110) gaze obtained through an eye-tracking camera (260-1). The wearable device (101) can render an image (or screen) displayed on at least one display (250) based on the position of the user's (110) eyes. For example, the visual quality of a first region related to the gaze within an image and the visual quality of a second region distinct from the first region (e.g., resolution, brightness, saturation, grayscale, PPI (pixels per inch)) may differ from each other.In the present disclosure, the term “resolution” is used to refer to the density of pixels of an image and / or display. The density of pixels and / or resolution may be measured based on units of PPI and / or dpi (dots per inch) or may be parameterized. The wearable device (101) may acquire an image having a visual quality of a first region and a visual quality of a second region that match the gaze of the user (110) using foveated rendering. For example, if the wearable device (101) supports an iris recognition function, user authentication may be performed based on iris information acquired using an eye-tracking camera (260-1). An example in FIG. 2b in which the eye tracking camera (260-1) is positioned toward the right eye of the user (110) is shown, but the embodiment is not limited thereto, and the eye tracking camera (260-1) may be positioned solely toward the left eye of the user (110) or toward both eyes.

[0081] In an embodiment of the present disclosure, a camera (260-4) can capture a real image or background to be matched with a virtual image in order to implement augmented reality or mixed reality content. The camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The camera (260-4) can capture an image of a specific object located at a position viewed by the user (110) and provide the image to at least one display (250). The at least one display (250) can display a single image in which information regarding a real image or background including the image of the specific object acquired using the camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (101) can compensate for depth information (e.g., distance between the wearable device (101) and an external object obtained through a depth sensor) using an image obtained through a shooting camera (260-4). The wearable device (101) can perform object recognition using an image obtained through a shooting camera (260-4). The wearable device (101) can perform a function of focusing on an object (or subject) within an image (e.g., auto focus (AF)) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using a shooting camera (260-4). The wearable device (101) can perform a pass-through function to display an image obtained through a shooting camera (260-4) superimposed on at least a portion of the screen while displaying a screen representing a virtual space on at least one display (250). In an embodiment of the present disclosure, the camera (260-4) may be placed on a bridge (203) positioned between the first rim (201) and the second rim (202).

[0082] The eye tracking camera (260-1) can achieve more realistic augmented reality by matching the user's (110) gaze with visual information provided to at least one display (250) by tracking the gaze of a user (110) wearing a wearable device (101). For example, when the user (110) looks straight ahead, the wearable device (101) can naturally display environmental information related to the user's (110) front on at least one display (250) at the location where the user (110) is situated. The eye tracking camera (260-1) can be configured to capture an image of the user's (110) pupil to determine the user's (110) gaze. For example, the eye tracking camera (260-1) can receive a gaze detection light reflected from the user's (110) pupil and track the user's (110) gaze based on the position and movement of the received gaze detection light. In an embodiment of the present disclosure, the eye-tracking camera (260-1) may be positioned at a location corresponding to the left and right eyes of the user (110). For example, the eye-tracking camera (260-1) may be positioned to face the direction in which the user (110) wearing the wearable device (101) is located within the first rim (201) and / or the second rim (202).

[0083] A motion recognition camera (260-2, 260-3) can provide a specific event to a screen provided on at least one display (250) by recognizing the movement of the user (110)'s entire body or part thereof, such as the user's torso, hands, or face. The motion recognition camera (260-2, 260-3) can recognize the user's (110) gesture, acquire a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display (250). A processor can identify the signal corresponding to the gesture and, based on the identification, perform a designated function. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for a 6-degrees-of-freedom pose (6 dof pose). The processor can perform gesture recognition and / or object tracking functions using motion recognition cameras (260-2, 260-3). In an embodiment of the present disclosure, the motion recognition cameras (260-2, 260-3) may be placed on the first rim (201) and / or the second rim (202).

[0084] The camera (260) included in the wearable device (101) is not limited to the eye-tracking camera (260-1) and motion recognition camera (260-2, 260-3) described above. For example, the wearable device (101) can identify external objects included within the field of view (FoV) by using a camera positioned toward the field of view (FoV) of the user (110). The identification of external objects by the wearable device (101) can be performed based on a sensor for identifying the distance between the wearable device (101) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus (AF) function and / or an optical image stabilization (OIS) function. For example, the wearable device (101) may include a camera (260) (e.g., a face tracking camera) positioned toward the face to acquire an image including the face of a user (110) wearing the wearable device (101).

[0085] Although not illustrated, according to one embodiment, the wearable device (101) may further include a light source (e.g., an LED (light emitting diode)) that emits light toward a subject (e.g., the user's (110) eyes, face, and / or an object outside the FoV) being photographed using a camera (260). The light source may include an LED of infrared wavelength. The light source may be placed in at least one of the frame (200) and hinge units (206, 207).

[0086] According to an embodiment of the present disclosure, the battery module (270) can supply power to the electronic components of the wearable device (101). In an embodiment of the present disclosure, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may each be disposed in the first temple (204) and the second temple (205). In an embodiment of the present disclosure, the battery module (270) may be disposed at the end of the first temple (204) and / or the second temple (205).

[0087] The antenna module (275) can transmit a signal or power to the outside of the wearable device (101) or receive a signal or power from the outside. In an embodiment of the present disclosure, the antenna module (275) may be placed within the first temple (204) and / or the second temple (205). For example, the antenna module (275) may be placed close to one side of the first temple (204) and / or the second temple (205).

[0088] The speaker (255) can output an acoustic signal to the outside of the wearable device (101). The acoustic output module may be referred to as the speaker. In an embodiment of the present disclosure, the speaker (255) may be placed within a first temple (204) and / or a second temple (205) to be placed adjacent to the ear of a user (110) wearing the wearable device (101). For example, the speaker (255) may include a second speaker (255-2) placed adjacent to the left ear of the user (110) by being placed within the first temple (204), and a first speaker (255-1) placed adjacent to the right ear of the user (110) by being placed within the second temple (205).

[0089] A light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (101) to the user (110). For example, when the wearable device (101) requires charging, it may emit red light at a constant frequency. In an embodiment of the present disclosure, the light-emitting module may be placed on a first rim (201) and / or a second rim (202).

[0090] Referring to FIG. 2b, a wearable device (101) according to one embodiment may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of a first temple (204) or a second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardware components included in the wearable device (101) (e.g., hardware components illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (101) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0091] According to an embodiment of the present disclosure, a wearable device (101) may include at least one of a gyroscope sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the wearable device (101) and / or the posture of a body part (e.g., head) of a user (110) wearing the wearable device (101). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to an embodiment of the present disclosure, the wearable device (101) can identify motions and / or gestures of a user (110) performed to execute or stop specific functions of the wearable device (101) based on an IMU.

[0092]

[0093] FIGS. 3a and 3b illustrate the appearance of a wearable device according to various embodiments of the present disclosure.

[0094] The wearable device (101) of FIG. 3a and FIG. 3b may be an example of the electronic device (101) of FIG. 1 or the wearable device (101) of FIG. 2a and FIG. 2b. An example of the appearance of a first surface (310) of the housing of the wearable device (101) according to an embodiment of the present disclosure may be shown in FIG. 3a, and an example of the appearance of a second surface (320) opposite to the first surface (310) may be shown in FIG. 3b.

[0095] Referring to FIG. 3a, a first surface (310) of a wearable device (101) according to an embodiment of the present disclosure may have a shape that is attachable to a body part of a user (110) (e.g., the face of the user (110)). Although not illustrated, the wearable device (101) may further include a strap for securing to a body part of the user (110) and / or one or more temples (e.g., a first temple (204) and / or a second temple (205) of FIG. 2a and FIG. 2b). A first display (250-1) for outputting an image to the left eye among the two eyes of the user (110) and a second display (250-2) for outputting an image to the right eye among the two eyes may be disposed on the first surface (310). The wearable device (101) may further include rubber or silicone packing formed on the first surface (310) to prevent interference by light different from light emitted from the first display (250-1) and the second display (250-2) (e.g., ambient light).

[0096] According to an embodiment of the present disclosure, a wearable device (101) may include cameras (260-1) for photographing and / or tracking both eyes of a user (110) adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referenced to the eye-tracking camera (260-1) of FIG. 2B. According to an embodiment of the present disclosure, a wearable device (101) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of a user (110). The cameras (260-5, 260-6) may be referenced to FT cameras. The wearable device (101) can control an avatar representing the user (110) in a virtual space based on the motion of the user's (110) face identified using cameras (260-5, 260-6). For example, the wearable device (101) can change the texture and / or shape of a part of the avatar (e.g., a part of the avatar representing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of the user (110) wearing the wearable device (101).

[0097] Referring to FIG. 3b, on a second surface (320) opposite to the first surface (310) of FIG. 3a, a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)), and / or a sensor (e.g., a depth sensor (330)) may be placed to acquire information related to the external environment of the wearable device (101). For example, cameras (260-7, 260-8, 260-9, 260-10) may be placed on the second surface (320) to recognize external objects. The cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2b.

[0098] For example, using cameras (260-11, 260-12), the wearable device (101) can acquire images and / or videos to be transmitted to each of the user's (110) eyes. Camera (260-11) may be placed on the second surface (320) of the wearable device (101) to acquire an image to be displayed through a second display (250-2) corresponding to the right eye among the two eyes. Camera (260-12) may be placed on the second surface (320) of the wearable device (101) to acquire an image to be displayed through a first display (250-1) corresponding to the left eye among the two eyes. Cameras (260-11, 260-12) may be referenced to the shooting camera (260-4) of FIG. 2B.

[0099] According to an embodiment of the present disclosure, a wearable device (101) may include a depth sensor (330) disposed on a second surface (320) to identify the distance between the wearable device (101) and an external object. Using the depth sensor (330), the wearable device (101) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user (110) wearing the wearable device (101). Although not illustrated, a microphone may be disposed on the second surface (320) of the wearable device (101) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0100] Hereinafter, with reference to FIG. 4, the hardware or software configuration of the wearable device (101) will be described.

[0101] FIG. 4 illustrates a block diagram of a wearable device according to an embodiment of the present disclosure.

[0102] The wearable device (101) of FIG. 4 may be an example of the electronic device (101) of FIG. 1, and the wearable device (101) of FIG. 2a, 2b, 3a, and 3b.

[0103] Referring to FIG. 4, a wearable device (101) according to one embodiment may include a processor (410), memory (415), a display (250) (e.g., a first display (250-1) and / or a second display (250-2) of FIG. 2a, 2b, 3a, and 3b), and / or a sensor (420). The processor (410), memory (415), display (250) and / or sensor (420) may be electrically and / or operationally connected to each other by electronic components such as a communication bus (402). In the present disclosure, the operational connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (101) are not limited to those shown in FIG. 4. For example, the wearable device (101) may include only some of the electronic components shown in FIG. 4.

[0104] A processor (410) of a wearable device (101) according to an embodiment of the present disclosure may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In an embodiment of the present disclosure, the wearable device (101) may include one or more processors. According to an embodiment of the present disclosure, the structure of the processor (410) is not limited to one embodiment of the present disclosure, and at least one circuit may be formed as a separate processor (e.g., an embedded secure element (eSE), a secure processor) physically separated from the processor (410). The processor (410) may have a structure of a multi-core processor such as a dual core, quad core, hexa core, and / or octa core. The multi-core processor structure of the processor (410) may include a structure based on multiple core circuits (e.g., big-little structure) distinguished by power consumption, clock, and / or computational amount per unit time. In one embodiment comprising a processor (410) having a multi-core processor structure, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (410).

[0105] A memory (415) of a wearable device (101) according to an embodiment of the present disclosure may include electronic components for storing data and / or instructions that are input to or output from a processor (410). The memory (415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multi-media card (eMMC). In an embodiment of the present disclosure, the memory (415) may be referred to as storage.

[0106] In an embodiment of the present disclosure, a display (250) of a wearable device (101) can output visualized information to a user (110) of the wearable device (101). A display (250) arranged in front of the eyes of a user (110) wearing the wearable device (101) may be placed in at least a part of the housing of the wearable device (101) (e.g., a first display (250-1) and / or a second display (250-2) of FIG. 2a, FIG. 2b, FIG. 3a, and FIG. 3b). For example, the display (250) may be controlled by a processor (410) including circuits such as a CPU, a GPU (graphic processing unit), and / or a DPU (display processing unit) to output visualized information to the user (110). The display (250) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (250) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (250) may include a projector (or projection assembly) for projecting light onto said lens. In embodiments of the present disclosure, the display (250) may be referred to as a display panel and / or a display module. The pixels included in the display (250) can be positioned toward either of the user's (110) eyes when the wearable device (101) is worn by the user (110).For example, the display (250) may include display areas (or active areas) corresponding to each of the user's (110) two eyes.

[0107] In an embodiment of the present disclosure, a sensor (420) of a wearable device (101) may generate electrical information that can be processed by a processor (410) and / or a memory (415) from non-electronic information associated with the wearable device (101). For example, the sensor (420) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (101). In addition to the GPS method, the sensor (420) may generate information indicating the geographic location of the wearable device (101) based on a global navigation satellite system (GNSS), such as Galileo or Beidou (compass). This information may be stored in the memory (415), processed by the processor (410), and / or transmitted to another electronic device distinct from the wearable device (101) via a communication circuit.

[0108] Referring to FIG. 4, an image sensor (421) and / or a motion sensor (422) are illustrated as examples of sensors (420) included in a wearable device (101). The sensor (420) may include one or more light sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate electrical signals representing the color and / or brightness of light. The image sensor (421) may be referred to as a camera. A plurality of light sensors included in the image sensor (421) may be arranged in the form of a two-dimensional grid (2 dimensional array). The image sensor (421) may acquire the electrical signals of each of the plurality of light sensors substantially simultaneously to generate two-dimensional frame data corresponding to the light reaching the light sensors of the two-dimensional grid. For example, photo data captured using the image sensor (421) may refer to one (a) two-dimensional frame data acquired from the image sensor (421). For example, video data captured using an image sensor (421) may refer to a sequence of multiple two-dimensional frame data obtained from the image sensor (421) along a frame rate. The image sensor (421) may further include a flash light for outputting light in the direction in which the image sensor (421) receives light.

[0109] According to an embodiment of the present disclosure, the wearable device (101) may include a plurality of image sensors arranged toward different directions, as an example of an image sensor (421). As described above with reference to FIGS. 2A, 2B, 3A, and 3B, the plurality of image sensors may include an eye-tracking camera (e.g., the eye-tracking camera (260-1) of FIGS. 2B and 3A) configured to be arranged toward the eyes of a user (110) wearing the wearable device (101). The plurality of image sensors may include an outward camera. A processor (410) may identify the direction of the user's (110) gaze using an image and / or video obtained from the eye-tracking camera. The eye-tracking camera may include an infrared (IR) sensor. The eye-tracking camera may be referred to as an eye sensor and / or an eye tracker.

[0110] An external camera may be positioned facing the front of a user (110) wearing the wearable device (101) (e.g., a direction in which both eyes may face). The wearable device (101) may include a plurality of external cameras. The embodiments are not limited thereto, and the external camera may be positioned facing the external space. Using images and / or videos obtained from the external camera, the processor (410) may identify external objects. For example, the processor (410) may identify the position, shape, and / or gesture (e.g., hand gesture) of the hand of the user (110) wearing the wearable device (101) based on images and / or videos obtained from the external camera. Using images and / or videos of the external environment obtained from the external camera, the processor (410) may recognize or track one or more objects within the external environment.

[0111] According to an embodiment of the present disclosure, the motion sensor (422) may output an electrical signal representing gravitational accelerations, accelerations, and / or angular velocities of a plurality of axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other and are based on a designated origin within the wearable device (101) and / or the motion sensor (422). For example, the processor (410) may repeatedly receive or acquire sensor data from the motion sensor (422), including accelerations, angular velocities, and / or magnitudes of the number of axes, based on a designated period (e.g., 1 millisecond). In an embodiment of the present disclosure, the motion sensor (422) may be referred to as an inertial measurement unit (IMU). The sensor (420) included in the wearable device (101) is not limited to the above and may include a grip sensor, a proximity sensor, a heart rate sensor, a fingerprint sensor, an ambient light sensor, and / or a ToF sensor. By using a motion sensor (422), the processor (410) can detect motion of the wearable device (101) (e.g., motion of the wearable device (101) caused by a user (110) wearing the wearable device (101).

[0112] In an embodiment of the present disclosure, a communication circuit (not shown) of a wearable device (101) may include a hardware component for supporting the transmission and / or reception of a signal between the wearable device (101) and an external electronic device (e.g., an external electronic device (102) or an external electronic device (104)). The communication circuit may include, for example, at least one of a modem, an antenna, and an O / E (optic / electronic) converter. The communication circuit (430) may support the transmission and / or reception of an electrical signal based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), WiFi (wireless fidelity), Bluetooth, BLE (Bluetooth low energy), Zigbee, LTE (long term evolution), and 5G NR (new radio).

[0113] In the memory (415) of a wearable device (101) according to an embodiment of the present disclosure, one or more instructions (or commands) representing data to be processed by the processor (410) of the wearable device (101), calculations to be performed, and / or operations to be performed may be stored. A set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, sub-routine, and / or software application (hereinafter, application). For example, the wearable device (101), and / or processor (410) may perform at least one of the operations of FIG. 7, FIG. 9, and FIG. 11 when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, program, and / or software application is executed. In the following, the statement that a software application is installed in a wearable device (101) may mean that one or more instructions provided in the form of a software application (or package) are stored in memory (415), and that the one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the wearable device (101)) by the processor (410). For example, the application may include a program and / or library related to a service provided to a user (110).

[0114] Referring to FIG. 4, programs installed on a wearable device (101) may be included in any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, within the hardware abstraction layer (480) (e.g., an Android system HAL, and / or an XR HAL), programs (e.g., modules, or drivers) designed to target the hardware of the wearable device (101) (e.g., a display (250), and / or a sensor (420)) may be included. The framework layer (450) may be referred to as an XR framework layer in that it includes one or more programs for providing XR (extended reality) services. For example, the layers illustrated in FIG. 4 are logically (or for convenience of explanation) separated, and may not imply that the address space of memory (415) is separated by said layers.

[0115] For example, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a location tracker (471), a spatial recognizer (472), a gesture tracker (473), an eye tracker (474), a face tracker (475), and / or a renderer (490)) may be included. The programs included in the framework layer (450) may provide an application programming interface (API) that is executable (or invokeable) based on other programs.

[0116] For example, within the application layer (440), a program designed to target a user (110) of the wearable device (101) may be included. Examples of programs included in the application layer (440) include an XR (extended reality) system UI (user interface) (441) and / or an XR application (442), but embodiments are not limited thereto. For example, programs included in the application layer (440) (e.g., software applications) may call an API to cause the execution of a function supported by programs included in the framework layer (450).

[0117] For example, the wearable device (101) may display one or more visual objects on the display (250) to perform interaction with the user (110) based on the execution of the XR system UI (441). A visual object may mean an object that can be placed on the screen for the transmission of information and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user (110) with functions available in a virtual space based on the execution of the XR system UI (441).

[0118] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plugin (444) are depicted within the XR system UI (441), but are not limited thereto. For example, based on the XR system UI (441), the processor (410) may execute a lightweight renderer (443) and / or an XR plugin (444) within the framework layer (450).

[0119] For example, a wearable device (101) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute a rendering pipeline, which is permitted to be partially modified, based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline, which is permitted to be partially modified. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, the wearable device (101) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute the entire rendering pipeline based on the execution of an XR plugin (444). The XR plugin (444) may be referred to as an open XR native client in terms of defining (or setting) the entire rendering pipeline.

[0120] For example, the wearable device (101) may display a screen representing at least a portion of a virtual space on the display (250) based on the execution of the XR application (442). The XR plugin (444-1) included in the XR application (442) may include instructions that support functions similar to the XR plugin (444) of the XR system UI (441). Descriptions of the XR plugin (444-1) that overlap with descriptions of the XR plugin (444) may be omitted. The wearable device (101) may trigger the execution of the virtual space manager (451) based on the execution of the XR application (442).

[0121] For example, the wearable device (101) may display an image on the display (250) in a virtual space based on the execution of an application (445). The application (445) may be configured to output image information for displaying a two-dimensional image. The wearable device (101) may trigger the execution of a virtual space manager (451) based on the execution of the application (445). The wearable device (101) may generate dual image information to display the two-dimensional image in a three-dimensional virtual space based on the execution of the application (445). Here, the dual image information may include a first image information for the left eye and a second image information for the right eye based on binocular parallax. To display the two-dimensional image in a three-dimensional virtual space, the wearable device (101) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0122] According to an embodiment of the present disclosure, a wearable device (101) can provide a virtual space service based on the execution of a virtual space manager (451). For example, the virtual space manager (451) may include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (451), the wearable device (101) can identify a virtual space formed based on the location of a user (110) indicated by data acquired through a sensor (420), and can display at least a portion of the virtual space on a display (250). The virtual space manager (451) may be referred to as a composition presentation manager (CPM).

[0123] For example, the virtual space manager (451) may include a runtime service (452). For example, the runtime service (452) may be referred to as an OpenXR runtime module (or OpenXR runtime program). The wearable device (101) may execute at least one of a pose prediction function, a frame timing function, and / or a spatial input function of the user (110) based on the execution of the runtime service (452). For example, the wearable device (101) may perform rendering for the virtual space service for the user (110) based on the execution of the runtime service (452). For example, a virtual space-related function that is executable by the application layer (440) based on the execution of the runtime service (452) may be supported.

[0124] For example, the virtual space manager (451) may include a pass-through manager (453). The wearable device (101) may display an image and / or video representing the real space acquired through an external camera superimposed on at least a portion of the screen while displaying a screen representing the virtual space on the display (250) based on the execution of the pass-through manager (453).

[0125] For example, the virtual space manager (451) may include an input manager (454). The wearable device (101) may identify acquired data (e.g., sensor data) by executing one or more programs included within the recognition service layer (470) based on the execution of the input manager (454). The wearable device (101) may identify user inputs associated with the wearable device (101) using the acquired data. The user inputs may be associated with motions (e.g., hand gestures), gazes, and / or speech of the user (110) identified by a sensor (420) (e.g., an image sensor (421) such as an external camera). The user inputs may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0126] For example, the perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). In terms of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. As an example, the perception abstract layer (460) can be referred to as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0127] According to an embodiment of the present disclosure, the recognition service layer (470) may include one or more programs for processing data obtained from a sensor (420). The one or more programs may include at least one of a position tracker (471), a spatial recognizer (472), a gesture tracker (473), an eye tracker (474), a face tracker (475), and / or a renderer (490). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those shown in FIG. 4.

[0128] For example, the wearable device (101) can identify the posture of the wearable device (101) using the sensor (420) based on the execution of the position tracker (471). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using an external camera (e.g., image sensor (421)) and / or an IMU (e.g., motion sensor (422) including a gyroscope, accelerometer, and / or geomagnetic sensor) based on the execution of the position tracker (471). The position tracker (471) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0129] For example, the wearable device (101) may acquire information to provide a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user (110) of the wearable device (101)) based on the execution of the spatial recognizer (472). The wearable device (101) may reproduce the surrounding environment of the wearable device (101) in three dimensions using data acquired using an external camera (e.g., image sensor (421)) based on the execution of the spatial recognizer (472). The wearable device (101) may identify at least one of a plane, an incline, and a staircase based on the surrounding environment of the wearable device (101) reproduced in three dimensions based on the execution of the spatial recognizer (472). The spatial recognizer (472) may be referred to as a scene understanding (SU) module (or scene understanding program).

[0130] For example, the wearable device (101) can identify (or recognize) the pose and / or gesture of the user (110) of the wearable device (101) based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user (110)'s hand using data acquired from an external camera (e.g., image sensor (421)) based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user (110)'s hand based on data (or images) acquired using an external camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0131] For example, the wearable device (101) can identify (or track) the eye movements of the user (110) of the wearable device (101) based on the execution of the eye tracker (474). For example, the wearable device (101) can identify the eye movements of the user (110) using data obtained from an eye tracking camera (e.g., image sensor (421)) based on the execution of the eye tracker (474). The eye tracker (474) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0132] For example, the recognition service layer (470) of the wearable device (101) may further include a face tracker (475) for tracking the face of the user (110). For example, the wearable device (101) may identify (or track) the movement of the user (110)'s face and / or the facial expression of the user (110) based on the execution of the face tracker (475). The wearable device (101) may estimate the facial expression of the user (110) based on the movement of the user (110)'s face based on the execution of the face tracker (475). For example, the wearable device (101) can identify the movement of the user's (110) face and / or the user's (110) facial expressions based on data (e.g., images and / or videos) acquired using a camera (e.g., a face tracking (FT) camera, a camera facing at least a part of the user's (110) face, an image sensor (421)) based on the execution of the face tracker (475). The face tracker (475) may be referred to as face tracking (FT) (or face tracking program) and / or a face tracking module.

[0133] Referring to FIG. 4, the renderer (490) may include instructions for rendering images in a three-dimensional virtual space. The processor (410) executing the renderer (490) may obtain at least one image to be displayed at least partially in a display area of ​​the display (250) in a software application. For example, the processor (410) executing the renderer (490) may determine the location of the area where an application (e.g., XR application (442), application (445)) will be rendered. The processor (410) executing the renderer (490) may generate an image of said application to be displayed on the display (250). The renderer (490) may synthesize images to generate a composite image to be displayed on the display (250).

[0134] For example, a processor (410) that executes a renderer (490) can divide the display area of ​​a display (250) into a foveated portion (or may be referred to as a foveated area) and a peripheral portion (or may be referred to as a residual area) using a gaze position calculated using a position tracker (471) and / or a gaze tracker (474). For example, a processor (410) that detects coordinate values ​​of the gaze position can determine the portion of the display area containing said coordinate values ​​as the foveated area. A DPU that executes a renderer (490) can acquire at least one image corresponding to each of said foveated area and said residual area, having a size smaller than the size of the entire display area of ​​the display (250) or having a resolution less than the resolution of the display area.

[0135] For example, the processor (410) that executes the renderer (490) can obtain or generate a composite image to be displayed on the display (250) by synthesizing an image corresponding to the foveated area and an image corresponding to the surrounding area. For example, the processor (410) can perform upscaling to enlarge the image corresponding to the surrounding area to the size of the entire display area of ​​the display (250). On the enlarged image, the processor (410) can combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (250). Along the boundary line of the image corresponding to the foveated area, the processor (410) can mix the enlarged image and the image corresponding to the foveated area by applying a visual effect such as blur.

[0136] FIG. 5 shows an example of a block diagram of a wearable device for displaying an image in a virtual space according to an embodiment of the present disclosure (e.g., the electronic device (101) of FIG. 1, the wearable device (101) of FIG. 2a to FIG. 4).

[0137] In FIG. 5, an example is described in which multiple programs / instructions for displaying an image in a virtual space are executed. The multiple programs / instructions may all be executed on a single processor (e.g., AP) or may be executed by multiple processors (e.g., AP, GPU (graphic processing unit), NPU (neural processing unit)). The meaning of being able to be executed by multiple processors is that some programs / instructions may be executed by a first processor and other programs / instructions may be executed by a second processor different from the first processor.

[0138] Referring to FIG. 5, a wearable device (101) may execute a virtual space manager (550) (e.g., virtual space manager (451) of FIG. 4, CPM) to render an image in a virtual space. For the virtual space manager (550), at least some of the descriptions of the virtual space manager (451) of FIG. 4 may be referenced. The virtual space manager (550) may include a platform for supporting virtual space services. The virtual space manager (550) may include a runtime service (551) (e.g., OpenXR Runtime), a panel renderer (552) (e.g., 2D (two-dimensional) Panel Render), and an XR composite unit (553) (XR Compositor). Based on the execution of the runtime service (551), the wearable device (101) may execute at least one of a pose prediction function, a frame timing function, and / or a spatial input function of a user (110). For the runtime service (551), at least some of the descriptions of the runtime service (452) of FIG. 4 may be referenced. The wearable device (101) may display at least one image (video) on a panel (e.g., a 2D panel) to enable the implementation of a virtual space through a display, based on the execution of the panel renderer (552). For example, the wearable device (101) may display a rendering image corresponding to RGB information (566) for a panel from the spatialization manager (540) described later through a display (e.g., a display (250)). The wearable device (101) may composite an image of a real area (hereinafter, a pass-through image) captured through a camera in a virtual space with an image of a virtual area, based on the execution of the XR composite unit (553) (XR Compositor). For example, the wearable device (101) can generate a composite image by merging the pass-through image and the virtual region image based on the execution of the XR synthesis unit (553).The wearable device (101) can transmit the generated composite image to a display buffer so that the composite image is displayed. The wearable device (101) can identify a virtual space through a virtual space manager (550) and display at least a portion of the virtual space on a display (250). The virtual space manager (550) may be referred to as CPM. The wearable device (101) can execute the virtual space manager (550) to render an image corresponding to at least a portion of the virtual space.

[0139] According to an embodiment of the present disclosure, a wearable device (101) can execute a spatialization manager (540). The spatialization manager (540) can perform processes for displaying an image in a three-dimensional virtual space. The wearable device (101) can perform preprocessing based on the execution of the spatialization manager (540) so that an image can be rendered in a three-dimensional virtual space through a virtual space manager (550). For example, the wearable device (101) can perform at least some of the functions of the renderer (490) of FIG. 4 based on the execution of the spatialization manager (540). The wearable device (101) can process image information provided by an application (e.g., an XR application (510), a non-XR general 2D screen providing application (520), a system UI providing application (530)) based on the execution of the spatialization manager (540).

[0140] For example, a spatialization manager (540) (e.g., Space Flinger) may include a system screen manager (541) (e.g., System scene), an input manager (542) (e.g., Input Routing), and a lightweight rendering engine (543) (e.g., Impress Engine). The system screen manager (541) may be executed to display the system UI (530). System UI-related information (564) may be transmitted to the system screen manager (541) from a program (e.g., API) that provides the system UI (530). System UI-related information (564) may be obtained through a spatializer API and / or a Same-process private API. The spatialization manager (540) may determine the layout (e.g., position, display order) of the system UI (530) screen in three-dimensional space through pre-allocated resources. The system screen manager (541) may transmit image information (567) for rendering a screen of the system UI (530) according to the layout to the virtual space manager (550). The input manager (542) may be configured to process user input (e.g., user input on a system screen or app screen). The input manager (542) may map user input recognized by a sensor (420) of the wearable device (101) to at least one of one or more software applications (e.g., an XR application (510), an application providing a non-XR general 2D screen (520), and an application providing a system UI (530)) mapped to a virtual space by the spatialization manager (540). For example, the mapping of user input may include an action of executing instructions (e.g., sub-routines and / or event handlers) of a software application for processing user input. The lightweight rendering engine (543) may be a renderer for image generation (e.g., lightweight renderer (443)).For example, a lightweight rendering engine (543) may be used to display a system UI (530). According to an embodiment of the present disclosure, the spatialization manager (540) may include a lightweight rendering engine (543) for rendering the system UI. According to an embodiment of the present disclosure, if the lightweight rendering engine (543) does not have sufficient resources to render an avatar used in an HMD, at least one external rendering engine may be used. In this case, to address compatibility issues with external rendering (e.g., a third-party engine), an external rendering engine support module may be added within the spatialization manager (540).

[0141] According to an embodiment of the present disclosure, a wearable device (101) may execute an application. For example, in response to the execution of an XR application (510) (e.g., an XR application (442), a three-dimensional game, an XR map, or other immersive application), a virtual space manager (550) may be executed. The wearable device (101) may provide dual image information (561) provided from the XR application (510) to the virtual space manager (550). To display images in three-dimensional space, the dual image information (561) may include two image information that takes into account binocular parallax. For example, the dual image information (561) may include a first image information for the left eye of the user (110) and a second image information for the right eye of the user (110) to render in three-dimensional virtual space. Hereinafter, the term dual image information is used in the present disclosure as a term referring to image information for displaying images for both eyes in three-dimensional space. In addition to the dual image information, the above dual image information may utilize binocular image information, dual image data, dual image, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimension conversion image data, binocular parallax image data, and / or equivalent technical terms. The wearable device (101) can generate a composite image by merging image layers through a virtual space manager (550). The wearable device (101) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (250) of the wearable device (101).

[0142] According to an embodiment of the present disclosure, a wearable device (101) may execute at least one application among an XR application (510) and other applications (520) (e.g., a first application (520-1), a second application (520-2), ..., a Nth application (520-N)). According to an embodiment of the present disclosure, the application (520) may be configured to output image information for displaying a two-dimensional image. In other words, the application (520) may provide a two-dimensional (2D) image (e.g., a window, and / or an activity). As an example, the application (520) may be a video application, a schedule application, or an internet browser application. If, in response to the execution of the application (520), image information (562) provided by the application (520) is provided to the virtual space manager (550), the image information (562) has only x and y coordinates within a two-dimensional plane, so it may be difficult to consider the sequential relationship between other applications centered on the user (110) (i.e., distance from the user (110)). The wearable device (101) may execute a spatialization manager (540) to provide dual image information to the virtual space manager (550) even when displaying an application (520) that provides a general 2D screen. For example, based on the execution of the spatialization manager (540), the wearable device (101) may receive application-related information (563) from the first application (520-1). For example, application-related information (563) may include image information representing a two-dimensional image of the first application (520-1) (e.g., information including RGB per pixel) and / or content information in the first application (520-1) (e.g., characteristics of content running in the first application, type of content). Application-related information (563) may be obtained through a spatializer API.Based on the execution of the spatialization manager (540), the wearable device (101) can identify information regarding the location of the area to be rendered and the size of the area to be rendered (hereinafter, location information). Based on the execution of the spatialization manager (540), the wearable device (101) can generate dual image information (565, e.g., RGBx2) that takes into account the binocular parallax of the user (110) through the image information and the location information. Based on the execution of the spatialization manager (540), the wearable device (101) can provide the dual image information (565) to the virtual space manager (550). By converting a simple two-dimensional image into dual image information (565), the problem caused by the image information (562) being directly transmitted to the virtual space manager (550) can be resolved (addressed). Additionally, as at least some of the functions for displaying images in virtual space are performed by the spatialization manager (540) instead of the virtual space manager (550), the burden on the virtual space manager (550) may be reduced.

[0143] FIG. 6 is a block diagram illustrating the operation of a wearable device that performs object tracking and hand tracking according to an embodiment of the present disclosure.

[0144] Referring to FIG. 6, a wearable device (101) according to one embodiment can perform hand tracking (610) and object tracking (620). For example, the wearable device (101) can display a hand object within a virtual screen displayed through a display (250) based on data about the user's (110) hand (e.g., hand position data, hand angle data, hand joint position data, hand joint angle data, hand gesture data, hand shape data, or a combination thereof) obtained through hand tracking (610) using a camera (260) and / or a sensor (420). For example, a wearable device (101) can display an object (or a virtual object) within a virtual screen displayed through a display (250) based on data about an object (e.g., position data of the object, angle data of the object, type data of the object, or a combination thereof) obtained by object tracking (620) using a camera (260) and / or a sensor (420).

[0145] If hand tracking (610) and object tracking (620) are performed independently, problems may arise in which the user's (110) personal information (e.g., password) is leaked or unintended actions are performed by the user (110). In one example, when a user (110) wearing a wearable device (101) enters a password through the input means of an automated teller machine (ATM), the user's (110) password may be leaked to a third party by hand tracking (610) and object tracking (620). In one example, while the user (110) wearing a wearable device (101) is controlling a real keyboard, unintended actions (e.g., virtual keyboard control) may be performed by the user (110) by a hand object displayed on a virtual screen.

[0146] To prevent personal information leakage or unintended actions by the user (110), information about the object obtained during object tracking (620) (e.g., object type, attributes) may be used for hand tracking (610). Below, a device and method for preventing personal information leakage or unintended actions by the user (110) by omitting or changing the process of hand tracking (610) according to the information about the object obtained during object tracking (620) are described.

[0147] FIG. 7 is a flowchart illustrating the operation of a wearable device for restricting the display of a user's hand object within a virtual screen according to an embodiment of the present disclosure.

[0148] The operations of FIG. 7 may be performed by the electronic device (101) of FIG. 1 and the wearable device (101) of FIG. 2a through FIG. 4. For example, at least some of the operations may be controlled by the processor (410) of the wearable device (101). In the following embodiments of the disclosure, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed. For example, at least two operations may be performed in parallel.

[0149] Referring to FIG. 7, in operation 701, a wearable device (101) according to one embodiment can identify an object included in a virtual screen displayed through a display (250).

[0150] In an embodiment of the present disclosure, a wearable device (101) may acquire an image of the surrounding environment using a camera (260) and / or a sensor (420). The image may include an image of an object outside the wearable device (101) (or, surrounding object, external object). The wearable device (101) may perform object recognition based on the image acquired using the camera (260) and / or the sensor (420). For example, object recognition may be performed based on identifying features of an object included in the image (e.g., boundary, corner, edge).

[0151] In an embodiment of the present disclosure, the wearable device (101) may perform object tracking for an identified object. For example, the wearable device (101) may generate object tracking data based on the object tracking. For example, the object tracking data may include position data of the object, angle data of the object, type data of the object, or a combination thereof.

[0152] In operation 702, a wearable device (101) according to one embodiment can identify whether an object included in a virtual screen corresponds to a type for restricting the display (or representation) of the user's (110) hand object.

[0153] In an embodiment of the present disclosure, the wearable device (101) can identify the type of object included in a virtual screen displayed through a display (250) based on object type data. For example, the object type may mean a criterion for classifying objects based on a category. In one example, the object type may correspond to one of the object types: an automated teller machine (ATM), a door lock, a keyboard, or a mouse. However, this is merely an example and the present disclosure is not limited thereto. For example, the object type may further include object types belonging to other categories, and the exemplified object types (e.g., keyboard and mouse) may be classified into one object type (e.g., input means) based on a higher category. For example, the object type may be replaced with object classification, object category, object kind, object group, object class, or other terms having an equivalent technical meaning.

[0154] In an embodiment of the present disclosure, the wearable device (101) may generate attribute data of an object based on an object type. For example, the attribute data may indicate one of the attributes associated with an interaction between the object and the user's (110) hand object. In one example, the attribute may be specified for the object type. In one example, the attribute may be identified based on an artificial intelligence model of the wearable device (101) that uses the object type as input data. For example, the attributes may include a first attribute, a second attribute, and a third attribute. However, the present disclosure is not limited thereto. For example, the attributes may include only some of the first attribute, the second attribute, and the third attribute.

[0155] In an embodiment of the present disclosure, the first attribute may indicate that interaction between the user (110)'s hand object and the object is prohibited. In one example, the object type having the first attribute may include an ATM and a door lock. However, the present disclosure is not limited thereto. The object type having the first attribute may include an object type in which the user's (110) personal information (or security information, password) may be leaked when hand tracking data and / or object tracking data are provided to an application, a system, and / or an external electronic device. To prevent the leakage of the user's (110) personal information, the object type having the first attribute may restrict the display of the user's (110) hand object within a virtual screen displayed through the display (250) of the wearable device (101).

[0156] In an embodiment of the present disclosure, the second attribute may indicate that interaction between the user (110)'s hand object and the object is restricted. In one example, the object type having the second attribute may include a keyboard and a mouse. However, the present disclosure is not limited thereto. The object type having the second attribute may include an object type capable of interacting with the user (110)'s hand (e.g., a real hand). For example, while the user (110) controls an input means (e.g., a real keyboard and / or a real mouse) using their hand, unintended actions (e.g., control of a virtual keyboard and / or a virtual mouse) may occur by the hand object displayed on the virtual screen of the wearable device (101). To prevent unintended actions by the user (110), the object type having the second attribute may restrict interaction with the user (110)'s hand object within the virtual screen displayed through the display (250) of the wearable device (101).

[0157] In an embodiment of the present disclosure, the third attribute may indicate that interaction between the user (110)'s hand object and the object is not restricted. An object type having the third attribute may be able to interact with the user (110)'s hand object within a virtual screen displayed through the display (250) of the wearable device (101).

[0158] In operation 703, if operation 702-yes (e.g., when the object is of a type for restricting the display of a hand object), a wearable device (101) according to one embodiment can identify whether a hand object is located within a threshold distance (e.g., 1 meter) from an object within a virtual screen. For example, the wearable device (101) can identify whether a hand object is located within a threshold distance from an object within a virtual screen based on the identification that an object included within the virtual screen corresponds to a type for restricting the display (or representation) of the user's (110) hand object (e.g., an object type having a first attribute).

[0159] In operation 704, if operation 703-yes (e.g., when a hand object is located within a threshold distance from an object), a wearable device (101) according to one embodiment may display a virtual screen through a display (250) that does not display the movement of the user (110)'s hand object according to hand tracking. For example, the wearable device (101) may display a virtual screen through a display (250) that does not display the movement of the hand object according to hand tracking, based on the identification that the hand object is located within a threshold distance from an object within the virtual screen.

[0160] In an embodiment of the present disclosure, the wearable device (101) may not perform hand tracking in order not to display the movement of a hand object according to hand tracking. Since hand tracking is not performed by the wearable device (101), hand tracking data may not be generated. Since hand tracking data is not generated, the hand object of the user (110) may not be displayed on the virtual screen of the wearable device (101). Since the hand object is not displayed on the virtual screen of the wearable device (101), the leakage of the user's (110) personal information displayed according to the hand movement of the user (110) may be prevented.

[0161] In an embodiment of the present disclosure, a wearable device (101) may generate hand tracking data based on hand tracking. The hand tracking data may include hand position data, hand angle data, hand joint position data, hand joint angle data, hand gesture data, hand shape data, or a combination thereof. However, the present disclosure is not limited thereto. For example, the hand tracking data may include only some of the data described above.

[0162] In an embodiment of the present disclosure, the wearable device (101) may refrain from providing hand tracking data generated according to hand tracking to an application, a system, and / or an external electronic device so as not to display the movement of a hand object according to hand tracking. For example, the wearable device (101) may identify, based on the attribute data of the object, that the display of a hand object located within a threshold distance from the object is restricted. The wearable device (101) may refrain from providing hand tracking data to an application, a system, and / or an external electronic device based on the identification that the display of a hand object within a virtual screen is restricted. Since the hand tracking data of the user (110) is not provided to an application, a system, and / or an external electronic device, the hand object may not be displayed through the display (250) of the wearable device (101). Since hand tracking data is not provided to applications, systems, and / or external electronic devices, the user's (110) personal information displayed according to the user's (110) hand movements can be prevented from being leaked.

[0163] In an embodiment of the present disclosure, the wearable device (101) may display a hand object having a default gesture on a virtual screen so as not to display the movement of the hand object according to hand tracking. For example, the wearable device (101) may identify, based on the attribute data of the object, that the display of a hand object located within a threshold distance from the object is restricted. The wearable device (101) may display a hand object having a default gesture on a virtual screen in accordance with the identification that the display of the hand object is restricted within the virtual screen. For example, the wearable device (101) may not use hand angle data, hand joint position data, hand joint angle data, hand gesture data, and hand shape data for hand tracking. Since the data is not used for hand tracking, the hand object displayed on the virtual screen may have a default gesture. In one example, the default gesture may correspond to a fist gesture. However, this is merely an example, and the present disclosure is not limited thereto. For example, a basic gesture may mean any gesture having a fixed hand posture (or shape). Since the hand object having the basic gesture does not display the user's (110) hand movements, the user's (110) personal information displayed according to the user's (110) hand movements can be prevented from being leaked.

[0164] In an embodiment of the present disclosure, the wearable device (101) may generate hand tracking data different from the movement of the user's (110) hand in order not to display the movement of the hand object according to hand tracking. For example, hand tracking data different from the movement of the user's (110) hand may be referred to as dummy data or other terms having an equivalent technical meaning. For example, the wearable device (101) may identify, based on the attribute data of the object, that the display of the hand object located within a threshold distance from the object is restricted. The wearable device (101) may generate different data different from the hand tracking data according to hand tracking in accordance with the identification that the display of the hand object within the virtual screen is restricted. For example, the wearable device (101) may generate different data different from the hand position data, hand angle data, hand joint position data, hand joint angle data, hand gesture data, and hand shape data generated according to hand tracking. In one example, since different data is generated that differs from the hand position data generated according to hand tracking, the position of the hand object displayed on the virtual screen may differ from the position of the user's (110) hand. In one example, since different data is generated that differs from the hand angle data, hand joint position data, hand joint angle data, hand gesture data, and hand shape data generated according to hand tracking, the posture (or shape) of the hand object displayed on the virtual screen of the wearable device (101) may differ from the posture (or shape) of the user's (110) hand. Since the position and / or posture of the user's (110) hand and the position and / or posture of the hand object displayed by the data generated by the wearable device (101) are different, the leakage of the user's (110) personal information displayed according to the user's (110) hand movements can be prevented.

[0165] In operation 705, if operation 702-No (e.g., the object is not of a type to restrict the display of the hand object) or operation 703-No (e.g., the hand object is located outside a threshold distance from the object), the wearable device (101) according to one embodiment may display a virtual screen containing the hand object of the user (110) based on hand tracking through the display (250). For example, the wearable device (101) may display a virtual screen containing the hand object of the user (110) based on hand tracking through the display (250) upon identification that the object included in the virtual screen does not correspond to a type to restrict the display (or representation) of the hand object of the user (110). In another example, the wearable device (101) can display a virtual screen containing the user's (110) hand object through a display (250) based on hand tracking, upon identification that the hand object is located outside a threshold distance from the object within the virtual screen.

[0166] In an embodiment of the present disclosure, the wearable device (101) can identify whether an object corresponds to a type in which interaction with the user (110)'s hand object is restricted. For example, the wearable device (101) may refrain from controlling a virtual object using a hand object displayed on a virtual screen according to hand tracking data, depending on the identification that the object corresponds to a type in which interaction with the user (110)'s hand object is restricted. In another example, the wearable device (101) may control a virtual object using a hand object displayed on a virtual screen according to hand tracking data, depending on the identification that the object does not correspond to a type in which interaction with the user (110)'s hand object is restricted.

[0167]

[0168] FIGS. 8a, 8b, and 8c illustrate the operation of a wearable device for limiting the display of a user's hand object within a virtual screen according to various embodiments of the present disclosure.

[0169] FIGS. 8a, FIGS. 8b, and FIGS. 8c explain that when a user (110) enters a password (e.g., 1523) through the input means of an ATM (automated teller machine) (802), the movement of a hand object according to hand tracking is not displayed within the virtual screen.

[0170] Referring to FIG. 8a, the user (110) can input the first number of the password (e.g., 1) at the first time, the second number of the password (e.g., 5) at the second time, the third number of the password (e.g., 2) at the third time, and the fourth number of the password (e.g., 3) at the fourth time through the input means of the ATM (802) by moving the hand (801). For example, the wearable device (101) can display an image of the ATM (802) obtained using a camera (260) and / or a sensor (420) within a virtual screen. In another example, the ATM (802) can be shown to the user (110) through a lens included in the display (250) of the wearable device (101). The wearable device (101) can identify attributes specified for the object type of the ATM (802). An attribute specified for the object type of ATM (802) may indicate that the display of the hand object is restricted within a virtual screen displayed through the display (250) of the wearable device (101). For example, the wearable device (101) may not perform hand tracking based on the identification that the ATM (802) corresponds to an object type that restricts the display (or representation) of the hand object. In another example, the wearable device (101) may refrain from providing hand tracking data generated by hand tracking to an application and / or system based on the identification that the ATM (802) corresponds to an object type that restricts the display of the hand object. For example, the hand tracking data generated by hand tracking may be processed in a trusted execution environment (TEE). For example, the hand tracking data may be processed in a secure area. In one example, the security area may be a separate area configured for security within the processor (410).In another example, the secure area may be a separate area configured for security outside the processor (410) (e.g., an embedded secure element (eSE), a secure processor). In yet another example, the secure area may be implemented based on software (e.g., a hypervisor). Since hand tracking is not performed or hand tracking data is not provided to the application and / or system, the wearable device (101) may not display a hand object within a virtual screen displayed through the display (250). For example, the wearable device (101) may not display a hand object (812) pressing the first number (e.g., 1) of a password in a first virtual screen (811) displayed through the display (250). For example, the wearable device (101) may not display a hand object (814) pressing the second number (e.g., 5) of the password on a second virtual screen (813) displayed through the display (250). For example, the wearable device (101) may not display a hand object (816) pressing the third number (e.g., 2) of the password on a third virtual screen (815) displayed through the display (250). For example, the wearable device (101) may not display a hand object (818) pressing the fourth number (e.g., 3) of the password on a fourth virtual screen (817) displayed through the display (250). Since hand objects are not displayed on the display (250) while the user (110) is entering a password using the hand (801), the leakage of the user's (110) password displayed according to the movement of the user's (110) hand (801) can be prevented.

[0171] Referring to FIG. 8b, the user (110) can input the first number of the password (e.g., 1) at the first time, the second number of the password (e.g., 5) at the second time, the third number of the password (e.g., 2) at the third time, and the fourth number of the password (e.g., 3) at the fourth time through the input means of the ATM (802) by moving the hand (801). For example, the wearable device (101) can display an image of the ATM (802) obtained using a camera (260) and / or a sensor (420) within a virtual screen. In another example, the ATM (802) can be shown to the user (110) through a lens included in the display (250) of the wearable device (101). The wearable device (101) can identify attributes specified for the object type of the ATM (802). The attribute specified for the object type of ATM (802) may indicate that the display of a hand object is restricted within a virtual screen displayed through the display (250) of the wearable device (101). The wearable device (101) may display a hand object having a default gesture (or a fixed gesture) on the virtual screen according to the identification that the ATM (802) corresponds to the object type that restricts the display of the hand object. The hand object having the default gesture may differ from the gesture (or posture, shape) of the user's (110) hand (801). For example, the wearable device (101) may display a hand object (822) having a default gesture that differs from the gesture of the user's (110) hand (801) on the first virtual screen (821) while the user (110) presses the first number (e.g., 1) of the password using the hand (801). For example, the wearable device (101) may display a hand object (824) having a basic gesture different from the gesture of the user's (110) hand (801) on a second virtual screen (823) while the user (110) presses the second number (e.g., 5) of the password using the hand (801).For example, the wearable device (101) may display a hand object (826) having a basic gesture different from the gesture of the user (110)'s hand (801) on a third virtual screen (825) while the user (110) presses the third number (e.g., 2) of the password using the hand (801). For example, the wearable device (101) may display a hand object (828) having a basic gesture different from the gesture of the user (110)'s hand (801) on a fourth virtual screen (827) while the user (110) presses the fourth number (e.g., 3) of the password using the hand (801). Since hand objects having basic gestures different from the gesture of the user (110)'s hand (801) entering the password are displayed on the virtual screen, leakage of the password displayed according to the user (110)'s hand movements can be prevented.

[0172] Referring to FIG. 8c, the user (110) can input the first number of the password (e.g., 1) at the first time, the second number of the password (e.g., 5) at the second time, the third number of the password (e.g., 2) at the third time, and the fourth number of the password (e.g., 3) at the fourth time through the input means of the ATM (802) by moving the hand (801). For example, the wearable device (101) can display an image of the ATM (802) obtained using a camera (260) and / or a sensor (420) within a virtual screen. In another example, the ATM (802) can be shown to the user (110) through a lens included in the display (250) of the wearable device (101). The wearable device (101) can identify attributes specified for the object type of the ATM (802). The attribute specified for the object type of ATM (802) may indicate that the display of a hand object is restricted within a virtual screen displayed through the display (250) of the wearable device (101). The wearable device (101) may generate hand tracking data (or dummy data) that differs from the movement of the user (110)'s hand (801) based on the identification that the ATM (802) corresponds to the object type that restricts the display of the hand object. For example, the wearable device (101) may identify, based on hand tracking, that the user (110) presses the first number (e.g., 1) of a password using the hand (801). Based on the identification, the wearable device (101) may display a hand object (832) pressing a different number (e.g., 2) that differs from the first number on the virtual screen (831). For example, the wearable device (101) can identify that the user (110) is pressing the second number (e.g., 5) of the password using the hand (801) based on hand tracking.Based on the identification, the wearable device (101) may display a hand object (834) pressing a different number (e.g., 1) from the second number on a virtual screen (833). For example, the wearable device (101) may identify, based on hand tracking, that the user (110) presses the third number (e.g., 2) of the password using the hand (801). Based on the identification, the wearable device (101) may display a hand object (836) pressing a different number (e.g., 4) from the third number on a virtual screen (835). For example, the wearable device (101) may identify, based on hand tracking, that the user (110) presses the fourth number (e.g., 3) of the password using the hand (801). The wearable device (101) can display a hand object (838) pressing a different number (e.g., 1) that is different from the fourth number on a virtual screen (837) based on the identification. Since hand objects pressing a password different from the password entered by the user's (110) hand (801) are displayed on the virtual screen of the wearable device (101), leakage of the password displayed according to the user's (110) hand movements can be prevented.

[0173] FIG. 9 is a flowchart illustrating the operation of a wearable device for restricting the interaction of a user's hand object within a virtual screen according to an embodiment of the present disclosure.

[0174] The operations of FIG. 9 may be performed by the electronic device (101) of FIG. 1 or the wearable device (101) of FIG. 2a through FIG. 4. For example, at least some of the operations may be controlled by the processor (410) of the wearable device (101). In the following embodiments of the disclosure, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed. For example, at least two operations may be performed in parallel.

[0175] Referring to FIG. 9, in operation 901, a wearable device (101) according to one embodiment can identify an object included in a virtual screen displayed through a display (250).

[0176] In an embodiment of the present disclosure, a wearable device (101) may acquire an image of the surrounding environment using a camera (260) and / or a sensor (420). The image may include an image of an object outside the wearable device (101) (or, surrounding object, external object). The wearable device (101) may perform object recognition based on the image acquired using the camera (260) and / or the sensor (420). For example, object recognition may be performed based on identifying features of an object included in the image (e.g., boundary, corner, edge).

[0177] In an embodiment of the present disclosure, the wearable device (101) may perform object tracking for an identified object. For example, the wearable device (101) may generate object tracking data based on the object tracking. For example, the object tracking data may include position data of the object, angle data of the object, type data of the object, or a combination thereof.

[0178] In operation 902, a wearable device (101) according to one embodiment can identify whether an object included in a virtual screen corresponds to a type for restricting the interaction of the user (110)'s hand object.

[0179] In an embodiment of the present disclosure, the wearable device (101) can identify the type of object included in a virtual screen displayed through a display (250) based on object type data. For example, the object type may mean a criterion for classifying objects based on a category. In one example, the object type may correspond to one of the object types: an automated teller machine (ATM), a door lock, a keyboard, or a mouse. However, this is merely an example and the present disclosure is not limited thereto. For example, the object type may further include object types belonging to other categories, and the exemplified object types (e.g., keyboard and mouse) may be classified into one object type (e.g., input means) based on a higher category. For example, the object type may be replaced with object classification, object category, object kind, object group, object class, or other terms having an equivalent technical meaning.

[0180] In an embodiment of the present disclosure, the wearable device (101) may generate attribute data of an object based on an object type. For example, the attribute data may indicate one of the attributes associated with an interaction between the object and the user's (110) hand object. In one example, the attribute may be specified for the object type. In one example, the attribute may be identified based on an artificial intelligence model of the wearable device (101) that uses the object type as input data. For example, the attributes may include a first attribute, a second attribute, and a third attribute. However, the present disclosure is not limited thereto. For example, the attributes may include only some of the first attribute, the second attribute, and the third attribute.

[0181] In an embodiment of the present disclosure, the first attribute may indicate that interaction between the user (110)'s hand object and the object is prohibited. In one example, the object type having the first attribute may include an ATM and a door lock. However, the present disclosure is not limited thereto. The object type having the first attribute may include an object type in which the user's (110) personal information (or security information, password) may be leaked when hand tracking data and / or object tracking data are provided to an application, a system, and / or an external electronic device. To prevent the leakage of the user's (110) personal information, the object type having the first attribute may restrict the display of the user's (110) hand object within a virtual screen displayed through the display (250) of the wearable device (101).

[0182] In an embodiment of the present disclosure, the second attribute may indicate that interaction between the user (110)'s hand object and the object is restricted. In one example, the object type having the second attribute may include a keyboard and a mouse. However, the present disclosure is not limited thereto. The object type having the second attribute may include an object type capable of interacting with the user (110)'s hand (e.g., a real hand). For example, while the user (110) controls an input means (e.g., a real keyboard and / or a real mouse) using their hand, an unintended action (e.g., control of a virtual keyboard and / or a virtual mouse) may occur by the hand object displayed on the virtual screen. To prevent unintended actions by the user (110), the object type having the second attribute may restrict interaction with the user (110)'s hand object within the virtual screen displayed through the display (250) of the wearable device (101).

[0183] In an embodiment of the present disclosure, the third attribute may indicate that interaction between the user (110)'s hand object and the object is not restricted. An object type having the third attribute may be able to interact with the user (110)'s hand object within a virtual screen displayed through the display (250) of the wearable device (101).

[0184] In operation 903, if operation 902-yes (e.g., when the object is of a type for restricting the interaction of the hand object), a wearable device (101) according to one embodiment can identify whether the hand object is located within a threshold distance (e.g., 10 centimeters) from the object within a virtual screen. For example, the wearable device (101) can identify whether the hand object is located within a threshold distance from the object within a virtual screen based on the identification that the object included within the virtual screen corresponds to a type for restricting the interaction of the hand object (e.g., an object type having a second attribute).

[0185] In operation 904, if operation 903-yes (e.g., when a hand object is located within a threshold distance from an object), the wearable device (101) according to one embodiment may refrain from controlling a virtual object using a hand object displayed through a display (250). For example, the wearable device (101) may refrain from controlling a virtual object using a hand object displayed through a display (250) upon identification that a hand object is located within a threshold distance from an object within a virtual screen. By refraining from controlling a virtual object using a hand object, unintended actions by the user (110) can be prevented.

[0186] In operation 905, if operation 902-No (e.g., the object is not of a type for restricting interaction with the hand object) or operation 903-No (e.g., the hand object is located outside a threshold distance from the object), the wearable device (101) according to one embodiment can control a virtual object using a hand object displayed through the display (250). For example, the wearable device (101) can control a virtual object using a hand object displayed through the display (250) based on the identification that an object included in the virtual screen does not correspond to a type for restricting interaction with the user (110)'s hand object. In another example, the wearable device (101) can control a virtual object using a hand object displayed in the virtual screen based on the identification that a hand object is located outside a threshold distance from the object within the virtual screen.

[0187] FIG. 10 illustrates the limitation of interaction by a user's hand object within a virtual screen according to an embodiment of the present disclosure.

[0188] FIG. 10 explains that while a user (110) controls a real keyboard (1001) using a hand (1003), a virtual keyboard (1002) is not controlled by a hand object (1004) displayed on a virtual screen.

[0189] Referring to FIG. 10, a user (110) can control a real keyboard (1001) through the movement of a hand (1003). A wearable device (101) can display images of the real keyboard (1001) and a virtual keyboard (1002) obtained using a camera (260) and / or a sensor (420) within a virtual screen. The wearable device (101) can identify attributes specified for the real keyboard (1001). Attributes specified for the object type of the keyboard may indicate that interaction by the hand object (1004) displayed on the virtual screen is restricted. The wearable device (101) can identify whether the hand object (1004) is located within a threshold distance (e.g., 10 centimeters) from the actual keyboard (1001), based on the identification that the actual keyboard (1001) corresponds to an object type for restricting interaction with the hand object. For example, the wearable device (101) can refrain from controlling the virtual keyboard (1002) using the hand object (1004) based on the identification that the hand object (1004) is located within a threshold distance from the actual keyboard (1001). In another example, the wearable device (101) can control the virtual keyboard (1002) using the hand object (1004) based on the identification that the hand object (1004) is located outside a threshold distance from the actual keyboard (1001).

[0190] FIG. 11 is a flowchart illustrating the operation of a wearable device for processing a hand object within a virtual screen according to the type of object within the virtual screen according to an embodiment of the present disclosure.

[0191] The operations of FIG. 11 may be performed by the electronic device (101) of FIG. 1 or the wearable device (101) of FIG. 2a through FIG. 4. For example, at least some of the operations may be controlled by a processor (410) of the wearable device (101). In the following embodiments of the disclosure, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed. For example, at least two operations may be performed in parallel.

[0192] Referring to FIG. 11, in operation 1101, a wearable device (101) according to one embodiment can identify an object included in a virtual screen displayed through a display (250).

[0193] In an embodiment of the present disclosure, a wearable device (101) may acquire an image of the surrounding environment using a camera (260) and / or a sensor (420). The image may include an image of an object outside the wearable device (101) (or, surrounding object, external object). The wearable device (101) may perform object recognition based on the image acquired using the camera (260) and / or the sensor (420). For example, object recognition may be performed based on identifying features of an object included in the image (e.g., boundary, corner, edge).

[0194] In an embodiment of the present disclosure, the wearable device (101) may perform object tracking for an identified object. For example, the wearable device (101) may generate object tracking data based on the object tracking. For example, the object tracking data may include position data of the object, angle data of the object, type data of the object, or a combination thereof.

[0195] In operation 1102, a wearable device (101) according to one embodiment can identify whether the type of an object corresponds to the first attribute.

[0196] In an embodiment of the present disclosure, the wearable device (101) can identify the type of object included in a virtual screen displayed through a display (250) based on object type data. For example, the object type may mean a criterion for classifying objects based on a category. In one example, the object type may correspond to one of the object types: an automated teller machine (ATM), a door lock, a keyboard, or a mouse. However, this is merely an example and the present disclosure is not limited thereto. For example, the object type may further include object types belonging to other categories, and the exemplified object types (e.g., keyboard and mouse) may be classified into one object type (e.g., input means) based on a higher category. For example, the object type may be replaced with object classification, object category, object kind, object group, object class, or other terms having an equivalent technical meaning.

[0197] In an embodiment of the present disclosure, the wearable device (101) may generate attribute data of an object based on an object type. For example, the attribute data may indicate one of the attributes associated with an interaction between the object and the user's (110) hand object. In one example, the attribute may be specified for the object type. In one example, the attribute may be identified based on an artificial intelligence model of the wearable device (101) that uses the object type as input data. For example, the attributes may include a first attribute, a second attribute, and a third attribute. However, the present disclosure is not limited thereto. For example, the attributes may include only some of the first attribute, the second attribute, and the third attribute.

[0198] In an embodiment of the present disclosure, the first attribute may indicate that interaction between the user (110)'s hand object and the object is prohibited. In one example, the object type having the first attribute may include an ATM and a door lock. However, the present disclosure is not limited thereto. The object type having the first attribute may include an object type in which the user's (110) personal information (or security information, password) may be leaked when hand tracking data and / or object tracking data according to hand tracking are provided to an application, a system, and / or an external electronic device. To prevent the leakage of the user's (110) personal information, the object type having the first attribute may restrict the display (or representation) of the user's (110) hand object within a virtual screen displayed through the display (250) of the wearable device (101).

[0199] In an embodiment of the present disclosure, the second attribute may indicate that interaction between the user (110)'s hand object and the object is restricted. In one example, the object type having the second attribute may include a keyboard and a mouse. However, the present disclosure is not limited thereto. The object type having the second attribute may include an object type capable of interacting with the user (110)'s hand (e.g., a real hand). For example, while the user (110) controls an input means (e.g., a real keyboard and / or a real mouse) using the hand, an unintended action (e.g., control of a virtual keyboard and / or a virtual mouse) may occur by the hand object displayed through the display (250) of the wearable device (101). To prevent unintended action by the user (110), the object type having the second attribute may restrict interaction with the user (110)'s hand object within a virtual screen displayed through the display (250) of the wearable device (101).

[0200] In an embodiment of the present disclosure, the third attribute may indicate that interaction between the user (110)'s hand object and the object is not restricted. An object type having the third attribute may be able to interact with the user (110)'s hand object within a virtual screen displayed through the display (250) of the wearable device (101).

[0201] In operation 1103, if operation 1102-yes (e.g., when the type of the object is the first attribute), the wearable device (101) according to one embodiment may display a virtual screen through the display (250) that does not display the movement of the hand object according to hand tracking.

[0202] In an embodiment of the present disclosure, the wearable device (101) can identify whether a hand object is located within a first threshold distance (e.g., 1 meter) from an object having a first attribute, based on the identification that the type of object corresponds to a first attribute. For example, the wearable device (101) can display a virtual screen through a display (250) that does not display the movement of the hand object according to hand tracking, based on the identification that the hand object is located within the first threshold distance from an object having a first attribute. In another example, the wearable device (101) can display a virtual screen through a display (250) that displays the movement of the hand object according to hand tracking, based on the identification that the hand object is located outside the first threshold distance from an object having a first attribute.

[0203] In an embodiment of the present disclosure, the wearable device (101) may not perform hand tracking in order not to display the movement of a hand object according to hand tracking. Since hand tracking is not performed by the wearable device (101), hand tracking data may not be generated. Since hand tracking data is not generated, the hand object of the user (110) may not be displayed through the display (250) of the wearable device (101). Since the hand object is not displayed through the display (250) of the wearable device (101), the leakage of the user's (110) personal information displayed according to the hand movement of the user (110) may be prevented.

[0204] In an embodiment of the present disclosure, a wearable device (101) may generate hand tracking data based on hand tracking. The hand tracking data may include hand position data, hand angle data, hand joint position data, hand joint angle data, hand gesture data, hand shape data, or a combination thereof. However, the present disclosure is not limited thereto. For example, the hand tracking data may include only some of the data described above.

[0205] In an embodiment of the present disclosure, the wearable device (101) may refrain from providing hand tracking data generated according to hand tracking to an application, a system, and / or an external electronic device so as not to display the movement of a hand object according to hand tracking. For example, the wearable device (101) may identify, based on the attribute data of the object, that the display of a hand object located within a threshold distance from the object is restricted. The wearable device (101) may refrain from providing hand tracking data to an application, a system, and / or an external electronic device based on the identification that the display of a hand object within a virtual screen is restricted. Since the hand tracking data of the user (110) is not provided to an application, a system, and / or an external electronic device, the hand object may not be displayed through the display (250) of the wearable device (101). Since hand tracking data is not provided to applications, systems, and / or external electronic devices, the user's (110) personal information displayed according to the user's (110) hand movements can be prevented from being leaked.

[0206] In an embodiment of the present disclosure, the wearable device (101) may display a hand object having a default gesture through the display (250) so as not to display the movement of the hand object according to hand tracking. For example, the wearable device (101) may identify, based on the attribute data of the object, that the display of a hand object located within a threshold distance from the object is restricted. The wearable device (101) may display a hand object having a default gesture through the display (250) in accordance with the identification that the display of the hand object is restricted within the virtual screen. For example, the wearable device (101) may not use hand angle data, hand joint position data, hand joint angle data, hand gesture data, and hand shape data for hand tracking. Since the data is not used for hand tracking, the hand object displayed through the display (250) of the wearable device (101) may have a default gesture. In one example, the basic gesture may correspond to a fist gesture. However, this is merely an example, and the present disclosure is not limited thereto. For example, the basic gesture may mean any gesture having a fixed hand posture (or shape). Since the hand object having the basic gesture does not display the hand movements of the user (110), the leakage of the user's (110) personal information displayed according to the hand movements of the user (110) can be prevented.

[0207] In an embodiment of the present disclosure, the wearable device (101) may generate hand tracking data different from the movement of the user's (110) hand in order not to display the movement of the hand object according to hand tracking. For example, hand tracking data different from the movement of the user's (110) hand may be referred to as dummy data or other terms having an equivalent technical meaning. For example, the wearable device (101) may identify, based on the attribute data of the object, that the display of the hand object located within a threshold distance from the object is restricted. The wearable device (101) may generate different data different from the hand tracking data according to hand tracking in accordance with the identification that the display of the hand object within the virtual screen is restricted. For example, the wearable device (101) may generate different data different from the hand position data, hand angle data, hand joint position data, hand joint angle data, hand gesture data, and hand shape data generated according to hand tracking. In one example, since different data is generated that differs from the hand position data generated according to hand tracking, the position of the hand object displayed through the display (250) of the wearable device (101) may differ from the position of the user's (110) hand. In one example, since different data is generated that differs from the hand angle data, hand joint position data, hand joint angle data, hand gesture data, and hand shape data generated according to hand tracking, the posture (or shape) of the hand object displayed through the display (250) of the wearable device (101) may differ from the posture (or shape) of the user's (110) hand. Since the position and / or posture of the user's (110) hand and the position and / or posture of the hand object displayed by the data generated by the wearable device (101) are different, the leakage of the user's (110) personal information displayed according to the user's (110) hand movements can be prevented.

[0208] In operation 1104, if operation 1102 is no (e.g., the type of the object is not the first attribute), the wearable device (101) according to one embodiment can identify whether the type of the object corresponds to the second attribute. For example, the wearable device (101) can identify whether the type of the object corresponds to the second attribute based on the identification that the type of the object does not correspond to the first attribute.

[0209] In operation 1105, if operation 1104-yes (e.g., when the type of the object is a second attribute), the wearable device (101) according to one embodiment may refrain from controlling the virtual object using the hand object displayed on the virtual screen. For example, the wearable device (101) may identify whether the hand object is located within a second threshold distance (e.g., 10 centimeters) from the object having the second attribute, based on the identification that the type of the object corresponds to the second attribute. For example, the wearable device (101) may refrain from controlling the virtual object using the hand object displayed on the virtual screen based on the identification that the hand object is located within the second threshold distance from the object having the second attribute. In another example, the wearable device (101) may control the virtual object using the hand object displayed on the virtual screen based on the identification that the hand object is located outside the second threshold distance from the object having the second attribute.

[0210] In operation 1106, if operation 1104-No (e.g., the type of the object is not the second attribute), the wearable device (101) according to one embodiment can control a virtual object using a hand object displayed on a virtual screen. For example, the wearable device (101) can control a virtual object using a hand object displayed on a virtual screen based on the identification that the type of the object does not correspond to the second attribute.

[0211] FIG. 12 is a flowchart illustrating the operation of a wearable device for processing a hand object within a virtual screen according to the type of object within the virtual screen according to an embodiment of the present disclosure.

[0212] The operations of FIG. 12 may be performed by the electronic device (101) of FIG. 1 or the wearable device (101) of FIG. 2a through FIG. 4. For example, at least some of the operations may be controlled by a processor (410) of the wearable device (101). In the following embodiments of the disclosure, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed. For example, at least two operations may be performed in parallel.

[0213] Referring to FIG. 12, in operation 1201, a wearable device (101) according to one embodiment can acquire object information data.

[0214] In an embodiment of the present disclosure, a wearable device (101) may acquire an image of the surrounding environment using a camera (260) and / or a sensor (420). The image may include an image of an object outside the wearable device (101) (or, surrounding object, external object). The wearable device (101) may perform object recognition based on the image acquired using the camera (260) and / or the sensor (420). For example, object recognition may be performed based on identifying features of an object included in the image (e.g., boundary, corner, edge).

[0215] In an embodiment of the present disclosure, a wearable device (101) may acquire object information data based on object recognition. For example, the object information data may include at least one of location data, type (or kind) data, or attribute data of an object identified based on object recognition. For example, the location data of an object may indicate the distance between the object and the user (110), whether the object (or wearable device (101)) is located at a designated place, and / or whether the object exists within a spatially designated tracking range by the user (110). For example, the type (or kind) data may indicate the type of an object. The object type may refer to a criterion for classifying objects based on a category. In one example, the object type may correspond to one of an automated teller machine (ATM), a door lock, a keyboard, or a mouse. However, this is merely an example and the present disclosure is not limited thereto. For example, attribute data may represent one of the attributes associated with the interaction between the object and the user's (110) hand object. Attribute data may correspond to one of the first attribute data, second attribute data, or third attribute data. The first attribute data may represent a first attribute in which the subject or scope of tracking is not restricted, or the object information data and tracking data are processed without restriction by the processor (410). The second attribute data may represent a second attribute in which the subject or scope of tracking is partially restricted, or at least one of the object information data or tracking data is processed with partial restriction by the processor (410).The third attribute data may represent a third attribute in which the target or scope of tracking is entirely restricted, or at least one of the object information data or tracking data is processed with entire restriction by the processor (410).

[0216] In operation 1202, a wearable device (101) according to one embodiment can acquire tracking data. For example, the wearable device (101) can acquire tracking data for a body part of a user (110) by performing hand tracking. The tracking data may include location data of a body part of the user (110).

[0217] In operation 1203, a wearable device (101) according to one embodiment may determine a target or range for tracking a body part based on identification that a specified condition is met.

[0218] In an embodiment of the present disclosure, the specified conditions may include at least one of whether an object identified based on object recognition corresponds to a specified object type, the distance between the object and a body part of the user (110), whether the object (or wearable device (101)) is located at a specified location, whether the current time is within a specified time range, whether the object is located within a spatially specified tracking range by the user (110), or whether the current situation corresponds to a situation specified by an artificial intelligence model. For example, the wearable device (101) may identify that the specified conditions are satisfied based on the identification that an object identified based on object recognition corresponds to a specified object type (e.g., ATM). For example, the wearable device (101) may identify that the specified conditions are satisfied based on the identification that the distance between the object and a body part of the user (110) is within a threshold distance. For example, the wearable device (101) can identify that a specified condition is satisfied based on the identification that the current time is within a specified time range. For example, the wearable device (101) can identify that a specified condition is satisfied based on the identification that an object is located within a spatially specified tracking range by the user (110). For example, the wearable device (101) can identify that a specified condition is satisfied based on the identification that the current situation corresponds to a situation specified by an artificial intelligence model.

[0219] In an embodiment of the present disclosure, the wearable device (101) may determine the target or scope of tracking based on the identification that a specified condition is met. For example, the target of tracking may be a body part (e.g., a hand) of the user (110). For example, the wearable device (101) may refrain from tracking the body part of the user (110) based on the identification that a specified condition is met. For example, the scope of tracking may be associated with the processing of tracking data obtained through tracking the body part of the user (110). For example, the wearable device (101) may refrain from transmitting tracking data regarding the body part of the user (110) to an application based on the identification that a specified condition is met. In another example, the wearable device (101) may restrict the interaction between the body part of the user (110) and an object based on the identification that a specified condition is met. For example, tracking data regarding a body part of the user (110) can be processed in a secure area. In one example, the secure area may be a separate area configured for security within the processor (410). In another example, the secure area may be a separate area configured for security outside the processor (410) (e.g., an embedded secure element (eSE), a secure processor). In yet another example, the secure area may be implemented based on software (e.g., a hypervisor).

[0220] In an embodiment of the present disclosure, the wearable device (101) may display a part of the user's (110) body in a virtual screen through a display (250) when a specified condition is met for an object having a first attribute. The part of the user's (110) body displayed in the virtual screen may be able to interact with the object.

[0221] In an embodiment of the present disclosure, the wearable device (101) may display a part of the user's (110) body within a virtual screen through a display (250) when a specified condition is met for an object having a second attribute. The part of the user's (110) body displayed within the virtual screen may be restricted from interacting with the object.

[0222] In an embodiment of the present disclosure, the wearable device (101) may not perform hand tracking in order not to display the movement of the hand object according to hand tracking when a specified condition is met for an object having a third attribute. Since hand tracking is not performed by the wearable device (101), hand tracking data may not be generated. Since hand tracking data is not generated, the hand object of the user (110) may not be displayed through the display (250) of the wearable device (101). Since the hand object is not displayed through the display (250) of the wearable device (101), the leakage of the user's (110) personal information displayed according to the hand movement of the user (110) may be prevented.

[0223] In an embodiment of the present disclosure, the wearable device (101) may refrain from providing hand tracking data generated according to hand tracking to an application, a system, and / or an external electronic device so as not to display the movement of the hand object according to hand tracking when a specified condition is met for an object having a third attribute. Since the hand tracking data of the user (110) is not provided to the application, a system, and / or an external electronic device, the hand object may not be displayed through the display (250) of the wearable device (101). Since the hand tracking data is not provided to the application, a system, and / or an external electronic device, the leakage of the user's (110) personal information displayed according to the user's (110) hand movement may be prevented.

[0224] In an embodiment of the present disclosure, the wearable device (101) may display a hand object having a default gesture through the display (250) so as not to display the movement of the hand object according to hand tracking when a specified condition is met for an object having a third attribute. For example, the wearable device (101) may not use hand angle data, hand joint position data, hand joint angle data, hand gesture data, and hand shape data for hand tracking. Since the data is not used for hand tracking, the hand object displayed through the display (250) of the wearable device (101) may have a default gesture. In one example, the default gesture may correspond to a fist gesture. However, this is merely an example and the present disclosure is not limited thereto. For example, the default gesture may mean any gesture having a fixed hand posture (or shape). Since the hand object with the basic gesture does not display the user's (110) hand movements, the user's (110) personal information displayed according to the user's (110) hand movements can be prevented from being leaked.

[0225] In an embodiment of the present disclosure, the wearable device (101) may generate hand tracking data different from the movement of the user's (110) hand in order not to display the movement of the hand object according to hand tracking when a specified condition is met for an object having a third attribute. For example, hand tracking data different from the movement of the user's (110) hand may be referred to as dummy data or other terms having an equivalent technical meaning. For example, the wearable device (101) may generate data different from the hand position data, hand angle data, hand joint position data, hand joint angle data, hand gesture data, and hand shape data generated according to hand tracking. In one example, since data different from the hand position data generated according to hand tracking is generated, the position of the hand object displayed through the display (250) of the wearable device (101) may be different from the position of the user's (110) hand. In one example, since different data is generated based on hand tracking, such as hand angle data, hand joint position data, hand joint angle data, hand gesture data, and hand shape data, the posture (or shape) of the hand object displayed through the display (250) of the wearable device (101) may differ from the posture (or shape) of the user's (110) hand. Since the position and / or posture of the user's (110) hand and the position and / or posture of the hand object displayed by the data generated by the wearable device (101) are different, the leakage of the user's (110) personal information displayed based on the user's (110) hand movements can be prevented.

[0226] A wearable device (101) (or electronic device) according to the present disclosure can prevent the leakage of a user's (110) personal information (e.g., password) by restricting the display (or representation) of a hand object displayed on a virtual screen in the vicinity of an object requiring security. A wearable device (101) (or electronic device) according to the present disclosure can prevent unintended actions by the user (110) by refraining from controlling a virtual object using a hand object displayed on a virtual screen in the vicinity of an object that serves as an input means.

[0227] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0228] The wearable device (101) described above may include at least one camera (260). The wearable device (101) may include a display (250). The wearable device (101) may include a memory (415) that stores instructions and includes one or more storage media. The wearable device (101) may include at least one processor (410) that includes a processing circuit. When the instructions are executed individually or collectively by the at least one processor (410), the wearable device (101) may cause the wearable device (101) to identify an object included in a virtual screen displayed through the display (250) using the at least one camera (260). When the above instructions are executed individually or collectively by the at least one processor (410), the wearable device (101) may cause the wearable device (101) to determine whether the identified object corresponds to a type for restricting the representation of the user's hand object. When the above instructions are executed individually or collectively by the at least one processor (410), the wearable device (101) may cause the wearable device (101) to display a virtual screen containing the hand object through the display (250) based on hand tracking, upon the determination that the identified object does not correspond to the type. When the above instructions are executed individually or collectively by the at least one processor (410), the wearable device (101) may cause the display (250) to display a virtual screen that does not display the movement of the hand object according to the hand tracking, based on identifying that the hand object is located within a threshold distance from the object according to the determination that the identified object corresponds to the type.

[0229] For example, when the instructions are executed individually or collectively by the at least one processor, the wearable device may be caused to identify the location of the hand object based on other data different from hand tracking data for the user's hand generated based on the hand tracking, according to a determination that the identified object corresponds to the type. When the instructions are executed individually or collectively by the at least one processor, the wearable device may be caused to display the hand object through a display at the identified location.

[0230] For example, when the above instructions are executed individually or collectively by the at least one processor, the wearable device may cause the hand object having a default gesture to be displayed on the display according to the determination that the identified object corresponds to the type.

[0231] For example, when the above instructions are executed individually or collectively by the at least one processor, the wearable device may be prevented from providing hand tracking data according to the hand tracking to an application executed by the wearable device, based on a determination that the identified object corresponds to the type.

[0232] For example, when the above instructions are executed individually or collectively by the at least one processor, the wearable device may be caused to refrain from generating hand tracking data based on the hand tracking, depending on the determination that the identified object corresponds to the type.

[0233] For example, when the above instructions are executed individually or collectively by the at least one processor, the wearable device may be caused to use an artificial intelligence model to determine the type for restricting the representation of the hand object and the first type for restricting the interaction of the hand object.

[0234] For example, when the instructions are executed individually or collectively by the at least one processor, the wearable device may be caused to determine whether the identified object corresponds to a first type for restricting the interaction of the hand object. When the instructions are executed individually or collectively by the at least one processor, the wearable device may be caused to generate hand tracking data indicating the restriction of interaction by the hand object based on the determination that the identified object corresponds to the first type. When the instructions are executed individually or collectively by the at least one processor, the wearable device may be caused to refrain from controlling a virtual object using the hand object indicated by the hand tracking data based on identifying that the hand object is located within a first threshold distance from the identified object.

[0235] A method performed by a wearable device (101) comprising at least one camera (260) and a display (250) as described above may include an action of identifying an object included in a virtual screen displayed through the display (250) using the at least one camera (260). The method may include an action of determining whether the identified object corresponds to a type for restricting the representation of a user's hand object. The method may include an action of displaying a virtual screen containing the hand object through the display based on hand tracking, in accordance with the determination that the identified object does not correspond to the type. The method may include an action of displaying a virtual screen through the display that does not display the movement of the hand object according to the hand tracking, in accordance with the determination that the identified object corresponds to the type, in accordance with identifying that the hand object is located within a threshold distance from the object.

[0236] For example, the method may include an operation of identifying the location of the hand object based on other data different from hand tracking data for the user's hand generated based on the hand tracking, according to a determination that the identified object corresponds to the type. The method may include an operation of displaying the hand object through a display at the identified location.

[0237] For example, the above method may include the action of displaying the hand object having a default gesture through a display, based on the determination that the identified object corresponds to the type.

[0238] For example, the above method may include an action of refraining from providing hand tracking data according to the hand tracking to an application executed by the wearable device, based on a determination that the identified object corresponds to the type.

[0239] For example, the above method may include an action of refraining from generating hand tracking data based on the hand tracking, depending on the determination that the identified object corresponds to the type.

[0240] For example, the above method may include an operation to determine a type for restricting the representation of the hand object and a first type for restricting the interaction of the hand object using an artificial intelligence model.

[0241] For example, the method may include an operation to determine whether the identified object corresponds to a first type for restricting the interaction of the hand object. The method may include an operation to generate hand tracking data indicating the restriction of interaction by the hand object based on the determination that the identified object corresponds to the first type. The method may include an operation to refrain from controlling a virtual object using the hand object indicated by the hand tracking data based on identifying that the hand object is located within a first threshold distance from the identified object.

[0242] A non-transient computer-readable storage medium as described above may store one or more programs. The one or more programs may include instructions that cause the wearable device (101) to identify an object contained within a virtual screen displayed through the display (250) using the at least one camera (260) when executed by at least one processor (410) of the wearable device (101) which includes a display (250) and at least one camera (260). The one or more programs may include instructions that cause the wearable device (101) to determine whether the identified object corresponds to a type for restricting the representation of the user's hand object when executed by at least one processor (410) of the wearable device (101). The above one or more programs may include instructions that cause the wearable device (101) to display a virtual screen containing the hand object based on hand tracking through the display (250), based on a determination that the identified object does not correspond to the type when executed by at least one processor (410) of the wearable device (101). The above one or more programs may include instructions that cause the wearable device (101) to display a virtual screen that does not display the movement of the hand object according to the hand tracking through the display (250), based on identifying that the hand object is located within a threshold distance from the object, based on a determination that the identified object corresponds to the type when executed by at least one processor (410) of the wearable device (101).

[0243] For example, the one or more programs may include instructions that cause the wearable device to identify the location of the hand object based on other data different from hand tracking data for the user's hand generated based on the hand tracking, according to a determination that the identified object corresponds to the type when executed by at least one processor of the wearable device. The one or more programs may include instructions that cause the wearable device to display the hand object through a display at the identified location when executed by at least one processor of the wearable device.

[0244] For example, the one or more programs may include instructions that cause the wearable device to display the hand object having a default gesture on the display, depending on the determination that the identified object corresponds to the type when executed by at least one processor of the wearable device.

[0245] For example, the above one or more programs may include instructions that cause the wearable device to refrain from providing hand tracking data according to the hand tracking to an application executed by the wearable device, depending on the determination that the identified object corresponds to the type when executed by at least one processor of the wearable device.

[0246] For example, the one or more programs may include instructions that cause the wearable device to refrain from generating hand tracking data based on hand tracking, upon a determination that the identified object corresponds to the type when executed by at least one processor of the wearable device.

[0247] For example, the above one or more programs may include instructions that cause the wearable device to use an artificial intelligence model to determine the type for restricting the representation of the hand object and the first type for restricting the interaction of the hand object when executed by at least one processor of the wearable device.

[0248] For example, the one or more programs may include instructions that cause the wearable device to determine whether the identified object corresponds to a first type for restricting interaction with the hand object when executed by at least one processor of the wearable device. The one or more programs may include instructions that cause the wearable device to generate hand tracking data indicating restriction of interaction by the hand object when executed by at least one processor of the wearable device, based on the determination that the identified object corresponds to the first type. The one or more programs may include instructions that cause the wearable device to refrain from controlling a virtual object using the hand object indicated by the hand tracking data, based on identifying that the hand object is located within a first threshold distance from the identified object when executed by at least one processor of the wearable device.

[0249] A wearable device as described above may include at least one display. The wearable device may include at least one sensor. The wearable device may include at least one processor including a processing circuit. The wearable device may include a memory that stores instructions and includes one or more storage media. When the instructions are executed individually or collectively by the at least one processor, the wearable device may cause the wearable device to acquire object information data regarding an object identified by the at least one sensor. When the instructions are executed individually or collectively by the at least one processor, the wearable device may cause the wearable device to acquire tracking data by tracking a part of the body identified by the at least one sensor. When the instructions are executed individually or collectively by the at least one processor, the wearable device may identify whether a specified condition is satisfied. When the above instructions are executed individually or collectively by the at least one processor, the wearable device may cause the wearable device to determine the target or range of tracking based on the identification, or to determine the target or range of the tracking data processed by the at least one processor.

[0250] For example, the object information data may include at least one of location data, type data, or attribute data for each object identified by the at least one sensor. The specified condition may include at least one of whether the object corresponds to a specified type, the distance between the object and the part of the body, whether the object or the wearable device is located at a specified place, whether the current time is within a specified time range, whether the object exists within a spatially specified tracking range by the part of the body, or whether it corresponds to a situation specified by an artificial intelligence model.

[0251] For example, the attribute data may include at least two of the following: first attribute data associated with a first attribute in which the target or scope of the tracking is not limited or the object information data and the tracking data are processed without limitation by the at least one processor; second attribute data associated with a second attribute in which, when at least one of the specified conditions is satisfied, the target or scope of the tracking is partially limited or at least one of the object information data and the tracking data is processed with partial limitation by the at least one processor; or third attribute data associated with a third attribute in which, when at least one of the specified conditions is satisfied, the target or scope of the tracking is fully limited or at least one of the object information data or the tracking data is processed with full limitation by the at least one processor.

[0252] For example, the at least one processor may include a virtual space manager configured to receive sensor data from the at least one sensor and generate a visual image to support a virtual space service based on the sensor data. The at least one application stored in the memory may cause the at least one display to display a screen displaying at least a part of a virtual space or a screen displaying a two-dimensional image based on at least one of the sensor data, the object information data, or the tracking data.

[0253] For example, when the above instructions are executed individually or collectively by the at least one processor, the wearable device may cause the virtual space manager to refrain from processing the received sensor data or to process the sensor data in a limited manner based on the determined target or range of the tracking data.

[0254] For example, when the above instructions are executed individually or collectively by the at least one processor, the wearable device may cause the virtual space manager to refrain from transmitting at least some of the sensor data, the object information data, or the tracking data to the at least one application based on the determined target or range of the tracking data.

[0255] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0256] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0257] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the embodiments described in the claims or specification of this disclosure. The one or more programs may be provided as a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). ™ It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0258] Such programs (software modules, software) may be stored in random access memory, non-volatile memory including flash memory, read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic disc storage devices, compact disc-ROM (CD-ROM), digital versatile discs (DVDs), or other forms of optical storage devices, magnetic cassettes. Alternatively, they may be stored in memory composed of some or all of these. Additionally, each constituent memory may include multiple units.

[0259] Additionally, the program may be stored on an attachable storage device that can be accessed via a communication network such as the Internet, Intranet, LAN (local area network), WAN (wide area network), or SAN (storage area network), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.

[0260] In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression is selected to suit the situation presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural form, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.

[0261] According to various embodiments of the present disclosure, one or more of the aforementioned components or operations may be omitted, or one or more other components or operations may be added. Generally or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components in the same or similar manner as those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments of the present disclosure, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0262] It will be understood that various embodiments of the present disclosure according to the claims and description of the specification may be realized in the form of hardware, software, or a combination of hardware and software.

[0263] Such software may be stored on a non-transient computer-readable storage medium. A non-transient computer-readable storage medium stores one or more computer programs (software modules), and one or more computer programs include computer-executable instructions that cause said electronic device to perform the method of the present disclosure when executed by one or more processors of an electronic device.

[0264] Such software may be stored in the form of volatile or non-volatile storage (e.g., a storage device such as ROM (read-only memory), regardless of whether it is erasable or rewritable), in the form of memory (e.g., RAM (random access memory), a memory chip, a device, or an integrated circuit), or in an optical or magnetic readable medium (e.g., a CD (compact disc), a DVD (digital versatile disc), a magnetic disc, or a magnetic tape). It will be understood that the storage device and storage medium are various embodiments of non-transient machine-readable storage suitable for storing computer program(s) containing instructions that implement various embodiments of the present disclosure at execution. Accordingly, various embodiments provide a program that stores code for implementing the device or method claimed in any one of the claims of this specification, and a non-transient machine-readable storage that stores such a program.

[0265] Although the present disclosure has been illustrated and described with reference to various embodiments, those skilled in the art will understand that various changes in form and detail are possible without departing from the scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. In a wearable device, At least one camera; display; Memory for storing instructions and including one or more storage media; and It includes a processing circuit and at least one processor that is communicatively connected to the at least one camera, the display, and the memory, and When the above instructions are executed individually or collectively by the at least one processor, the wearable device, Using at least one camera, an object included within a virtual screen displayed through the display is identified, and Determine whether the above-mentioned identified object corresponds to a type for restricting the representation of the user's hand object, and Based on the determination that the identified object does not correspond to the type, a virtual screen including the hand object is displayed through the display based on hand tracking, and Causing to display a virtual screen through the display that does not display the movement of the hand object according to the hand tracking, based on identifying that the hand object is located within a threshold distance from the object according to the determination that the identified object corresponds to the type. Wearable device.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the wearable device, Based on the determination that the above-identified object corresponds to the above type, the location of the hand object is identified based on hand tracking data for the user's hand generated based on the hand tracking and other data different from the hand tracking data generated based on the hand tracking, and At the above identified location, causing the hand object to be displayed through the display, further Wearable device.

3. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the wearable device, In accordance with the determination that the above-identified object corresponds to the above type, causing the hand object having a default gesture to be displayed through the display, Wearable device.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the wearable device, In accordance with the determination that the above-identified object corresponds to the above type, further causing to refrain from providing hand tracking data according to the hand tracking to an application executed by the wearable device, Wearable device.

5. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the wearable device, In accordance with the determination that the above-identified object corresponds to the above type, causing further to refrain from generating hand tracking data based on the hand tracking, Wearable device.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the wearable device, Using an artificial intelligence model to determine the type for restricting the representation of the hand object and the first type for restricting the interaction of the hand object, further causing Wearable device.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the wearable device, Determining whether the identified object corresponds to a first type for restricting the interaction of the hand object, Based on the determination that the identified object corresponds to the first type, hand tracking data indicating a restriction on interaction by the hand object is generated, and Based on identifying that the hand object is located within a first threshold distance from the identified object, further causing to refrain from controlling a virtual object using the hand object displayed according to the hand tracking data. Wearable device.

8. A method performed by a wearable device comprising at least one camera and a display, wherein An operation of identifying an object included within a virtual screen displayed through the display using at least one camera; An operation to determine whether the above-mentioned identified object corresponds to a type for restricting the representation of the user's hand object; An operation of displaying a virtual screen including the hand object through the display based on hand tracking, in accordance with the determination that the identified object does not correspond to the type; and Based on identifying that the hand object is located within a threshold distance from the object according to a determination that the identified object corresponds to the type, the operation of displaying a virtual screen through the display that does not display the movement of the hand object according to the hand tracking is included. method.

9. In Paragraph 8, An operation to identify the location of the hand object based on different data different from hand tracking data for the user's hand generated based on the hand tracking, in accordance with a determination that the identified object corresponds to the type; and Further including the action of displaying the hand object through a display at the identified location, method.

10. In Paragraph 8, A method further comprising the action of displaying the hand object having a default gesture through a display, based on the determination that the identified object corresponds to the type. method.

11. In Paragraph 8, Further including the action of refraining from providing hand tracking data according to the hand tracking to an application executed by the wearable device, based on the determination that the identified object corresponds to the type. method.

12. In Paragraph 8, A further operation comprising refraining from generating hand tracking data based on the hand tracking, in accordance with the determination that the identified object corresponds to the type. method.

13. In Paragraph 8, A method further comprising, using an artificial intelligence model, determining the type for restricting the representation of the hand object and the first type for restricting the interaction of the hand object. method.

14. In Paragraph 8, An operation to determine whether the identified object corresponds to a first type for restricting the interaction of the hand object; The operation of generating hand tracking data indicating a restriction on interaction by the hand object, based on the determination that the identified object corresponds to the first type; and A further operation comprising refraining from controlling a virtual object using the hand object displayed according to the hand tracking data, based on identifying that the hand object is located within a first threshold distance from the identified object. method.

15. A non-transient computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed individually or collectively by at least one processor of a wearable device comprising a display and at least one camera, Using at least one camera, an object included within a virtual screen displayed through the display is identified, and Determine whether the above-mentioned identified object corresponds to a type for restricting the representation of the user's hand object, and Based on the determination that the identified object does not correspond to the type, a virtual screen including the hand object is displayed through the display based on hand tracking, and Based on the determination that the identified object corresponds to the type, and based on identifying that the hand object is located within a threshold distance from the object, a virtual screen that does not display the movement of the hand object according to the hand tracking is displayed through the display. Including instructions that cause the above-mentioned wearable device, Non-transient computer-readable storage media.

Images