Wearable device, method, and non-transitory computer-readable storage medium for recognizing data of electronic device as input

Abstract

This wearable device is provided. The wearable device may comprise: a display assembly including at least one display; a communication circuit; a memory for storing instructions; and at least one processor including processing circuitry. The instructions, when executed individually or collectively by the at least one processor, may cause the wearable device to: transmit, to an electronic device via the communication circuit, mirroring data for a screen provided through the display assembly; while providing the mirroring data to the electronic device to display the screen according to the mirroring data on the electronic device, receive, from the electronic device via the communication circuit, a message including images acquired via a camera of the electronic device; identify a user input according to the movement of an external object in the images on the basis of the images included in the message; and display a visual object corresponding to the user input via the display assembly.

Description

Wearable device, method, and non-transitory computer-readable storage medium for recognizing data from an electronic device as input

[0001] The present disclosure relates to a wearable device, a method, and a non-transitory computer-readable storage medium for recognizing data of an electronic device as input.

[0002] A wearable device may include a display and communication circuitry. The wearable device may be utilized as a tool for implementing virtual reality (VR), augmented reality (AR), and mixed reality (MR). The wearable device may display a three-dimensional (3D) space on the display. The wearable device may stream the 3D space to an electronic device by transmitting data about the 3D space to the electronic device via the communication circuitry.

[0003] The above information is provided solely as background information to aid in understanding the present disclosure. No determination is made, and no claim is made, regarding whether any of the above constitutes prior art relating to the present disclosure.

[0004] Aspects of the present disclosure address at least the problems and / or disadvantages mentioned above and provide at least the advantages described below. Accordingly, one aspect of the present disclosure provides a wearable device, a method, and a non-transitory computer-readable storage medium for recognizing data from an electronic device as input.

[0005] Additional aspects will be described in some of the subsequent descriptions, and some will be apparent from the description or may be understood by practice of the embodiments set forth.

[0006] According to an aspect of the present disclosure, a wearable device is provided. The wearable device may include a display assembly including at least one display, a communication circuit, a memory storing instructions and including one or more storage media, and at least one processor including processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to transmit, through the communication circuitry, mirroring data for a screen representing at least a portion of a three-dimensional space provided through the display assembly to an electronic device; while providing the mirroring data to the electronic device so that a screen according to the mirroring data is displayed on the electronic device, receive, through the communication circuitry, a message including images acquired through a camera of the electronic device from the electronic device; identify, based on the images included in the message, a user input according to a movement of an external object within the images; and display, through the display assembly, a visual object corresponding to the user input.

[0007] According to another aspect of the present disclosure, a method is provided for use in a wearable device including a display assembly including at least one display and a communication circuit. The method may include: transmitting, to an electronic device through the communication circuit, mirroring data for a screen representing at least a portion of a three-dimensional space provided through the display assembly; receiving, from the electronic device through the communication circuit, a message including images acquired through a camera of the electronic device while providing the mirroring data to the electronic device so that the screen according to the mirroring data is displayed on the electronic device; identifying, based on the images included in the message, a user input according to a movement of an external object within the images; and displaying, through the display assembly, a visual object corresponding to the user input.

[0008] According to another aspect of the present disclosure, one or more non-transitory computer-readable storage media are provided storing one or more programs storing one or more computer-executable instructions that, when individually or collectively executed by at least one processor of a wearable device, cause the wearable device to perform operations of the wearable device. The above operations may include an operation of transmitting mirroring data for a screen representing at least a part of a three-dimensional space provided through a display assembly of the wearable device to an electronic device through a communication circuit of the wearable device, an operation of receiving a message including images acquired through a camera of the electronic device from the electronic device through the communication circuit while providing the mirroring data to the electronic device so that the electronic device displays a screen according to the mirroring data when executed by the wearable device, an operation of identifying a user input according to a movement of an external object within the images based on the images included in the message when executed by the wearable device, and an operation of displaying a visual object corresponding to the user input through the display assembly when executed by the wearable device.

[0009] According to another aspect of the present disclosure, an electronic device is provided. The electronic device may include a display, a camera, a communication circuit, a memory storing instructions and including one or more storage media, and at least one processor including processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive mirroring data for a screen displayed on the wearable device from a wearable device through the communication circuitry, display a screen according to the received mirroring data through the display, acquire images including an external object through the camera while the screen according to the mirroring data is displayed on the display of the electronic device, and transmit a message including the images to the wearable device through the communication circuitry. The images included in the message may be used to display a visual object based on the images on the wearable device.

[0010] According to another aspect of the present disclosure, a method performed in an electronic device is provided. The method may include receiving mirroring data for a screen displayed on a wearable device from a wearable device through a communication circuit of the electronic device, displaying a screen according to the received mirroring data through a display of the electronic device, acquiring images including an external object through a camera of the electronic device while the screen according to the mirroring data is displayed on the display of the electronic device, and transmitting a message including the images to the wearable device through the communication circuit. The images included in the message may be used to display a visual object based on the images on the wearable device.

[0011] According to another aspect of the present disclosure, one or more non-transitory computer-readable storage media are provided, storing one or more programs storing one or more computer-executable instructions that, when individually or collectively executed by at least one processor of an electronic device, cause the electronic device to perform operations. The operations may include receiving, from a wearable device, mirroring data for a screen displayed on the wearable device through a communication circuit of the electronic device, displaying a screen according to the received mirroring data through a display of the electronic device, acquiring images including an external object through a camera of the electronic device while the screen according to the mirroring data is displayed on the display of the electronic device, and transmitting, through the communication circuit, a message including the images to the wearable device. The images included in the message may be used to display a visual object based on the images on the wearable device.

[0012] According to another aspect of the present disclosure, a wearable device is provided. The wearable device may include a display assembly including at least one display, a communication circuit, a memory storing instructions and including one or more storage media, and at least one processor including processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to transmit, through the communication circuitry, mirroring data for a screen representing at least a portion of a three-dimensional space provided through the display assembly to an electronic device, and to provide the mirroring data to the electronic device so that a screen according to the mirroring data is displayed on the electronic device, while receiving a message including a control command from the electronic device through the communication circuitry, and to execute a function according to the control command included in the message. The message may indicate that the control command corresponds to a user input acquired through a camera of the electronic device.

[0013] According to another aspect of the present disclosure, a method performed in a wearable device is provided. The method may include: transmitting mirroring data for a screen representing at least a portion of a three-dimensional space provided through a display assembly of the wearable device to an electronic device through a communication circuit of the wearable device; receiving a message including a control command from the electronic device through the communication circuit while providing the mirroring data to the electronic device so that the screen according to the mirroring data is displayed on the electronic device; and executing a function according to the control command included in the message. The message may indicate that the control command corresponds to a user input acquired through a camera of the electronic device.

[0014] According to another aspect of the present disclosure, one or more non-transitory computer-readable storage media are provided, storing one or more programs storing one or more computer-executable instructions that, when individually or collectively executed by at least one processor of a wearable device, cause the wearable device to perform operations. The operations may include transmitting, to an electronic device via a communication circuit of the wearable device, mirroring data for a screen representing at least a portion of a three-dimensional space provided through a display assembly of the wearable device; receiving, from the electronic device via the communication circuit, a message including a control command while providing the mirroring data to the electronic device so that the electronic device displays a screen according to the mirroring data; and executing a function according to the control command included in the message. The message may indicate that the control command corresponds to a user input acquired via a camera of the electronic device.

[0015] The important features, advantages, and other aspects of the present disclosure will become apparent to those skilled in the art from the following detailed description taken in conjunction with the accompanying drawings.

[0016] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0017] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments of the present disclosure;

[0018] FIG. 2A illustrates an example of a perspective view of a wearable device according to one embodiment of the present disclosure;

[0019] FIG. 2b illustrates an example of one or more hardware elements arranged within a wearable device according to one embodiment of the present disclosure;

[0020] FIGS. 3A and 3B illustrate examples of the appearance of a wearable device according to various embodiments of the present disclosure;

[0021] FIG. 4 illustrates an example of a block diagram of a wearable device according to one embodiment of the present disclosure;

[0022] FIG. 5 illustrates an example block diagram of an electronic device for displaying an image in a virtual space according to one embodiment of the present disclosure;

[0023] FIG. 6 illustrates an example of a structure of multiple layers according to one embodiment of the present disclosure;

[0024] FIG. 7 illustrates an example of a guide mode of a wearable device displaying visual objects corresponding to data of an electronic device according to one embodiment of the present disclosure;

[0025] FIG. 8 illustrates an example of a control mode of a wearable device that executes a function corresponding to data of an electronic device according to one embodiment of the present disclosure;

[0026] FIG. 9 illustrates an example of operations performed between a wearable device and an electronic device to display a visual object corresponding to data acquired from an electronic device on a display of the wearable device according to one embodiment of the present disclosure;

[0027] FIG. 10 illustrates an example of capability negotiation performed between a wearable device and an electronic device according to one embodiment of the present disclosure;

[0028] FIG. 11 illustrates an example of a message transmitted from an electronic device according to one embodiment of the present disclosure; and

[0029] FIG. 12 illustrates examples of operations of a wearable device that executes a function according to a control command of an electronic device according to one embodiment of the present disclosure.

[0030] It will be understood that like reference numerals throughout the drawings refer to like parts, components, and structures.

[0031] The following description, with reference to the attached drawings, is provided to assist in a comprehensive understanding of the various embodiments of the disclosure defined by the claims and their equivalents. While it includes numerous specific details to aid understanding, these should be considered merely exemplary. Accordingly, those skilled in the art will recognize that various modifications and variations of the various embodiments described herein can be made without departing from the spirit and scope of the present disclosure. Furthermore, descriptions of well-known functions and structures may be omitted for clarity and brevity.

[0032] The terms and words used in the following description and claims are not to be limited to their dictionary meanings, but have been used by the inventors to ensure a clear and consistent understanding of the present disclosure. Accordingly, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only, and is not intended to limit the present disclosure, which is defined by the appended claims and their equivalents.

[0033] Unless the context clearly dictates otherwise, the singular forms "a," "an," and "the" should be understood to include plural references. For example, the expression "a component surface" implies that there may be one or more such surfaces.

[0034] The terms used in this disclosure are used merely to describe specific embodiments and may not be intended to limit the scope of various embodiments. Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0035] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0036] In the following description, terms referring to data (e.g., data, information, signal, data packet, message, mirroring data, movement data, external data, input data, payload data), terms referring to instructions (e.g., module, service, manager, engine, application, app, function, part, layer, API (application programming interface)), terms referring to values ​​(e.g., threshold, reference information, reference gesture information, designated parameter, value, octet, bit, parameter), terms for operation states (e.g., operation, process), terms referring to network entities, terms referring to components of a device, etc. are examples for convenience of description. Therefore, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings may be used.

[0037] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled, but this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." A condition described as "more than" may be replaced with "more than," a condition described as "less than" may be replaced with "less than," and a condition described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of elements from A (including A) to B (including B). hereinafter, "C" and / or "D" mean at least one of "C" or "D," that is, including {"C", "D", "C" and "D"}.

[0038] It should be noted that the blocks and combinations of flowcharts in each flowchart can be executed by one or more computer programs containing instructions. The one or more computer programs may be stored entirely in a single memory device, or the one or more computer programs may be divided into different parts and stored in multiple different memory devices.

[0039] Any of the functions or operations described in this document may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing and may include an application processor (AP) (e.g., a central processing unit (CPU), a communication processor (CP) (e.g., a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi™ (wireless fidelity) chip, a Bluetooth™ chip, a global positioning system (GPS) chip, a near field communication (NFC) chip, connectivity chips, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio codec chip, a universal serial bus (USB) controller, a camera controller, an image processing integrated circuit, a microprocessor unit (MPU), a system on a chip (SoC), an integrated circuit, or a similar circuit.

[0040] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments of the present disclosure.

[0041] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0042] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0043] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0044] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0045] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0046] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0047] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0048] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0049] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0050] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0051] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0052] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0053] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0054] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0055] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0056] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0057] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently from the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module may be a first network (198) (e.g., Bluetooth™, WiFi). TMThe wireless communication module (192) can communicate with an external electronic device (104) via a short-range communication network such as (wireless fidelity) direct or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a fifth-generation (5G) network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network) such as a local area network (LAN) or a wide area network (WAN). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can use subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

[0058] The wireless communication module (192) can support a 5G network and next-generation communication technologies following the 4G (fourth-generation) network, for example, NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may be configured to achieve a Peak data rate (e.g., 20 Gbps (gigabits per second) or higher) for eMBB implementation, a Loss Coverage (e.g., 164 dB (decibels) or lower) for mMTC implementation, or a U-plane latency (e.g., 0 for downlink (DL) and uplink (UL) respectively) for URLLC implementation.It can support 5ms (milliseconds) or less, or round trip 1ms or less.

[0059] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0060] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0061] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0062] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (e.g., the electronic device (102), the electronic device (104), and the server (108)). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (104) or server (108) may be included in the second network (199). The electronic device (101) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0063] In embodiments of the present disclosure, an electronic device (e.g., electronic device (101) of FIG. 1) for displaying an image in a virtual space may be a wearable device. The electronic device (101) (e.g., wearable device (101)) may include a head-mounted display (HMD) that is wearable on a user's head. The wearable device (101) may be referred to as a head-mounted device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of a hardware configuration included in the wearable device (101) is described with reference to FIG. 4. An example of the structure of a wearable device (101) that can be worn on a user's head is described with reference to FIGS. 2A, 2B, 3A, and / or 3B. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) for attaching to a user's head to form an HMD.

[0064] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when a user wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user may see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0065] According to one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). For example, when a user wears the wearable device (101), the wearable device (101) may include a housing that covers the user's eyes. The wearable device (101), in this state, may include a display disposed on a first side of the housing facing the eyes. The wearable device (101) may include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (101) may acquire images and / or videos representing ambient light. The wearable device (101) may output the images and / or videos within the display disposed on the first side, thereby allowing the user to perceive the ambient light through the display. The displaying area (or displaying region) (or active area or active region) of the display arranged on the first surface may be formed by one or more pixels included in the display. The wearable device (101) may synthesize a virtual object into an image and / or video output through the display, thereby allowing the user to recognize the virtual object together with a real object recognized by ambient light.

[0066] According to one embodiment, the wearable device (101) can identify or recognize a position (or location) and / or direction (or orientation) of the wearable device (101) based on an image (and / or video) obtained or acquired using a camera. The wearable device (101) can obtain information about the external space using one or more cameras and / or one or more sensors. The information can include a geographic location (e.g., global positioning system (GPS) coordinates) of the external space identified from one or more sensors. The information can include images and / or videos of the external space identified from one or more cameras. The wearable device (101) can perform object recognition on the images and / or videos to identify external objects included in the external space from the images and / or videos.

[0067] Below, an example of a hardware configuration of a wearable device (101) is described with reference to FIGS. 2a, 2b, 3a, 3b, and 4.

[0068] FIG. 2A illustrates an example of a perspective view of a wearable device according to an embodiment of the present disclosure. FIG. 2B illustrates an example of one or more hardware elements disposed within a wearable device according to an embodiment of the present disclosure. According to an embodiment, the wearable device (101) may have a form of glasses that are wearable on a body part (e.g., head) of a user. The wearable device (101) of FIGS. 2A and 2B may be an example of the electronic device (101) of FIG. 1. The wearable device (101) may include a head-mounted display (HMD). For example, the housing of the wearable device (101) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (101) may include one or more straps capable of being twined around the user's head, and / or one or more temples attachable to the ears of the head.

[0069] Referring to FIG. 2A, according to one embodiment, a wearable device (101) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0070] According to one embodiment, a wearable device (101) can be worn on a part of a user's body. The wearable device (101) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (101). For example, the wearable device (101) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0071] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0072] Referring to FIG. 2B, at least one display (250) can provide visual information transmitted from external light to the user through a lens included in the at least one display (250), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When a user wears the wearable device (101), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0073] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (101) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0074] The wearable device (101) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (101) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (101) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (101) can view an image displayed on at least one display (250).

[0075] According to one embodiment, the frame (200) may be formed as a physical structure that allows the wearable device (101) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that, when the user wears the wearable device (101), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0076] Referring to FIG. 2A, the frame (200) may include a region (220) that is in contact with at least a portion of a user's body when the user wears the wearable device (101). For example, the region (220) of the frame (200) that is in contact with a portion of the user's body may include a region that is in contact with a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (101) makes contact with. According to one embodiment, the frame (200) may include a nose pad (210) that is in contact with a portion of the user's body. When the wearable device (101) is worn by the user, the nose pad (210) may be in contact with a portion of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another portion of the user's body that is distinct from the portion of the user's body.

[0077] For example, the frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (101) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200).

[0078] According to one embodiment, the wearable device (101) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (200).

[0079] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (101) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B, but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B. When the number of microphones (265) included in the wearable device (101) is two or more, the wearable device (101) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (200).

[0080] According to one embodiment, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or may be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (101) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0081] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (101). For example, the wearable device (101) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (101) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (101) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first area related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI (pixels per inch)) of a second area distinguished from the first area may be different from each other. In this disclosure, the term “resolution” is used to refer to the density of pixels of an image and / or display (250).The density and / or resolution of pixels can be measured or parameterized based on units of PPI and / or dpi (dots per inch). The wearable device (101) can obtain an image having a visual quality of a first area matching the user's gaze and a visual quality of a second area using foveated rendering. For example, if the wearable device (101) supports an iris recognition function, user authentication can be performed based on iris information obtained using the gaze tracking camera (260-1). Although an example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, the embodiment is not limited thereto, and the gaze tracking camera (260-1) can be positioned solely toward the user's left eye, or toward both eyes.

[0082] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (101) can compensate for depth information (e.g., the distance between the wearable device (101) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (260-4). The wearable device (101) can perform object recognition through an image acquired using the capturing camera (260-4). The wearable device (101) can perform a function (e.g., auto focus (AF)) for focusing on an object (or subject) in an image and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capturing camera (260-4). The wearable device (101) can perform a pass-through function for displaying an image acquired through the capturing camera (260-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the capturing camera (260-4) can be disposed on a bridge (203) disposed between the first rim (201) and the second rim (202).

[0083] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing a wearable device (101), thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (101) looks straight ahead, the wearable device (101) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (101) is positioned.

[0084] The motion recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The motion recognition camera (260-2, 260-3) can recognize the user's motion (gesture recognition), obtain a signal corresponding to the motion, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the motion, and perform a designated function based on the identification. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor can perform a gesture recognition function and / or an object tracking function using the motion recognition camera (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) may be positioned on the first rim (201) and / or the second rim (202).

[0085] The camera (260) included in the wearable device (101) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (101) can identify an external object included in the user's field of view (FoV) using a camera positioned toward the FoV. The wearable device (101) can identify an external object based on a sensor for identifying the distance between the wearable device (101) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus (AF) function and / or an optical image stabilization (OIS) function. For example, the wearable device (101) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (101).

[0086] Although not shown, in one embodiment, the wearable device (101) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame (200) and the hinge units (206, 207).

[0087] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (101). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0088] The antenna module (275) can transmit signals or power to the outside of the wearable device (101), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0089] The speaker (255) can output an acoustic signal to the outside of the wearable device (101). The acoustic output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (101). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus positioned adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus positioned adjacent to the user's right ear.

[0090] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (101) to the user. For example, when the wearable device (101) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0091] Referring to FIG. 2B, according to one embodiment, a wearable device (101) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (101) (e.g., hardwares illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (101) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0092] According to one embodiment, a wearable device (101) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (101) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (101). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (101) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (101) based on the IMU.

[0093] FIGS. 3A and 3B illustrate an example of an exterior appearance of a wearable device (e.g., the wearable device (101) of FIGS. 2A and 2B) according to various embodiments of the present disclosure. The wearable device (101) of FIGS. 3A and 3B may be an example of the electronic device (101) of FIG. 1 and the wearable device (101) of FIGS. 2A and 2B. An example of an exterior appearance of a first side (310) of a housing of the wearable device (101) according to one embodiment is illustrated in FIG. 3A, and an example of an exterior appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B.

[0094] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (101) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (101) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (101) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0095] According to one embodiment, the wearable device (101) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (or eye tracking camera) (260-1) of FIG. 2B. According to one embodiment, the wearable device (101) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (101) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (101) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar expressing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (101).

[0096] Referring to FIG. 3B, a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor (330)) for obtaining information related to the external environment of the wearable device (101) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0097] Using cameras (260-11, 260-12), the wearable device (101) can acquire images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be positioned on the second face (320) of the wearable device (101) to acquire an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be positioned on the second face (320) of the wearable device (101) to acquire an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0098] According to one embodiment, the wearable device (101) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (101) and an external object. Using the depth sensor (330), the wearable device (101) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (101). Although not illustrated, a microphone may be disposed on the second face (320) of the wearable device (101) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0099] Hereinafter, with reference to FIG. 4, the hardware or software configuration of the wearable device (101) is described.

[0100] FIG. 4 illustrates an example of a block diagram of a wearable device (e.g., the wearable device (101) of FIGS. 2A and 2B) according to one embodiment of the present disclosure. The wearable device (101) of FIG. 4 may be an example of the electronic device (101) of FIG. 1 or the wearable devices (101) of FIGS. 2A to 3B.

[0101] Referring to FIG. 4, a wearable device (101) according to one embodiment may include a processor (410), a memory (415), a display (250) (e.g., the first display (250-1) and / or the second display (250-2) of FIGS. 2a, 2b, 3a, and 3b), a sensor (420) (e.g., an image sensor (421) and / or a motion sensor (422)), and / or a communication circuit (430) (e.g., including at least a portion of the communication module (190) of FIG. 1). The processor (410), the memory (415), the display (250), the sensor (420), and / or the communication circuit (430) may be electrically and / or operatively connected to each other by electronic components such as a communication bus (402). In the present disclosure, the operational connection of electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components, such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (101) is not limited to those illustrated in FIG. 4. For example, the wearable device (101) may include only some of the electronic components illustrated in FIG. 4.

[0102] According to one embodiment, the processor (410) of the wearable device (101) may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. According to one embodiment, the structure of the processor (410) is not limited to one embodiment of the present disclosure, and at least one circuit may be formed as a separate processor that is physically separated from the processor. The processor (410) may have a structure of a multi-core processor, such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor architecture of the processor (410) may include a architecture based on multiple core circuits (e.g., a big-little architecture) that are distinguished by power consumption, clock frequency, and / or computational amount per unit time. In one embodiment including the processor (410) having a multi-core processor architecture, the operations and / or functions of the present disclosure may be individually or collectively performed by one or more cores included in the processor (410).

[0103] According to one embodiment, the memory (415) of the wearable device (101) may include electronic components for storing data and / or instructions input to and / or output from the processor (410). The memory (415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC). In one embodiment, the memory (415) may be referred to as storage.

[0104] In one embodiment, the display (250) of the wearable device (101) can output visualized information to the user of the wearable device (101). The display (250), which is arranged in front of the eyes of the user wearing the wearable device (101), can be arranged on at least a portion of the housing of the wearable device (101) (e.g., the first display (250-1) and / or the second display (250-2) of FIGS. 2A, 2B, 3A, and 3B). For example, the display (250) can be included in a display assembly. For example, the display (250) can be controlled by a processor (410) including circuits such as a CPU (411), a GPU (graphics processing unit) (412), and / or a DPU (display processing unit) (413), to output visualized information to the user. The display (250) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (250) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). The embodiment is not limited thereto, and for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (250) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (250) may be referred to as a display panel and / or a display module.The pixels included in the display (250) may be arranged to face either of the user's two eyes when the wearable device (101) is worn by the user. For example, the display (250) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0105] In one embodiment, the sensor (420) of the wearable device (101) may generate electrical information that may be processed by the processor (410) and / or the memory (415) from non-electronic information related to the wearable device (101). For example, the sensor (420) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (101). In addition to the GPS method, the sensor (420) may generate information indicating the geographic location of the wearable device (101) based on a global navigation satellite system (GNSS) such as, for example, Galileo or Beidou (compass). The information may be stored in the memory (415), processed by the processor (410), and / or transmitted to another electronic device distinct from the wearable device (101) via a communication circuit.

[0106] Referring to FIG. 4, an image sensor (421) and / or a motion sensor (422) are illustrated as examples of a sensor (420) included in a wearable device (101). The sensor (420) may include one or more optical sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate electrical signals representing the color and / or brightness of light. The image sensor (421) may be referred to as a camera. A plurality of optical sensors included in the image sensor (421) may be arranged in the form of a two-dimensional grid (2-dimensional array). The image sensor (421) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional grid. For example, photographic data captured using the image sensor (421) may mean one (a) two-dimensional frame data acquired from the image sensor (421). For example, video data captured using the image sensor (421) may mean a sequence of a plurality of two-dimensional frame data acquired from the image sensor (421) according to a frame rate. The image sensor (421) may be arranged toward the direction in which the image sensor (421) receives light and may further include a flash light for outputting light toward the direction.

[0107] According to one embodiment, the wearable device (101) may include a plurality of image sensors, as an example of an image sensor (421), arranged in different directions. As described above with reference to FIGS. 2A, 2B, 3A, and 3B, the plurality of image sensors may include gaze tracking cameras (e.g., eye tracking camera (260-1) of FIGS. 2B and 3A) configured to be arranged toward the eyes of a user wearing the wearable device (101). The plurality of image sensors may include outward cameras. The processor (410) may identify the direction of the user's gaze using images and / or videos acquired from the gaze tracking cameras. The gaze tracking cameras may include infrared (IR) sensors. The gaze tracking cameras may be referred to as eye sensors and / or eye trackers.

[0108] The external camera may be positioned facing the front of a user wearing the wearable device (101) (e.g., in a direction that both eyes may face). The wearable device (101) may include multiple external cameras. The embodiment is not limited thereto, and the external camera may be positioned facing an external space. Using images and / or videos acquired from the external cameras, the processor (410) may identify external objects. For example, the processor (410) may identify the position, shape, and / or gesture (e.g., hand gesture) of a hand of a user wearing the wearable device (101) based on images and / or videos acquired from the external cameras. Using images and / or videos of the external environment acquired from the external cameras, the processor (410) may recognize or track one or more objects within the external environment.

[0109] In one embodiment, the motion sensor (422) may output electrical signals representing gravitational accelerations, accelerations, and / or angular velocities of a plurality of axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other and based on a designated origin within the wearable device (101) and / or the motion sensor (422). For example, the processor (410) may repeatedly receive or acquire sensor data including accelerations, angular velocities, and / or magnitudes of magnetic fields of the plurality of axes from the motion sensor (422) based on a designated period (e.g., 1 millisecond). In one embodiment, the motion sensor (422) may be referred to as an inertial measurement unit (IMU). The sensor (420) included in the wearable device (101) is not limited to those described above, and may include a grip sensor, a proximity sensor, a heart rate sensor, a fingerprint sensor, an ambient light sensor, and / or a ToF sensor. Using the motion sensor (422), the processor (410) can detect motion of the wearable device (101) (e.g., motion of the wearable device (101) caused by a user wearing the wearable device (101).

[0110] In one embodiment, the communication circuit (430) of the wearable device (101) may include hardware components for supporting transmission and / or reception of signals between the wearable device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), the electronic device (601) of FIG. 6, and the electronic device (701) of FIG. 7). The communication circuit (430) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (430) may include a communication circuit (430) that supports Ethernet, a local area network (LAN), a wide area network (WAN), and WiFi. TM (wireless fidelity), BluetoothTM , BLE(Bluetooth TM low energy), zigbee TM , can support transmission and / or reception of electrical signals based on various types of protocols such as LTE (long term evolution), and 5G NR (new radio).

[0111] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (410) of the wearable device (101) may be stored in the memory (415) of the wearable device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (101) and / or the processor (410) may perform at least one of the operations of FIGS. 6, 7, 8, 9, 10, 11, and 12 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, a program, and / or a software application is executed. Hereinafter, the fact that a software application is installed in a wearable device (101) may mean that one or more instructions provided in the form of a software application (or package) are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (101)). For example, the application may include a program and / or a library related to a service provided to a user.

[0112] Referring to FIG. 4, programs installed in the wearable device (101) may be included in any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (250), and / or the sensor (420)) of the wearable device (101) may be included in the hardware abstraction layer (480) (e.g., the android system HAL, and / or the XR HAL). The framework layer (450) may be referred to as an XR framework layer from the perspective of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 4 may be logically (or for convenience of explanation) separated, and may not mean that the address space of the memory (415) is separated by the layers.

[0113] Within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a position tracker (471), a space recognizer (472), a gesture tracker (473), an eye tracker (474), a face tracker (475), and / or a renderer (490)) may be included. The programs included in the framework layer (450) may provide an API (application programming interface) that is executable (or callable) based on other programs.

[0114] The application layer (440) may include programs designed to target users of the wearable device (101). Examples of programs included in the application layer (440) include an extended reality (XR) system user interface (UI) (441) and / or an XR application (442), but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (440) may call APIs to cause execution of functions supported by programs included in the framework layer (450).

[0115] The wearable device (101) may display one or more visual objects on the display (250) for performing interaction with the user based on the execution of the XR system UI (441). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user with functions available within a virtual space based on the execution of the XR system UI (441).

[0116] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plug-in (444) are illustrated to be included within the XR system UI (441), but are not limited thereto. For example, based on the XR system UI (441), the processor (410) may execute a lightweight renderer (443) and / or an XR plug-in (444) within the framework layer (450).

[0117] The wearable device (101) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline from the perspective of defining a rendering pipeline that allows partial changes. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (444). The XR plug-in (444) may be referred to as an open XR native client from the perspective of defining (or configuring) an entire rendering pipeline.

[0118] The wearable device (101) may display a screen representing at least a portion of a virtual space on the display (250) based on the execution of the XR application (442). The XR plug-in (441-1) included in the XR application (442) may include instructions that support functions similar to those of the XR plug-in (444) of the XR system UI (441). Descriptions of the XR plug-in (441-1) that overlap with those of the XR plug-in (444) may be omitted. The wearable device (101) may cause the execution of the virtual space manager (451) based on the execution of the XR application (442).

[0119] The wearable device (101) can display an image on the display (250) in a virtual space based on the execution of the application (445). The application (445) can be configured to output image information for displaying a two-dimensional image. The wearable device (101) can cause the execution of the virtual space manager (451) based on the execution of the application (445). The wearable device (101) can generate dual image information to display the two-dimensional image in a three-dimensional virtual space based on the execution of the application (445). Here, the dual image information can include first image information for the left eye and second image information for the right eye, taking into account binocular disparity. In order to display the two-dimensional image in the three-dimensional virtual space, the wearable device (101) can generate the dual image information based on the image information for displaying the two-dimensional image.

[0120] According to one embodiment, the wearable device (101) can provide a virtual space service based on the execution of the virtual space manager (451). For example, the virtual space manager (451) can include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (451), the wearable device (101) can identify a virtual space formed based on the user's location indicated by data acquired through the sensor (420), and display at least a portion of the virtual space on the display (250). The virtual space manager (451) can be referred to as a composition presentation manager (CPM).

[0121] The virtual space manager (451) may include a runtime service (452). For example, the runtime service (452) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (101) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (452). For example, the wearable device (101) may perform rendering for a virtual space service for the user based on the execution of the runtime service (452). For example, a function related to a virtual space, executable by the application layer (440), may be supported based on the execution of the runtime service (452).

[0122] The virtual space manager (451) may include a pass-through manager (453). Based on the execution of the pass-through manager (453), the wearable device (101) may display an image and / or video representing an actual space acquired through an external camera on at least a portion of the screen while displaying a screen representing a virtual space on the display (250).

[0123] The virtual space manager (451) may include an input manager (454). The wearable device (101) may identify data (e.g., sensor data) obtained by executing one or more programs included in the recognition service layer (470) based on the execution of the input manager (454). The wearable device (101) may use the obtained data to identify user input related to the wearable device (101). The user input may be related to a motion (e.g., a hand gesture), gaze, and / or speech of the user identified by a sensor (420) (e.g., an image sensor (421) such as an external camera). The user input may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0124] The perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). From the perspective of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referenced as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0125] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data acquired from the sensor (420). The one or more programs may include at least one of a position tracker (471), a space recognizer (472), a gesture tracker (473), an eye tracker (474), a face tracker (475), and / or a renderer (490). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those illustrated in FIG. 4.

[0126] The wearable device (101) can identify the posture of the wearable device (101) using the sensor (420) based on the execution of the position tracker (471). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using an external camera (e.g., an image sensor (421)) and / or an IMU (e.g., a motion sensor (422) including a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (471). The position tracker (471) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0127] The wearable device (101) can obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of the space recognizer (472). The wearable device (101) can reproduce the surrounding environment of the wearable device (101) in three dimensions using data obtained using an external camera (e.g., an image sensor (421)) based on the execution of the space recognizer (472). The wearable device (101) can identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (101) reproduced in three dimensions based on the execution of the space recognizer (472). The space recognizer (472) can be referred to as a scene understanding (SU) module (or a scene recognition program).

[0128] The wearable device (101) can identify (or recognize) the pose and / or gesture of the user's hand based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user's hand using data acquired from an external camera (e.g., an image sensor (421)) based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user's hand based on data (or images) acquired using the external camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0129] The wearable device (101) can identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (474). For example, the wearable device (101) can identify eye movements of the user using data acquired from a gaze tracking camera (e.g., an image sensor (421)) based on the execution of the gaze tracker (474). The gaze tracker (474) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0130] The recognition service layer (470) of the wearable device (101) may further include a face tracker (475) for tracking the user's face. For example, the wearable device (101) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (475). The wearable device (101) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (475). As an example, the wearable device (101) may identify the movement of the user's face and / or the user's expression based on data (e.g., images and / or videos) acquired using an FT camera (e.g., a camera facing at least a portion of the user's face, an image sensor (421)) based on the execution of the face tracker (475). The face tracker (475) may be referred to as a face tracking (FT) (or face tracking program), and / or a face tracking module.

[0131] Referring to FIG. 4, examples of a processor (410) include a CPU (411), a GPU (graphics processing unit) (412), and / or a DPU (display processing unit) (413). A renderer (490) may include instructions for rendering images in a three-dimensional virtual space. A processor (410) (e.g., DPU (413)) executing the renderer (490) may obtain at least one image to be at least partially displayed in a display area of ​​a display (250) from a software application (e.g., a software application executed by the CPU (411) and / or GPU (412)). For example, a processor (410) executing the renderer (490) may determine a location of an area in which an application (e.g., an XR application (442), an application (445)) is to be rendered. The processor (410) executing the renderer (490) can generate an image of the application to be displayed on the display (250). The renderer (490) can synthesize images to generate a composite image to be displayed on the display (250).

[0132] The processor (410) executing the renderer (490) can divide the display area of ​​the display (250) into a foveated portion (or may be referred to as a foveated area) and a peripheral portion (or may be referred to as a residual area) using the gaze position calculated using the position tracker (471) and / or the gaze tracker (474). For example, the processor (410) detecting the coordinate values ​​of the gaze position can determine the portion of the display area including the coordinate values ​​as the foveated area. The DPU (413) executing the renderer (490) can obtain at least one image corresponding to each of the foveated area and the residual area, and having a size smaller than the size of the entire display area of ​​the display (250) or a resolution smaller than the resolution of the display area.

[0133] The processor (410) executing the renderer (490) may obtain or generate a composite image to be displayed on the display (250) by synthesizing an image corresponding to the foveated area and an image corresponding to the peripheral area. For example, the processor (410) may perform upscaling to enlarge the image corresponding to the peripheral area to the size of the entire display area of ​​the display (250). On the enlarged image, the processor (410) may combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (250). Along the boundary line of the image corresponding to the foveated area, the processor (410) may apply a visual effect, such as blur, to blend the enlarged image and the image corresponding to the foveated area.

[0134] FIG. 5 illustrates an example of a block diagram of an electronic device (e.g., the electronic device 101 of FIG. 1, the wearable device 101 of FIGS. 2A to 4) for displaying an image in a virtual space according to an embodiment of the present disclosure. In FIG. 5, an example of executing a plurality of programs (or instructions) for displaying an image in a virtual space is described. The plurality of programs (or instructions) may all be executed in one processor (e.g., an AP) or may be executed by a plurality of processors (e.g., an AP, a GPU (graphics processing unit), an NPU (neural processing unit)). The meaning of being executed by the plurality of processors means that some programs (or instructions) may be executed by a first processor, and other programs (or instructions) may be executed by a second processor different from the first processor.

[0135] Referring to FIG. 5, the electronic device (101) may execute a virtual space manager (550) (e.g., the virtual space manager (451) of FIG. 4, CPM) to render an image in a virtual space. For the virtual space manager (550), at least some of the descriptions of the virtual space manager (451) of FIG. 4 may be referred to. The virtual space manager (550) may include a platform for supporting a virtual space service. The virtual space manager (550) may include a runtime service (551) (e.g., OpenXR Runtime), a panel rendering (552) (e.g., a 2D (two-dimensional) Panel Render), and an XR composition unit (553) (XR Compositor). The electronic device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (551). For the runtime service (551), at least some of the descriptions of the runtime service (452) of FIG. 4 may be referred to. The electronic device (101) may display at least one image (video) on a panel (e.g., a 2D panel) to implement a virtual space through the display (250) based on the execution of the panel rendering (552). For example, the electronic device (101) may display a rendering image corresponding to RGB (red, green, blue) information (566) for the panel from the spatialization manager (540) described below through the display (e.g., the display (250)). The electronic device (101) may synthesize an image of an actual area captured by a camera in the virtual space (hereinafter, a pass-through image) with a virtual area image based on the execution of the XR synthesis unit (553) (XR Compositor). For example, the electronic device (101) can generate a composite image by merging the pass-through image and the virtual area image based on the execution of the XR synthesis unit (553).The electronic device (101) can transmit the generated composite image to the display buffer so that the composite image is displayed. The electronic device (101) can identify a virtual space through a virtual space manager (550) and display at least a portion of the virtual space on the display (250). The virtual space manager (550) may be referred to as a CPM. The electronic device (101) can execute the virtual space manager (550) to render an image corresponding to at least a portion of the virtual space.

[0136] According to one embodiment, the electronic device (101) may execute a spatialization manager (540). The spatialization manager (540) may perform processes for displaying an image in a three-dimensional virtual space. The electronic device (101) may perform preprocessing based on the execution of the spatialization manager (540) so that the image can be rendered in a three-dimensional virtual space through the virtual space manager (550). For example, the electronic device (101) may perform at least some of the functions of the renderer (490) of FIG. 4 based on the execution of the spatialization manager (540). The electronic device (101) may process image information provided by an application (e.g., an XR application (510), an application (520) that provides a general 2D screen other than XR, and an application that provides a system UI (530)) based on the execution of the spatialization manager (540). A spatialization manager (540) (e.g., Space Flinger) may include a system scene manager (541) (e.g., System scene), an input manager (542) (e.g., Input Routing), and a lightweight rendering engine (543) (e.g., Impress Engine). The system scene manager (541) may be executed to display a system UI (530). System UI-related information (564) may be transmitted to the system scene manager (541) from a program (e.g., API) that provides the system UI (530). The system UI-related information (564) may be obtained through a spatializer API and / or a same-process private API. The spatialization manager (540) may determine the layout (e.g., location, display order) of the screen of the system UI (530) in a three-dimensional space through pre-allocated resources. The system screen manager (541) can transmit image information (567) for rendering the screen of the system UI (530) to the virtual space manager (550) according to the above layout.The input manager (542) may be configured to process user input (e.g., user input on a system screen or an app screen). The input manager (542) may map user input recognized by the sensor (420) of the electronic device (101) to at least one of one or more software applications (e.g., an XR application (510), an application (520) that provides a general 2D screen other than XR, and an application that provides a system UI (530)) mapped to a virtual space by the spatialization manager (540). For example, mapping the user input may include an operation of executing instructions (e.g., a subroutine and / or an event handler) of a software application for processing the user input. The lightweight rendering engine (543) may be a renderer for generating an image (e.g., a lightweight renderer (443)). For example, the lightweight rendering engine (543) may be used to display the system UI (530).

[0137] In one embodiment, the spatialization manager (540) may include a lightweight rendering engine (543) for rendering the system UI. In one embodiment, if the lightweight rendering engine (543) does not have sufficient resources to render an avatar used in the HMD, at least one external rendering engine may be used. In this case, to resolve compatibility issues with external rendering (e.g., a 3rd party engine), an external rendering engine support module may be added within the spatialization manager (540).

[0138] According to one embodiment, the electronic device can execute an application. For example, in response to the execution of an XR application (510) (e.g., an XR application (442), a 3D (three-dimensional) game, an XR map, or other immersive application), the electronic device can execute a virtual space manager (550). The electronic device (101) can provide dual image information (561) provided from the XR application (510) to the virtual space manager (550). In order to display an image in a three-dimensional space, the dual image information (561) can include two pieces of image information that take binocular parallax into account. For example, the dual image information (561) can include first image information for the user's left eye and second image information for the user's right eye for rendering in a three-dimensional virtual space. Hereinafter, in the present disclosure, dual image information is used as a term referring to image information for displaying images for both eyes in a three-dimensional space. In addition to dual image information, the above dual image information may also include binocular image information, dual image data, dual images, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimensional conversion image data, binocular parallax image data, and / or equivalent technical terms. The electronic device (101) can generate a composite image by merging image layers through a virtual space manager (550). The electronic device (101) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (250) of the electronic device (101).

[0139] According to one embodiment, the electronic device can execute at least one application among an XR application (510) and other applications (520) (e.g., a first application (520-1), a second application (520-2), ..., an Nth application (520-N)). According to one embodiment, the application (520) can be configured to output image information for displaying a two-dimensional (2D) image. In other words, the application (520) can provide a two-dimensional (2D) image (e.g., a window and / or an activity). For example, the application (520) can be a video application, a schedule application, or an Internet browser application. If, in response to the execution of the application (520), the image information (562) provided from the application (520) is provided to the virtual space manager (550), the image information (562) has only x-coordinates and y-coordinates within a two-dimensional plane, so it may be difficult to consider the chronological relationship (i.e., the distance from the user) between other applications centered on the user. Even when displaying an application (520) that provides a general 2D screen, the electronic device (101) may execute the spatialization manager (540) to provide dual image information to the virtual space manager (550). For example, based on the execution of the spatialization manager (540), the electronic device (101) may receive application-related information (563) from the first application (520-1). For example, application-related information (563) may include image information representing a two-dimensional image of the first application (520-1) (e.g., information including RGB for each pixel) and / or content information in the first application (520-1) (e.g., characteristics of content executed in the first application, type of content). The application-related information (563) may be obtained through a spatializer API.Based on the execution of the spatialization manager (540), the electronic device (101) can identify information (hereinafter, location information) about the location of the area to be rendered and the size of the area to be rendered by the first application (520-1). Based on the execution of the spatialization manager (540), the electronic device (101) can generate dual image information (565, e.g., RGBx2) that takes into account the user's binocular disparity through the image information and the location information. Based on the execution of the spatialization manager (540), the electronic device (101) can provide the dual image information (565) to the virtual space manager (550). By converting a simple two-dimensional image into the dual image information (565), a problem that occurs when the image information (562) is directly transmitted to the virtual space manager (550) can be resolved. Additionally, since at least some of the functions for displaying images in a virtual space are performed by the spatialization manager (540) instead of the virtual space manager (550), the burden on the virtual space manager (550) can be reduced.

[0140] FIG. 6 illustrates an example of a structure of multiple layers according to one embodiment of the present disclosure.

[0141] Referring to FIG. 6, programs installed in the wearable device (101) can be classified into one of the platform layer (610), the recognition service layer (620) (e.g., the recognition service layer (470) of FIG. 4), and the sensor service layer (630). For example, the wearable device (101) can operate based on the platform layer (610), the recognition service layer (620), and the sensor service layer (630).

[0142] According to one embodiment, the platform layer (610) may be configured for an XR service. For example, the platform layer (610) may include a platform (e.g., an Android platform) for supporting an XR service. For example, the platform layer (610) may include a virtual space manager (550) of FIG. 5. The platform layer (610) may include a runtime service (611). For the runtime service (611), reference may be made to the descriptions of the runtime service (551) of FIG. 5 and the descriptions of the runtime service (452) of FIG. 4. As an example, the runtime service (611) may be referred to as an OpenXR runtime module. The runtime service (611) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (101). As an example, the runtime service (611) may be used to perform rendering for an XR service to a user. For example, an application (e.g., unity or OpenXR native application) can be implemented based on the runtime service (611).

[0143] The perception abstraction layer (612) can be used for data exchange between the platform layer (610) and the perception service layer (620). For the perception abstraction layer (612), the descriptions of the perception abstraction layer (460) of FIG. 4 can be referenced. For example, the perception abstraction layer (612) can be referenced as OpenPX. The perception abstraction layer (612) can be used for a perception client and a perception service.

[0144] According to one embodiment, the recognition service layer (620) may include a service module (621), a recognition plug-in layer (622), a sensor management module (623), a playback module (624), and / or an external data management module (625). For example, the recognition service layer (620) may include at least one of the service module (621), the recognition plug-in layer (622), the sensor management module (623), the playback module (624), and / or the external data management module (625). For example, at least some of the service module (621), the recognition plug-in layer (622), the sensor management module (623), the playback module (624), and the external data management module (625) may be omitted.

[0145] The service module (621) can manage input data of the wearable device (101). The service module (621) can be used to manage data (e.g., gesture information) acquired from a plurality of recognition modules included in the recognition plug-in layer (622). For example, the service module (621) can be referred to as SxrDataService. The service module (621) can manage data (640) received from an electronic device (601) managed by an external data management module (625).

[0146] The service module (621) can interface with an upper layer (e.g., the platform layer (610) or the runtime service (611)). The service module (621) can exchange data with the upper layer (e.g., the platform layer (610) or the runtime service (611)) through the recognition abstraction layer (612). For example, the recognition abstraction layer (612) can be referred to as OpenPX. According to an embodiment, the service module (621) can support not only OpenPX but also OpenXR Extension. The service module (621) can be used to exchange data (e.g., gesture information) between a plurality of recognition modules. The service module (621) can be configured to manage data processed in the recognition service layer (620). The service module (621) can select data to be recognized as input of the wearable device (101) from among the above data. The above data may include data acquired from multiple recognition modules and data acquired through an external data management module (625). The service module (621) may manage data to be used in the recognition abstraction layer (612). The service module (621) may select data to be recognized as input from the wearable device (101) among the above data and provide the data to the recognition abstraction layer (612).

[0147] The recognition plug-in layer (622) may include multiple recognition modules. The multiple recognition modules may be referred to as a plurality of perception solutions.

[0148] For example, the plurality of recognition modules may include at least one of a head tracking (HeT) module (622-1) (e.g., a position tracker (471)), a scene understanding (SU) (or environment recognition module) module (622-2) (e.g., a space recognizer (472)), a hand tracking (HaT) module (622-3) (e.g., a gesture tracker (473)), an eye tracking (ET) module (622-4) (e.g., a gaze tracker (474)), and a face tracking (FT) module (622-5) (e.g., a face tracker (475)). Each of the plurality of recognition modules included in the recognition plug-in layer (622) may include a common interface for connection (or interworking) with the sensor management module (623). Each of the plurality of recognition modules may include a common interface for connection (or interworking) with the sensor management module (623).

[0149] The head tracking module (622-1) can identify the posture of the wearable device (101) using at least one sensor of the wearable device (101). For example, the head tracking module (622-1) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) based on data acquired using a camera (e.g., image sensor (421) of FIG. 4) and an IMU.

[0150] The environment recognition module (622-2) can be used to construct the surrounding environment of the wearable device (101) (or the user of the wearable device (101)) into a three-dimensional virtual space. The environment recognition module (622-2) can be used to reconstruct the surrounding environment of the wearable device (101) in three dimensions based on data acquired using a camera (e.g., the image sensor (421) of FIG. 4). The environment recognition module (622-2) can identify at least one of a plane, a slope, and stairs based on the three-dimensionally reconstructed surrounding environment of the wearable device (101).

[0151] The hand tracking module (622-3) may be used to identify (or recognize) the pose and / or gesture of the hand of a user of the wearable device (101). For example, the hand tracking module (622-3) may identify the pose and / or gesture of the user's hand based on data acquired from at least one sensor. For example, the hand tracking module (622-3) may identify the pose and / or gesture of the user's hand based on data (e.g., an image) acquired using a camera.

[0152] The eye tracking module (622-4) can be used to identify (or track) eye movements of a user of the wearable device (101). For example, the eye tracking module (622-4) can identify eye movements of the user based on data acquired from at least one sensor. For example, the eye tracking module (622-4) can identify eye movements of the user based on data acquired using a camera (e.g., the eye tracking camera (260-1) of FIGS. 2B and 3A) and / or an infrared light emitting diode (IR LED).

[0153] The face tracking module (622-5) can be used to identify (or track) the user's facial movements and / or facial expressions. The face tracking module (622-5) can estimate the user's facial expressions based on the user's facial movements. For example, the face tracking module (622-5) can identify the user's facial movements and / or facial expressions based on data (e.g., images) acquired using a camera (e.g., the camera (260) of FIGS. 2A and 2B ).

[0154] For example, a plurality of recognition modules (e.g., a head tracking module (622-1), an environment recognition module (622-2), a hand tracking module (622-3), an eye tracking module (622-4), and a face tracking module (622-5)) included in the recognition plug-in layer (622) may be configured in a plug-in structure. For example, some of the plurality of recognition modules may be replaced with other modules regardless of the sensor service layer (630) and the platform layer (610), which are lower layers of the recognition service layer (620).

[0155] In one embodiment, the sensor management module (623) may be used to provide (or transmit) data to each of a plurality of recognition modules through a common interface. For example, the sensor management module (623) may be used to separate (or remove) the dependency between the sensor service layer (630), which is a lower layer, and the recognition plug-in layer (622), which is an upper layer. For example, the sensor management module (623) may be referred to as SxrSensorSeviceManger.

[0156] The sensor management module (623) can support various modules (or sensor services) of the sensor service layer (630). The plurality of recognition modules may not directly interface with the sensor service layer (630). The plurality of recognition modules may receive data (e.g., sensor data) through the sensor management module (623). Therefore, even if a module of the sensor service layer (630) is changed, the plurality of recognition modules may not be affected.

[0157] The sensor management module (623) may further include a load balancing module. The load balancing module may identify data provided from the sensor service layer (630). The load balancing module may identify at least some of the plurality of recognition modules based on the data provided from the sensor service layer (630). The load balancing module may provide data to at least some of the identified recognition modules. For example, the load balancing module may distribute data to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable device (101). For example, the load balancing module may filter the data provided to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable device (101). Depending on the embodiment, the load balancing module may be configured independently from the sensor management module (623). The load balancing module may be referred to as SxrPerceptionLoadBalancer.

[0158] The playback module (624) can be used to provide a stored dataset to at least one of a plurality of recognition modules in real time through playback. For example, the dataset can be stored through the playback module (624) based on a specified standard. The dataset can include first data acquired from the sensor service layer (630) as well as second data (e.g., virtual object data or synthetic data) acquired based on the first data acquired from the sensor service layer (630). For example, the first data can be referred to as sensor data. The second data can be referred to as virtual data.

[0159] According to an embodiment, the wearable device (101) may receive data from an external electronic device (e.g., the electronic device (601)). For example, the data received from the external electronic device may include first data obtained from a service layer included in the external electronic device and / or second data obtained based on the first data. The wearable device (101) may perform playback (or a playback function) using the data received from the external electronic device. The wearable device (101) may transmit the result of performing the playback (or the playback function) to the external electronic device. For example, the wearable device (101) may be used to process data obtained from the external electronic device on its behalf. The wearable device (101) may receive data obtained from at least one sensor of the external electronic device. The wearable device (101) can acquire information (e.g., information on 6 degrees of freedom posture) based on the received data through the playback module (624) (or multiple recognition modules). The wearable device (101) can transmit the acquired information to an external electronic device. The external electronic device can provide an XR service based on the acquired information.

[0160] The playback module (624) can perform playback (or a playback function) based on at least one of the first data and the second data. In some embodiments, the playback module (624) can perform playback by combining (or mixing) real-time data (e.g., runtime data) and pre-stored data.

[0161] For example, playback may refer to a function that utilizes data (or gesture information) stored based on the operation of a wearable device (101). For example, playback may refer to a function that identifies a value for the performance of an XR service by comparing gesture information acquired based on a specified operation related to the XR service with reference gesture information based on the specified operation.

[0162] For example, playback may refer to a function for obtaining performance information of an XR service provided to a user of a wearable device (101). The playback module (624) may identify information (e.g., gesture information) about a user who performed a specified action (e.g., a mission) regarding the XR service. The playback module (624) may identify reference information about the specified action. The reference information may refer to information for determining the completion of performance of the specified action. The playback module (624) may identify the similarity between the information about the user who performed the specified action and the reference information. Based on the similarity, the playback module (624) may identify whether the performance of the action specified by the user has been completed.

[0163] In some embodiments, the playback module (624) may be included in the sensor management module (623). For example, the playback module (624) may perform playback through the sensor management module (623) without changing multiple recognition modules.

[0164] The external data management module (625) may be used to manage data (e.g., data (640)) acquired through an external electronic device (e.g., electronic device (601)) (or at least one sensor of the external electronic device) connected to the wearable device (101). The external data management module (625) may be represented as a module for using data (640) acquired through the electronic device (601) within the wearable device (101). The electronic device (601) may correspond to a smart watch, a smartphone, a smart TV, a smart monitor, or a tablet PC. However, the present invention is not limited thereto. As an example, the electronic device (601) may include or correspond to at least a part of the electronic device (101) of FIG. 1.

[0165] According to one embodiment, the external data management module (625) can improve the accuracy of the plurality of recognition modules by using the data (640) acquired from the electronic device (601). For example, the external data management module (625) can correct the data (or gesture information) acquired from the plurality of recognition modules by using the data (640) acquired from the electronic device (601). Depending on the embodiment, the external data management module (625) may not be included in the recognition service layer (620). Depending on the embodiment, the external data management module (625) may be included in the recognition abstraction layer (612).

[0166] The external data management module (625) can receive data (640) included in a UIBC (user input back channel) message from the electronic device (601). The data (640) is a Wi-Fi display standard (Wi-Fi TMAccording to the display specification, the data (640) may be encapsulated in a message (or data packet). The wearable device (101) may recognize the data (640) as an input to the wearable device (101). The wearable device (101) may execute a function corresponding to the data (640). The data (640) may include data related to a touch input to the display of the electronic device (601). For example, the data (640) may include a coordinate value of a point where an external object is in contact with the display of the electronic device (601). The external data management module (625) may provide the data (640) to the recognition service layer (620). For example, the recognition service layer (620) may provide the data (640) to the recognition abstraction layer (612), thereby enabling the data (640) to be utilized within the wearable device (101). For example, the wearable device (101) can recognize data (640) as input to the wearable device (101).

[0167] According to one embodiment, the wearable device (101) may give priority to data of the external data management module (625) among data (e.g., data (640)) of the external data management module (625) and data acquired by the recognition plug-in layer (622). When recognizing data (640) as input of the wearable device (101), the recognition abstraction layer (612) may refrain from receiving or bypass data acquired by the recognition plug-in layer (622). The wearable device (101) may recognize data (640) among data acquired by the recognition plug-in layer (622) and data (640) as input of the wearable device (101) based on the external data management module (625) receiving data (640). The service module (621) can select data (640) from among data acquired by the recognition plug-in layer (622) and data (640).

[0168] The sensor service layer (630) may be used to control at least one sensor (e.g., a camera, an IMU, a time of flight (TOF) sensor). For example, the sensor service layer (630) may be used to provide a service for accessing at least one sensor. For example, the sensor service layer (630) may include at least one of a module for a VR service (e.g., QVRservice), a module for an XR service (e.g., SxrSensorService), a sensor API (e.g., android sensor API), and a sensor hardware abstraction layer (sensor HAL).

[0169] According to one embodiment, the sensor management module (623) can provide sensor data to the recognition plug-in layer (622) through a common interface. For example, the sensor management module (623) can provide sensor data to each of a plurality of recognition modules (e.g., a head tracking module (622-1), an environment recognition module (622-2), a hand tracking module (622-3), an eye tracking module (622-4), and a face tracking module (622-5)) through the same interface. For example, the sensor management module (623) can provide sensor data according to the operation of the recognition module to the recognition module without changing the configuration information of the recognition plug-in layer (622) based on changing (or modifying) the configuration information (e.g., a configuration file) regarding the sensor management module (623).

[0170] The sensor management module (623) can identify sensor data for at least one recognition module based on the operation of at least one recognition module among a plurality of recognition modules. The sensor management module (623) can provide the identified sensor data to the at least one recognition module.

[0171] According to one embodiment, when the head tracking module (622-1) is driven, the sensor management module (623) may obtain camera data and IMU data through at least one of a module for VR service, a module for XR service, a sensor API, and a sensor hardware abstraction layer in the sensor service layer (630). The sensor management module (623) may provide the camera data and IMU data to the head tracking module (622-1). Depending on the embodiment, the camera data and the IMU data may be obtained through different modules.

[0172] In one embodiment, when the environment recognition module (622-2) is operated in playback mode, the sensor management module (623) can identify stored camera data and stored posture data. The sensor management module (623) can provide the camera data and posture data to the environment recognition module (622-2).

[0173] In one embodiment, the service module (621) may be configured to eliminate dependency on a higher layer of the recognition plugin layer (622). For example, the higher layer of the recognition plugin layer (622) may include a platform layer (610) (e.g., Android XR) and / or an application layer (e.g., the application layer (440) of FIG. 4).

[0174] The service module (621) can manage input data of the wearable device (101). The service module (621) can be configured to integrate and manage information (e.g., gesture information or tracking data) obtained from multiple recognition modules. The service module (621) can convert information (e.g., gesture information or tracking data) according to the requirements of a higher layer without changing the multiple recognition modules, and then provide the converted information to the higher layer.

[0175] For example, the service module (621) can obtain information on a 6-degree-of-freedom posture from the head tracking module (622-1). The information on the 6-degree-of-freedom posture obtained from the head tracking module (622-1) can be configured in a quaternion format. On the other hand, a higher layer (e.g., the platform layer (610)) can request information on the 6-degree-of-freedom posture configured in an axis-angle representation format. The service module (621) can change (or convert) the information on the 6-degree-of-freedom posture configured in a quaternion format into information on the 6-degree-of-freedom posture configured in an axis-angle representation format. The service module (621) can provide the information on the 6-degree-of-freedom posture configured in an axis-angle representation format to the higher layer (e.g., the platform layer (610)). However, the present invention is not limited thereto. For example, the service module (621) can change (or convert) information about a 6-degree-of-freedom pose configured in an axis-angle representation format into information about a 6-degree-of-freedom pose configured in a quaternion format and provide it to an upper layer.

[0176] For example, the service module (621) can obtain information about hand movements from the hand tracking module (622-3). The information about hand movements can be obtained based on the movements of a first number of joints. On the other hand, a higher layer (e.g., the platform layer (610)) can request information about hand movements obtained based on the movements of a second number of joints. The service module (621) can perform either a joint interpolation procedure or a simplification procedure. The service module (621) can support the structure of the joints required by the higher layer based on performing either a joint interpolation procedure or a simplification procedure.

[0177] According to one embodiment, the wearable device (101) may include an immersive device that blocks the user's line of sight when worn by the user. For example, the wearable device (101) may include a head mounted device (HMD). For example, a user wearing the wearable device (101) may have limited or difficult touch input to a display (e.g., display (250)). A user wearing the wearable device (101) may control the wearable device (101) using gesture input and / or gaze input. The frequency of use of gesture input in the wearable device (101) may be greater than the frequency of use of touch input in the display (250). The number of functions executable by touch input in the wearable device (101) may be less than the number of functions executable by gesture input in the wearable device (101). The frequency of use of gaze input in the wearable device (101) may be greater than the frequency of use of touch input for the display (250). The number of functions executable by touch input in the wearable device (101) may be less than the number of functions executable by gaze input in the wearable device (101).

[0178] The wearable device (101) can use data (640) received from the electronic device (601) using an external data management module (625). The wearable device (101) can be controlled by data (640) using the external data management module (625). The data (640) can include coordinate data of a touch input acquired by the electronic device (601). The wearable device (101) can execute a function according to the coordinate data of the touch input acquired by the electronic device (601) using the external data management module (625). When the data (640) includes coordinate data of a touch input acquired by the electronic device (601), the number of functions that can be executed by the touch input of the wearable device (101) is relatively small, which may cause inconvenience to the user. A method of using data (640) including gesture input data and / or gaze input data may be required to control relatively many functions of a wearable device (101).

[0179] In the present disclosure, a technology may be described in which a wearable device (101) is controlled by multimodal input (e.g., gesture input and / or gaze input) obtained from an electronic device (601). Data (640) obtained through the electronic device (601) may be obtained by using a camera (e.g., a camera module, an image sensor) of the electronic device (601) and / or a sensor (e.g., a sensor module, a motion sensor) of the electronic device (601). As a non-limiting example, data (640) may be generated by performing preprocessing on data obtained by using the camera of the electronic device (601) and / or the sensor of the electronic device (601). For example, data (640) may include movement data (or movement information) of an external object (e.g., a part of the body, a hand, a pupil). For example, data (640) may include voice data acquired through a microphone of the electronic device (601).

[0180] According to an embodiment of the present disclosure, a wearable device (101) can execute a method of utilizing data (640) including multimodal input using an external data management module (625). For example, the wearable device (101) can execute a function according to data (640) including multimodal input. For example, the wearable device (101) can display a visual object according to the data (640) on a display (250). This method will be described and exemplified in FIGS. 7 to 12 .

[0181] FIG. 7 illustrates an example of a guide mode of a wearable device (101) displaying visual objects corresponding to data (e.g., data included in a message (720)) of an electronic device (e.g., electronic device (601)) according to one embodiment of the present disclosure. The wearable device (101) of FIG. 7 may be an example of the electronic device (101) of FIG. 1 and the wearable devices (101) of FIGS. 2A, 3A, and 3B.

[0182] Referring to FIG. 7, the electronic device (701) may be one of various forms of mobile devices, such as a laptop, smartphones with various form factors (e.g., bar-type smartphones, foldable smartphones, or rollable smartphones), tablets, cellular phones, and other similar computing devices. However, the present invention is not limited thereto. The electronic device (701) may include a fixed electronic device, such as a desktop computer or a TV. For example, the electronic device (701) may be referred to as a user device, a multi-function device, or a portable device. For example, the electronic device (701) may be an example of the electronic device (101) of FIG. 1. The electronic device (701) may include a communication circuit (not shown), a camera (702) (e.g., including at least a portion of a camera module (180)), a display (703) (e.g., including at least a portion of a display module (160)), a memory (e.g., memory (130)), and a processor (e.g., processor (120)).

[0183] Referring to FIG. 7, the wearable device (101) can display a screen (730) via the display (250). The screen (730) can represent at least a portion of a three-dimensional space. For example, the screen (730) can include an executable object (or visual object) (e.g., an application). The screen (730) can provide a virtual space (or virtual reality) to the user of the wearable device (101).

[0184] The wearable device (101) can execute a designated function based on receiving an input for executing the function. For example, the input may include a gesture input and / or a gaze input, but is not limited thereto. For example, the wearable device (101) can receive a gesture input by executing a gesture tracker (e.g., a gesture tracker (473)) based on data (e.g., hand tracking data) acquired through a camera (e.g., an image sensor (421)). For example, the wearable device (101) can receive a gaze input by executing a gaze tracker (474) based on data (e.g., eye tracking data) acquired through a camera (e.g., an image sensor (421)).

[0185] The wearable device (101) can transmit or provide mirroring data (710) to the electronic device (701) via the communication circuit (430). The mirroring data (710) can be described as data for mirroring (or streaming) a screen (e.g., screen (730), screen (830) of FIG. 8) displayed on the display (250) of the electronic device (101) to the electronic device (701). The mirroring data (710) can be described as data that causes a screen (740) to be displayed on the display (703) of the electronic device (701). The electronic device (701) can display the screen (740) on the display (703) based on receiving the mirroring data (710).

[0186] Screen (730) may correspond to screen (740). For example, screen (740) may be identical to or similar to screen (730). For example, the size of screen (740) may be different from that of screen (730). For example, screen (740) may be identical to or similar to at least a portion of screen (730). For example, screen (740) may include some of the plurality of window pop-ups within screen (730). For example, screen (740) may include some of the plurality of contents within screen (730).

[0187] The electronic device (701) can transmit a message (720) including input data (e.g., data (640)) to the wearable device (101) through a communication circuit. The wearable device (101) can receive the message (720) including the input data from the electronic device (701) through the communication circuit (430). The wearable device (101) can recognize the input data in the message (720) as input to the wearable device (101). The wearable device (101) can recognize the input data in the message (720) as input to the wearable device (101) by using an external data management module (e.g., an external data management module (625)). The wearable device (101) being controlled by input data within a message (720) received from an external electronic device (701) may be referred to as a UIBC (user input back channel). The wearable device (101) may be referred to as a source device. The electronic device (701) may be referred to as a sink device.

[0188] According to one embodiment, the electronic device (701) can receive a touch input (or tap input) on the display (703). Based on receiving the touch input through the display (703), the electronic device (701) can transmit a message (720) including coordinate data of a contact point to the wearable device (101). Based on receiving the message (720) including the coordinate data through the communication circuit (430), the wearable device (101) can recognize the coordinate data as an input to the wearable device (101). The wearable device (101) can execute the same function as the function of a touch input (or tap input) for a point within the display (250) corresponding to the coordinate data.

[0189] According to an embodiment of the present disclosure, an electronic device (701) can capture an external object (721) (e.g., a part of a body) through a camera (702). The electronic device (701) can capture a moving external object (721) through the camera (702) while displaying a screen (740) through a display (703). The electronic device (701) can acquire images representing the movement of the external object (721). The electronic device (701) can transmit a message (720) including images representing the movement of the external object (721) to a wearable device (101) through a communication circuit.

[0190] The wearable device (101) can obtain movement data of an external object (721) using at least some of the images in the received message (720). For example, the external object (721) may be a hand of a user of the electronic device (701). The movement data may be referred to as hand tracking data. The wearable device (101) can identify a gesture input based on the movement data. For example, the external object (721) may be one or more eyes of the user. The movement data may be referred to as eye tracking data. The wearable device (101) can identify a gaze input based on the movement data. The gesture input and / or the gaze input may be referred to as user input.

[0191] The wearable device (101) can display a visual object (731) corresponding to the movement data of the external object (721) through the display (250). The mode in which the wearable device (101) displays a visual object (731) corresponding to the movement data of the external object (721) through the display (250) can be referred to as a guide mode. The visual object (731) can be included in the screen (730). The visual object (731) can include a visual guide. For example, the visual object (731) can be represented as a visual object to assist a user input of a user wearing the wearable device (101). The visual object (731) can guide a user input corresponding to the movement data in the message (720) to the user of the wearable device (101). The wearable device (101) may execute a function corresponding to movement data of an external object (721) based on receiving a user input corresponding to a visual object (731). For example, the function may be the same as a function that the user of the electronic device (701) intends to execute through the movement of the external object (721).

[0192] According to one embodiment, when the movement data of the external object (721) corresponds to the user's hand moving in one direction, the visual object (731) may indicate the one direction. For example, the visual object (731) may include an indicator pointing in the direction in which the external object (721) moves. The wearable device (101) may identify a part of the body (e.g., the user's hand) moving in the one direction through a camera (e.g., the image sensor (421)) while the visual object (731) is displayed. The wearable device (101) may change the screen (730) displayed on the display (250) based on the identification of the part of the body moving in the one direction. For example, an executable object within the screen (730) may move in the one direction. For example, a background within the screen (730) may move in the one direction. However, the present invention is not limited thereto.

[0193] In FIG. 7, the visual object (731) is illustrated as a visual object indicating a direction, but this is merely an example. The visual object (731) may have a different shape than that illustrated in FIG. 7. The shape of the visual object (731) may vary depending on the movement data of the external object (721) within the message (720). According to one embodiment, if the movement data of the external object (721) within the message (720) corresponds to a user's hand performing a pressing action (or tapping action) within the space, the wearable device (101) may display a visual object highlighting an executable object (e.g., an application) within the screen (730). For example, the visual object highlighting an executable object may include a pointer indicating the executable object.

[0194] According to an embodiment of the present disclosure, while the wearable device (101) provides a guide mode, the wearable device (101) may display a screen (730) including a visual object (731) that assists a user of the wearable device (101) based on images in a message (720). The display of the visual object (731) may be controlled by an external object (721). With the assistance of the external object (721), the user of the wearable device (101) may execute a function of the wearable device (101) that the user was not aware of. The wearable device (101) may enhance the user experience by executing a function of the wearable device (101) that the user was not aware of.

[0195] The wearable device (101) can obtain movement data of an external object (721) based on providing data (e.g., images) within a message (720) from an external data management module (e.g., an external data management module (625)) to a service module (e.g., a service module (621)). For example, the wearable device (101) can identify user input (e.g., a gesture input, a gaze input) using the service module (621).

[0196] In one embodiment, the wearable device (101) can use the recognition abstraction layer (612) to identify user input (e.g., gesture input, gaze input) from images within a message (720). The wearable device (101) can use the recognition abstraction layer (612) to obtain movement data of external objects (721) within the images.

[0197] According to one embodiment, the wearable device (101) may provide movement data of an external object (721) to a virtual space manager (e.g., the virtual space manager (451) of FIG. 4, the virtual space manager (550) of FIG. 5, CPM). The wearable device (101) may provide the movement data of the external object (721) from the virtual space manager to a lightweight rendering engine (e.g., the lightweight rendering engine (543) of FIG. 5). The lightweight rendering engine (543) may be referred to as an impress engine. The wearable device (101) may display a visual object (731) corresponding to the movement data of the external object (721) through the display (250) using the lightweight rendering engine (543). The wearable device (101) may provide data of the visual object (731) generated through the lightweight rendering engine (543) to an application that provides a system UI (e.g., the system UI (530)). A visual object (731) may be an example of a system UI (530). The wearable device (101) may display a visual object (731) through the display (250) according to the execution of an application providing the system UI (530). The visual object (731) may be referred to as a guide user interface. For example, the visual object (731) may include shapes such as arrows, circles, squares, and stars. For example, the visual object (731) may be composed of dotted lines and / or solid lines.

[0198] According to one embodiment, while the wearable device (101) provides a guide mode, the wearable device (101) may display a screen (730) including a visual object (732). The visual object (732) may be described as a visual object for indicating to a user of the wearable device (101) that the guide mode is being provided. For example, the visual object (732) may include text. For example, the visual object (732) may change color. For example, the visual object (732) may be displayed at the border of the screen (730). For example, the visual object (732) may not include text and may be an object that changes color.

[0199] According to one embodiment, the wearable device (101) may provide a visitor mode according to the user's settings. The visitor mode may be described as a mode for users who are not registered within the wearable device (101). The visitor mode may be referred to as a mode for users who have no experience operating the wearable device (101). The visitor mode may be referred to as a guest mode. While providing the visitor mode, the wearable device (101) may mirror (or stream) the screen (730) to the electronic device (701) according to the user's input. For example, the electronic device (701) may be owned by a visitor (e.g., a user who has no experience operating the wearable device (101) or a user who is not registered within the wearable device (101). The wearable device (101) may, while providing a visitor mode, display a screen (730) including visual objects (731) that assist a visitor of the wearable device (101) based on images (e.g., included in a message (720)) acquired through the visitor's electronic device (e.g., electronic device (701)).

[0200] FIG. 8 illustrates an example of a control mode of a wearable device that executes a function corresponding to data (e.g., a command included in a message (720)) of an electronic device (e.g., the electronic device (601) of FIG. 6 or the electronic device (701) of FIG. 7) according to one embodiment of the present disclosure. The wearable device (101) of FIG. 8 may be an example of the electronic device (101) of FIG. 1 or the wearable devices (101) of FIGS. 2A to 3B.

[0201] Referring to FIG. 8, the wearable device (101) can display a screen (830) through the display (250). The screen (830) can include an image (831). The wearable device (101) can mirror (or stream) the screen (830) to an electronic device (701). To perform mirroring, the wearable device (101) can transmit mirroring data (710) to the electronic device (701) through a communication circuit (e.g., a communication circuit (430)). For the mirroring data (710), reference may be made to the descriptions of the mirroring data (710) of FIG. 7.

[0202] The electronic device (701) can display a screen (840) through the display (703). The screen (840) can correspond to the screen (830). For example, the screen (840) can be identical to or similar to the screen (830). For example, the screen (840) can be identical to a reduced screen (830). For example, the screen (840) can include an image corresponding to the image (831). For the screen (830) and the screen (840), reference can be made to the descriptions of the screen (730) and the screen (740) of FIG. 7.

[0203] The electronic device (701) can obtain movement data of an external object (721) through a camera (702). The electronic device (701) can identify a gesture input based on the movement data of the external object (721). The electronic device (701) can identify a function of the wearable device (101) corresponding to the identified gesture input. The electronic device (701) can transmit a message (720) including a control command (e.g., data (640)) for executing a function of the wearable device (101) according to the gesture input to the wearable device (101) through a communication circuit (not shown). The control command can cause the wearable device (101) that receives the control command to execute a function according to the control command. For example, the control command can include a gesture input.

[0204] In one embodiment, the message (720) may include a control command. The message (720) may indicate that the control command corresponds to a user input (e.g., a gesture input, a gaze input) obtained through the camera (702) of the electronic device (701). For example, indicating that the control command in the message (720) is obtained through the camera (702) will be described and exemplified in more detail with reference to FIG. 11.

[0205] The wearable device (101) can receive a message (720) including a control command according to an external object (721) through the communication circuit (430). The wearable device (101) can execute a function corresponding to the control command based on receiving the message (720) through the communication circuit (430). The mode in which the wearable device (101) executes the function according to the control command in the message (720) can be referred to as a control mode. For example, if the movement data of the external object (721) indicates the external object (721) moving in one direction, the control command can cause content (e.g., an image (831), a pop-up window, a user interface) to move in one direction within the screen (830) of the wearable device (101). The wearable device (101) can move an image (831) in the above direction within the screen (830) based on receiving a message (720) including a control command. However, the present disclosure is not limited thereto.

[0206] In one embodiment, while the wearable device (101) provides a control mode, the wearable device (101) may display a screen (830) including a visual object (832). The visual object (832) may indicate to a user of the wearable device (101) that the control mode is being provided. The visual object (832) may indicate to a user of the wearable device (101) that the wearable device (101) is being controlled according to a control command received from the electronic device (701) (e.g., a control command included in a message (720). For example, the visual object (832) may include text. As a non-limiting example, the visual object (832) may not include text, but may be represented by an object (e.g., an icon) whose color changes.

[0207] According to one embodiment, the electronic device (701) can capture images by capturing an environment including users through a camera (702). The electronic device (701) can identify a user occupying the largest area within the images. Based on the images acquired through the camera (702), the electronic device (701) can identify the user's hand (e.g., external object (721)) occupying the largest area. The electronic device (701) can assign an identifier to a visual object corresponding to the external object (721) within the images. The identifier can be referred to as a hand ID (identifier). The electronic device (701) can obtain movement data of the identified hand using the images. The electronic device (701) can transmit a message (720) including the movement data to the wearable device (101) through a communication circuit (not shown).

[0208] According to one embodiment, the electronic device (701) may obtain images including a plurality of hands through the camera (702). For example, the plurality of hands may be two. The electronic device (701) may assign a first identifier to a first hand based on the images. The electronic device (701) may assign a second identifier to a second hand based on the images. The identifiers (e.g., the first identifier and the second identifier) ​​may be referred to as hand IDs. The electronic device (701) may obtain coordinate information of the first hand and movement information of the second hand using the images. For example, the coordinate information of the first hand may include coordinates indicating a path of a pointer corresponding to the first hand. For example, the coordinate information of the first hand may be referred to as movement information of a pointer corresponding to the first hand. For example, the coordinate information of the first hand may include movement information of a point pointed by a finger of the first hand. For example, the movement information of the second hand may include a gesture input according to the movement data of the second hand.

[0209] The electronic device (701) can transmit a message (720) including coordinate information of a first hand and movement information of a second hand to the wearable device (101) through a communication circuit (not shown). The wearable device (101) can execute a function corresponding to the coordinate information of the first hand and the movement information of the second hand in the received message (720). For example, the wearable device (101) can move a pointer (not shown) in a screen (830) according to the coordinate information of the first hand. For example, the wearable device (101) can identify a gesture corresponding to the movement information of the second hand. For example, the wearable device (101) can execute a function corresponding to the gesture. For example, the wearable device (101) can cause the pointer to be positioned on content (e.g., a visual object, an image (831)) according to the coordinate information of the first hand. The wearable device (101) can cause content to be selected or moved using a gesture based on movement information of the second hand while the pointer is positioned on the content.

[0210] FIG. 9 illustrates an example of operations performed between a wearable device and an electronic device to display a visual object (e.g., a visual object (731) of FIG. 7) corresponding to data (e.g., images included in a message (720) of FIG. 7) obtained from an electronic device (e.g., an electronic device (601) of FIG. 6) according to one embodiment of the present disclosure on a display of the wearable device (e.g., a display (250) of FIG. 2A).

[0211] Referring to FIG. 9, in operation 901, the wearable device (101) (e.g., the processor (410)) may transmit or provide mirroring data (e.g., the mirroring data (710)) to the electronic device (701) through a communication circuit (e.g., the communication circuit (430)) while displaying a screen (e.g., the screen (730)) through a display assembly (e.g., an assembly including the display (250)). The screen (730) may represent at least a portion of a three-dimensional space. The mirroring data (710) may cause a screen corresponding to the screen of the wearable device (101) to be displayed through a display (e.g., the display (703)) of the electronic device (701). The electronic device (701) may receive the mirroring data (710) from the wearable device (101).

[0212] In operation 903, the wearable device (101) (e.g., the processor (410)) may display a screen (e.g., the screen (730)) representing at least a portion of a three-dimensional space through a display assembly. The display assembly may include a display (e.g., the display (250)). In FIG. 9, operation 903 is depicted as being performed after operation 901, but this is merely an example. For example, operation 903 may be performed before operation 901. For example, while the wearable device (101) is performing operation 903, operations 901, 909, 911, and 913 may be performed.

[0213] In operation 905, the electronic device (701) can display a screen (e.g., screen (740)) according to the mirroring data (710) through a display (e.g., display (703)). By displaying the screen (740) according to the mirroring data (710) through the display (703), the user of the wearable device (101) can share a user experience with the user of the electronic device (701).

[0214] In operation 907, the electronic device (701) may acquire images including an external object (e.g., an external object (721)) through a camera (e.g., a camera (702)) while the screen (740) is displayed. For example, the images may represent movements of the external object (721). For example, the images may include gesture inputs expressed by the external object (721). For example, the images may include gaze inputs expressed by the external object (721).

[0215] In operation 909, the electronic device (701) can transmit a message including images (e.g., message (720)) to the wearable device (101) via a communication circuit (not shown) of the electronic device (701). The wearable device (101) can receive the message (720) from the electronic device (701) via the communication circuit (430). The wearable device (101) can receive the message (720) from the electronic device (701) via the communication circuit (430) while displaying the screen (730) via the display assembly. The wearable device (101) can receive the message (720) (or images within the received message (720)) from the electronic device (701) via the communication circuit (430) while providing mirroring data (710) to the electronic device (701). The wearable device (101) may provide the received message (720) (or images within the received message (720)) to an external data management module of the wearable device (101) (e.g., the external data management module (625) of FIG. 6). For example, the images within the message (720) may represent the movement of an external object (721).

[0216] According to one embodiment, the wearable device (101) may receive a message (720) from the electronic device (701) via a communication channel. The communication channel may be referred to as a user input back channel (UIBC). The message (720) may be transmitted via transmission control protocol / internet protocol (TCP / IP).

[0217] In operation 911, the wearable device (101) (e.g., processor (410)) can identify a user input based on images in a message (720). The wearable device (101) can provide the images from an external data management module (625) to a recognition service layer (e.g., recognition service layer (470), recognition service layer (620)). The wearable device (101) can identify the user input by processing the images in a service module (e.g., service module (621)) in the recognition service layer. The wearable device (101) can provide the identified user input to a virtual space manager (e.g., virtual space manager (550) of FIG. 5) from the recognition service layer (620) through a recognition abstraction layer (e.g., recognition abstraction layer (460), recognition abstraction layer (612)).

[0218] In one embodiment, the wearable device (101) can process images in a message (720) using the recognition abstraction layer (612). For example, the wearable device (101) can obtain movement data of an external object (721) in the images using the recognition abstraction layer (612). For example, the external object (721) may be a hand of a user of the electronic device (701). The wearable device (101) can identify a gesture input from the images using the recognition abstraction layer (612). As a non-limiting example, the external object may be one or more eyes of the user. The wearable device (101) can identify a gaze input from the images using the recognition abstraction layer (612). The wearable device (101) can provide the identified user input (e.g., gesture input, gaze input) using the recognition abstraction layer (612) to the virtual space manager (550).

[0219] In operation 913, the wearable device (101) (e.g., the processor (410)) can display a visual object (e.g., the visual object (731)) corresponding to the user input through the display assembly. The wearable device (101) can provide the user input to a lightweight rendering engine (e.g., the lightweight rendering engine (543)) through the virtual space manager (550). The wearable device (101) can use the lightweight rendering engine (543) to render the visual object (731) corresponding to the user input. The visual object (731) can assist the operation of a user whose field of vision is blocked. By displaying the visual object (731), the wearable device (101) can guide an operation method to a user who has no experience in operating the wearable device (101). A user of a wearable device (101) can learn how to operate the device more quickly when there is a visual object (731) than when there is no visual object (731). Since the wearable device (101) is controlled by the user of the electronic device (701) while mirroring the screen, the user of the wearable device (101) can share a user experience with the user of the electronic device (701).

[0220] FIG. 10 illustrates an example of capability negotiation performed between a wearable device and an electronic device according to an embodiment of the present disclosure. The capability negotiation of FIG. 10 may be performed between the wearable device (101) and the electronic device (701) prior to operation 901 of FIG. 9 . For example, the capability negotiation illustrated in FIG. 10 may be referred to as UIBC (user input back channel) capability negotiation. The capability negotiation may be referred to as a setup process (or configuration process) for the electronic device (701) to transmit data obtained from the electronic device (701) (e.g., data included in a message (720)) to the wearable device (101). The capability negotiation may be performed to determine a set of parameters to be used in subsequent communication processes between the wearable device (101) and the electronic device (701). During capability negotiation, the wearable device (101) may determine whether to recognize a user input (e.g., gesture input, gaze input) received from the electronic device (701) as input for the wearable device (101). The capability negotiation may be performed to exchange information used to transmit and receive content between the wearable device (101) and the electronic device (701). The information used to transmit and receive content may be used to ensure compatibility between the wearable device (101) and the electronic device (701). The information used to transmit and receive content may be used to perform optimal operations between the wearable device (101) and the electronic device (701). For example, the information used to transmit and receive content may include a video compression method and / or an audio compression method. For example, the video compression method may include a video codec such as H.264 and / or HEVC (high efficiency video coding). For example, the audio compression scheme may include an audio codec such as advanced audio coding (AAC).After the wearable device (101) and the electronic device (701) perform the operations illustrated in FIG. 10, the data obtained from the electronic device (701) can be used in the wearable device (101).

[0221] Referring to FIG. 10, in operation 1001, a wearable device (101) (e.g., a processor (410)) may transmit a first parameter request message to an electronic device (701) through a communication circuit (e.g., a communication circuit (430)). The first parameter request message may be a Wi-Fi display standard (Wi-Fi TM In the display specification, the wearable device (101) may be referred to as an M3 request. The wearable device (101) may transmit a first parameter request message to the electronic device (701) to obtain (or determine) a list of parameters representing the capabilities of the electronic device (701). The first parameter request message may include a request to transmit data of a list of parameters supported by the electronic device (701).

[0222] In operation 1003, the electronic device (701) may transmit a first parameter response message to the wearable device (101) (e.g., processor (410)) via a communication circuit. The first parameter response message may be a Wi-Fi Display Standard (Wi-Fi TMIn the display specification, the M3 response may be referred to. The electronic device (701) may transmit a first parameter response message to the wearable device (101) in response to the first parameter request message. The first parameter response message may include a response to the parameters specified in the first parameter request message. For example, the first parameter response message may include 'input category list= HIDC (human interface device class)', 'generic cap list= none', 'HIDC cap list= mouse / BT (Bluetooth TM )', 'RemoteControl / infrared', and 'port=none'. For example, the first parameter response message may include responses to parameters such as 'input category list= generic', 'generic cap list= mouse', 'singletouch', 'HIDC cap list= none', and 'port=none'.

[0223] In operation 1005, the wearable device (101) (e.g., processor (410)) may transmit a second parameter request message to the electronic device (701) via a communication circuit (e.g., communication circuit (430)). The second parameter request message may be in accordance with the Wi-Fi Display Standard (Wi-Fi TMIn the display specification, the wearable device (101) may be referred to as an M4 request and / or an M14 request. The wearable device (101) may transmit a second parameter request message to the electronic device (701) to determine (or set) values ​​of parameters of the electronic device (701). The second parameter request message may include values ​​of parameters to be used in a subsequent process. The types of the parameters may be based on the first parameter response message. For example, the second parameter request message may include 'input category list= HIDC', 'generic cap list= none', 'HIDC cap list= mouse / BT (Bluetooth TM )', 'RemoteControl / infrared', and 'port=1000'. For example, the second parameter request message may include responses to parameters such as 'input category list= generic', 'generic cap list= mouse', 'singletouch', 'HIDC cap list= none', and 'port=1000'.

[0224] In operation 1007, the electronic device (701) may transmit a second parameter response message to the wearable device (101) (e.g., the processor (410)) via the communication circuit. The second parameter response message may be referred to as an M4 response or an M14 response in the Wi-Fi display specification. The second parameter response message may indicate whether setting values ​​of parameters of the electronic device (701) according to the second parameter request message is successful.

[0225] FIG. 11 illustrates an example of a message (e.g., message (720)) transmitted from an electronic device (e.g., electronic device (701)) according to one embodiment of the present disclosure.

[0226] The message (1110) illustrated in FIG. 11 may include the messages illustrated in FIGS. 7, 8, 9, and 12.

[0227] Referring to FIG. 11, a message (1110) may include a version (1121), a timestamp flag (1122), reserved (1123), an input category (1124), a length (1125), a timestamp (1126), and an input body (1130). The message (1110) may be referred to as a data packet.

[0228] The version (1121) may indicate the version of a specific communication protocol being implemented in an electronic device (e.g., electronic device (701)). The version (1121) may be represented by a 3-bit field. The electronic device (701) may be referred to as a sink device.

[0229] The timestamp flag (1122) may indicate the presence or absence of a timestamp (1126). The timestamp (1126) in the message (1110) may be optional. If the timestamp (1126) is present, the timestamp flag (1122) may contain "1." If the timestamp (1126) is not present, the timestamp flag (1122) may contain "0." The timestamp flag (1122) may be represented as a 1-bit field.

[0230] Reserved (1123) may be represented as an 8-bit field that does not contain information used when extracting data (e.g., payload data) from a message (1110). Reserved (1123) may be used to extract data from a message (1110) according to the data of the version (1121). Reserved (1123) may include additional data without changing the format of the message (1110).

[0231] The input category (1124) can be described as a 4-bit field for identifying the input category according to the input data included in the input body (1130). The input category (1124) can indicate the format type of the input body (1130).

[0232] A wearable device (e.g., wearable device (101)) can extract data from an input body (1130) using data indicated by an input category (1124). The wearable device (101) can be referred to as a source device.

[0233] The length (1125) can be represented as a 16-bit field indicating the length of the message (1110). The wearable device (101) can identify the end of the message (1110) using the data indicated by the length (1125).

[0234] The timestamp (1126) may be represented as a 16-bit field for identifying the frame of the screen being displayed on the electronic device (701) when the user input data of the input body (1130) within the message (1110) was acquired.

[0235] The input body (1130) may include data representing user input (e.g., gesture input, gaze input) obtained from the electronic device (701). The input body (1130) may be referred to as payload data. The input body (1130) may include images obtained through a camera (e.g., camera (702)) of the electronic device (701). The input body (1130) may include control commands generated based on the images.

[0236] The input body (1130) may be referred to as a HIDC (human interface device class) input body. The format of the HIDC input body may be referred to in [Table 1], [Table 2], and [Table 3] below.

[0237]

[0238]

[0239]

[0240] Table 1 may represent the format of the HIDC input body. Table 2 may represent the path of the user input acquired by the electronic device (701). Table 3 may represent the type of the user input acquired by the electronic device (701). According to one embodiment, if the input body (1130) in the message (1110) includes images acquired using a camera (e.g., camera (702)), the value of the HID Type may represent 'camera' or 'reserved'. For example, if the input body (1130) in the message (1110) includes the images, the value indicated by the HID Type may be '5'. For example, if the input body (1130) in the message (1110) includes the images, the value indicated by the HID Type may be a designated value. The designated value may indicate the images.

[0241] According to one embodiment, if the input body (1130) within the message (1110) includes a control command, the value of HID Type may indicate 'reserved'. For example, if the input body (1130) within the message (1110) includes a control command, the value indicated by HID Type may be a designated value. The designated value may indicate the control command.

[0242] FIG. 12 illustrates an example of operations of a wearable device (e.g., a wearable device (101) of FIG. 2A) executing a function according to a control command (e.g., a control command included in a message (720) of FIG. 7) of an electronic device (e.g., an electronic device (701) of FIG. 7, an electronic device (601) of FIG. 6) according to one embodiment of the present disclosure.

[0243] Referring to FIG. 12, in operation 1201, a wearable device (101) (e.g., a processor (410)) may transmit mirroring data (e.g., mirroring data (710)) for a screen provided through a display assembly to an electronic device (701) through a communication circuit (e.g., a communication circuit (430)). Operation 1201 may correspond to operation 901 of FIG. 9.

[0244] In operation 1203, the wearable device (101) may receive a message (e.g., message (720)) including a control command from the electronic device (701) through the communication circuit (430) while providing mirroring data (710) to the electronic device (701). The control command may be generated in the electronic device (701). The control command may be generated based on images including an external object (e.g., external object (721)) acquired through a camera (e.g., camera (702)) of the electronic device (701). The electronic device (701) may identify a user input (e.g., gesture input, gaze input) using at least some of the images. The electronic device (701) may generate a control command based on the user input.

[0245] According to one embodiment, a control command may be generated in an electronic device (701) using images including an external object (721) acquired through a camera (702). The control command may be described as data based on user input identified from the images. Since the control command is data obtained by processing images in the electronic device (701), the wearable device (101) may utilize the control command with relatively little computation.

[0246] According to one embodiment, the control command may include movement information of a pointer. The pointer may be described as an identifier corresponding to an external object (721). The pointer may be referred to as a cursor, an identifier, and / or an indicator. The pointer may be expressed as a dot on a screen provided through a display (e.g., display (250)) of the wearable device (101), but is not limited thereto. The pointer may be expressed as a visual object corresponding to a hand. The pointer may not be displayed on the screen. The movement information of the pointer may correspond to movement data of the external object (721) in the images. The movement information of the pointer may be referenced as a coordinate displacement value. The wearable device (101) may cause the pointer to move according to the movement information of the pointer on the screen.

[0247] The message (720) may indicate that the control command corresponds to a user input obtained through the camera (702) of the electronic device (701). The electronic device (701) may set the HID Type in the message (720) to a specified value. The wearable device (101) may use the value of the HID type in the message (720) to identify that the control command corresponds to a user input obtained through the camera (702).

[0248] In operation 1205, the wearable device (101) can execute a function according to a control command. The wearable device (101) can execute a function according to the control command based on providing the control command from an external data management module (e.g., an external data management module (625)) to a service module (e.g., a service module (621)).

[0249] According to one embodiment, when a control command includes pointer movement information, the wearable device (101) may execute a function according to the pointer movement information on the screen while displaying the screen through the display (250). For example, the wearable device (101) may cause the screen to change in a direction according to the pointer movement information. For example, the wearable device (101) may move a visual object within the screen in a direction according to the pointer movement information. For example, the wearable device (101) may identify a gesture input corresponding to the pointer movement information. For example, the wearable device (101) may execute a function mapped to the gesture input.

[0250] In an embodiment according to the present disclosure, a wearable device (e.g., electronic device (101)) may receive a message (e.g., message (720)) from the electronic device (701) while mirroring (or streaming) a screen (e.g., screen (730)) to the electronic device (e.g., electronic device (601), electronic device (701)). The wearable device (101) may identify a user input (e.g., gesture input) based on images in the message. The images may be acquired through a camera (e.g., camera (702)) of the electronic device (701). The wearable device (101) may display a visual object (e.g., visual object (731)) based on the user input. The visual object (731) may guide a user who has no (or lacks) experience in operating the wearable device (101) on an operating method. A user of a wearable device (101) can easily learn how to operate the wearable device (101) by operating the wearable device according to a visual object (731). A user of an electronic device (701) can control the wearable device (101) based on images acquired through a camera (702) of the electronic device (701). A user of an electronic device (701) can share a user experience of the wearable device (101).

[0251] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0252] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0253] The various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include any one of the items listed together in the corresponding phrase among the phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish the corresponding element from other corresponding elements and do not limit the corresponding elements in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0254] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0255] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101) of FIG. 1). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0256] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0257] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0258] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0259] As described above, a wearable device (e.g., the electronic device (101) of FIG. 1, the wearable devices (101) of FIGS. 2A, 2B, 3A, 3B, and 4, the electronic device (101) of FIG. 5, and the wearable devices (101) of FIGS. 7 to 10) may include a display assembly including at least one display (e.g., the display (250)). The wearable device may include communication circuitry (e.g., the communication circuitry (430)). The wearable device may include a memory (e.g., the memory (415)) that stores instructions and includes one or more storage media. The wearable device may include at least one processor (e.g., the processor (410)) that includes processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to transmit, through the communication circuit, mirroring data (e.g., mirroring data (710)) for a screen (e.g., screen (730)) representing at least a portion of a three-dimensional space provided through the display assembly, to an electronic device (e.g., electronic device (701)). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive, through the communication circuit, a message (e.g., message (720), message (1110)) including images acquired through a camera (e.g., camera (702)) of the electronic device while providing the mirroring data to the electronic device so that the electronic device displays a screen (e.g., screen (740)) according to the mirroring data.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a user input based on movement of an external object (e.g., external object (721)) within the images within the message. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display assembly, a visual object (e.g., visual object (731)) corresponding to the user input.

[0260] In one embodiment, the images may represent the gaze of one or more eyes.

[0261] In one embodiment, the images may represent gestures for parts of the user's body.

[0262] In one embodiment, the visual object may include an indicator pointing in the direction in which the external object within the images is moving.

[0263] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify the user input based on providing the images in the message from an external data management module of the wearable device (e.g., an external data management module (625)) to a service module (e.g., a service module (621)) that manages input data of the wearable device.

[0264] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to display the visual object through the display assembly by providing the user input to a lightweight rendering engine of the wearable device via a virtual space manager of the wearable device.

[0265] In one embodiment, the message may include UIBC (user input back channel) data. A value indicated by the HID (human interface device) type of the UIBC data may represent the images including the external object.

[0266] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to transmit, to the electronic device via the communication circuit, a first message for UIBC (user input back channel) capability negotiation prior to transmitting the mirroring data to the electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to establish a connection between the wearable device and the electronic device based on receiving, from the electronic device via the communication circuit, a second message indicating a response to the first message prior to transmitting the mirroring data to the electronic device.

[0267] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the visual object corresponding to the user input through the display assembly. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to execute a function corresponding to the movement of the external object within the images based on receiving another input corresponding to the visual object while displaying the visual object.

[0268] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display assembly, the visual object corresponding to the user input. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display assembly, another visual object representing a guide mode for displaying the visual object based on the images.

[0269] A method performed in a wearable device (e.g., the electronic device (101) of FIG. 1, the wearable device (101) of FIGS. 2A, 2B, 3A, 3B, and FIG. 4, the electronic device (101) of FIG. 5, and the wearable device (101) of FIGS. 7 to 10) having a display assembly including at least one display (e.g., the display (250)) and a communication circuit (e.g., the communication circuit (430)) as described above may include an operation of transmitting mirroring data (e.g., the mirroring data (710)) for a screen (e.g., the screen (730)) representing at least a part of a three-dimensional space provided through the display assembly to the electronic device (e.g., the electronic device (701)) through the communication circuit. The method may include an operation of receiving, from the electronic device through the communication circuit, a message (e.g., message (720), message (1110)) including images acquired through a camera (e.g., camera (702)) of the electronic device while providing the mirroring data to the electronic device so that the electronic device displays a screen (e.g., screen (740)) according to the mirroring data. The method may include an operation of identifying, based on the images in the message, a user input according to a movement of an external object (e.g., external object (721)) in the images. The method may include an operation of displaying, through the display assembly, a visual object (e.g., visual object (731)) corresponding to the user input.

[0270] In one embodiment, the images may represent the gaze of one or more eyes.

[0271] In one embodiment, the images may represent gestures for parts of the user's body.

[0272] In one embodiment, the visual object may include an indicator pointing in the direction in which the external object within the images is moving.

[0273] According to one embodiment, the method may include an action of identifying the user input based on providing the images within the message from an external data management module of the wearable device (e.g., an external data management module (625)) to a service module (e.g., a service module (621)) that manages input data of the wearable device.

[0274] In one embodiment, the method may include displaying the visual object through the display assembly by providing the user input to a lightweight rendering engine of the wearable device through a virtual space manager of the wearable device.

[0275] In one embodiment, the message may include UIBC (user input back channel) data. A value indicated by the HID (human interface device) type of the UIBC data may represent the images including the external object.

[0276] According to one embodiment, the method may include an operation of transmitting, to the electronic device via the communication circuit, a first message for UIBC (user input back channel) capability negotiation before transmitting the mirroring data to the electronic device. The method may include an operation of establishing a connection between the wearable device and the electronic device based on receiving, from the electronic device via the communication circuit, a second message indicating a response to the first message before transmitting the mirroring data to the electronic device.

[0277] In one embodiment, the method may include an operation of displaying the visual object corresponding to the user input through the display assembly. The method may include an operation of executing a function corresponding to the movement of the external object within the images based on receiving another input corresponding to the visual object while displaying the visual object.

[0278] In one embodiment, the method may include an operation of displaying the visual object corresponding to the user input through the display assembly. The method may include an operation of displaying another visual object, through the display assembly, indicating a guide mode for displaying the visual object based on the images.

[0279] In a computer-readable storage medium having one or more programs stored thereon as described above, the one or more programs may include instructions that cause a wearable device (e.g., the electronic device (101) of FIG. 1, the wearable device (101) of FIGS. 2A, 2B, 3A, 3B, and FIG. 4, the electronic device (101) of FIG. 5, and the wearable device (101) of FIGS. 7 to 10) to transmit mirroring data (e.g., the mirroring data (710)) for a screen (e.g., the screen (730)) representing at least a part of a three-dimensional space provided through the display assembly to an electronic device (e.g., the electronic device (701)) through the communication circuit when the wearable device is executed. The one or more programs may include instructions that cause the wearable device, when executed by the wearable device, to receive, from the electronic device through the communication circuit, a message (e.g., message (720), message (1110)) including images acquired through a camera (e.g., camera (702)) of the electronic device while providing the mirroring data to the electronic device so as to display a screen (e.g., screen (740)) according to the mirroring data. The one or more programs may include instructions that cause the wearable device, when executed by the wearable device, to identify a user input according to a movement of an external object (e.g., external object (721)) within the images based on the images within the message. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display a visual object (e.g., visual object (731)) corresponding to the user input through the display assembly.

[0280] In one embodiment, the images may represent the gaze of one or more eyes.

[0281] In one embodiment, the images may represent gestures for parts of the user's body.

[0282] In one embodiment, the visual object may include an indicator pointing in the direction in which the external object within the images is moving.

[0283] According to one embodiment, the one or more programs may include instructions that cause the wearable device to identify the user input based on providing the images in the message from an external data management module of the wearable device (e.g., an external data management module (625)) to a service module (e.g., a service module (621)) that manages input data of the wearable device when executed by the wearable device.

[0284] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object through the display assembly by providing the user input to a lightweight rendering engine of the wearable device via a virtual space manager of the wearable device.

[0285] In one embodiment, the message may include UIBC (user input back channel) data. A value indicated by the HID (human interface device) type of the UIBC data may represent the images including the external object.

[0286] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to transmit, to the electronic device via the communication circuit, a first message for UIBC (user input back channel) capability negotiation before transmitting the mirroring data to the electronic device. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to establish a connection between the wearable device and the electronic device based on receiving, from the electronic device via the communication circuit, a second message indicating a response to the first message before transmitting the mirroring data to the electronic device.

[0287] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object corresponding to the user input through the display assembly. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to execute a function corresponding to the movement of the external object within the images based on receiving another input corresponding to the visual object while displaying the visual object.

[0288] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display, through the display assembly, the visual object corresponding to the user input. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display, through the display assembly, another visual object representing a guide mode for displaying the visual object based on the images.

[0289] As described above, an electronic device (e.g., electronic device 701) may include a display (e.g., display 703). The electronic device may include a camera (e.g., camera 702). The electronic device may include communication circuitry. The electronic device may include a memory that stores instructions and includes one or more storage media. The electronic device may include at least one processor (e.g., processor 120) that includes processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive, through the communication circuitry, mirroring data (e.g., mirroring data 710) for a screen (e.g., screen 730) displayed on a wearable device (e.g., wearable device 101)) from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a screen (e.g., screen (740)) according to the received mirroring data through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to acquire, through the camera, images including an external object (e.g., external object (721)) while the screen (e.g., screen (740)) for the mirroring data is displayed on the electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit, through the communication circuit, a message (e.g., message (720), message (1110)) including the images to the wearable device. The images within the message may be used to display a visual object based on the images on the wearable device.

[0290] In one embodiment, the images may represent the gaze of one or more eyes.

[0291] In one embodiment, the images may represent gestures for parts of the user's body.

[0292] In one embodiment, the message may include UIBC (user input back channel) data. A value indicated by the HID (human interface device) type of the UIBC data may represent the images including the external object.

[0293] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive, from the electronic device through the communication circuitry, a first message for UIBC (user input back channel) capability negotiation prior to receiving the mirroring data from the wearable device through the communication circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to establish a connection between the electronic device and the wearable device based on transmitting, to the wearable device through the communication circuitry, a second message indicating a response to the first message prior to receiving the mirroring data from the wearable device through the communication circuitry.

[0294] As described above, a method performed in an electronic device (e.g., electronic device 701) having a display (e.g., display (703)), a camera (e.g., camera (702)), and a communication circuit may include an operation of receiving, from a wearable device, mirroring data (e.g., mirroring data (710)) for a screen (e.g., screen (730)) displayed on the wearable device through the communication circuit. The method may include an operation of displaying, through the display, a screen (e.g., screen (740)) according to the received mirroring data. The method may include an operation of acquiring, through the camera, images including an external object (e.g., external object (721)) while the screen (e.g., screen (740)) for the mirroring data is displayed on the electronic device. The method may include an operation of transmitting, through the communication circuit, a message (e.g., message (720), message (1110)) including the images to the wearable device. The images within the above message can be used to display visual objects based on the images on the wearable device.

[0295] In one embodiment, the images may represent the gaze of one or more eyes.

[0296] In one embodiment, the images may represent gestures for parts of the user's body.

[0297] In one embodiment, the message may include UIBC (user input back channel) data. A value indicated by the HID (human interface device) type of the UIBC data may represent the images including the external object.

[0298] According to one embodiment, the method may include an operation of receiving, from the electronic device through the communication circuit, a first message for UIBC (user input back channel) capability negotiation before receiving the mirroring data from the wearable device through the communication circuit. The method may include an operation of establishing a connection between the electronic device and the wearable device based on transmitting, to the wearable device through the communication circuit, a second message indicating a response to the first message before receiving the mirroring data from the wearable device through the communication circuit.

[0299] In a computer-readable storage medium having one or more programs stored thereon as described above, the one or more programs, when executed by an electronic device having a display (e.g., display (703)), a camera (e.g., camera (702)), and a communication circuit, may include instructions that cause the electronic device, when executed by the electronic device, to receive mirroring data (e.g., mirroring data (710)) for a screen (e.g., screen (730)) displayed on the wearable device from a wearable device through the communication circuit. The one or more programs may include instructions that cause the electronic device, when executed by the electronic device, to display a screen (e.g., screen (740)) according to the received mirroring data through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to acquire, through the camera, images including an external object (e.g., external object (721)) while a screen (e.g., screen (740)) for the mirroring data is displayed on the electronic device. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to transmit, through the communication circuit, a message (e.g., message (720), message (1110)) including the images to the wearable device. The images in the message may be used to display a visual object based on the images on the wearable device.

[0300] In one embodiment, the images may represent the gaze of one or more eyes.

[0301] In one embodiment, the images may represent gestures for parts of the user's body.

[0302] In one embodiment, the message may include UIBC (user input back channel) data. A value indicated by the HID (human interface device) type of the UIBC data may represent the images including the external object.

[0303] In one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to receive, from the electronic device through the communication circuit, a first message for UIBC (user input back channel) capability negotiation before receiving the mirroring data from the wearable device through the communication circuit. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to establish a connection between the electronic device and the wearable device based on transmitting, to the wearable device through the communication circuit, a second message indicating a response to the first message before receiving the mirroring data from the wearable device through the communication circuit.

[0304] As described above, a wearable device may include a display assembly including at least one display. The wearable device may include communication circuitry. The wearable device may include a memory storing instructions and including one or more storage media. The wearable device may include at least one processor including processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to transmit, to an electronic device through the communication circuitry, mirroring data for a screen representing at least a portion of a three-dimensional space provided through the display assembly. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive, from the electronic device through the communication circuitry, a message including a control command while providing the mirroring data to the electronic device so that the electronic device displays a screen according to the mirroring data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to perform a function in accordance with the control command within the message. The message may indicate that the control command corresponds to a user input obtained through a camera of the electronic device.

[0305] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to execute the function in accordance with the control command based on providing the control command within the message from an external data management module of the wearable device to a service module that manages input data of the wearable device.

[0306] In one embodiment, the message may include UIBC (user input back channel) data. A value indicated by the HID (human interface device) type of the UIBC data may represent the control command.

[0307] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to transmit, to the electronic device via the communication circuit, a first message for UIBC (user input back channel) capability negotiation prior to transmitting the mirroring data to the electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to establish a connection between the wearable device and the electronic device based on receiving, from the electronic device via the communication circuit, a second message indicating a response to the first message prior to transmitting the mirroring data to the electronic device.

[0308] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to perform the function in accordance with the control command within the message. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, via the display assembly, a visual object indicating a control mode for performing the function in accordance with the control command.

[0309] In one embodiment, the shape of a visual object may change based on motion data included in the message. The motion data may correspond to the movement of an external object.

[0310] In one embodiment, when the motion data of the external object indicates that the external object is moving in one direction, the control command may cause content to move in the one direction within the screen.

[0311] A method performed in a wearable device having a display assembly including at least one display and a communication circuit as described above may include an operation of transmitting, to an electronic device through the communication circuit, mirroring data for a screen representing at least a portion of a three-dimensional space provided through the display assembly. The method may include an operation of receiving, from the electronic device through the communication circuit, a message including a control command while providing the mirroring data to the electronic device so that the electronic device displays a screen according to the mirroring data. The method may include an operation of executing a function according to the control command in the message. The message may indicate that the control command corresponds to a user input acquired through a camera of the electronic device.

[0312] According to one embodiment, the method may include an operation of executing the function according to the control command based on providing the control command in the message from an external data management module of the wearable device to a service module that manages input data of the wearable device.

[0313] In one embodiment, the message may include UIBC (user input back channel) data. A value indicated by the HID (human interface device) type of the UIBC data may represent the control command.

[0314] According to one embodiment, the method may include an operation of transmitting, to the electronic device via the communication circuit, a first message for UIBC (user input back channel) capability negotiation before transmitting the mirroring data to the electronic device. The method may include an operation of establishing a connection between the wearable device and the electronic device based on receiving, from the electronic device via the communication circuit, a second message indicating a response to the first message before transmitting the mirroring data to the electronic device.

[0315] In one embodiment, the method may include an action of executing the function according to the control command within the message. The method may include an action of displaying, through the display assembly, a visual object indicating a control mode for executing the function according to the control command.

[0316] In one embodiment, the shape of a visual object may change based on motion data included in the message. The motion data may correspond to the movement of an external object.

[0317] In one embodiment, when the motion data of the external object indicates that the external object is moving in one direction, the control command may cause content to move in the one direction within the screen.

[0318] In a computer-readable storage medium having one or more programs stored thereon, as described above, the one or more programs may include instructions that, when executed by a wearable device having a display assembly including at least one display and a communication circuit, cause the wearable device to transmit, through the communication circuit, mirroring data for a screen representing at least a portion of a three-dimensional space provided through the display assembly, to an electronic device. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to receive, through the communication circuit, a message including a control command from the electronic device while providing mirroring data to the electronic device so that a screen according to the mirroring data is displayed on the electronic device. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to execute a function according to the control command in the message. The message may indicate that the control command corresponds to a user input acquired through a camera of the electronic device.

[0319] According to one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to execute the function according to the control command based on providing the control command in the message from an external data management module of the wearable device to a service module that manages input data of the wearable device.

[0320] In one embodiment, the message may include UIBC (user input back channel) data. A value indicated by the HID (human interface device) type of the UIBC data may represent the control command.

[0321] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to transmit, to the electronic device via the communication circuit, a first message for UIBC (user input back channel) capability negotiation before transmitting the mirroring data to the electronic device. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to establish a connection between the wearable device and the electronic device based on receiving, from the electronic device via the communication circuit, a second message indicating a response to the first message before transmitting the mirroring data to the electronic device.

[0322] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to execute the function in accordance with the control command within the message. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display, through the display assembly, a visual object indicating a control mode for executing the function in accordance with the control command.

[0323] In one embodiment, the shape of a visual object may change based on motion data included in the message. The motion data may correspond to the movement of an external object.

[0324] In one embodiment, when the motion data of the external object indicates that the external object is moving in one direction, the control command may cause content to move in the one direction within the screen.

[0325] It will be appreciated that the various embodiments of the present disclosure, as described in the claims and specification, may be implemented in hardware form, software form, or a combination of hardware and software form.

[0326] Such software may be stored on non-transitory computer-readable storage media. The non-transitory computer-readable storage media store one or more computer programs (software modules), wherein the one or more computer programs include computer-executable instructions that, when executed alone or collectively by one or more processors of the electronic device, cause the electronic device to perform the method of the present disclosure.

[0327] Such software may be stored in a form of volatile or non-volatile storage. For example, it may be stored in a storage device such as read-only memory (ROM), whether erasable or rewritable, or in a memory form such as random access memory (RAM), memory chips, devices, or integrated circuits (ICs), or in an optical or magnetic readable medium such as a compact disc (CD), a digital versatile disc (DVD), a magnetic disk, or a magnetic tape. It will be appreciated that such storage devices and storage media are various embodiments of non-transitory machine-readable storage media suitable for storing a computer program or a plurality of computer programs comprising instructions that, when executed, implement various embodiments of the present disclosure. Accordingly, various embodiments of the present disclosure provide a program comprising code for implementing an apparatus or method according to any one of the claims of the present specification, and a non-transitory machine-readable storage medium storing such a program.

[0328] Although the present disclosure has been described with reference to various embodiments, it will be apparent to those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present disclosure.

Claims

1. In a wearable device (101), A display assembly comprising at least one display (250); Communication circuit (430); A memory (415) storing instructions and including one or more storage media; and At least one processor (410) comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor (410), cause the wearable device (101) to: Transmitting mirroring data (710) for a screen (730) representing at least a part of a three-dimensional space provided through the display assembly to an electronic device (701) through the communication circuit (430), While providing the mirroring data (710) to the electronic device (701) so that the screen (740) according to the mirroring data (710) is displayed on the electronic device (701), a message (720, 1110) including images acquired through the camera (702) of the electronic device (701) is received from the electronic device (701) through the communication circuit (430), Based on the images included in the above message (720, 1110), user input is identified based on the movement of an external object (721) within the images, and Causing the display assembly to display a visual object (731) corresponding to the user input; Wearable device (101).

2. In claim 1, The above images represent the gaze of one or more eyes of the user. Wearable device (101).

3. In claim 1, The above images represent gestures for parts of the user's body. Wearable device (101).

4. In claim 1, The above visual object (731) includes an indicator pointing to the direction of movement of the external object (721) identified according to the images. Wearable device (101).

5. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (410), cause the wearable device (101) to: Based on providing the images included in the message (720, 1110) from the external data management module (625) of the wearable device (101) to the service module (621) managing the input data of the wearable device (101), causing the user input to be identified. Wearable device (101).

6. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (410), cause the wearable device (101) to: By providing the user input to the lightweight rendering engine (543) of the wearable device (101) through the virtual space manager (550) of the wearable device (101), thereby causing the visual object (731) to be displayed through the display assembly, Wearable device (101).

7. In claim 1, The above message (720, 1110) includes UIBC (user input back channel) data, and The value indicated by the HID (human interface device) type of the above UIBC data represents the images including the external object (721). Wearable device (101).

8. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (410), cause the wearable device (101) to: Before transmitting the above mirroring data (710) to the electronic device (701): Transmitting a first message (720, 1110) for UIBC (user input back channel) capability negotiation to the electronic device (701) through the communication circuit (430), and Causing to establish a connection between the wearable device (101) and the electronic device (701) based on receiving a second message (720, 1110) representing a response to the first message (720, 1110) from the electronic device (701) through the communication circuit (430). Wearable device (101).

9. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (410), cause the wearable device (101) to: Displaying the visual object (731) corresponding to the user input through the display assembly, and Causing a function corresponding to the movement of the external object (721) within the images to be executed based on receiving another input corresponding to the visual object (731) while displaying the visual object (731). Wearable device (101).

10. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (410), cause the wearable device (101) to: Displaying the visual object (731) corresponding to the user input through the display assembly, and Causing another visual object (732) to be displayed through the display assembly, indicating a guide mode for displaying the visual object (731) based on the above images; Wearable device (101).

11. In the electronic device (701), display (703); Camera (702); communication circuit; A memory storing instructions and including one or more storage media; and At least one processor (120) comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (701) to: Receive mirroring data (710) for a screen (730) displayed on the wearable device (101) from the wearable device (101) through the communication circuit, Displaying a screen (740) according to the mirroring data (710) received from the wearable device through the display (703), While the screen (740) according to the mirroring data (710) is displayed on the display of the electronic device (701), images including an external object (721) are acquired through the camera (702), and Causing a message (720, 1110) including the above images to be transmitted to the wearable device (101) through the communication circuit, and The images included in the above message (720, 1110) are used to display a visual object (731) based on the images in the wearable device (101). Electronic device (701).

12. In claim 11, The above images represent the gaze of one or more eyes of the user. Electronic device (701).

13. In claim 11, The above images represent gestures for parts of the user's body. Electronic device (701).

14. In claim 11, The above message (720, 1110) includes UIBC (user input back channel) data, and The value indicated by the HID (human interface device) type of the above UIBC data represents the images including the external object (721). Electronic device (701).

15. In claim 11, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (701) to: Before receiving the above mirroring data (710) from the wearable device (101) through the communication circuit: Receive a first message (720, 1110) for UIBC (user input back channel) capability negotiation from the electronic device (701) through the communication circuit, and Causing a connection to be established between the electronic device (701) and the wearable device (101) based on transmitting a second message (720, 1110) representing a response to the first message (720, 1110) to the wearable device (101) through the communication circuit. Electronic device (701).

Images