Wearable device for identifying user input by using one or more sensors, and method therefor

Abstract

A processor of a wearable device, according to one embodiment, can identify a first input for a point, spaced apart from all of a plurality of virtual objects, by using a first sensor from among a plurality of sensors. The at least one processor can: identify at least one virtual object adjacent to the point from among the plurality of virtual objects in response to a second input for the point, identified on the basis of the first sensor within a designated period after the first input is identified; and display, on a display, a visual object that can be identified by a second sensor from among the plurality of sensors and indicates a third input corresponding to a function of the at least one virtual object.

Description

Wearable device and method for identifying user input using one or more sensors

[0001] The present disclosure relates to a wearable device and method for identifying user input using one or more sensors.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] According to an embodiment, a wearable device may include a plurality of sensors, a display, a memory for storing instructions, and at least one processor for executing the instructions. The at least one processor may be configured to identify a first input to a point spaced apart from all of a plurality of virtual objects using a first sensor among the plurality of sensors. The at least one processor may be configured to identify, within a designated period of time after identifying the first input, at least one virtual object adjacent to the point among the plurality of virtual objects in response to a second input to the point identified based on the first sensor. The at least one processor may be configured to display, on the display, a visual object representing a third input identifiable by a second sensor among the plurality of sensors and corresponding to a function of the at least one virtual object.

[0005] In one embodiment, a method of a wearable device may include an operation of identifying a first input for a point spaced from all of a plurality of virtual objects using a first sensor among a plurality of sensors of the wearable device. The method may include an operation of identifying at least one virtual object adjacent to the point among the plurality of virtual objects in response to a second input for the point, the second input being identified based on the first sensor within a specified period of time after identifying the first input. The method may include an operation of displaying, on a display of the wearable device, a visual object representing a third input identifiable by a second sensor among the plurality of sensors and corresponding to a function of the at least one virtual object.

[0006] In one embodiment, a wearable device may include one or more sensors, a display, a memory, and at least one processor. The at least one processor may be configured to determine, based on identifying a first input for a first point in a virtual space provided through the display using the one or more sensors, whether a virtual object has been identified at the first point. The at least one processor may be configured to, in a first state in which the first virtual object has been identified at the first point, execute a first function assigned to the first virtual object and indicated by the first input. The at least one processor may be configured to, in a second state different from the first state, identify a second virtual object positioned at a second point in the virtual space adjacent to the first point, based on identifying a second input for the first point. The at least one processor may be configured to display, on the display, a visual object including text for invoking a second function executable based on the second virtual object.

[0007] In one embodiment, a method of a wearable device may include an operation of identifying a first input for a first point in a virtual space provided through a display of the wearable device using one or more sensors of the wearable device, and obtaining whether the virtual object has been identified at the first point based on the identification of the first input. The method may include an operation of executing a first function assigned to the first virtual object and indicated by the first input within a first state in which the first virtual object has been identified at the first point. The method may include an operation of identifying a second virtual object disposed at a second point in the virtual space adjacent to the first point based on the identification of a second input for the first point within a second state different from the first state. The method may include an operation of displaying a visual object on the display including text for invoking a second function executable based on the second virtual object.

[0008] FIG. 1 illustrates an exemplary user interface (UI) provided by a wearable device according to one embodiment.

[0009] FIGS. 2A and 2B illustrate an example block diagram of a wearable device, according to one embodiment.

[0010] FIG. 3 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0011] Figure 4 illustrates an exemplary operation of a wearable device for identifying user input.

[0012] FIG. 5 illustrates a UI displayed by a wearable device that identifies user input based on hand gestures.

[0013] FIG. 6 illustrates a UI displayed by a wearable device that identifies user input based on an external electronic device such as a remote controller.

[0014] Figure 7 illustrates a UI displayed by a wearable device that identifies user input based on gaze.

[0015] FIG. 8 illustrates a UI displayed by a wearable device that identifies user input based on hand gestures.

[0016] FIG. 9 illustrates a UI displayed by a wearable device that identifies user input based on hand gestures.

[0017] FIG. 10 illustrates a UI displayed by a wearable device that identifies user input based on hand gestures.

[0018] FIG. 11 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0019] FIG. 12 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0020] FIG. 13A illustrates an example of a perspective view of a wearable device, according to one embodiment.

[0021] FIG. 13b illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.

[0022] FIGS. 14A and 14B illustrate an example of an appearance of a wearable device according to one embodiment.

[0023] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0024] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0025] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0026] FIG. 1 illustrates an exemplary user interface (UI) provided by a wearable device (101), according to one embodiment. The wearable device (101) may include a head-mounted display (HMD) that is wearable on the head of a user (110). The wearable device (101) may be referred to as a head-mounted device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) or visible see-through device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of a hardware configuration included in the wearable device (101) is exemplarily described with reference to FIG. 2A and / or FIG. 2B. An example of the structure of a wearable device (101) that can be worn on the head of a user (110) is described with reference to FIGS. 13A, 13B, 14A, and / or 14B. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) for attaching to the head of a user to form an HMD.

[0027] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) may see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0028] In one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include a housing that covers the eyes of the user (110). The wearable device (101), in this state, may include a display disposed on a first side of the housing facing the eyes. The wearable device (101) may include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (101) may acquire images and / or videos representing ambient light. The wearable device (101) can output the image and / or video within the display disposed on the first surface, thereby allowing the user (110) to recognize the ambient light through the display. The displaying area (or active area) of the display disposed on the first surface can be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object into the image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with a real object recognized by the ambient light.

[0029] Referring to FIG. 1, an embodiment of a wearable device (101) that provides an interactive user interface (UI) and / or user experience (UX) to a user (110) is illustrated. The wearable device (101) can identify a motion (or action) of a user (110), including a hand gesture of the user (110) (e.g., a gesture performed by a hand (112). Based on the motion of the user (110), the wearable device (101) can estimate the intention of the user (110) with respect to the motion. Based on the repeatedly identified motions, the wearable device (101) can display a UI for checking the intention related to the motions.

[0030] Referring to FIG. 1, a wearable device (101) can display a screen (130) to a user (110) wearing the wearable device (101). In one embodiment of the wearable device (101) displaying a screen (130) based on VST, the wearable device (101) can display a screen (130) including a visual object (132) corresponding to a real object (e.g., a speaker (120)). The visual object (132) can be a portion of an image and / or video identified through a camera facing the front direction of the wearable device (101) (e.g., the image sensor (230-1) of FIG. 2A and / or FIG. 2B). The wearable device (101) can display a screen (130) including the image and / or the video, thereby allowing a user (110) wearing the wearable device (101) to recognize the surrounding environment.

[0031] In a state where a visual object (132) corresponding to a speaker (120), which is a real object, is displayed, the wearable device (101) may display a virtual object (virtual object and / or imaginary object) (140) within the screen (130). The virtual object (140) may include a graphical object provided by a software application (hereinafter, “application”) executed by the wearable device (101). The virtual object (140) may have a two-dimensional panel (or square, rectangular parallelepiped) shape, such as a window (or activity) provided by the software application. The embodiment is not limited thereto, and the virtual object (140) may have a three-dimensional shape based on three-dimensional rendering. Referring to FIG. 1, a virtual object (140) in the form of a calendar provided by a software application (e.g., a calendar application) that supports a calendar-related function is illustrated as an example, but the embodiment is not limited thereto.

[0032] Referring to FIG. 1, the wearable device (101) can provide a user (110) with a mixed reality of a real object (e.g., a speaker (120)) and a virtual object (140) by substantially simultaneously displaying a screen (130) including a virtual object (140) and a visual object (132). Based on the mixed reality, the wearable device (101) can support spatial computing. The user (110) viewing the screen (130) can obtain information that cannot be identified through external space by using the virtual object (140) having the form of a calendar. For example, in an external space that only includes a speaker (120), the wearable device (101) can provide the user (110) with a user experience similar to viewing a calendar by using the virtual object (140) within the screen (130). The wearable device (101) can execute at least one function supported by a software application corresponding to the virtual object (140) based on a user input related to the virtual object (140).

[0033] For example, the wearable device (101) can remotely control the speaker (120) based on a screen (130) that includes a visual object (132) corresponding to the speaker (120), which is a real object. For example, even if the user (110) does not directly operate the speaker (120), the wearable device (101) can wirelessly transmit a signal related to the user input to the speaker (120) based on identifying the user input related to the visual object (132). In one embodiment, the wearable device (101) can display an interactable object (e.g., a virtual object (140) and / or a visual object (132)).

[0034] According to one embodiment, a wearable device (101) may, while displaying an interactable object, execute at least one function corresponding to the interactable object in response to a user input. An example of a hardware configuration of a wearable device (101) for identifying a user input is described with reference to FIG. 2A and / or FIG. 2B. An operation of the wearable device (101) identifying a user input and executing at least one function related to the interactable object is described with reference to FIG. 3. Types (e.g., types and / or categories) of user inputs identifiable by the wearable device (101) are described with reference to FIG. 4.

[0035] In one embodiment, the wearable device (101) may, while displaying a plurality of interactable objects, such as virtual objects (140) and / or visual objects (132) within a screen (130), execute a function of one of the plurality of interactable objects based on a user input. The wearable device (101) that identifies a pointing gesture performed by a hand (112) of pointing in a direction (h1) with an index finger may execute a function of one of the virtual objects (140) and / or visual objects (132) located in the direction (h1). Referring to FIG. 1, the wearable device (101) that identifies a hand (112) may display a virtual object (150) corresponding to the hand (112) within the screen (130). The wearable device (101) that has identified the pointing gesture may display a virtual object (150) having a shape corresponding to the pointing gesture on the screen (130) and display a line (152) extending from the virtual object (150) along the direction (h1) of the pointing gesture. The line (152) may be referred to as a ray (or hand ray). The embodiment is not limited thereto, and the line (152) may be referred to as a cursor and / or a pointer.

[0036] Within the exemplary state of FIG. 1, the wearable device (101) that has identified a pointing gesture can identify a user input for selecting one of the objects (e.g., a virtual object (140) and / or a visual object (132)) displayed on the screen (130) based on the direction (h1) of the pointing gesture. If any one object located in the direction (h1) is identified, the wearable device (101) can execute a function mapped to the identified object and the entire pointing gesture. For example, if the direction (h1) of the pointing gesture is identified toward a virtual object (140) that supports different functions for different hand gestures, the wearable device (101) can execute a specific function of the virtual object (140) corresponding to the pointing gesture. If no object is positioned in the above direction (h1), the wearable device (101) may not execute any function assigned to objects within the screen (130).

[0037] A user (110) may perform the pointing gesture to interact with any of the objects (e.g., virtual objects (140) and / or visual objects (132)) displayed on the screen (130). If the user (110) incorrectly performs the pointing gesture, the wearable device (101) may, in response to the pointing gesture, perform a function different from the user's (110) intention, or may not perform any function. If a function different from the user's (110) intention is performed, the user (110) may additionally perform a user input to cancel the execution of the function. If no function is performed, the user (110) may perform the pointing gesture again. According to one embodiment, the wearable device (101) may perform an action to more accurately identify the intention of the user (110) based on a user input to cancel execution of a function and / or repeatedly performed user inputs (particularly, identified while not executing any function).

[0038] Referring to FIG. 1, a wearable device (101) that identifies a pointing gesture directed toward a direction (h1) and / or a point (v1) away from all objects displayed on a screen (130) (e.g., a virtual object (140) and / or a visual object (132)) may not perform any function because the pointing gesture does not overlap with any object. When the wearable device (101) additionally identifies a pointing gesture based on the direction (h1) and / or the point (v1), the wearable device (101) may request the user (110) to perform a motion different from the pointing gesture in order to more accurately identify the user's (110) intention. For example, despite repeatedly performing pointing gestures, the wearable device (101) may guide the user (110) to perform a type of motion different from the pointing gesture in a state in which no function is executed. For example, the wearable device (101) may display visual objects (162, 164) within the screen (130) that guide voice input based on pointing gestures that are repeatedly identified and do not correspond to any function.

[0039] Referring to FIG. 1, the wearable device (101) may display visual objects (162, 164) corresponding to each of the interactable objects (e.g., the virtual object (140) and / or the visual object (132)). The visual objects (162, 164) are illustrated as having bubble shapes and displayed at points within the screen (130) adjacent to each of the virtual object (140) and the visual object (132), but the embodiment is not limited thereto. The wearable device (101) may display text (e.g., "Turn on the music") related to an executable function associated with a real object (e.g., a speaker (120)) corresponding to the visual object (132) within the visual object (162) displayed adjacent to the visual object (132). The wearable device (101) may display text (e.g., “Register for schedule”) for calling a function supported by the virtual object (140) within a visual object (164) displayed adjacent to the virtual object (140).

[0040] Using visual objects (162, 164), the wearable device (101) can recommend a voice input to the user (110) as a user input instead of a hand gesture, such as a pointing gesture. While the visual objects (162, 164) are displayed, the wearable device (101) can identify the user's (110) utterance through an audio sensor, such as a microphone. By performing speech-to-text (STT) on the utterance, the wearable device (101) can obtain a text corresponding to the utterance. Based on a natural language sentence represented by the text, the wearable device (101) can identify a voice input represented by the utterance. For example, the wearable device (101) can perform voice recognition by executing a function for processing or analyzing a natural language sentence (e.g., natural language understating (NLU)). The wearable device (101) can execute a function corresponding to a voice input identified based on voice recognition. For example, a user (110) viewing visual objects (162, 164) through a screen (130) can read text included in the visual object (164), such as “register schedule.” The wearable device (101) that identifies the text based on an audio sensor can execute a function (e.g., register schedule) of a virtual object (140). Similarly, the wearable device (101) that identifies a voice input corresponding to text included in a visual object (162), such as “turn on music,” can control a speaker (120) corresponding to the visual object (162) to execute a function (e.g., play music) corresponding to the text.

[0041] As described above, according to one embodiment, the wearable device (101) can identify a first input to a point (v1) spaced apart from all of a plurality of objects (e.g., virtual objects (140) and / or visual objects (132)) using a first sensor (e.g., an image sensor for identifying a hand (112)). Based on the first input, the wearable device (101) may not perform any function. Within a designated period of time after identifying the first input, in response to a second input to the point (v1) identified based on the first sensor, the wearable device (101) can display a visual object (e.g., visual objects (162, 164)) related to a function of at least one virtual object among the plurality of objects. In one embodiment, the wearable device (101) may display a visual object representing a third input (e.g., a voice input) identifiable by a second sensor (e.g., an audio sensor including one or more microphones) different from the first sensor using the visual object. Using the visual object, the wearable device (101) may recommend a third input identifiable by a second sensor distinct from the first sensor, thereby causing the user (110) to perform a user input related to the second sensor. Using the second sensor, the wearable device (101) may more accurately identify the intention of the user (110) who performed the first input and / or the second input.

[0042] Below, an example of a hardware configuration included in a wearable device (101) to perform the operations described with reference to FIG. 2a and / or FIG. 2b is described.

[0043] FIGS. 2A and 2B illustrate an example of a block diagram of a wearable device (101), according to one embodiment. The wearable device (101) described with reference to FIG. 1 may include at least one of the hardware components of the wearable device (101) distinguished by different blocks in FIGS. 2A and / or 2B.

[0044] Referring to FIG. 2A, a wearable device (101) according to one embodiment may include at least one of a processor (210), a memory (215), a display (220), a sensor (230), and a communication circuit (235). The processor (210), the memory (215), the display (220), the sensor (230), and / or the communication circuit (235) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (202). The type and / or number of hardware components included in the wearable device (101) is not limited to those illustrated in FIG. 2A. For example, the wearable device (101) may include only some of the hardware components illustrated in FIG. 2A.

[0045] According to one embodiment, the processor (210) of the wearable device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. The processor (210) may have a multi-core processor structure, such as a dual core, a quad core, a hexa core, and / or an octa core.

[0046] The memory (215) of the wearable device (101) according to one embodiment may include a hardware component for storing data and / or instructions input to and / or output from the processor (210). The memory (215) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC). In one embodiment, the memory (215) may be referred to as storage.

[0047] In one embodiment, the display (220) of the wearable device (101) can output visualized information (e.g., the screen (130) of FIG. 1) to a user of the wearable device (101) (e.g., the user (110) of FIG. 1). For example, the display (220) can be configured to visualize information provided from a processor (210) including a circuit such as a graphic processing unit (GPU). The display (220) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). Embodiments are not limited thereto, and for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (220) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (220) may be referred to as a display panel and / or a display module.

[0048] According to one embodiment, a sensor (230) of a wearable device (101) may generate electrical information from non-electronic information related to the wearable device (101), which may be processed and / or stored by a processor (210) and / or a memory (215) of the wearable device (101). The information may be referred to as sensor data. The sensor (230) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor (or a ToF camera) for detecting a geographic location of the wearable device (101), an inertial measurement unit (IMU) (e.g., an acceleration sensor, a geomagnetic sensor, a gravity sensor, or any combination thereof) for detecting a physical motion of the wearable device (101).

[0049] Referring to FIG. 2A, the sensor (230) may include an image sensor (230-1), an eye sensor (230-2), an audio sensor (230-3), or a combination thereof. The image sensor (230-1) may include one or more optical sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate electrical signals representing the color and / or brightness of light. The image sensor (230-1) may be referred to as a camera and may be included in the sensor (230) of FIG. 2A. The plurality of optical sensors included in the image sensor (230-1) may be arranged in the form of a two-dimensional grid. The image sensor (230-1) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional grid. For example, photographic data captured using the image sensor (230-1) may mean one (a) two-dimensional frame data acquired from the image sensor (230-1). For example, video data captured using the image sensor (230-1) may mean a sequence of a plurality of two-dimensional frame data acquired from the image sensor (230-1) according to a frame rate. The image sensor (230-1) may further include a flash light that is arranged toward the direction in which the image sensor (230-1) receives light and outputs light toward the direction.

[0050] According to one embodiment, the image sensor (230-1) of the wearable device (101) may be positioned facing the front of a user wearing the wearable device (101) (e.g., a direction in which both eyes may face). Using the image and / or video acquired from the image sensor (230-1), the processor (210) may identify an external object (e.g., a speaker (120) of FIG. 1). The embodiment is not limited thereto, and the processor (210) may identify a position, shape, and / or gesture (e.g., a hand gesture) of a hand (e.g., a hand (112) of FIG. 1) based on the image and / or video acquired from the image sensor (230-1). The sensor included in the wearable device (101) to identify an external object is not limited to the image sensor (230-1). For example, to identify external objects, the wearable device (101) may include a LiDAR (light detection and ranging), a depth sensor (and / or a depth camera).

[0051] Referring to FIG. 2A, the eye sensor (230-2) may be configured to be positioned toward at least one of the user's two eyes while worn by the user (e.g., the user (110) of FIG. 1). The eye sensor (230-2) may include one or more optical sensors. The embodiment is not limited thereto, and the eye sensor (230-2) may include an infrared (IR) sensor and / or an IR light source synchronized with the IR sensor for acquiring an image and / or video of at least one of the user's two eyes based on an infrared wavelength band. For example, based on the image and / or video identified using the eye sensor (230-2), the processor (210) may identify the direction of at least one of the two eyes. The eye sensor (230-2) may be referred to as a gaze tracker and / or an eye tracker.

[0052] Referring to FIG. 2A, the audio sensor (230-3) can output an electrical signal representing vibrations in the atmosphere. The audio sensor (230-3) can include at least one microphone. For example, if the audio sensor (230-3) includes multiple microphones, the multiple microphones can be arranged in an array (e.g., a microphone array). The embodiment is not limited thereto, and the multiple microphones can be placed at points spaced apart from each other on the wearable device (101).

[0053] In one embodiment, the communication circuit (235) of the wearable device (101) may include circuitry for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device. The external electronic device capable of communicating with the wearable device (101) may include the speaker (120) of FIG. 1 and / or the external electronic devices described with reference to FIG. 4. The communication circuit (235) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (235) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), 6G and / or above-6G. In one embodiment, the communication circuit (235) may be referred to as a communication processor and / or a communication module.

[0054] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (210) of the wearable device (101) may be stored in the memory (215) of the wearable device (101). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (101) and / or the processor (210) may perform at least one of the operations of FIG. 3 and FIG. 11 to FIG. 12 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (101) may mean that one or more instructions provided in the form of an application are stored in a memory (215), and that the one or more applications are stored in a format executable by the processor (210) (e.g., a file having an extension specified by the operating system of the wearable device (101)). For example, an application may include a program and / or a library related to a service provided to a user.

[0055] Referring to FIG. 2A, programs installed in the wearable device (101) may be classified into any one of different layers, including an application layer (240), a framework layer (250), and / or a hardware abstraction layer (HAL) (280), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (220), and / or the sensor (230)) of the wearable device (101) may be classified within the hardware abstraction layer (280). The framework layer (250) may be referred to as an XR framework layer in terms of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 2A may be logically (or for convenience of explanation) separated, and may not mean that the address space of the memory (215) is separated by the layers.

[0056] For example, within the framework layer (250), programs designed to target at least one of the hardware abstraction layer (280) and / or the application layer (240) (e.g., a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye-gaze tracker (274), a face tracker (275), and / or an interface handler (276)) may be classified. Programs classified within the framework layer (250) may provide an executable API (application programming interface) based on other programs.

[0057] For example, within the application layer (240), programs designed to target users of the wearable device (101) may be classified. As examples of programs classified into the application layer (240), an extended reality (XR) system user interface (UI) (241) and / or an XR application (242) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) classified into the application layer (240) may call an application programming interface (API) to cause execution of functions supported by programs classified into the framework layer (250).

[0058] For example, the wearable device (101) may display one or more visual objects on the display (220) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (241). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user with functions available within the virtual space based on the execution of the XR system UI (241).

[0059] Referring to FIG. 2A, a lightweight renderer (243) and / or an XR plug-in (244) are illustrated to be included within the XR system UI (241), but are not limited thereto. For example, based on the XR system UI (241), the processor (210) may execute a lightweight renderer (243) and / or an XR plug-in (244) within the framework layer (250).

[0060] For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (243). The lightweight renderer (243) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (243) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (244). The XR plugin (244) can be referred to as an open XR native client from the perspective of defining (or setting up) the entire rendering pipeline.

[0061] For example, the wearable device (101) may display a screen representing at least a portion of a virtual space on the display (220) based on the execution of the XR application (242). The XR plug-in (244-1) included in the XR application (242) may include instructions that support functions similar to those of the XR plug-in (244) of the XR system UI (241). Descriptions of the XR plug-in (244-1) that overlap with those of the XR plug-in (244) may be omitted. The wearable device (101) may cause the execution of the virtual space manager (251) based on the execution of the XR application (242).

[0062] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the virtual space manager (251). For example, the virtual space manager (251) may include a platform for supporting the virtual space service. The wearable device (101) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (230) based on the execution of the virtual space manager (251), and may display at least a portion of the virtual space on the display (220). The virtual space manager (251) may be referred to as a composition presentation manager (CPM).

[0063] For example, the virtual space manager (251) may include a runtime service (252). As an example, the runtime service (252) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (252). As an example, the wearable device (101) may perform rendering for a virtual space service to the user based on the execution of the runtime service (252). For example, a function related to a virtual space, executable by the application layer (240), may be supported based on the execution of the runtime service (252).

[0064] For example, the virtual space manager (251) may include a pass-through manager (253). Based on the execution of the pass-through manager (253), the wearable device (101) may display a screen representing a virtual space on the display (220), while overlaying another screen representing an actual space acquired through the image sensor (230-1) on at least a portion of the screen.

[0065] For example, the virtual space manager (251) may include an input manager (254). The wearable device (101) may identify data (e.g., sensor data) obtained by executing one or more programs included in the recognition service layer (270) based on the execution of the input manager (254). The wearable device (101) may use the obtained data to identify user input related to the wearable device (101). The user input may be related to a motion (e.g., hand gesture), gaze, and / or speech of the user identified by the sensor (230).

[0066] For example, the perception abstract layer (260) can be used for data exchange between the virtual space manager (251) and the perception service layer (270). From the perspective of being used for data exchange between the virtual space manager (251) and the perception service layer (270), the perception abstract layer (260) can be referred to as an interface. For example, the perception abstract layer (260) can be referenced as OpenPX. The perception abstract layer (260) can be used for a perception client and a perception service.

[0067] According to one embodiment, the recognition service layer (270) may include one or more programs for processing data acquired from a sensor (230) (e.g., an image sensor (230-1) and / or an eye sensor (230-2)). The one or more programs may include at least one of a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye tracker (274), a face tracker (275), and / or an interface processor (276). The type and / or number of the one or more programs included in the recognition service layer (270) are not limited to those illustrated in FIG. 2A.

[0068] For example, the wearable device (101) can identify the pose of the wearable device (101) using the sensor (230) based on the execution of the position tracker (271). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using the image sensor (230-1) and / or the IMU based on the execution of the position tracker (271). The position tracker (271) can be referred to as a head tracking (HeT) module (or head tracking program).

[0069] For example, the wearable device (101) may obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of the space recognizer (272). The wearable device (101) may reconstruct the surrounding environment of the wearable device (101) in three dimensions using data obtained using the image sensor (230-1) based on the execution of the space recognizer (272). The wearable device (101) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (101) reconstructed in three dimensions based on the execution of the space recognizer (272). The space recognizer (272) may be referred to as a scene understanding (SU) module (or a scene understanding program).

[0070] For example, the wearable device (101) can identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (101) based on the execution of the gesture tracker (273). As an example, the wearable device (101) can identify a pose and / or gesture of a hand of a user using data acquired from an image sensor (230-1) based on the execution of the gesture tracker (273). As an example, the wearable device (101) can identify a pose and / or gesture of a hand of a user based on data (or images) acquired using the image sensor (230-1) based on the execution of the gesture tracker (273). The gesture tracker (273) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0071] For example, the wearable device (101) can identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (274). As an example, the wearable device (101) can identify eye movements of the user using data acquired from the eye sensor (230-2) based on the execution of the gaze tracker (274). The gaze tracker (274) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0072] For example, the recognition service layer (270) of the wearable device (101) may further include a face tracker (275) for tracking the user's face. For example, the wearable device (103) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (275). The wearable device (101) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (275). As an example, the wearable device (101) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a sensor (230) (e.g., an image sensor directed at at least a portion of the user's face) based on the execution of the face tracker (275).

[0073] According to one embodiment, the processor (210) of the wearable device (101) may execute an interface handler (276) to process a user input. For example, while the interface handler (276) is executed, the processor (210) may identify a user input identified by a gesture tracker (273) and / or an eye tracker (274). Based on the user input, the processor (210) may identify an input event of a software application executed by the processor (210). Based on the input event, the processor (210) may execute instructions (e.g., an event handler) corresponding to the input event among instructions of the software application.

[0074] According to one embodiment, the processor (210) of the wearable device (101) may identify a user input for interacting with a virtual object provided by another program, such as a virtual space manager (251), based on the interface handler (276). If no function is executed based on the user input, the processor (210) may display a visual object (e.g., the visual objects (162, 164) of FIG. 1) for guiding a user input of a different type than the user input, based on the interface handler (276). Hereinafter, with reference to FIG. 2B, one or more instructions and / or programs related to the interface handler (276) described with reference to FIG. 2A are described.

[0075] Referring to FIG. 2B, instructions and / or programs included in the interface processor (276) are illustrated as being divided into different blocks according to corresponding functions. The interface processor (276) may include an input event detector (291), an error detector (292), an object analyzer (293), an interface determiner (294), a recommended action determiner (296), and / or a recommended action outputter (297). The interface processor (276) may include interface information (295) related to at least one of the above-described programs (e.g., the interface determiner (294)).

[0076] Based on the execution of the input event detector (291), the processor of the wearable device (101) (e.g., the processor (210) of FIG. 2A) can detect a user input. For example, the wearable device (101) can identify an input event related to the user input. The wearable device (101) can identify the type of the input event and / or the user input (e.g., a hand gesture, a touch input, a gaze input, a voice input, and / or a remote input by an external electronic device such as a remote controller). The wearable device (101) can identify a software application related to the input event and / or a coordinate value related to the input event (e.g., a three-dimensional coordinate value within a virtual space and / or an external space managed by the wearable device (101)).

[0077] Based on the execution of the error detector (292), the processor of the wearable device (101) can identify an error in the user input and / or input event identified by the input event detector (291). The processor of the wearable device (101) can identify whether the user input is an input intended by the user based on the execution of the error detector (292). The processor of the wearable device (101) can identify an error in the user input by comparing the three-dimensional coordinates of the point where the user input is identified with the three-dimensional coordinates of a virtual object (e.g., the virtual object (140) of FIG. 1) provided by the wearable device (101). For example, the error may occur when the point where the input event is identified is spaced apart from all interactable objects (e.g., the virtual object (140) of FIG. 1 and / or the visual object (132)) provided by the wearable device (101). For example, the above error may occur if, based on the user input, no function supported by an interactive object (e.g., a virtual object (140) and / or a visual object (132) of FIG. 1) provided by the wearable device (101) is executed.

[0078] Based on the execution of the error detector (292), the wearable device (101) can identify the reliability of the hand gesture identified by the input event detector (291). The reliability may include, and / or be determined by, probabilities that the hand gesture corresponds to any one of the designated gestures (e.g., a pointing gesture, a pinch gesture, and / or a grab gesture). The wearable device (101) can identify an error in the hand gesture by comparing each of the probabilities with a threshold probability.

[0079] Based on the execution of the error detector (292), the wearable device (101) can identify repeatedly performed user inputs and / or input events. If input events are repeatedly generated within a specified period of time (e.g., several seconds), but the input events are not associated with any virtual objects provided by the wearable device (101), the wearable device (101) can generate an error based on the error detector (292).

[0080] Based on the occurrence of an error by the error detector (292), the wearable device (101) can generate error information related to the repeatedly generated input events. The error information can include three-dimensional coordinates of the input events. The error information can include a sensor (e.g., sensor (230) of FIG. 2A) used to identify the input events and / or types of the input events. For example, a wearable device (101) that detects an error in an input event generated by a hand gesture can generate error information indicating the occurrence of an error based on the hand gesture. For example, a wearable device (101) that detects an error generated by a voice input can generate error information indicating that the error was generated by the voice input. For example, if an error is caused by a gaze input, the wearable device (101) may generate error information including the direction of the gaze input and / or the three-dimensional coordinates of a point related to the gaze input and / or parameters (e.g., constants and / or flags) indicating an error based on the gaze input. The error information may be referred to as error log information.

[0081] Using the error information obtained based on the execution of the error detector (292), the wearable device (101) can execute the object analyzer (293) and / or the interface determiner (294). For example, based on the execution of the object analyzer (293) and / or the interface determiner (294), the wearable device (101) can process the error information.

[0082] Based on the execution of the object analyzer (293), the wearable device (101) can identify at least one virtual object adjacent to the input event based on error information. For example, the wearable device (101) can identify distances between the plurality of virtual objects and the point based on three-dimensional coordinate values ​​assigned to the plurality of virtual objects and three-dimensional coordinate values ​​of the point where the input event is identified. The wearable device (101) can identify at least one virtual object disposed adjacent to the point where the input event occurs based on the distances.

[0083] For example, the wearable device (101) can scan an external space and / or a virtual space in three dimensions based on a point where an input event is identified, and identify at least one virtual object adjacent to the point. For example, the wearable device (101) can identify at least one virtual object adjacent to the point using a three-dimensional spherical scan area centered on the point. By gradually expanding the scan area, the wearable device (101) can identify one or more virtual objects included in the scan area. The wearable device (101) can expand the scan area until a specified number of virtual objects (e.g., 1 to 3) are included in the scan area. The embodiment is not limited thereto, and the scan area may have a specified radius.

[0084] A wearable device (101) that identifies at least one virtual object adjacent to a point associated with a user input and / or an input event may identify and / or analyze attribute information of the at least one virtual object. The attribute information may include information about a software application that provides the corresponding virtual object, its layout, and / or UI (e.g., whether the virtual object includes visual objects such as buttons, text inputs, date pickers, sliders, and / or check boxes). The attribute information may include flags and / or parameters indicating whether the virtual object is a three-dimensionally rendered object (or a two-dimensionally rendered object). The attribute information may include information about one or more functions executable by the at least one virtual object and / or user input mapped to the one or more functions.

[0085] Based on the execution of the interface determiner (294), the wearable device (101) can determine the type of user input to be recommended to the user based on error information generated by the error detector (292), attribute information identified by the object analyzer (293), and / or interface information (295). The interface information (295) can indicate the type of user input and / or input event identifiable by the wearable device (101). For example, the interface information (295) can include parameters and / or data regarding whether the wearable device (101) can detect hand gestures, whether the wearable device (101) can track the user's gaze, whether the wearable device (101) can identify voice input, and / or one or more external electronic devices (e.g., a remote controller) connected to the wearable device (101).

[0086] For example, when an error is detected for a first user input (e.g., a gaze input based on a direction of the user's gaze) based on a first sensor (e.g., an eye sensor (230-2) of FIG. 2A), the wearable device (101) may, based on the execution of the interface determiner (294), identify a second sensor (e.g., an image sensor (230-1) and / or an audio sensor (230-3) of FIG. 2A) to be used for receiving and / or identifying a second user input of a different type from the first user input. The wearable device (101) may identify the second sensor based on attribute information of at least one virtual object adjacent to a point associated with the first user input.

[0087] Based on the execution of the recommended action determiner (296), the wearable device (101) can determine a user input to be recommended to the user within the type determined by the interface determiner (294). For example, when recommending a user input based on a voice input, the wearable device (101) can obtain a natural language sentence to be recommended (or guided) to the user based on the execution of the recommended action determiner (296). For example, when recommending a hand gesture to the user, the wearable device (101) can obtain a form of the hand gesture to be recommended to the user.

[0088] Based on the execution of the recommended action output device (297), the wearable device (101) can display a visual object (e.g., visual objects (162, 164) of FIG. 1) representing the user input determined by the execution of the recommended action determiner (296) on the display (e.g., display (220) of FIG. 2A).

[0089] In one embodiment, the wearable device (101) may determine the type of user input to be recommended to the user based on Table 1.

[0090] Types of user input Property information of virtual objects Types of user input supported by virtual objects Recommended types for users Remote controller (or gesture controller) Gaze and hand gestures 2D objects - z-axis input supported Vertical touch (e.g., press), button input Voice input 2D objects - x, y-axis input supported Continuously adjustable values ​​(e.g., sliders) External electronic devices (e.g., watches and / or phones) 3D objects (including windows with depth values) Dimensionless manipulation (e.g., rotation input) Hand gestures and / or remote controller Physical objects N / AN / AVoice input 2D objects - z-axis input supported Voice input with text, hover and selection gestures Hand gestures and / or remote controller 2D objects - x, y-axis input supported Voice input related to input External electronic devices (e.g., watches and / or phones) 3D objects (including windows with depth values) Voice input related to input Hand gestures and / or controller Physical objects Voice input related to real objects Voice input related to real objects

[0091] Table 1 is exemplary and the embodiment is not limited thereto. For example, when an error in an input event based on a remote controller is identified, the wearable device (101) can identify attribute information of a virtual object adjacent to the input event. The wearable device (101) that has identified a two-dimensional virtual object supporting x, y axis input based on the attribute information can, based on Table 1, recommend a user input based on an external electronic device (e.g., a watch and / or a mobile phone) connected to the wearable device (101). For example, when an error in a user input based on a combination of gaze and hand gestures is identified and a three-dimensional virtual object adjacent to the user input is identified, the wearable device (101) can, based on Table 1, recommend a user input based on a hand gesture and / or a remote controller connected to the wearable device (101).

[0092] As described above, according to one embodiment, the wearable device (101) can guide or recommend a user input of a different type from the types of input events based on repeatedly occurring input events and errors caused by the input events. For example, based on a user input at a point away from an interactable object provided by the wearable device (101), the wearable device (101) can display a UI for receiving another user input of a different type from the user input based on attribute information of the object. Based on the UI, the user can perform the intended input more accurately.

[0093] Hereinafter, with reference to FIG. 3, an exemplary operation of the wearable device (101) described with reference to FIG. 1, FIG. 2a and / or FIG. 2b is described.

[0094] FIG. 3 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operations of the wearable device described with reference to FIG. 3. The operations of the wearable device described with reference to FIG. 3 may be related to the interface processor (276) of FIG. 2A.

[0095] Referring to FIG. 3, in operation (310), according to one embodiment, a processor of a wearable device may identify a user input via a first sensor. The wearable device may identify the user input of operation (310) while displaying one or more interactable objects (e.g., virtual objects (140) and / or visual objects (132) of FIG. 1 ). The interactable objects may be provided by the wearable device (e.g., provided by a software application executed by the wearable device), such as, for example, the virtual objects (140) of FIG. 1 , and may be associated with points in virtual space. The interactable objects may be associated with points of real objects (e.g., speaker (120) of FIG. 1 ) identified by the wearable device, such as, for example, the visual objects (132) of FIG. 1 .

[0096] The first sensor of the action (310) may be included in the sensor (230) of FIG. 2A and / or FIG. 2B. For example, the wearable device may identify a user input based on a hand gesture using an image sensor (e.g., the image sensor (230-1) of FIG. 2A and / or FIG. 2B). For example, the wearable device may identify a user input based on a gaze using an eye sensor (e.g., the eye sensor (230-2) of FIG. 2A and / or FIG. 2B). Within the above example, the wearable device may identify a user input based on a combination of a hand gesture identified by the image sensor and a direction of the gaze. For example, the wearable device may identify a voice input based on a user's speech using an audio sensor (e.g., the audio sensor (230-3) of FIG. 2A and / or FIG. 2B). The embodiment is not limited thereto, and the wearable device (101) may use a communication circuit (e.g., the communication circuit (235) of FIG. 2A and / or FIG. 2B) to identify a user input identified by an external electronic device (e.g., a remote controller) connected to the wearable device (101).

[0097] Referring to FIG. 3, in operation (320), a processor of a wearable device according to an embodiment may determine whether a function corresponding to a user input has been identified. While displaying one or more virtual objects, the processor that has identified the user input of operation (310) may identify a virtual object corresponding to the user input. Among the functions supported by the virtual object, the processor may identify a function corresponding to the user input. If the virtual object and / or function corresponding to the user input has been identified (320 - Yes), the processor may perform operation (330). If not all virtual objects and functions corresponding to the user input have been identified (320 - No), the processor may perform operation (340).

[0098] Referring to FIG. 3, in operation (330), according to one embodiment, the processor of the wearable device may execute an identified function. For example, the wearable device may execute the function of operation (330) based on the execution of a software application that provides a virtual object corresponding to a user input. If the user input identified by operation (310) corresponds to a function of a specific virtual object, the wearable device may perform operation (330). Based on operation (330), a user wearing the wearable device may recognize the result of executing the function of the wearable device corresponding to the user input.

[0099] Referring to FIG. 3, in operation (340), according to one embodiment, the processor of the wearable device may determine whether the same user input has been identified within a specified period of time. The specified period of time may be, for example, less than several seconds. The embodiment is not limited thereto, and the period of operation (340) may be empirically determined. If no function corresponding to the user input is executed, the user of the wearable device may repeatedly perform the user input. If the processor does not identify a user input identical to the user input identified based on operation (310) within the specified period of time (340-No), the processor may continue to monitor the user input based on operation (320).

[0100] If a user input substantially identical to the user input identified based on operation (310) within a specified period of time is identified (340-yes), the processor may perform operation (350). The processor may perform operation (340) based on whether an additional user input is identified within a specified range (e.g., an error range) from a point associated with the user input identified by operation (310).

[0101] Referring to FIG. 3, in operation (350), a processor of a wearable device according to one embodiment may determine whether a virtual object adjacent to a point corresponding to a user input has been identified. The virtual object of operation (350) may include not only the virtual object (140) of FIG. 1, but also an object configured to support interaction with a user, such as the visual object (132) of FIG. 1. The wearable device may identify at least one virtual object adjacent to the point of operation (350) based on the scan area described with reference to FIG. 2B. If no virtual object adjacent to the point of operation (350) is identified (350-No), the processor may perform operation (360). If at least one virtual object adjacent to the point of operation (350) is identified (350-Yes), the processor may perform operation (370).

[0102] Referring to FIG. 3 , in operation (360), according to one embodiment, a processor of a wearable device may display a visual object for confirming a user input. The visual object of operation (360) may include text (or a prompt) requesting additional user input (e.g., “How may I help you?”) that is different from the user input of operation (310). The visual object of operation (360) may include text to guide that the user input of operation (310) has not been accurately identified and / or that a function corresponding to the user input has not been identified. If the user input of operation (310) is a voice input, the visual object displayed based on operation (360) may include text expressing a natural language sentence included in the voice input. Based on the visual object of operation (360), the processor may notify the user of the wearable device of a failure (or error) in recognizing the user input of operation (310).

[0103] Referring to FIG. 3, in operation (370), the processor of the wearable device according to one embodiment may determine whether attribute information of a virtual object identified based on operation (350) has been identified. The attribute information of operation (370) may indicate an identifiable user input based on the virtual object of operation (370), as shown in Table 1. The attribute information of operation (370) may indicate a sensor required to identify the user input together with the user input. If the attribute information of operation (370) is not identified (370-No), the processor may perform operation (360) described above. If the attribute information of operation (370) is identified (370-Yes), the processor may perform operation (380).

[0104] Referring to FIG. 3 , in operation (380), according to one embodiment, the processor of the wearable device may recommend another user input based on a second sensor different from the first sensor based on attribute information. The processor may identify a user input that is identifiable using the second sensor different from the first sensor of operation (310) based on attribute information of operation (370) and supported by a virtual object of operation (370). The processor may display a visual object to describe the identified user input. For example, the processor may display a visual object including text, an image, an icon, and / or a video to describe the user input identified based on operation (380), such as visual objects (162, 164) of FIG. 1 .

[0105] The second sensor of the action (380) may be included in the sensor (230) of FIG. 2A and / or FIG. 2B. In an exemplary case where the first sensor of the action (310) is the image sensor (230-1) of FIG. 2A and / or FIG. 2B, and the processor identifies a first user input based on a hand gesture using the first sensor, the processor may, based on the attribute information, display a visual object for recommending a second user input (e.g., a user's voice input) associated with a second sensor (e.g., an audio sensor (230-3) of FIG. 2A and / or FIG. 2B) different from the first sensor. The visual object may include text expressing an utterance (or natural language sentence) associated with the second user input.

[0106] As described above, a wearable device that identifies a user input that cannot perform any function through the first sensor can recommend a user input based on a second sensor different from the first sensor. Below, with reference to FIG. 4, the types of user input that can be identified by the wearable device are exemplarily described.

[0107] FIG. 4 illustrates an exemplary operation of a wearable device (101) for identifying user input. The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operation of the wearable device described with reference to FIG. 3. The operation of the wearable device described with reference to FIG. 3 may be related to the interface processor (276) of FIG. 2A.

[0108] Referring to FIG. 4, according to one embodiment, a wearable device (101) may acquire images (410) of two eyes of a user (e.g., user (110) of FIG. 1) wearing the wearable device (101) using the eye sensor (230-2) of FIG. 2A. The images (410) may include an image (411) corresponding to the user's left eye and / or an image (412) corresponding to the user's right eye. Based on the images (410), the wearable device (101) may identify a direction of the user's gaze. A first type of user input that the wearable device (101) can identify may include a user input (e.g., gaze input) based on the direction identified by at least one of the images (410).

[0109] Referring to FIG. 4, according to one embodiment, a wearable device (101) can acquire an image (420) of a hand of a user wearing the wearable device (101) (e.g., hand (112) of FIG. 1) using the image sensor (230-1) of FIG. 2A. To acquire an image of the hand, image sensors having different directions and / or fields-of-views (FoVs) may be arranged on the front surface of the wearable device (101). Within an image (420) acquired from at least one of the image sensors, the wearable device (101) can identify a visual object (422) corresponding to the hand. Based on the shape and / or gesture of the hand identified using the visual object (422) (e.g., a pointing gesture illustrated with reference to image (420)), the wearable device (101) can detect a hand gesture performed by the hand. A second type of user input that the wearable device (101) can identify can include a user input (e.g., a gesture input) corresponding to a hand gesture identified based on the image (420).

[0110] According to one embodiment, the wearable device (101) can obtain an audio signal including a speech of a user wearing the wearable device (101) using the audio sensor (230-3) of FIG. 2A. By performing STT on the audio signal, the wearable device (101) can obtain a text corresponding to the speech. When a designated keyword for calling the wearable device (101) (e.g., “Hey,” “Bixby,” and / or a keyword designated by the user) is identified from the text, the wearable device (101) can identify a natural language sentence having a timing adjacent to the designated keyword. For example, the wearable device (101) can identify a natural language sentence following the designated keyword, such as “Hey Bixby, tell me today’s weather.” The embodiment is not limited thereto, and the wearable device (101) can identify a natural language sentence preceding a specified keyword, such as "What's my schedule for tomorrow, Bixby." A third type of user input that the wearable device (101) can identify may include a user input (e.g., voice input) represented by the natural language sentence.

[0111] Referring to FIG. 4, according to one embodiment, a wearable device (101) may communicate with an external electronic device (e.g., a speaker (120) of FIG. 1) using the communication circuit (235) of FIG. 2A. The wearable device (101) may receive a signal from the external electronic device, the signal representing a user input for controlling the wearable device (101). A fourth type of user input that the wearable device (101) can identify may include a user input (e.g., a remote control input) indicated by the signal received from the external electronic device.

[0112] Referring to FIG. 4, external electronic devices capable of identifying the fourth type of user input while connected (or paired) with the wearable device (101) are exemplified. For example, the wearable device (101) may communicate with the watch (430) to identify user input received through the touch screen, buttons, dials, and / or digital crown of the watch (430). For example, the user input may include a hand gesture of rotating the digital crown of the watch (430). For example, the wearable device (101) may communicate with the remote controller (432) to identify user input based on buttons, a joystick, and / or motion of the remote controller (432) (e.g., motion of a user's hand holding the remote controller (432). For example, the wearable device (101) can communicate with a glove (434) (or smart glove) to identify user input based on motion of a hand wearing the glove (434). For example, the wearable device (101) can communicate with a mouse (436) to identify user input based on buttons, wheels, and / or motions of the mouse (436). For example, the wearable device (101) can communicate with a keyboard (438) to identify user input indicated by pressing one or more buttons included in the keyboard (438).

[0113] In one embodiment, a virtual object provided by the wearable device (101) may be associated with attribute information that supports multiple types of user input from among the first to fourth types described above. For example, a first virtual object provided by the wearable device (101) may be capable of identifying all of the first type of gaze input and / or the third type of voice input, and a second virtual object may be capable of identifying all of the second type of gesture input and / or the fourth type of remote control input. The wearable device (101) may determine, select, or identify one type of user input to be recommended to the user from among the types of user input identifiable by the virtual object. For example, based on a context of one or more interactable objects provided to the user, the wearable device (101) may determine the type of user input to be recommended to the user.

[0114] As described above, according to one embodiment, when the wearable device (101) identifies first user inputs in which an error occurred (or in which no function was executed), it can recommend to the user a different type of user input that is distinct from the type of the first user inputs. For example, a user who has been repeatedly performing a gesture input including a specific hand gesture may, based on the recommendation of the wearable device (101), perform a different type of user input (e.g., voice input, gaze input, and / or remote input) specified by the recommendation. In the above example, the user can more accurately convey the user's intention related to the hand gesture to the wearable device (101) by using the user input that is distinct from the hand gesture.

[0115] Hereinafter, the operation of a wearable device (101) for recommending user inputs distinguished by the types described above with reference to FIG. 4 is described with reference to FIG. 5, FIG. 6, FIG. 7, FIG. 8, FIG. 9 and / or FIG. 10.

[0116] FIG. 5 illustrates a UI displayed by a wearable device (101) that identifies a user input based on a hand gesture. The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operation of the wearable device (101) described with reference to FIG. 5. The operation of the wearable device (101) described with reference to FIG. 5 may be performed based on the execution of the interface processor (276) of FIG. 2A.

[0117] Referring to FIG. 5, an exemplary state of a wearable device (101) displaying a screen (130) including a plurality of virtual objects (511, 512, 513) is illustrated. The virtual objects (511, 512, 513) may be provided from at least one software application and / or framework executed by the wearable device (101). The wearable device (101) may display the virtual objects (511, 512, 513) in the screen (130) corresponding to a portion of the virtual space based on three-dimensional coordinates in the virtual space. Each of the virtual objects (511, 512, 513) may be mapped with information (e.g., attribute information) including a function supported by the corresponding virtual object and a combination of user inputs set to execute the function.

[0118] According to one embodiment, a wearable device (101) may identify a user input for a specific point in a virtual space provided through a display using one or more sensors (e.g., sensor (230) of FIG. 2A and / or FIG. 2B ). Referring to FIG. 5 , an exemplary state of a wearable device (101) that has identified a pointing gesture performed based on a hand (112) pointing toward a point (v1) along a direction (h1) is illustrated. The wearable device (101) that has identified a first user input including the pointing gesture may display a line (152) extending along the direction (h1) of the pointing gesture to the point (v1) within a screen (130). While the line (152) is illustrated as having a straight shape, the embodiment is not limited thereto. For example, the shape of the line (152) may be curved. The wearable device (101) that has identified the first user input can determine whether a virtual object has been identified at the point (v1). In a state where the virtual object has been identified at the point (v1), the wearable device (101) can execute a first function assigned to the identified virtual object and indicated by the first user input. The first user input can be identified based on an image sensor of the wearable device (101) (e.g., the image sensor (230-1) of FIG. 2A and / or FIG. 2B).

[0119] Referring to the exemplary state of FIG. 5, in a state where no virtual object is placed at the point (v1), the wearable device (101) may not execute any function in response to the first user input. In one embodiment, the wearable device (101) may identify, scan, or search for at least one virtual object adjacent to the point (v1) based on identifying a second user input for the point (v1) related to the first user input. The second user input may be repeatedly performed by the user who performed the first user input (e.g., the user (110) of FIG. 1). The wearable device (101) may initiate identifying at least one virtual object adjacent to the point (v1) based on identifying the second user input within a specified period of time after identifying the first user input. For example, the wearable device (101) can identify at least one virtual object adjacent to the point (v1) by using a scan area (520) in the shape of a sphere centered at the point (v1) and having a radius (r). The radius (r) may have a fixed length, or may increase or decrease based on the number of virtual objects included in the scan area (520). For example, the wearable device (101) can gradually increase the radius (r) of the scan area (520) until a specified number of virtual objects are included in the scan area (520).

[0120] According to one embodiment, the wearable device (101) may, based on identifying at least one virtual object within a scan area (520), display a visual object representing a third user input, the visual object corresponding to a function of the at least one virtual object and identifiable by a second sensor different from the first sensor of the first user input and / or the second user input (e.g., the image sensor (230-1) of FIGS. 2A and / or 2B). Referring to FIG. 5, the wearable device (101) that has identified a plurality of virtual objects (511, 512) within the scan area (520) may display, within the screen (130), visual objects (531, 532) that include text for invoking executable functions based on each of the virtual objects (511, 512). The wearable device (101) can display a visual object (531) at a point within the screen (130) adjacent to a virtual object (512), and / or can display a visual object (532) at a point within the screen (130) adjacent to a virtual object (511).

[0121] In one embodiment, the third user input guided by the visual objects (531, 532) may be determined to be a different type from the first user input and / or the second user input. The wearable device (101) that has identified the first user input and the second user input of the second type related to the hand gesture may guide a user input of another type (e.g., a third type based on a voice input) different from the second type. The wearable device (101) may identify the other type based on attribute information assigned to the virtual objects (511, 512) included in the scan area (520).

[0122] For example, the wearable device (101) may display a visual object (531) that includes text describing a voice input corresponding to a function of enlarging a virtual object (512) (e.g., "Enlarge window B"). For example, the wearable device (101) may display a visual object (532) that includes text representing a voice input corresponding to a function of moving a virtual object (532) (e.g., "Show window A closer"). While visual objects (531, 532) are illustrated as examples that include texts that include names (e.g., window A and / or window B) uniquely assigned to each of the virtual objects (511, 512), the embodiment is not limited thereto. The wearable device (101) may display texts that include numerical values ​​(e.g., indices) uniquely assigned to each of the virtual objects (511, 512) on each of the visual objects (531, 532).

[0123] Referring to FIG. 5, in response to the repeatedly performed first user input and second user input, within a state in which visual objects (531, 532) are displayed, the wearable device (101) may activate a sensor (e.g., audio sensor (230-3) of FIG. 2A and / or FIG. 2B) corresponding to a third user input (e.g., voice input) guided by the visual objects (531, 532). Using the activated sensor, the wearable device (101) may identify the third user input. Within the exemplary state of FIG. 5, the wearable device (101) may determine, based on audio data acquired using the audio sensor switched to the active state, whether an utterance related to text included in any one of the visual objects (531, 532) has been received. When an utterance related to text contained in any one of the visual objects (531, 532) is received, the wearable device (101) can execute a function corresponding to the utterance.

[0124] As described above, according to one embodiment, the wearable device (101) may identify a virtual object related to the pointing gesture among the plurality of virtual objects (511, 512, 513) based on a line (152) (e.g., a hand ray) displayed based on the hand (112) performing the pointing gesture and distances of each of the plurality of virtual objects (511, 512, 513) within the screen (130). If none of the plurality of virtual objects (511, 512, 513) is related to the pointing gesture and the user (110) repeatedly performs the pointing gesture, the wearable device (101) may guide the user to use a different means (e.g., voice and / or speech) than the hand (112) performing the pointing gesture. The above guide may include an action of displaying visual objects (531, 532) within the screen (130).

[0125] Although exemplary operations of a wearable device (101) that recommends voice input in response to a hand (112)-based gesture input have been described, the embodiments are not limited thereto. Below, with reference to FIG. 6, operations of a wearable device (101) that identifies user input through an external electronic device are described.

[0126] Fig. 6 illustrates a UI displayed by a wearable device (101) that identifies a user input based on an external electronic device such as a remote controller (432). The wearable device (101) of Figs. 1, 2a, and 2b and / or the processor (210) of Fig. 2a can perform the operation of the wearable device (101) described with reference to Fig. 6. The operation of the wearable device (101) described with reference to Fig. 6 can be performed based on the execution of the interface processor (276) of Fig. 2a.

[0127] Referring to FIG. 6, an exemplary state of a wearable device (101) displaying a screen (130) for playing a video is illustrated. The wearable device (101) can display the screen (130) by executing a software application for playing multimedia content such as a video. Within the screen (130), the wearable device (101) can display virtual objects (621, 622, 623, 624, 625, 630) for controlling the playback of the video together with the video. Through the virtual objects (621, 622, 623, 624, 625) having the form of buttons, the wearable device (101) can identify a user input for executing a designated function related to the video being played through the screen (130). For example, based on a user input associated with a virtual object (622), the wearable device (101) may pause a video being played on the screen (130). For example, based on a user input associated with at least one of the virtual objects (621, 623), the wearable device (101) may change or switch a video being played on the screen (130) to another video in a playlist that includes the video. For example, based on a user input associated with a virtual object (624), the wearable device (101) may reduce the volume of audio corresponding to the video being played on the screen (130) to substantially 0 (e.g., mute). For example, based on a user input associated with a virtual object (625), the wearable device (101) may display a menu for adjusting settings (e.g., subtitles) associated with the video being played on the screen (130). The above menu can be superimposed on the video within the screen (130).

[0128] Referring to the exemplary state of FIG. 6, the wearable device (101) may display a virtual object (630) in the form of a navigation bar for controlling the timing of a video being played through the screen (130). Based on a handle (632) that is movable on the virtual object (630), the wearable device (101) may visualize the timing of the video being played through the screen (130) within the entire time interval of the video. The virtual object (630) may be referred to as a navigation bar, a seek bar, and / or a progress bar.

[0129] According to one embodiment, the wearable device (101) can identify a user input while displaying the screen (130). Referring to FIG. 6, in an exemplary state where a user holds a remote controller (432), the wearable device (101) can receive a signal representing a user input identified by the remote controller (432) from the remote controller (432) through a communication circuit (e.g., the communication circuit (235) of FIG. 2A). The signal can be received wirelessly based on a wireless communication protocol such as Bluetooth. The embodiment is not limited thereto, and the signal can be received through a cable connecting the wearable device (101) and the remote controller (432) based on a wired communication protocol such as a universal serial bus (USB).

[0130] Referring to FIG. 6, in an exemplary state where a user input is identified through a remote controller (432), the wearable device (101) may execute a function corresponding to the user input. The wearable device (101) that has identified the user input may display, within the screen (130), a line (610) extending toward a reference direction identified by the remote controller (432). The line (610) may correspond to the line (152) of FIG. 1 and / or FIG. 5 . In one embodiment, the line (610) may be referred to as a ray. The user input identified through the remote controller (432) may be associated with a three-dimensional point (v1) corresponding to an end of the line (610). For example, a user (110) wearing a wearable device (101) can control a remote controller (432) to move a point (v1) of a line (610). The user (110) can perform a user input to select a virtual object by pressing a button of the remote controller (432) while the point (v1) overlaps with a virtual object intended by the user (110). The user input can be included in the fourth type, as described above with reference to FIG. 4.

[0131] Referring to Fig. 6, the wearable device (101) that identifies a user input based on a remote controller (432) can execute a function of a virtual object overlapping a point (v1) specified by the user input. Referring to Fig. 6, the wearable device (101) that identifies a first user input for a point (v1) that is spaced apart from all of the virtual objects (621, 622, 623, 624, 625, 630) within the screen (130) can not execute any function based on the user input. In response to a second user input repeatedly performed for the point (v1) within a specified period of time after the first user input, the wearable device (101) can recommend to the user (110) a different type of user input that is distinct from the remote controller (432).

[0132] For example, the wearable device (101) may recommend to the user (110) a user input corresponding to a function of a virtual object (624) adjacent to a point (v1). The wearable device (101) may further display a visual object for recommending the user input (e.g., voice input) of a type distinct from that of the remote controller (432). While displaying the visual object, the wearable device (101) may switch an audio sensor, such as a microphone (e.g., audio sensor (230-3) of FIG. 2A), from an inactive state to an active state.

[0133] For example, the wearable device (101) may recommend to the user (110) a user input corresponding to a function executable by a virtual object (630) adjacent to the point (v1). The wearable device (101) may recommend to the user (110) a user input identifiable through an external electronic device distinct from the remote controller (432) that identified the first user input and / or the second user input. For example, the wearable device (101) that identified the watch (430) worn on the wrist of the user (110) may recommend to the user (110) a user input based on the dial of the watch (430).

[0134] For example, the wearable device (101) may display a visual object (640) to guide the rotation of the dial of the watch (430) within the screen (130). The visual object (640) may include text, an image, and / or an icon representing the watch (430) connected (or paired) with the wearable device (101). The visual object (640) may include designated text to guide a user input based on the watch (430), such as “Wear the watch.” A user (110) viewing the visual object (640) may rotate the dial of the watch (430). The wearable device (101), which receives a signal indicating the rotation of the dial from the watch (430), may execute a function related to the virtual object (630) based on the signal. For example, the wearable device (101) can adjust the timing of the video being played through the screen (130). The wearable device (101) that has adjusted the timing can change the position of the handle (632) superimposed on the virtual object (630) based on the timing.

[0135] Although an embodiment of recommending user input based on the dial of the watch (430) has been described, the embodiment is not limited thereto. For example, the wearable device (101) may recommend user input based on the touch screen, buttons, and / or digital crown of the watch (430).

[0136] Fig. 7 illustrates a UI displayed by a wearable device (101) that identifies a user input based on gaze. The wearable device (101) of Figs. 1, 2a, and 2b and / or the processor (210) of Fig. 2a can perform the operation of the wearable device (101) described with reference to Fig. 7. The operation of the wearable device (101) described with reference to Fig. 7 can be performed based on the execution of the interface processor (276) of Fig. 2a.

[0137] Referring to FIG. 7, an exemplary state of a wearable device (101) displaying a screen (130) for playing a video is illustrated, similar to FIG. 6. While displaying the screen (130), the wearable device (101) may identify user input using one or more sensors, such as the eye sensor (230-2) of FIG. 2A. For example, the wearable device (101) may identify a direction of gaze toward a point (g1) within the screen (130) based on images (410) of two eyes of a user (110) wearing the wearable device (101) acquired using the eye sensor. Although an exemplary operation of the wearable device (101) identifying a point (g1) corresponding to a single coordinate and / or pixel within the screen (130) is described, the embodiment is not limited thereto. For example, the wearable device (101) can identify the gaze of the user (110) toward an area having a size, such as a circle having a radius.

[0138] Referring to FIG. 7, based on the direction of the gaze identified based on the images (410), the wearable device (101) can identify the gaze input of the user (110). For example, the user (110) wearing the wearable device (101) can gaze at a specific point (g1) for a specified period of time, or, while gazing at the specific point (g1), press a button of the wearable device (101), and / or perform a hand gesture (e.g., a pinch gesture). The wearable device (101) that identifies the motion of the user (110) gazing at the specific point (g1) for a specified period of time can identify a first type of user input related to the point (g1). While gazing at a specific point (g1), the wearable device (101) that identifies the pressing of the button and / or the hand gesture can identify a first type of user input (e.g., gaze input) related to the point (g1).

[0139] The wearable device (101) that identifies a first user input based on a gaze can execute a function assigned to a virtual object located at a point (g1) specified by the direction of the gaze and / or adjacent to the point (g1). In the exemplary state of FIG. 7 where the point (g1) is spaced apart from all of the virtual objects (621, 622, 623, 624, 625, 630) within the screen (130), the wearable device (101) may not execute any function based on the first user input. After the first user input, in response to a second user input related to the point (g1) (or within an area within an error range from the point (g1)) within a designated period of time, the wearable device (101) can recommend to the user (110) a different type of user input than the first type of the first user input and / or the second user input.

[0140] For example, the wearable device (101) can identify virtual objects (623, 625) adjacent to the point (g1). The wearable device (101) that has identified the virtual object (623) adjacent to the point (g1) can display a visual object (720) that guides a voice input that is different from the gaze input based on attribute information corresponding to the virtual object (623). Within the visual object (720), the wearable device (101) can display a natural language sentence (e.g., “Show me the next video”) that expresses a function supported by the virtual object (623) (e.g., a function of switching a video displayed through the screen (130).

[0141] In one embodiment, the wearable device (101) that identifies a virtual object (625) adjacent to a point (g1) may recommend a user input based on the user's (110) usage history for a menu (or options within the menu) provided through the virtual object (625). For example, within the menu provided through the virtual object (625), if the user (110) has set the subtitle language to French, the wearable device (101) may display a visual object (730) related to an input for setting the subtitle language to French. Within the visual object (730), the wearable device (101) may display a natural language sentence corresponding to the input (e.g., "Select French subtitles").

[0142] While displaying visual objects (720, 730), the wearable device (101) can obtain audio signals related to user's (110) utterances using an activated audio sensor (e.g., audio sensor (230-3) of FIG. 2A and / or FIG. 2B). For example, the user (110) viewing the visual objects (720, 730) can utter any one of the natural language sentences included in the visual objects (720, 730). Based on the user's (110) utterances identified based on the audio sensor and / or STT, the wearable device (101) can execute at least one function indicated by the utterance. While displaying visual objects (720, 730) for recommending voice input, if the microphone of the wearable device (101) is not switched to an active state or is occupied by another software application, the wearable device (101) may acquire the audio signal using a microphone of an external electronic device (e.g., the watch (430) and / or remote controller (432) of FIG. 4) connected to the wearable device (101).

[0143] Fig. 8 illustrates a UI displayed by a wearable device (101) that identifies a user input based on a hand gesture. The wearable device (101) of Figs. 1, 2a, and 2b and / or the processor (210) of Fig. 2a can perform the operation of the wearable device (101) described with reference to Fig. 8. The operation of the wearable device (101) described with reference to Fig. 8 can be performed based on the execution of the interface processor (276) of Fig. 2a.

[0144] Referring to FIG. 8, an exemplary state of a wearable device (101) displaying a screen (130) including a plurality of virtual objects (810, 820) is illustrated. While displaying the screen (130), the wearable device (101) may identify a posture, shape, and / or direction (h1) of a hand (112) of a user (110) using the image sensor (230-1) of FIG. 2A. Referring to FIG. 8, the wearable device (101) that identifies a first user input including a pointing gesture toward a point (v1) along the direction (h1) may execute a function assigned to at least one virtual object associated with the point (v1). Referring to FIG. 8, if no virtual object is placed at the point (v1), the wearable device (101) may not execute any function despite the first user input.

[0145] Within a specified period of time after the first user input, the wearable device (101) that has identified a second user input related to the point (v1) may not perform any function for any virtual object, similar to the first user input. Instead of performing any function, the wearable device (101) may perform an action (e.g., the action of FIG. 3) to identify the user's (110) intention. For example, the wearable device (101) may recommend a different type of user input than a hand-based (112) gesture input to more accurately identify the user's (110) intention.

[0146] In one embodiment, the wearable device (101) may identify virtual objects (810, 820) adjacent to points (v1) corresponding to the first user input and the second user input in response to repeatedly performed first user input and second user input. Based on attribute information of the virtual objects (810, 820), the wearable device (101) may identify a type of user input to be recommended to the user (110). When the attribute information corresponding to the virtual objects (810, 820) is set to receive user input of the type of the first user input and the second user input (e.g., the second type corresponding to a gesture input in the case of FIG. 8), the wearable device (101) may recommend a gesture input based on a hand (112) to the user (110) similar to the first user input and the second user input.

[0147] Referring to FIG. 8, an exemplary state is illustrated in which the wearable device (101) identifies virtual objects (810, 820) that match attribute information set to receive a gesture input related to the hand (112) in response to a first user input and a second user input repeatedly performed based on the hand (112). Within the state, the wearable device (101) may display visual objects (830, 840) related to functions of the virtual objects (810, 820) executable by the hand (112). For example, if the virtual objects (810, 820) are capable of performing three-dimensional movement of the virtual objects (810, 820) based on a gesture performed by the hand (112), the wearable device (101) may display visual objects (830, 840) for the three-dimensional movement.

[0148] Within the exemplary state of FIG. 8, each of the visual objects (830, 840) may include icons indicating that the virtual objects (810, 820) are movable. In response to a user input related to any one of the visual objects (830, 840), the wearable device (101) may change the three-dimensional coordinates of any one of the virtual objects (810, 820). Based on the changed three-dimensional coordinates, the wearable device (101) may change the position of the virtual object corresponding to the user input within the screen (130).

[0149] FIG. 9 illustrates a UI displayed by a wearable device (101) that identifies a user input based on a hand gesture. The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A can perform the operation of the wearable device (101) described with reference to FIG. 9. The operation of the wearable device (101) described with reference to FIG. 9 can be performed while the interface processor (276) of FIG. 2A is executed.

[0150] Referring to FIG. 9, an exemplary state of a wearable device (101) that identifies a hand gesture based on two hands (e.g., a right hand (911) and / or a left hand (912)) of a user (110) while displaying a screen (130) including a virtual object (930) is illustrated. The wearable device (101) that identifies the right hand (911) using the image sensor (230-1) of FIG. 2A may display a virtual object (921) representing the position, posture, and / or direction of the right hand (911) within the screen (130). Similarly, the wearable device (101) that identifies the left hand (912) using the image sensor may display a virtual object (922), which is a graphical object representing the left hand (912), within the screen (130). Virtual objects (921, 922) may be referred to as avatars corresponding to the user (110).

[0151] Referring to FIG. 9, a wearable device (101) that identifies two hands of a user (110) can identify a hand gesture based on at least one of the two hands. For example, the wearable device (101) can identify a pinch gesture performed based on the left hand (912) and related to a vertex (a) of a virtual object (930). While identifying a first pinch gesture based on the left hand (912), the wearable device (101) can further identify a second pinch gesture performed based on the right hand (911) and related to a point (c).

[0152] In the exemplary case of FIG. 9, a function for changing the size of a virtual object (930) (e.g., dimensions of the virtual object (930) such as width, height, and depth) may be matched to pinch gestures corresponding to each of the vertices (a, b) of the virtual object (930). The wearable device (101) executing the function may display a bounding box (940) having a rectangular parallelepiped shape surrounding the virtual object (930) within the screen (130).

[0153] In the above case, the wearable device (101) that has identified the first pinch gesture and the second pinch gesture may not execute the function because the point (c) related to the second pinch gesture is spaced apart from the virtual object (930). If the point (c) is spaced apart from all virtual objects displayed through the screen (130), the wearable device (101) may identify a virtual object (930) adjacent to the point (c) and / or a point (b) linked to the function related to the virtual object (930) in order to estimate the intention of the user (110) related to the second pinch gesture. While identifying the virtual object adjacent to the point (c), the wearable device (101) may gradually enlarge the virtual object (921) corresponding to the second pinch gesture and / or change the color and / or transparency of the virtual object (921).

[0154] In the exemplary case of FIG. 9, the wearable device (101) that identifies a vertex (b) of a virtual object (930) adjacent to a point (c) corresponding to a second pinch gesture may execute a function related to the vertex (b). For example, the wearable device (101) that identifies a first pinch gesture performed on vertex (a) and a second pinch gesture performed on point (c) may change the point (c) corresponding to the second pinch gesture to vertex (b), thereby operating similarly to identifying pinch gestures for each of the vertices (a, b). For example, the wearable device (101) may execute a function for adjusting the size of the virtual object (930) based on the first pinch gesture and the second pinch gesture. The wearable device (101) that executes the above function can display a bounding box (950) having a shape of a rectangular parallelepiped, including a vertex (a) linked to a virtual object (922) corresponding to the left hand (912) that performed the first pinch gesture and a point (c) within the screen (130) corresponding to the right hand (911) that performed the second pinch gesture. While displaying the bounding box (950), the wearable device (101) can change the size of the bounding box (950) and / or the virtual object (930) based on the distance between the two hands that maintain the pinch gesture.

[0155] Fig. 10 illustrates a UI displayed by a wearable device (101) that identifies a user input based on a hand gesture. The wearable device (101) of Figs. 1, 2a, and 2b and / or the processor (210) of Fig. 2a may perform the operation of the wearable device (101) described with reference to Fig. 10. The operation of the wearable device (101) described with reference to Fig. 10 may be related to the interface processor (276) of Fig. 2a.

[0156] Referring to FIG. 10, an exemplary state of a wearable device (101) identifying a hand gesture performed by a hand (112) of a user (110) while displaying a screen (130) including a virtual object (1010) is illustrated. The wearable device (101) can acquire an image and / or video related to the hand (112) using the image sensor (230-1) of FIG. 2A. Based on the position, shape, and / or direction (h1) of the hand (112) identified based on the image, the wearable device (101) can display a virtual object (150) corresponding to the hand (112) within the screen (130). Based on the direction (h1) of the hand (112), the wearable device (101) can display a line (152) corresponding to the direction (h1) within the screen (130).

[0157] Referring to FIG. 10, while displaying a virtual object (1010-1) at a first point within the screen (130), the wearable device (101) may identify a user input including a pointing gesture directed toward the point (v1). If no virtual object is placed at the point (v1), the wearable device (101) may not perform any function. Based on identifying one or more user inputs for the point (v1), the wearable device (101) may identify a virtual object (1010-1) adjacent to the point (v1).

[0158] The wearable device (101) that identifies a virtual object (1010-1) displayed at a first point spaced apart from a point (v1) where a user input is identified can move the virtual object (1010-1) to the point (v1). For example, the wearable device (101) can move the virtual object (1010-1) along the depth direction. Based on the movement, the wearable device (101) can display a virtual object (1010-2) located at the point (v1). The movement of the virtual object (1010) by the wearable device (101) can be performed when no type of user input other than a gesture input based on a hand (112) is set in the attribute information corresponding to the virtual object (1010). The embodiment is not limited thereto. For example, based on the above attribute information, within a state in which a virtual object (1010) capable of responding to a gesture input and / or a voice input related to a hand (112) is identified, if an audio sensor (e.g., audio sensor (230-3) of FIG. 2A and / or FIG. 2B) for receiving the voice input remains inactive or if the audio sensor is dedicated to a specific software application, the wearable device (101) can move the virtual object (1010) to a point (v1) where it is easy to receive a gesture input related to a hand (112).

[0159] FIG. 11 illustrates an example of a flowchart of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operations of the wearable device described with reference to FIG. 11. The operations of the wearable device described with reference to FIG. 11 may be related to the operations of FIG. 3.

[0160] Referring to FIG. 11, in operation (1110), according to one embodiment, a processor of a wearable device can identify a hand gesture for a three-dimensional point via a remote controller (e.g., remote controller (432) of FIG. 4). The wearable device can identify the hand gesture of operation (1110) based on the operation described with reference to FIG. 6.

[0161] Referring to FIG. 11, in operation (1120), according to one embodiment, the processor of the wearable device may identify a two-dimensional virtual object adjacent to a three-dimensional point. If a two-dimensional virtual object adjacent to the three-dimensional point of operation (1110) is identified (1120 - Yes), the processor of the wearable device may perform operation (1130). If a two-dimensional virtual object adjacent to the three-dimensional point of operation (1110) does not exist (1120 - No), or if a different virtual object than the two-dimensional virtual object is identified, the processor of the wearable device may perform operation (1160).

[0162] Referring to FIG. 11, in operation (1130), according to one embodiment, a processor of a wearable device may identify whether a two-dimensional virtual object supports x-axis control and / or y-axis control. The processor may use attribute information corresponding to the two-dimensional virtual object of operation (1130) to identify whether the two-dimensional virtual object supports x-axis control and / or y-axis control. The x-axis and / or the y-axis may correspond to a horizontal direction and a vertical direction, respectively, of a display of the wearable device (e.g., display (220) of FIG. 2A). If the two-dimensional virtual object supports x-axis control and / or y-axis control (1130 - Yes), the processor of the wearable device may perform operation (1140). If the two-dimensional virtual object does not support both x-axis control and y-axis control, or only supports z-axis control (1130-No), the processor of the wearable device may perform operation (1150).

[0163] Referring to FIG. 11, in operation (1140), a processor of a wearable device according to one embodiment may guide a user input based on a watch (e.g., watch (430) of FIG. 4) and / or an external electronic device, such as a mobile phone. Operation (1140) may be performed while identifying at least one virtual object wirelessly connected (or paired) with the wearable device. For example, the processor may display a visual object for guiding the watch-based user input, such as the visual object (640) of FIG. 6. Based on the guidance of operation (1140), the wearable device may cause the user to perform the user input based on the external electronic device of operation (1140). The processor, which has identified the user input guided by operation (1140), may move the two-dimensional virtual object along the direction of the x-axis and / or the y-axis.

[0164] Referring to FIG. 11, in operation (1150), according to one embodiment, a processor of a wearable device may control a two-dimensional virtual object along the z-axis based on a user's speech. To obtain the speech, in operation (1150), the processor of the wearable device may activate an audio sensor of the wearable device including a microphone (e.g., audio sensor (230-3) of FIG. 2A). The processor may perform speech recognition (e.g., speech recognition based on at least one of STT and / or NLU) on an audio signal output from the audio sensor to identify the user's speech. Based on the speech, the processor may move the two-dimensional virtual object along the z-axis.

[0165] Referring to FIG. 11, in operation (1160), according to one embodiment, a processor of a wearable device may identify a 3D virtual object adjacent to a 3D point. The processor may scan a virtual space including one or more virtual objects centered on the 3D point of operation (1110). In a state where a 3D virtual object of operation (1160) is identified (1160-Yes), the processor of the wearable device may perform operation (1170). If no 3D virtual object is identified based on operation (1160) (1160-No), the processor of the wearable device may perform operation (1180).

[0166] Referring to FIG. 11, in operation (1170), a processor of a wearable device according to one embodiment may guide a user input based on the properties of a three-dimensional virtual object identified based on operation (1160). For example, using property information corresponding to the three-dimensional virtual object of operation (1160), the processor may guide a different type of user input than the hand gesture of operation (1110). The processor may select a type of user input based on a priority indicated by the property information.

[0167] Referring to FIG. 11, in operation (1180), a processor of a wearable device according to one embodiment may guide a user input based on a hand gesture. If the processor fails to identify all two-dimensional virtual objects and three-dimensional virtual objects, the processor may guide a gesture input based on a hand gesture based on operation (1180). For example, the processor may guide the user to perform a hand gesture on a real object.

[0168] The operations described with reference to Fig. 11 may be at least partially related to Table 1. The operations described with reference to Fig. 11 may be expressed in the pseudo-code of Table 2.

[0169]

[0170] Referring to Table 2, the processor may change the sensor associated with the user input depending on the category of the virtual object (e.g., two-dimensional virtual object and / or three-dimensional virtual object) and / or depending on the functionality supported by the virtual object.

[0171] FIG. 12 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operations of the wearable device described with reference to FIG. 12. The operations of the wearable device described with reference to FIG. 12 may be related to the operations of FIG. 3 and / or FIG. 11.

[0172] Referring to FIG. 12, in operation (1210), according to one embodiment, a processor of a wearable device may identify a first input for a point spaced apart from all of a plurality of virtual objects using a first sensor. The first sensor may be included in sensor (230) of FIG. 2A. In a state where the first input for a point spaced apart from all of the virtual objects has been identified, the processor may not execute any of the functions assigned to each of the virtual objects. For example, based on the first input, the processor may refrain from executing all of the functions.

[0173] Referring to FIG. 12, in operation (1220), a processor of a wearable device according to an embodiment may identify at least one virtual object adjacent to a point where the first input was identified based on a second input identified based on a first sensor within a designated period of time after identifying a first input. The second input may be identified based on the point where the first input of operation (1210) was identified. Based on operation (1220), the processor may identify at least one virtual object of operation (1220) by using a scan area formed around the point where the first input was identified (e.g., scan area (520) of FIG. 5).

[0174] Referring to FIG. 12 , in operation (1230), a processor of a wearable device according to an embodiment may display a visual object representing a third input corresponding to a function of at least one virtual object. The processor may identify the function of operation (1230) using attribute information corresponding to at least one virtual object identified based on operation (1220). The processor may identify the third input identifiable by a second sensor different from the first sensor of operation (1210) using the attribute information. The second sensor may be included in the sensor (230) of FIG. 2A . The visual object of operation (1230) may include visual objects (162, 164) of FIG. 1 , visual objects (531, 532) of FIG. 5 , visual objects (640) of FIG. 6 , and / or visual objects (720, 730) of FIG. 7 .

[0175] Hereinafter, exemplary appearances of the wearable devices described with reference to FIGS. 1 to 12 are illustrated with reference to FIGS. 13A, 13B, 14A and / or 14B. The wearable device (1300) of FIG. 13A and / or 13B and / or the wearable device (1400) of FIG. 14A and / or 14B may be an example of the wearable device (101) of FIG. 1.

[0176] FIG. 13A illustrates an example of a perspective view of a wearable device, according to one embodiment. According to one embodiment, the wearable device (1300) may have the form of glasses that are wearable on a body part of a user (e.g., the head). The wearable device (1300) may include a head-mounted display (HMD). For example, the housing of the wearable device (1300) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (1300) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0177] Referring to FIG. 13A, according to one embodiment, a wearable device (1300) may include at least one display (1350) and a frame (1300) supporting at least one display (1350).

[0178] According to one embodiment, a wearable device (1300) may be worn on a part of a user's body. The wearable device (1300) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (1300). For example, the wearable device (1300) may display a virtual reality image provided from at least one optical device (1382, 1384) of FIG. 13B on at least one display (1350) in response to a user's designated gesture acquired through the motion recognition cameras (1360-2, 1360-3) of FIG. 13B.

[0179] According to one embodiment, at least one display (1350) may provide visual information to a user. For example, at least one display (1350) may include a transparent or translucent lens. At least one display (1350) may include a first display (1350-1) and / or a second display (1350-2) spaced apart from the first display (1350-1). For example, the first display (1350-1) and the second display (1350-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0180] Referring to FIG. 13B, at least one display (1350) can provide visual information transmitted from external light to a user through a lens included in the at least one display (1350), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (1350) can include a first surface (1331) and a second surface (1332) opposite to the first surface (1331). A display area can be formed on the second surface (1332) of the at least one display (1350). When a user wears the wearable device (1300), external light can be transmitted to the user by being incident on the first surface (1331) and transmitted through the second surface (1332). As another example, at least one display (1350) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (1382, 1384) on a real screen transmitted through external light, in a display area formed on the second surface (1332).

[0181] In one embodiment, at least one display (1350) may include at least one waveguide (1333, 1334) that diffracts light emitted from at least one optical device (1382, 1384) and transmits the diffracted light to a user. The at least one waveguide (1333, 1334) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (1333, 1334). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (1333, 1334) may be propagated to the other end of the at least one waveguide (1333, 1334) by the nano-pattern. At least one waveguide (1333, 1334) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1333, 1334) may be arranged within the wearable device (1300) to guide a screen displayed by at least one display (1350) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (1333, 1334).

[0182] The wearable device (1300) can analyze an object included in a real image collected through a shooting camera (1360-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1350). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (1300) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (1300) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using the multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (1300) can view an image displayed on at least one display (1350).

[0183] According to one embodiment, the frame (1300) may be configured as a physical structure that allows the wearable device (1300) to be worn on the user's body. According to one embodiment, the frame (1300) may be configured so that, when the user wears the wearable device (1300), the first display (1350-1) and the second display (1350-2) can be positioned corresponding to the user's left and right eyes. The frame (1300) may support at least one display (1350). For example, the frame (1300) may support the first display (1350-1) and the second display (1350-2) to be positioned corresponding to the user's left and right eyes.

[0184] Referring to FIG. 13A, the frame (1300) may include a region (1320) that at least partially contacts a portion of the user's body when the user wears the wearable device (1300). For example, the region (1320) of the frame (1300) that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (1300) makes contact with. According to one embodiment, the frame (1300) may include a nose pad (1310) that contacts a portion of the user's body. When the wearable device (1300) is worn by the user, the nose pad (1310) may contact a portion of the user's nose. The frame (1300) may include a first temple (1304) and a second temple (1305) that contact another part of the user's body that is distinct from the part of the user's body.

[0185] For example, the frame (1300) may include a first rim (1301) that surrounds at least a portion of the first display (1350-1), a second rim (1302) that surrounds at least a portion of the second display (1350-2), a bridge (1303) that is disposed between the first rim (1301) and the second rim (1302), a first pad (1311) that is disposed along a portion of the edge of the first rim (1301) from one end of the bridge (1303), a second pad (1312) that is disposed along a portion of the edge of the second rim (1302) from the other end of the bridge (1303), a first temple (1304) that extends from the first rim (1301) and is fixed to a portion of the wearer's ear, and a second temple (1305) that extends from the second rim (1302) and is fixed to a portion of the ear opposite the ear. There are. The first pad (1311) and the second pad (1312) can be in contact with a part of the user's nose, and the first temple (1304) and the second temple (1305) can be in contact with a part of the user's face and a part of the user's ear. The temples (1304, 1305) can be rotatably connected to the rim through the hinge units (1306, 1307) of FIG. 13B. The first temple (1304) can be rotatably connected to the first rim (1301) through the first hinge unit (1306) disposed between the first rim (1301) and the first temple (1304). The second temple (1305) can be rotatably connected to the second rim (1302) via a second hinge unit (1307) disposed between the second rim (1302) and the second temple (1305). In one embodiment, the wearable device (1300) can identify an external object (e.g., a user's fingertip) touching the frame (1300) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (1300).

[0186] According to one embodiment, the wearable device (1300) may include hardwares (e.g., hardwares described above based on the block diagrams of FIGS. 2A and / or 2B) that perform various functions. For example, the hardwares may include a battery module (1370), an antenna module (1375), at least one optical device (1382, 1384), speakers (e.g., speakers 1355-1, 1355-2), microphones (e.g., microphones 1365-1, 1365-2, 1365-3), a light-emitting module, and / or a printed circuit board (PCB) (1390) (e.g., a printed circuit board). The various hardwares may be disposed within the frame (1300).

[0187] According to one embodiment, microphones (e.g., microphones 1365-1, 1365-2, 1365-3) of the wearable device (1300) may be disposed on at least a portion of the frame (1300) to acquire sound signals. A first microphone (1365-1) disposed on the bridge (1303), a second microphone (1365-2) disposed on the second rim (1302), and a third microphone (1365-3) disposed on the first rim (1301) are illustrated in FIG. 13B , but the number and arrangement of the microphones (1365) are not limited to the embodiment of FIG. 13B . When the number of microphones (1365) included in the wearable device (1300) is two or more, the wearable device (1300) can identify the direction of a sound signal by using a plurality of microphones placed on different parts of the frame (1300).

[0188] According to one embodiment, at least one optical device (1382, 1384) may project a virtual object onto at least one display (1350) to provide various image information to a user. For example, at least one optical device (1382, 1384) may be a projector. At least one optical device (1382, 1384) may be disposed adjacent to at least one display (1350) or may be included within at least one display (1350) as a part of at least one display (1350). According to one embodiment, the wearable device (1300) may include a first optical device (1382) corresponding to a first display (1350-1) and a second optical device (1384) corresponding to a second display (1350-2). For example, at least one optical device (1382, 1384) may include a first optical device (1382) disposed at an edge of a first display (1350-1) and a second optical device (1384) disposed at an edge of a second display (1350-2). The first optical device (1382) may transmit light to a first waveguide (1333) disposed on the first display (1350-1), and the second optical device (1384) may transmit light to a second waveguide (1334) disposed on the second display (1350-2).

[0189] In one embodiment, the camera (1360) may include a recording camera (1360-4), an eye tracking camera (ET CAM) (1360-1), and / or a motion recognition camera (1360-2, 1360-3). The recording camera (1360-4), the eye tracking camera (1360-1), and the motion recognition cameras (1360-2, 1360-3) may be positioned at different locations on the frame (1300) and may perform different functions. The eye tracking camera (1360-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (1300). For example, the wearable device (1300) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (1360-1).

[0190] The wearable device (1300) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (1360-1). The wearable device (1300) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (1300) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (1360-1). The wearable device (1300) can render an image (or screen) displayed on at least one display (1350) based on the position of the user's eyes.

[0191] For example, the visual quality of a first region related to the gaze within an image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinct from the first region may be different from each other. The wearable device (1300) may obtain an image having the visual quality of the first region and the visual quality of the second region that match the user's gaze using foveated rendering. For example, if the wearable device (1300) supports an iris recognition function, user authentication may be performed based on iris information obtained using the gaze tracking camera (1360-1). Although an example in which the gaze tracking camera (1360-1) is positioned toward the user's right eye is illustrated in FIG. 13B, the embodiment is not limited thereto, and the gaze tracking camera (1360-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0192] In one embodiment, the capturing camera (1360-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (1360-4) can be used to obtain a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (1360-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1350). The at least one display (1350) can display a single image in which information about an actual image or background including an image of the specific object obtained using the capturing camera (1360-4) and a virtual image provided through at least one optical device (1382, 1384) are superimposed. The wearable device (1300) can compensate for depth information (e.g., the distance between the wearable device (1300) and an external object acquired through a depth sensor) using an image acquired through the capture camera (1360-4). The wearable device (1300) can perform object recognition using an image acquired using the capture camera (1360-4). The wearable device (1300) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capture camera (1360-4). The wearable device (1300) can perform a pass-through function to display an image acquired through the capture camera (1360-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (1350) while displaying a screen. In one embodiment, the shooting camera (1360-4) may be positioned on a bridge (1303) positioned between the first rim (1301) and the second rim (1302).

[0193] The gaze tracking camera (1360-1) can implement more realistic augmented reality by tracking the gaze of a user wearing a wearable device (1300) and matching the user's gaze with visual information provided to at least one display (1350). For example, when the wearable device (1300) looks straight ahead, the wearable device (1300) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1350). The gaze tracking camera (1360-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1360-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1360-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1360-1) may be positioned within the first rim (1301) and / or the second rim (1302) to face the direction in which the user wearing the wearable device (1300) is positioned.

[0194] The gesture recognition cameras (1360-2, 1360-3) can recognize the movement of the user's entire body, such as the user's torso, hands, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (1350). The gesture recognition cameras (1360-2, 1360-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1350). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition cameras (1360-2, 1360-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (1360-2, 1360-3). In one embodiment, the motion recognition cameras (1360-2, 1360-3) may be positioned on the first rim (1301) and / or the second rim (1302).

[0195] The camera (1360) included in the wearable device (1300) is not limited to the above-described gaze tracking camera (1360-1) and motion recognition cameras (1360-2, 1360-3). For example, the wearable device (1300) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (1300) can identify an external object based on a sensor for identifying the distance between the wearable device (1300) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1360) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (1300) may include a camera (1360) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (1300).

[0196] Although not shown, in one embodiment, the wearable device (1300) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (1360). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (1300) and the hinge units (1306, 1307).

[0197] According to one embodiment, the battery module (1370) may supply power to the electronic components of the wearable device (1300). In one embodiment, the battery module (1370) may be disposed within the first temple (1304) and / or the second temple (1305). For example, the battery module (1370) may be a plurality of battery modules (1370). The plurality of battery modules (1370) may be disposed within each of the first temple (1304) and the second temple (1305). In one embodiment, the battery module (1370) may be disposed at an end of the first temple (1304) and / or the second temple (1305).

[0198] The antenna module (1375) can transmit signals or power to the outside of the wearable device (1300), or receive signals or power from the outside. In one embodiment, the antenna module (1375) can be positioned within the first temple (1304) and / or the second temple (1305). For example, the antenna module (1375) can be positioned close to one surface of the first temple (1304) and / or the second temple (1305).

[0199] The speaker (1355) can output an audio signal to the outside of the wearable device (1300). The audio output module may be referred to as a speaker. In one embodiment, the speaker (1355) may be positioned within the first temple (1304) and / or the second temple (1305) so as to be positioned adjacent to the ear of a user wearing the wearable device (1300). For example, the speaker (1355) may include a second speaker (1355-2) positioned within the first temple (1304) and thus adjacent to the user's left ear, and a first speaker (1355-1) positioned within the second temple (1305) and thus adjacent to the user's right ear.

[0200] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (1300) to the user. For example, when the wearable device (1300) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (1301) and / or the second rim (1302).

[0201] Referring to FIG. 13B, according to one embodiment, a wearable device (1300) may include a printed circuit board (PCB) (1390). The PCB (1390) may be included in at least one of the first temple (1304) or the second temple (1305). The PCB (1390) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (1300) (e.g., hardwares illustrated by different blocks in FIG. 2A and / or FIG. 2B) may be disposed on the PCB (1390). The wearable device (1300) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0202] According to one embodiment, a wearable device (1300) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (1300) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (1300). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (1300) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (1300) based on the IMU.

[0203] Figures 14a and 14b illustrate an example of an exterior appearance of a wearable device according to one embodiment. The wearable device (1400) of Figures 14a and 14b may include at least a portion of the hardware of the wearable device (1300) described with reference to Figures 13a and / or 13b. An example of an exterior appearance of a first side (1410) of a housing of the wearable device (1400) according to one embodiment is illustrated in Figure 14a, and an example of an exterior appearance of a second side (1420) opposite to the first side (1410) may be illustrated in Figure 14b.

[0204] Referring to FIG. 14A, a first surface (1410) of a wearable device (1400) according to one embodiment may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1400) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1304) and / or the second temple (1305) of FIGS. 13A and 13B). A first display (1350-1) for outputting an image to a left eye among the user's two eyes, and a second display (1350-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (1410). The wearable device (1400) may be formed on the first surface (1410) and may further include a rubber or silicone packing to prevent interference from light (e.g., ambient light) different from the light emitted from the first display (1350-1) and the second display (1350-2).

[0205] According to one embodiment, a wearable device (1400) may include cameras (1360-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (1350-1) and the second display (1350-2). The cameras (1360-1) may be referred to as the gaze tracking camera (1360-1) of FIG. 13B. According to one embodiment, a wearable device (1400) may include cameras (1360-5, 1360-6) for photographing and / or recognizing a face of a user. The cameras (1360-5, 1360-6) may be referred to as FT cameras. The wearable device (1400) can control an avatar representing the user in a virtual space based on the facial motion of the user identified using cameras (1360-5, 1360-6). For example, the wearable device (1400) can change the texture and / or shape of a portion of the avatar (e.g., a portion of the avatar representing a human face) using information obtained by cameras (1360-5, 1360-6) (e.g., FT cameras) and representing the facial expression of the user wearing the wearable device (1400).

[0206] Referring to FIG. 14B, a camera (e.g., cameras (1360-7, 1360-8, 1360-9, 1360-10, 1360-11, 1360-12)) and / or a sensor (e.g., a depth sensor (1430)) for obtaining information related to the external environment of the wearable device (1400) may be disposed on a second surface (1420) opposite to the first surface (1410) of FIG. 14A. For example, the cameras (1360-7, 1360-8, 1360-9, 1360-10) may be disposed on the second surface (1420) to recognize external objects. The cameras (1360-7, 1360-8, 1360-9, 1360-10) of FIG. 14b can correspond to the motion recognition cameras (1360-2, 1360-3) of FIG. 13b.

[0207] For example, using cameras (1360-11, 1360-12), the wearable device (1400) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1360-11) can be placed on the second face (1420) of the wearable device (1400) to obtain an image to be displayed through the second display (1350-2) corresponding to the right eye among the two eyes. The camera (1360-12) can be placed on the second face (1420) of the wearable device (1400) to obtain an image to be displayed through the first display (1350-1) corresponding to the left eye among the two eyes. The cameras (1360-11, 1360-12) can correspond to the shooting camera (1360-4) of FIG. 13B.

[0208] According to one embodiment, the wearable device (1400) may include a depth sensor (1430) disposed on a second face (1420) to identify a distance between the wearable device (1400) and an external object. Using the depth sensor (1430), the wearable device (1400) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (1400). Although not illustrated, a microphone may be disposed on the second face (1420) of the wearable device (1400) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0209] In one embodiment, a method may be required to more accurately identify the intent of a user who has performed a user input that does not correspond to a function. According to one embodiment, a wearable device (e.g., the wearable device (101) of FIG. 1) as described above includes a plurality of sensors (e.g., the sensor (230) of FIG. 2A), a display (e.g., the display (220) of FIG. 2A), a memory for storing instructions (e.g., the memory (215) of FIG. 2A) and at least one processor for executing the instructions (e.g., the processor (210) of FIG. 2A), wherein the at least one processor uses a first sensor among the plurality of sensors to generate a plurality of virtual objects (e.g., the virtual object (140) of FIG. 1, the virtual objects (511, 512, 513) of FIG. 5, the virtual objects (621, 622, 623, 624, 625, 630) of FIGS. 6 and 7, the virtual objects (810, 820) of FIG. 8). The at least one processor may be configured to identify a first input for a point spaced apart from all of the virtual objects (930 of FIG. 9, 1010 of FIG. 10). The at least one processor may be configured to identify, within a designated period of time after identifying the first input, at least one virtual object adjacent to the point among the plurality of virtual objects in response to a second input for the point identified based on the first sensor. The at least one processor may be configured to display, on the display, a visual object (e.g., visual objects (162, 164) of FIG. 1, visual objects (531, 532) of FIG. 1, visual object (640) of FIG. 6, visual objects (720, 730) of FIG. 7, visual objects (830, 840) of FIG. 8) indicating a third input identifiable by a second sensor among the plurality of sensors and corresponding to a function of the at least one virtual object.

[0210] For example, the at least one processor may be configured to identify, based on attribute information of the at least one virtual object, the second sensor among the plurality of sensors to be used for receiving the third input based on the second sensor.

[0211] For example, the at least one processor may be configured, in response to the second input, to compare distances between the point and the plurality of virtual objects to identify the at least one virtual object adjacent to the point.

[0212] For example, the at least one processor may be configured to identify the distances based on three-dimensional coordinate values ​​assigned to the plurality of virtual objects and three-dimensional coordinate values ​​of the point. The at least one virtual object may be positioned adjacent to the point on the display based on the three-dimensional coordinate values ​​assigned to the at least one virtual object.

[0213] For example, the at least one processor may be configured to identify the first input, which includes a hand gesture of a user wearing the wearable device, using sensor data from the first sensor, which is an image sensor. The at least one processor may be configured to display the visual object, which includes text corresponding to a utterance associated with the function, based on the second sensor, which is a microphone.

[0214] For example, the at least one processor may be configured to, within a state of identifying the hand gesture based on sensor data of the first sensor, display on the display a line (e.g., line (152) of FIG. 1) extending from a hand performing the hand gesture to the point.

[0215] For example, the at least one processor may be configured to display the visual object at a point adjacent to the at least one virtual object displayed on the display.

[0216] For example, the at least one processor may be configured to, after identifying the first input for the point, which is a first point, display the visual object in response to the second input, which is identified at a second point within a specified range from the first point and spaced apart from all of the plurality of virtual objects.

[0217] According to one embodiment, a method of a wearable device as described above may include an operation of identifying a first input for a point spaced from all of a plurality of virtual objects using a first sensor among a plurality of sensors of the wearable device (e.g., operation 1210 of FIG. 12 ). The method may include an operation of identifying at least one virtual object adjacent to the point among the plurality of virtual objects in response to a second input for the point identified based on the first sensor within a specified period of time after identifying the first input (e.g., operation 1220 of FIG. 12 ). The method may include an operation of displaying, on a display of the wearable device, a visual object representing a third input identifiable by a second sensor among the plurality of sensors and corresponding to a function of the at least one virtual object.

[0218] For example, the displaying operation may include an operation of identifying the second sensor to be used for receiving the third input among the plurality of sensors based on attribute information of the at least one virtual object.

[0219] For example, the act of identifying the at least one virtual object may include, in response to the second input, an act of comparing distances between the point and the plurality of virtual objects to identify the at least one virtual object adjacent to the point.

[0220] For example, the operation of identifying the at least one virtual object may include an operation of identifying the distances based on three-dimensional coordinate values ​​assigned to the plurality of virtual objects and three-dimensional coordinate values ​​of the point. The at least one virtual object may be positioned adjacent to the point on the display based on the three-dimensional coordinate values ​​assigned to the at least one virtual object.

[0221] For example, the action of identifying the first input may include an action of identifying the first input including a hand gesture of a user wearing the wearable device using sensor data of the first sensor, which is an image sensor. The action of displaying may include an action of displaying the visual object including text corresponding to a utterance linked to the function based on the second sensor, which is a microphone.

[0222] For example, the act of identifying the first input may include, within a state of identifying the hand gesture based on sensor data of the first sensor, displaying, on the display, a line extending from a hand performing the hand gesture to the point.

[0223] For example, the displaying action may include displaying the visual object at a point adjacent to the at least one virtual object displayed on the display.

[0224] For example, the displaying action may include, after identifying the first input for the point, which is a first point, displaying the visual object in response to the second input identified at a second point within a specified range from the first point and spaced apart from all of the plurality of virtual objects.

[0225] According to one embodiment, a wearable device as described above may include one or more sensors, a display, a memory, and at least one processor. The at least one processor may be configured to determine, based on identifying a first input for a first point in a virtual space provided through the display using the one or more sensors, whether a virtual object has been identified at the first point. The at least one processor may be configured to, in a first state in which the first virtual object has been identified at the first point, execute a first function assigned to the first virtual object and indicated by the first input. The at least one processor may be configured to, in a second state different from the first state, identify a second virtual object located at a second point in the virtual space adjacent to the first point, based on identifying a second input for the first point. The at least one processor may be configured to display a visual object on the display, the visual object including text for invoking a second function executable based on the second virtual object.

[0226] For example, the at least one processor may be configured to display the visual object based on identifying the second input within a specified period of time after identifying the first input.

[0227] For example, a wearable device may include a microphone. The at least one processor may be configured to determine whether a utterance including the text has been received based on audio data acquired using the microphone switched to an active state while displaying the visual object.

[0228] For example, the at least one processor may be configured to display the visual object at a third location on the display adjacent to the second virtual object.

[0229] According to one embodiment of the present invention, a method of a wearable device as described above may include an operation of determining whether a virtual object has been identified at a first point in a virtual space provided through a display of the wearable device based on identifying a first input for a first point using one or more sensors of the wearable device. The method may include an operation of executing a first function assigned to the first virtual object and indicated by the first input within a first state in which the first virtual object has been identified at the first point. The method may include an operation of identifying a second virtual object disposed at a second point in the virtual space adjacent to the first point based on identifying a second input for the first point within a second state different from the first state. The method may include an operation of displaying a visual object on the display including text for invoking a second function executable based on the second virtual object.

[0230] For example, the displaying action may include displaying the visual object based on identifying the second input within a specified period of time after identifying the first input.

[0231] For example, the displaying action may include an action of determining whether an utterance including the text has been received based on audio data obtained using a microphone of the wearable device that has been switched to an active state while displaying the visual object.

[0232] For example, the displaying action may include displaying the visual object at a third point of the display adjacent to the second virtual object.

[0233] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0234] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0235] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0236] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0237] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In a wearable device (101), Multiple sensors (230); display (220); A memory (215) storing instructions, including one or more storage media; and At least one processor (210) including a processing circuit, wherein the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Identifying a first input for a point spaced apart from all of the plurality of virtual objects (140; 511, 512, 513; 621, 622, 623, 624, 625, 630; 810, 820; 930; 1010) using a first sensor among the plurality of sensors; In response to a second input for the point identified based on the first sensor within a specified period of time after identifying the first input, identifying at least one virtual object adjacent to the point among the plurality of virtual objects; and Causing to display, on the display, a visual object (162, 164; 531, 532; 640; 720, 730; 830, 840) representing a third input, identifiable by a second sensor among the plurality of sensors and corresponding to a function of the at least one virtual object. Wearable devices.

2. In claim 1, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Based on the attribute information of the at least one virtual object, causing the second sensor to be used for receiving the third input based on the second sensor among the plurality of sensors to be identified. Wearable devices.

3. In claims 1 and 2, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: In response to said second input, comparing distances between said point and said plurality of virtual objects, thereby causing at least one virtual object adjacent to said point to be identified. Wearable devices.

4. In claims 1 to 3, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: To cause the distances to be identified based on the three-dimensional coordinate values ​​assigned to the plurality of virtual objects and the three-dimensional coordinate values ​​of the point, At least one virtual object of the above, Based on the three-dimensional coordinate values ​​assigned to at least one virtual object, positioned adjacent to the point on the display, Wearable devices.

5. In claims 1 to 4, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Using sensor data of the first sensor, which is an image sensor, identifying the first input including a hand gesture of a user wearing the wearable device; Causing said visual object to be displayed, said visual object including text corresponding to said utterance associated with said function, based on said second sensor being a microphone; Wearable devices.

6. In claims 1 to 5, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: In a state of identifying the hand gesture based on the sensor data of the first sensor, causing a line (152) to be displayed on the display, extending from the hand performing the hand gesture to the point. Wearable devices.

7. In claims 1 to 6, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: causing the visual object to be displayed at a point adjacent to the at least one virtual object displayed on the display; Wearable devices.

8. In claims 1 to 7, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: After identifying the first input for the point, which is the first point, in response to the second input identified at a second point within a specified range from the first point and spaced apart from all of the plurality of virtual objects, causing the visual object to be displayed. Wearable devices.

9. In the method of a wearable device, An operation (1210) of identifying a first input for a point spaced apart from all of a plurality of virtual objects using a first sensor among a plurality of sensors of the wearable device; An operation (1220) of identifying at least one virtual object adjacent to the point among the plurality of virtual objects in response to a second input for the point identified based on the first sensor within a specified period after identifying the first input; and An operation (1230) of displaying a visual object on a display of the wearable device, the visual object being identifiable by a second sensor among the plurality of sensors and indicating a third input corresponding to a function of the at least one virtual object. method.

10. In claim 9, the indicated action is: An operation of identifying the second sensor to be used for receiving the third input among the plurality of sensors based on the attribute information of the at least one virtual object, method.

11. In claims 9 to 10, the operation of identifying at least one virtual object comprises: In response to said second input, comprising an operation of comparing distances between said point and said plurality of virtual objects to identify at least one virtual object adjacent to said point; method.

12. In claims 9 to 11, the operation of identifying at least one virtual object comprises: An operation of identifying the distances based on the three-dimensional coordinate values ​​assigned to the plurality of virtual objects and the three-dimensional coordinate values ​​of the point, At least one virtual object of the above, Based on the three-dimensional coordinate values ​​assigned to at least one virtual object, positioned adjacent to the point on the display, method.

13. In claims 9 to 12, the operation of identifying the first input comprises: An operation of identifying the first input including a hand gesture of a user wearing the wearable device using sensor data of the first sensor, which is an image sensor, The actions indicated above are: Including an action of displaying said visual object including text corresponding to a utterance linked to said function based on said second sensor which is a microphone; method.

14. In claims 9 to 13, the operation of identifying the first input comprises: Including an action of displaying a line extended from a hand performing the hand gesture to the point on the display, within a state of identifying the hand gesture based on the sensor data of the first sensor. method.

15. In claims 9 to 14, the indicated action is: An action comprising displaying the visual object at a point adjacent to the at least one virtual object displayed on the display. method.

Images