Wearable device, method, and non-transitory computer-readable storage medium for interaction with user's gaze

Abstract

This wearable device may comprise at least one memory for storing instructions. The instructions may be configured to cause, when executed by at least one processor of the wearable device, the wearable device to, in response to generation of data for outputting an execution result of a function indicated by a speech input, identify whether a gaze is located within a first reference distance from a visual object related to an AI speech recognition model, which is displayed on a display of the wearable device, the instructions may be configured to cause, when executed by the at least one processor, the wearable device to, in response to identification that the gaze is located within the first reference distance from the visual object, output the execution result on the basis of the data for outputting the execution result, and the instructions may be configured to cause, when executed by the at least one processor, the wearable device to, in response to identification that the gaze is located outside the first reference distance from the visual object, delay the output of the execution result.

Description

Wearable device, method, and non-transitory computer-readable storage medium for interaction with user gaze

[0001] The following descriptions relate to a wearable device, method, and non-transitory computer readable storage medium for interaction with a user's gaze.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] A wearable device is disclosed. The wearable device may include at least one sensor that acquires data for identifying the gaze of a user wearing the wearable device. The wearable device may include a display capable of displaying a stereoscopic image. The wearable device may include at least one microphone. The wearable device may include at least one processor. The wearable device may include at least one memory storing instructions. The instructions, when executed by the at least one processor, may cause the wearable device to acquire a voice input from the user through the at least one microphone. The instructions, when executed by the at least one processor, may cause the wearable device to execute a function indicated by the voice input based on an artificial intelligence (AI) voice recognition model. The instructions, when executed by the at least one processor, may cause the wearable device to identify the gaze of the user based on the data acquired through the at least one sensor. The instructions, when executed by the at least one processor, may cause the wearable device, in response to data being generated for outputting an execution result of the function indicated by the voice input, to identify whether the gaze is located within a first reference distance from a visual object associated with the AI ​​voice recognition model displayed on the display. The instructions, when executed by the at least one processor, may cause the wearable device, in response to identifying that the gaze is located within the first reference distance from the visual object, to output the execution result based on the data for outputting the execution result.The instructions, when executed by the at least one processor, may cause the wearable device to postpone outputting the execution result in response to identifying that the gaze is located outside the first reference distance from the visual object.

[0004] A method is disclosed. The method can be executed on a wearable device comprising at least one sensor for acquiring data for identifying a gaze of a user wearing the wearable device, a display capable of displaying a stereoscopic image, and at least one microphone. The method can include an operation of acquiring a voice input of the user via the at least one microphone. The method can include an operation of executing a function indicated by the voice input based on an AI voice recognition model. The method can include an operation of identifying the gaze of the user based on the data acquired via the at least one sensor. The method can include an operation of identifying, in response to generating data for outputting an execution result of the function indicated by the voice input, whether the gaze is located within a first reference distance from a visual object associated with the AI ​​voice recognition model displayed on the display. The method can include an operation of outputting an execution result based on the data for outputting the execution result in response to identifying the gaze as being located within the first reference distance from the visual object. The method may include an action of postponing output of the execution result in response to identifying that the gaze has moved away from the visual object by the first reference distance.

[0005] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium can store a program including instructions. The instructions, when executed by a processor of the wearable device, the wearable device including at least one sensor for obtaining data for identifying the gaze of a user wearing the wearable device and a display capable of displaying a stereoscopic image, can cause the wearable device to obtain a voice input from the user through the at least one microphone. The instructions, when executed by the processor, can cause the wearable device to perform a function indicated by the voice input based on an AI voice recognition model. The instructions, when executed by the processor, can cause the wearable device to identify the gaze of the user based on the data obtained through the at least one sensor. The instructions, when executed by the processor, may cause the wearable device, in response to generating data for outputting an execution result of the function indicated by the voice input, to identify whether the gaze is located within a first reference distance from a visual object associated with the AI ​​voice recognition model displayed on the display. The instructions, when executed by the processor, may cause the wearable device, in response to identifying the gaze as being located within the first reference distance from the visual object, to output the execution result based on the data for outputting the execution result. The instructions, when executed by the processor, may cause the wearable device, in response to identifying the gaze as being located outside the first reference distance from the visual object, to postpone outputting the execution result.

[0006] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0007] FIG. 2A illustrates an example of a perspective view of a wearable device, according to one embodiment.

[0008] FIG. 2b illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.

[0009] FIG. 3A illustrates an example of an appearance of a wearable device according to one embodiment.

[0010] FIG. 3b illustrates an example of an appearance of a wearable device according to one embodiment.

[0011] FIG. 4 illustrates an example block diagram of a wearable device according to one embodiment.

[0012] FIG. 5A illustrates an example of a field of view (FOV) of a user wearing a wearable device in one embodiment.

[0013] FIG. 5b illustrates an example of a situation in which input from a user wearing a wearable device is obtained in one embodiment.

[0014] FIG. 6A illustrates an example of a screen displayed by a wearable device in one embodiment.

[0015] FIG. 6b illustrates an example of a screen displayed by a wearable device in one embodiment.

[0016] FIG. 6c illustrates an example of a screen displayed by a wearable device in one embodiment.

[0017] FIG. 6d illustrates an example of a screen displayed by a wearable device in one embodiment.

[0018] FIG. 6e illustrates an example of a screen displayed by a wearable device in one embodiment.

[0019] FIG. 7 illustrates an example of a screen displayed by a wearable device in one embodiment.

[0020] FIG. 8 illustrates an example of a screen displayed by a wearable device in one embodiment.

[0021] Figure 9 illustrates a flowchart of operations performed by a wearable device in one embodiment.

[0022] Figure 10 illustrates a flowchart of operations performed by a wearable device in one embodiment.

[0023] Figure 11 illustrates a flowchart of operations performed by a wearable device in one embodiment.

[0024] Figure 12 illustrates a flowchart of operations performed by a wearable device in one embodiment.

[0025] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0026] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0027] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0028] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0029] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0030] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0034] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0035] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0039] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0041] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 664 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip) for URLLC realization.

[0044] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0045] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0046] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048] FIG. 2A illustrates an example of a perspective view of a wearable device (200), according to one embodiment. FIG. 2B illustrates an example of one or more hardware elements disposed within a wearable device (200), according to one embodiment. The wearable device (200) of FIGS. 2A and 2B may correspond to the electronic device (101) of FIG. 1. As shown in FIG. 2A, the wearable device (200), according to one embodiment, may include at least one display (250) and a frame supporting the at least one display (250).

[0049] According to one embodiment, the wearable device (200) can be worn on a part of a user's body. The wearable device (200) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (200). For example, the wearable device (200) can output a virtual reality image to the user through at least one display (250) in response to a designated gesture of the user acquired through the motion recognition camera (240-2) of FIG. 2B.

[0050] According to one embodiment, at least one display (250) within a wearable device (200) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0051] Referring to FIG. 2B, at least one display (250) may form a display area on a lens to provide a user wearing the wearable device (200) with visual information contained in external light passing through the lens, along with other visual information distinct from the visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. The display area formed by at least one display (250) may be formed on the second surface (232) among the first surface (231) and the second surface (232) of the lens. When the user wears the wearable device (200), external light may be incident on the first surface (231) and transmitted through the second surface (232) to be transmitted to the user. As another example, at least one display (250) may display a virtual reality image to be combined with a real screen transmitted through the external light. The virtual reality image output from at least one display (250) can be transmitted to the user's eyes through one or more hardware (e.g., optical devices (282, 284), and / or at least one waveguide (233, 234)) included in the wearable device (200).

[0052] According to one embodiment, a wearable device (200) may include waveguides (233, 234) that diffract light transmitted from at least one display (250) and relayed by optical devices (282, 284) and transmit the diffracted light to a user. The waveguides (233, 234) may be formed based on at least one of glass, plastic, or polymer. Nanopatterns may be formed on at least a portion of the exterior or interior of the waveguides (233, 234). The nanopatterns may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the waveguides (233, 234) may be propagated to the other end of the waveguides (233, 234) by the nanopatterns. The waveguides (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, the waveguides (233, 234) may be arranged within the wearable device (200) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the waveguides (233, 234).

[0053] According to one embodiment, the wearable device (200) may analyze an object included in a real image collected through a capturing camera (240-1), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object may include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (200) may analyze the object based on a multi-camera, such as a stereo camera. For the object analysis, the wearable device (200) may execute time-of-flight (ToF) and / or simultaneous localization and mapping (SLAM) supported by the multi-camera. A user wearing the wearable device (200) may view an image displayed on at least one display (250).

[0054] According to one embodiment, the frame may be formed as a physical structure that allows the wearable device (200) to be worn on the user's body. According to one embodiment, the frame may be configured so that, when the user wears the wearable device (200), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame may support at least one display (250). For example, the frame may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0055] Referring to FIG. 2A, the frame may include a region (220) that at least partially contacts a part of the user's body when the user wears the wearable device (200). For example, the region (220) of the frame that contacts a part of the user's body may include a region that contacts a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (200) makes contact with. According to one embodiment, the frame may include a nose pad (210) that contacts a part of the user's body. When the wearable device (200) is worn by the user, the nose pad (210) may contact a part of the user's nose. The frame may include a first temple (204) and a second temple (205) that contact a part of the user's body that is distinct from the part of the user's body.

[0056] For example, the frame may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of an ear of the wearer, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of an ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (200) can identify an external object (e.g., a user's fingertip) touching the frame and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame.

[0057] According to one embodiment, the wearable device (200) may include hardwares that perform various functions (e.g., hardwares described above based on the block diagram of FIG. 1). For example, the hardwares may include a battery module (270), an antenna module (275), optical devices (282, 284), speakers (292-1, 292-2), microphones (294-1, 294-2, 294-3), a light-emitting module (not shown), and / or a printed circuit board (290). The various hardwares may be arranged within a frame.

[0058] According to one embodiment, the microphones (294-1, 294-2, 294-3) of the wearable device (200) may be arranged on at least a portion of the frame to acquire a sound signal. A first microphone (294-1) arranged on the nose pad (210), a second microphone (294-2) arranged on the second rim (202), and a third microphone (294-3) arranged on the first rim (201) are illustrated in FIG. 2B, but the number and arrangement of the microphones (294) are not limited to the embodiment of FIG. 2B. When the number of microphones (294) included in the wearable device (200) is two or more, the wearable device (200) may identify the direction of the sound signal by using a plurality of microphones arranged on different portions of the frame.

[0059] According to one embodiment, the optical devices (282, 284) can transmit a virtual object transmitted from at least one display (250) to the wave guides (233, 234). For example, the optical devices (282, 284) can be projectors. The optical devices (282, 284) can be positioned adjacent to at least one display (250) or can be included within at least one display (250) as a part of the at least one display (250). The first optical device (282) can correspond to the first display (250-1), and the second optical device (284) can correspond to the second display (250-2). The first optical device (282) can transmit light output from the first display (250-1) to the first waveguide (233), and the second optical device (284) can transmit light output from the second display (250-2) to the second waveguide (234).

[0060] In one embodiment, the camera (240) may include an eye tracking camera (ET CAM) (240-1), a motion recognition camera (240-2), and / or a recording camera (240-3). The recording camera, the eye tracking camera (240-1), and the motion recognition camera (240-2) may be positioned at different locations on the frame and may perform different functions. The eye tracking camera (240-1) may output data representing the gaze of a user wearing the wearable device (200). For example, the wearable device (200) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (240-1). An example in which the gaze tracking camera (240-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (240-1) may be positioned solely toward the user's left eye, or may be positioned toward both eyes.

[0061] In one embodiment, the capturing camera (240-3) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about the actual image or background including the image of the specific object acquired using the capturing camera is superimposed on a virtual image provided through optical devices (282, 284). In one embodiment, the capturing camera can be placed on a bridge (203) disposed between the first rim (201) and the second rim (202).

[0062] In one embodiment, the gaze tracking camera (240-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (200) and thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (200) looks straight ahead, the wearable device (200) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (240-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (240-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (240-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (240-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (200) is positioned.

[0063] In one embodiment, the gesture recognition camera (240-2) can provide a specific event on a screen provided on at least one display (250) by recognizing the movement of the user's entire body or a part of the user's body, such as the user's torso, hands, or face. The gesture recognition camera (240-2) can recognize the user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the gesture and perform a designated function based on the identification. In one embodiment, the gesture recognition camera (240-2) can be disposed on the first rim (201) and / or the second rim (202).

[0064] In one embodiment, the camera (240) included in the wearable device (200) is not limited to the gaze tracking camera (240-1) and the motion recognition camera (240-2) described above. For example, the wearable device (200) can identify an external object included in the FoV using the photographing camera (240-3) positioned toward the user's FoV. The wearable device (200) can identify an external object based on a sensor for identifying the distance between the wearable device (200) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (240) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (200) may include a camera (240) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (200).

[0065] Although not shown, in one embodiment, the wearable device (200) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (240). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame and hinge units (206, 207).

[0066] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (200). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0067] In one embodiment, the antenna module (275) can transmit signals or power to the outside of the wearable device (200), or receive signals or power from the outside. The antenna module (275) can be electrically and / or operatively connected to the communication circuit (250) of FIG. 2. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0068] According to one embodiment, the speakers (292-1, 292-2) may output audio signals to the outside of the wearable device (200). The audio output module may be referred to as a speaker. In one embodiment, the speakers (292-1, 292-2) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ears of a user wearing the wearable device (200). For example, the wearable device (200) may include a second speaker (292-2) positioned within the first temple (204) and thus adjacent to the user's left ear, and a first speaker (292-1) positioned within the second temple (205) and thus adjacent to the user's right ear.

[0069] In one embodiment, the light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state to visually provide information regarding a specific state of the wearable device (200) to the user. For example, when the wearable device (200) requires charging, the wearable device (200) may repeatedly emit red light at a designated time. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0070] Referring to FIG. 2B, according to one embodiment, a wearable device (200) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) and the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardware components included in the wearable device (200) may be disposed on the PCB (290). The wearable device (200) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0071] According to one embodiment, a wearable device (200) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (200) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (200) may identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable device (200) based on the IMU.

[0072] FIGS. 3A and 3B illustrate an example of an exterior appearance of a wearable device (300) according to one embodiment. The wearable device (300) of FIGS. 3A and 3B may be included in the electronic device (101) of FIG. 1. An example of an exterior appearance of a first side (310) of a housing of the wearable device (300) according to one embodiment is illustrated in FIG. 3A, and an example of an exterior appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B.

[0073] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (300) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (300) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (300) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0074] According to one embodiment, the wearable device (300) may include cameras (340-1, 340-2) for photographing and / or tracking the user's two eyes adjacent to the first display (250-1) and the second display (250-2), respectively. The cameras (340-1, 340-2) may be referred to as ET cameras. According to one embodiment, the wearable device (300) may include cameras (340-3, 340-4) for photographing and / or recognizing the user's face. The cameras (340-3, 340-4) may be referred to as FT cameras.

[0075] Referring to FIG. 3B, a camera (e.g., cameras (340-5, 340-6, 340-7, 340-8, 340-9, 340-10)) and / or a sensor (e.g., depth sensor (330)) for obtaining information related to the external environment of the wearable device (300) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A. For example, the cameras (340-5, 340-6, 340-7, 340-8, 340-9, 340-10) may be disposed on the second surface (320) to recognize an external object different from the wearable device (300). For example, using cameras (340-9, 340-10), the wearable device (300) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (340-9) can be placed on the second face (320) of the wearable device (300) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (340-10) can be placed on the second face (320) of the wearable device (300) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes.

[0076] According to one embodiment, the wearable device (300) may include a depth sensor (330) disposed on the second surface (320) to identify a distance between the wearable device (300) and an external object. Using the depth sensor (330), the wearable device (300) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (300).

[0077] Although not shown, a microphone may be placed on the second side (320) of the wearable device (300) to acquire sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0078] As described above, according to one embodiment, the wearable device (300) may have a form factor for being worn on a user's head. The wearable device (300), while worn on the head, may provide a user experience based on augmented reality, virtual reality, and / or mixed reality. Using cameras (340-5, 340-6, 340-7, 340-8, 340-9, 340-10) for recording video of an external space, the wearable device (300) and a server connected to the wearable device (300) (e.g., server (110) of FIG. 1) may provide an on-demand service that provides video of a location and / or place selected by the user, and / or a metaverse service.

[0079] According to one embodiment, the wearable device (300) can display frames acquired through cameras (340-9, 340-10) on the first display (250-1) and the second display (250-2), respectively. The wearable device (300) can provide a user with a mixed user experience of real objects and virtual objects (e.g., video see-through (VST)) by combining virtual objects within frames displayed through the first display (250-1) and the second display (150-2), which include real objects. The wearable device (300) can change the virtual objects based on information acquired by the cameras (340-1, 340-2, 340-3, 340-4, 340-5, 340-6, 340-7, 340-8) and / or the depth sensor (330). For example, if a visual object corresponding to a real object and a virtual object at least partially overlap within the frame, the wearable device (300) may stop displaying the virtual object based on detecting a motion for interacting with the real object. By stopping displaying the virtual object, the wearable device (300) may prevent the visibility of the real object from being reduced as the visual object corresponding to the real object is occluded by the virtual object.

[0080] FIG. 4 illustrates an example of a block diagram of a wearable device (401) according to one embodiment. The wearable device (401) of FIG. 4 may correspond to the electronic device (101) of FIG. 1. The wearable device (401) of FIG. 4 may correspond to the wearable device (200) of FIG. 2. The wearable device (401) of FIG. 4 may correspond to the wearable device (300) of FIG. 3.

[0081] Referring to FIG. 4, a wearable device (401) according to one embodiment may include at least one of a processor (410), a memory (415), a display (420), a camera (425), a sensor (430), or a communication circuit (435). The processor (410) of FIG. 4 may correspond to the processor (120) of FIG. 1. The memory (415) of FIG. 4 may correspond to the memory (130) of FIG. 1. The display (420) of FIG. 4 may correspond to the display module (160) of FIG. 1. The camera (425) of FIG. 4 may correspond to the camera module (180) of FIG. 1. The sensor (430) of FIG. 4 may correspond to the sensor module (176) of FIG. 1. The communication circuit (435) of FIG. 4 may correspond to the communication module (190) of FIG. 1.

[0082] The processor (410), memory (415), display (420), camera (425), sensor (430), and communication circuit (435) may be electrically and / or operably coupled with each other by electronic components such as a communication bus (402). The type and / or number of hardware components included in the wearable device (200) are not limited to those illustrated in FIG. 4. For example, the wearable device (401) may include only some of the hardware components illustrated in FIG. 4. Elements (e.g., layers and / or modules) within the memory described below may be logically separated, but are not limited thereto.

[0083] A processor (410) of a wearable device (401) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (410) may be one or more. For example, the processor (410) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0084] The memory (415) of the wearable device (401) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (410). The memory (415) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi media card (eMMC).

[0085] In one embodiment, the display (420) of the wearable device (401) can output visualized information to a user of the wearable device (401). For example, the display (420) can be controlled by a processor (410) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (420) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0086] In one embodiment, the camera (425) of the wearable device (401) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (425) may be arranged in the form of a two-dimensional array. The camera (425) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (425) may mean one (a) two-dimensional frame data acquired from the camera (425). For example, video data captured using the camera (425) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (425) according to a frame rate. The camera (425) may further include a flash light that is positioned toward the direction in which the camera (425) receives light and outputs light toward the direction.

[0087] According to one embodiment, the wearable device (401) may include a plurality of cameras, for example, cameras (425), arranged in different directions. A first camera among the plurality of cameras may be referred to as a motion recognition camera (e.g., motion recognition cameras 260-2 and 260-3 of FIG. 2B ), and a second camera may be referred to as a gaze tracking camera (e.g., gaze tracking camera 260-1 of FIG. 2B ). The wearable device (401) may identify a position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (401) may identify a direction of a gaze of a user wearing the wearable device (401) using an image acquired using the second camera. For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0088] According to one embodiment, a sensor (430) of a wearable device (401) may generate electrical information that may be processed by a processor (410) and / or a memory (415) of the wearable device (401) from non-electronic information related to the wearable device (401). The information may be referred to as sensor data. The sensor (430) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (401), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (401).

[0089] In one embodiment, the communication circuit (435) of the wearable device (401) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (401) and an external electronic device. The communication circuit (435) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (435) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0090] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (410) of the wearable device (401) may be stored in the memory (415) of the wearable device (401). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (401) and / or the processor (410) may perform at least one of the operations of FIG. 6 or FIG. 11 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (401) may mean that one or more instructions provided in the form of an application are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (401)). As an example, an application may include a program and / or a library related to a service provided to a user.

[0091] Referring to FIG. 4, programs installed in the wearable device (401) may be classified into any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (401) (e.g., the display (420), the camera (420), and / or the sensor (430)) may be classified within the hardware abstraction layer (480). The framework layer (450) may be referred to as an XR framework layer in that it includes one or more programs for providing an XR (extended reality) service. For example, FIG. 4 illustrates layers being divided within the memory (415), the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (415).

[0092] For example, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a position tracker (471), a space recognizer (472), a gesture tracker (473), and / or an eye tracker (474), a face tracker (475)) may be classified. Programs classified within the framework layer (450) may provide an executable API (application programming interface) based on other programs.

[0093] For example, within the application layer (440), programs designed to target users controlling wearable devices (401) may be classified. Examples of programs classified into the application layer (440) include, but are not limited to, an XR (extended reality) system UI (user interface) and / or an XR application (442). For example, programs (e.g., software applications) classified into the application layer (440) may call an API (application programming interface) to cause execution of functions supported by programs classified into the framework layer (450).

[0094] For example, the wearable device (401) may display one or more visual objects on the display (420) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (441). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (401) may provide a service for controlling functions available within a virtual space to the user based on the execution of the XR system UI (441).

[0095] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plug-in (444) are illustrated as being included within the XR system UI (441), but are not limited thereto. For example, the XR system UI (441) may cause execution of functions supported by the lightweight renderer (443) and / or the XR plug-in (444) included within the framework layer (450).

[0096] For example, the wearable device (401) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (442). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (443) may include a renderer that is built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (401) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (444). The XR plugin (444) may be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0097] For example, the wearable device (401) may display a screen representing at least a portion of a virtual space on the display (420) based on the execution of the XR application (442). The XR plug-in (444-1) included in the XR application (442) may be referenced by the XR plug-in (444) of the XR system UI (441). Any description of the XR plug-in (444-1) that overlaps with the description of the XR plug-in (444) may be omitted. The wearable device (401) may cause the execution of the virtual space manager (451) based on the execution of the XR application (442).

[0098] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the virtual space manager (451). For example, the virtual space manager (451) may include a platform (e.g., an Android platform) for supporting the virtual space service. The wearable device (401) may display the posture of a virtual object representing the user's posture rendered using data acquired through the sensor (430) on the display based on the execution of the virtual space manager (451). The virtual space manager (451) may be referred to as a composition presentation manager (CPM).

[0099] For example, the virtual space manager (451) may include a runtime service (452). As an example, the runtime service (452) may be referred to as an OpenXR runtime module. The wearable device (401) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (401) based on the execution of the runtime service (452). As an example, the wearable device (401) may be used to perform rendering for a virtual space service to a user based on the execution of the runtime service (452). For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the execution of the runtime service (452).

[0100] For example, the virtual space manager (451) may include a pass-through manager (453). Based on the execution of the pass-through manager (453), the wearable device (401) may display a screen representing a virtual space on the display (420), while superimposing another screen representing an actual space acquired through the camera (425) on at least a portion of the screen.

[0101] For example, the virtual space manager (451) may include an input manager (454). The wearable device (401) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (470) based on the execution of the input manager (454). The wearable device (401) may initiate execution of at least one of the functions of the wearable device (401) using the acquired data.

[0102] For example, the perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). From the perspective of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referenced as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0103] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data acquired from a sensor (430) (or a camera (425)). The one or more programs may include at least one of a position tracker (471), a space recognizer (472), a gesture tracker (473), an eye tracker (474), and / or a face tracker (473). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those illustrated in FIG. 4.

[0104] For example, the wearable device (401) can identify the pose of the wearable device (401) using the sensor (430) based on the execution of the position tracker (471). The wearable device (401) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (401) using data acquired using the camera (425) and the IMU based on the execution of the position tracker (471). The position tracker (471) can be referred to as a head tracking (HeT) module.

[0105] For example, the wearable device (401) may be used to construct a three-dimensional virtual space around the wearable device (401) (or a user of the wearable device (401)) based on the execution of the space recognizer (472). The wearable device (401) may reconstruct the three-dimensional surroundings of the wearable device (401) using data acquired using the camera (425) based on the execution of the space recognizer (472). The wearable device (401) may identify at least one of a plane, a slope, and stairs based on the three-dimensionally reconstructed surroundings of the wearable device (401) based on the execution of the space recognizer (472). The space recognizer (472) may be referred to as a scene understanding (SU) module.

[0106] For example, the wearable device (401) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (401) based on the execution of the gesture tracker (473). As an example, the wearable device (401) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (430) based on the execution of the gesture tracker (473). As an example, the wearable device (401) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using a camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0107] For example, the wearable device (401) can identify (or track) eye movements of a user of the wearable device (401) based on the execution of the gaze tracker (474). As an example, the wearable device (401) can identify eye movements of the user using data acquired from at least one sensor based on the execution of the gaze tracker (474). As an example, the wearable device (401) can identify eye movements of the user based on data acquired using a camera (e.g., the gaze tracking camera (260-1) of FIGS. 2A and 2B) and / or an infrared light emitting diode (IR LED) based on the execution of the gaze tracker (474). The gaze tracker (474) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0108] For example, the recognition service layer (470) of the wearable device (401) may further include a face tracker (475) for tracking the user's face. For example, the wearable device (103) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (475). The wearable device (401) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (475). As an example, the wearable device (401) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (475).

[0109] The memory (415) may include an artificial intelligence (AI) assistant (485). The AI ​​assistant (485) may be included in the application layer (440) and / or the framework layer (450). Here, the AI ​​assistant (485) may also be referred to as an AI speech recognition model. The AI ​​assistant (485) may also be referred to as an AI program and / or an AI module.

[0110] The AI ​​assistant (485) may include an artificial intelligence model (486) for interpreting the intent of the user's input. The AI ​​assistant (485) may include an artificial intelligence model (486) for establishing a plan corresponding to the intent. The AI ​​assistant (485) may include an artificial intelligence model (486) for providing the user with a result produced according to the plan corresponding to the intent. The AI ​​assistant (485) may include an automatic speech recognition (ASR) module (491), a natural language understanding (NLU) module (492), a planner module (493), a natural language generator (NLG) module (494), and a text-to-speech (TTS) module (495). Here, the user's input may include a voice input, but is not limited thereto. For example, the user's input may include a text input.

[0111] According to one embodiment, the automatic speech recognition module (491) can convert input obtained from the wearable device (401) into text data. According to one embodiment, the natural language understanding module (492) can identify the user's intention using the text data of the input. For example, the natural language understanding module (492) can identify the user's intention by performing syntactic analysis or semantic analysis. According to one embodiment, the natural language understanding module (492) can identify the meaning of words extracted from the text data of the input using linguistic features (e.g., grammatical elements) of morphemes or phrases, and can determine the user's intention by matching the meaning of the identified words to the intention.

[0112] According to one embodiment, the planner module (493) can generate a plan using the intent and parameters determined by the natural language understanding module (492). According to one embodiment, the planner module (493) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (493) can determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (493) can determine parameters necessary to execute the plurality of determined operations or result values ​​output by the execution of the plurality of operations. The parameters and the result values ​​can be defined as concepts related to a specified format (or class). Accordingly, the plan can include a plurality of operations and a plurality of concepts determined by the user's intent. The planner module (493) can determine the relationship between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module (493) can determine the execution order of a plurality of actions based on the user's intention based on a plurality of concepts. In other words, the planner module (493) can determine the execution order of a plurality of actions based on parameters required for the execution of the plurality of actions and results output by the execution of the plurality of actions. Accordingly, the planner module (493) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts. The planner module (493) can generate the plan using information stored in a capsule database that stores a set of relationships between concepts and actions.

[0113] According to one embodiment, the natural language generation module (494) can convert specified information into text format. The information converted into text format may be in the form of natural language speech. The text-to-speech conversion module (495) of one embodiment can convert information in text format into information in speech format.

[0114] According to an embodiment, at least some functions of the AI ​​assistant (485) may be performed externally (e.g., server (108)). For example, at least one function executed by at least one module of the automatic speech recognition module (491), the natural language understanding module (492), the planner module (493), the natural language generation module (494), or the text-to-speech conversion module (495) of the AI ​​assistant (485) may be performed externally (e.g., server (108)). When at least some functions of the AI ​​assistant (485) are performed externally (e.g., server (108)), the wearable device (401) may request execution of at least some functions from the external device (e.g., server (108)) and obtain execution results of at least some functions.

[0115] FIG. 5A illustrates an example of a field of view (FOV, 510) of a user (501) wearing a wearable device (401) in one embodiment. FIG. 5B illustrates an example of a situation in which input from a user wearing a wearable device is acquired in one embodiment.

[0116] The wearable device (401) of FIGS. 5A and 5B may correspond to the electronic device (101) of FIG. 1. The wearable device (401) of FIGS. 5A and 5B may correspond to the wearable device (200) of FIGS. 2A and 2B. The wearable device (401) of FIGS. 5A and 5B may correspond to the wearable device (300) of FIGS. 3A and 3B. The wearable device (401) of FIGS. 5A and 5B may correspond to the wearable device (401) of FIG. 4. FIGS. 5A and 5B may be described with reference to FIGS. 1, 2A, 2B, 3A, 3B, and 4.

[0117] Referring to FIG. 5A, a user (501) may wear a wearable device (401). The user (501) may view at least one content (520) within an FOV (510) through the wearable device (401). Here, the FOV (510) may refer to an area viewable by the user (501). The FOV (510) may refer to a display area of ​​the wearable device (401) viewable by the user (501). The FOV (510) may include one or more areas in which at least one content (520) is displayed.

[0118] The processor (410) of the wearable device (401) can display at least one content (520) through the display (420). Here, the at least one content (520) can be displayed as a stereoscopic image on the display (420). The stereoscopic image can be an image that takes into account the binocular disparity of the user (501). The stereoscopic image can be an image for providing the user (501) with a three-dimensional (3D) sense of space. Here, the at least one content (520) can represent the execution result of an application. The at least one content (520) can reflect an actual space. Here, reflecting an actual space can mean that at least one content (520) existing in an actual space can be shown to the user (501) through the display (420) through pass-through. Reflecting the actual space may mean that an image representing the actual space acquired through the camera (425) (or the front camera (240-3, 340-9, 340-10)) via VST (video see through) and / or pass-through is provided to the user. However, the present invention is not limited thereto. An area excluding the area of ​​the content (520) may reflect the actual space. For example, an area excluding the area of ​​the content (520) in the FOV (510) may reflect the actual space. For example, an area excluding the area of ​​the content (520) in the FOV (510) may display an image representing the actual space acquired through the camera (425) (or the front camera (240-3, 340-9, 340-10)).

[0119] The processor (410) can display a visual object (530) related to the AI ​​assistant (485).

[0120] The processor (410) may display a visual object (530) related to the AI ​​assistant (485) at a designated location. The processor (410) may display the visual object (530) at the designated location before the user (501) calls the AI ​​assistant (485). However, the present invention is not limited thereto. The processor (410) may not display the visual object (530) related to the AI ​​assistant (485). The processor (410) may not display the visual object (530) related to the AI ​​assistant (485) before the user (501) calls the AI ​​assistant (485). The processor (410) may display the visual object (530) related to the AI ​​assistant (485) after the user (501) calls the AI ​​assistant (485). Hereinafter, a visual object (530) related to an AI assistant (485) may be referred to as an AI object (530).

[0121] For example, the processor (410) may display a visual object (530) associated with the AI ​​assistant (485) at a fixed location within the FOV (510). Here, displaying at a fixed location within the FOV (510) may mean that the visual object (530) is displayed at a specific location even if the user (501) moves (e.g., even if the user (501) turns his / her head). Fixing the visual object (530) at a fixed location within the FOV (510) may also be referred to as a body lock.

[0122] For example, the processor (410) may display a visual object (530) related to the AI ​​assistant (485) at a fixed location on coordinates based on the user (501) (or, center, or origin). Here, displaying at a fixed location on coordinates based on the user (501) may mean that the display location of the visual object (530) changes according to the movement (and / or rotation) of the user (501). Displaying at a fixed location on coordinates based on the user (501) may also be referred to as world lock.

[0123] The processor (410) can identify (or track) the gaze (505). For example, the processor (410) can identify (or track) the eye movements of the user (501) of the wearable device (401) based on the execution of the gaze tracker (474). The processor (410) can identify a region of interest (ROI) based on the gaze (505). Here, the region of interest can be an area on the FOV (510) where the gaze (505) of the user (501) is located.

[0124] The processor (410) can process user input. The processor (410) can process the user input based on the object where the user's (501) gaze (505) is located. For example, the processor (410) can process the user input through an application related to the object where the user's (501) gaze (505) is located. Here, the user input may include the user's voice acquired through a microphone included in the input module (150), but is not limited thereto. For example, the user input may include a gesture input acquired through a gesture tracker (473). The user input may be acquired through the input module (150) (e.g., a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen)).

[0125] Referring to FIG. 5B, when the user's (501) gaze (505) is located at another visual object (521, 523) that is distinct from the AI ​​object (530), the processor (410) may process user input through an application associated with the other visual object (521, 523). For example, when the other visual object (521, 523) is an avatar of another user, the processor (410) may process user input through an application for interaction with the other user. For example, the processor (410) may process user input through an application for voice chatting and / or phone (or conference call) with the user of the other visual object (521). For example, the processor (410) may provide a voice chat service to the user of the wearable device (401) through an application for voice chat, in response to a user input requesting a voice chat with the user of another visual object (521) while the user's (501) gaze (505) is positioned on the other visual object (521).

[0126] The processor (410) can identify an interaction between the user's (501) gaze (505) and the AI ​​object (530). When the user's (501) gaze (505) is positioned at the AI ​​object (530), the processor (410) can process user input through an application related to the AI ​​object (530). Hereinafter, operations performed by the processor (410) according to the interaction between the user's (501) gaze (505) and the AI ​​object (530) will be described with reference to FIGS. 6a, 6b, 6c, 6d, 7, 8a, and 8b.

[0127] FIG. 6A illustrates an example of a screen (601, 602) displayed by a wearable device (401) in one embodiment.

[0128] The wearable device (401) of FIG. 6A may correspond to the electronic device (101) of FIG. 1. The wearable device (401) of FIG. 6A may correspond to the wearable device (200) of FIGS. 2A and 2B. The wearable device (401) of FIG. 6A may correspond to the wearable device (300) of FIGS. 3A and 3B. The wearable device (401) of FIG. 6A may correspond to the wearable device (401) of FIG. 4. FIG. 6A may be described with reference to FIGS. 1, 2A, 2B, 3A, 3B, and 4.

[0129] Referring to situation (601) of FIG. 6A, the processor (410) can identify that the user's (501) gaze (505) has moved. The processor (410) can identify that the user's (501) gaze (505) has moved based on the execution of the gaze tracker (474).

[0130] For example, the processor (410) can identify that the user's (501) gaze (505) moves from the content (520) to the AI ​​object (530). For example, the processor (410) can identify that the user's (501) gaze (505) toward the content (520) moves to the AI ​​object (530). For example, the processor (410) can identify that the user's (501) gaze (505) moves toward the AI ​​object (530). For example, the processor (410) can identify that the user's (501) area of ​​interest moves from the content (520) to the AI ​​object (530) as the gaze moves.

[0131] Referring to the situation (602) of FIG. 6A, the processor (410) may change the display of the AI ​​object (530) based on whether the user's (501's) gaze (505) is directed toward the AI ​​object (530). For example, the processor (410) may enlarge the size of the AI ​​object (530) based on whether the user's (501's) gaze (505) is directed toward the AI ​​object (530). However, the present invention is not limited thereto. The processor (410) may apply an image effect (e.g., sparkle) to the AI ​​object (530) based on whether the user's (501's) gaze (505) is directed toward the AI ​​object (530). The processor (410) may change the properties (e.g., shape, color) of the surrounding area of ​​the AI ​​object (530). The processor (410) may change the properties (e.g., shape, color) of the AI ​​object (530).

[0132] The processor (410) can obtain the input of the user (501) based on whether the user's (501) gaze (505) is directed toward the AI ​​object (530). The processor (410) can obtain the voice input of the user (501) based on whether the user's (501) gaze (505) is directed toward the AI ​​object (530). For example, the processor (410) can obtain the voice input of the user (501) through a microphone included in the input module (150). However, the present invention is not limited thereto.

[0133] FIG. 6b illustrates an example of a screen displayed by a wearable device (401) in one embodiment.

[0134] The wearable device (401) of FIG. 6B may correspond to the electronic device (101) of FIG. 1. The wearable device (401) of FIG. 6B may correspond to the wearable device (200) of FIGS. 2A and 2B. The wearable device (401) of FIG. 6B may correspond to the wearable device (300) of FIGS. 3A and 3B. The wearable device (401) of FIG. 6B may correspond to the wearable device (401) of FIG. 4. FIG. 6B may be described with reference to FIGS. 1, 2A, 2B, 3A, 3B, and 4.

[0135] Situations (603, 604) of Fig. 6b may illustrate situations subsequent to situations (601, 602) of Fig. 6a.

[0136] The processor (410) may activate the AI ​​assistant (485) based on the user's (501) gaze (505) being directed toward the AI ​​object (530). Here, activating the AI ​​assistant (485) may mean executing the AI ​​assistant (485). Activating the AI ​​assistant (485) may mean transmitting the user's (501) input (e.g., voice input) to the AI ​​assistant (485).

[0137] For example, the processor (410) may obtain an input such as “Check if tomorrow’s meeting attendees are available” through the input module (150). Here, the input may include at least one of a text input or a voice input.

[0138] The processor (410) can identify the intention of the user (501) through the AI ​​assistant (485) based on the input of the user (501). The processor (410) can establish a plan corresponding to the intention of the user (501) through the AI ​​assistant (485). The processor (410) can execute the established plan through the AI ​​assistant (485). The processor (410) can obtain an execution result of the established plan through the AI ​​assistant (485). Here, the execution result can include feedback to be provided to the user (501). The feedback can include at least one of voice feedback and visual feedback. Here, the visual feedback can be based on at least one of an image, a video, or text.

[0139] For example, the processor (410) can identify the intent of the input, "Check if the attendees of tomorrow's meeting are available," through the AI ​​assistant (485). The processor (410) can establish a plan corresponding to the intent of the input, "Check if the attendees of tomorrow's meeting are available," through the AI ​​assistant (485). For example, the processor (410) can identify the attendees of tomorrow's meeting through the AI ​​assistant (485). For example, the processor (410) can send a message to each of the attendees of tomorrow's meeting's contact information (e.g., email address, phone number) through the AI ​​assistant (485) asking whether they are available to attend. For example, the processor (410) can wait for a response to the message through the AI ​​assistant (485). For example, the processor (410) may identify whether tomorrow's meeting attendees are available based on the received response, when the AI ​​assistant (485) has completed receiving the response or when the set response time has elapsed. For example, the processor (410) may generate an execution result including the identification result based on the response, through the AI ​​assistant (485).

[0140] The processor (410) can determine whether the user's (501) input is sufficient. The processor (410) can determine, through the AI ​​assistant (485), whether the user's (501) input is sufficient to establish a plan according to the user's (501) intention. If the user's (501) input is insufficient, the processor (410) can provide a notification to the user (501). Here, the notification provided to the user (501) may be different from the feedback notifying the user (501) that the generation of the execution result has been completed. For example, the notification provided to the user (501) may change the display of the AI ​​object (530) in a different manner from the feedback notifying the user (501) that the generation of the execution result has been completed.

[0141] The processor (410) can identify the intention of the user (501) through the AI ​​assistant (485). The processor (410) can establish a plan corresponding to the intention of the user (501) through the AI ​​assistant (485). The processor (410) can execute the plan established through the AI ​​assistant (485). The processor (410) can obtain an execution result of the established plan through the AI ​​assistant (485). Here, the execution result can include feedback to be provided to the user (501). The feedback can include at least one of voice feedback and visual feedback. Here, the visual feedback can be based on at least one of an image, a video, or text.

[0142] The processor (410) may output a notification related to the AI ​​assistant (485) while obtaining an execution result based on the user's (501) input (e.g., voice input) through the AI ​​assistant (485). For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that it is working. For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that it is working through the audio output module (155) (e.g., speaker). For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that it is working through the display (420). For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that it is working through the audio output module (155) (e.g., speaker) and the display (420).

[0143] Referring to situation (603) of FIG. 6B, the processor (410) can identify that the user's (501) gaze (505) has moved. The processor (410) can identify that the user's (501) gaze (505) has moved based on the execution of the gaze tracker (474). While obtaining an execution result based on the user's (501) input (e.g., voice input) through the AI ​​assistant (485), the processor (410) can identify that the user's (501) gaze (505) has moved from the AI ​​object (530) to the content (520). The processor (410) can identify that the user's (501) gaze (505) has moved from the AI ​​object (530) to the content (520) before the acquisition of the execution result based on the user's (501) input is completed through the AI ​​assistant (485).

[0144] The processor (410) may restore the properties of the AI ​​object (530) and / or the properties surrounding the AI ​​object (530) based on the user's (501) gaze (505) moving away from the AI ​​object (530). Here, restoring the properties may mean changing the properties of the AI ​​object (530) and / or the properties surrounding the AI ​​object (530) to the properties they had before the gaze (505) turned toward the AI ​​object (530).

[0145] For example, the processor (410) can identify that an execution result for the input of the user (501) is generated. For example, the processor (410) can identify that an execution result for the input of the user (501) is generated through the AI ​​assistant (485). For example, the processor (410) can identify that data for outputting an execution result for the input of the user (501) is acquired through the AI ​​assistant (485). The processor (410) can identify that the execution result for the input of the user (501) is acquired after the user's (501) gaze (505) moves away from the AI ​​object (530). The processor (410) can identify that the acquisition of the execution result for the input of the user (501) is completed after the user's (501) gaze (505) moves away from the AI ​​object (530).

[0146] The processor (410) may notify the user (501) that the generation of the execution result is complete. The processor (410) may notify the user (501) that the generation of feedback related to the execution result is complete. The processor (410) may provide a notification to guide the gaze (505) of the user (501) to the AI ​​object (530). For example, the processor (120) may provide a notification to guide the gaze (505) of the user (501) to the AI ​​object (530) based on at least one of an audio output, a haptic output, or an image output (e.g., an image output including text). For example, the processor (120) may provide a notification to guide the gaze (505) of the user (501) to the AI ​​object (530) in different ways based on the type of content (520) or application (146) currently being displayed. For example, the processor (120) may provide a notification to guide the user's (501) gaze (505) to the AI ​​object (530) depending on the surrounding noise or lighting. For example, the processor (120) may provide a louder notification when the surrounding noise is loud.

[0147] Referring to situation (604) of FIG. 6B, the processor (410) may change the display of the AI ​​object (530) based on the completion of generation of the execution result. The processor (410) may change the display of the AI ​​object (530) in response to identifying that acquisition of the execution result for the input of the user (501) is completed after the user's (501) gaze (505) moves away from the AI ​​object (530). For example, the processor (410) may enlarge the size of the AI ​​object (530) based on the generation of the execution result. However, the present invention is not limited thereto.

[0148] The processor (410) may apply an image effect (e.g., sparkle) to the AI ​​object (530) based on the generation of the execution result. For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that the task is completed through the display (420) based on the generation of the execution result. For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that the task is completed based on the generation of the execution result. For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that the task is completed through the audio output module (155) (e.g., speaker) based on the generation of the execution result.

[0149] The processor (410) can determine whether the execution result is output based on the generation of the execution result. In response to identifying that acquisition of the execution result is complete, the processor (410) can determine whether the execution result is output. The processor (410) can determine whether the execution result is output based on the gaze (505) of the user (501). The processor (410) can determine whether the execution result is output based on whether the gaze (505) of the user (501) is located within a first reference distance from the AI ​​object (530). Here, the first reference distance may mean a distance at which the user (501) is recognized as interacting with the AI ​​object (530). The first reference distance may be set according to the size of the AI ​​object (530).

[0150] The processor (410) may not output the execution result if the user's (501) gaze (505) is identified as being outside the first reference distance from the AI ​​object (530). The processor (410) may suspend the output of the execution result in response to the user's (501) gaze (505) being identified as being outside the first reference distance from the AI ​​object (530).

[0151] The processor (410) may output an execution result when the user's (501) gaze (505) is identified as being located within a first reference distance from the AI ​​object (530). The processor (410) may output an execution result based on data for outputting the execution result in response to the user's (501) gaze (505) being identified as being located within the first reference distance from the AI ​​object (530). When the user's (501) gaze (505) is identified as being located within the first reference distance from the AI ​​object (530), the processor (410) may output the execution result through the audio output module (155). However, the present invention is not limited thereto. When the user's (501) gaze (505) is identified as being located within the first reference distance from the AI ​​object (530), the processor (410) may output the execution result through the display (420). For example, if the processor (410) identifies that the user's (501) gaze (505) is located within a first reference distance from the AI ​​object (530), the processor (410) may output a screen indicating an execution result through the display (420). Here, the screen indicating the execution result may indicate the execution result based on at least one of an image, a video, or text. If the processor (410) identifies that the user's (501) gaze (505) is located within a first reference distance from the AI ​​object (530), the processor (410) may output an audio signal indicating the execution result through the audio output module (155), and output a screen indicating the execution result through the display (420).

[0152] The processor (410) can determine whether the output of the execution result is interrupted while outputting the execution result. The processor (410) can determine whether the output of the execution result is interrupted based on the gaze (505) of the user (501). The processor (410) can determine whether the output of the execution result is interrupted based on whether the gaze of the user (501) is located within a second reference distance from the AI ​​object (530). Here, the second reference distance may refer to a distance at which the user (501) is recognized as releasing the interaction with the AI ​​object (530). The second reference distance may be longer than the first reference distance, but is not limited thereto. For example, the second reference distance may be the same as the first reference distance.

[0153] The processor (410) can continue to output the execution result based on whether the user's (501) gaze is located within a second reference distance from the AI ​​object (530).

[0154] The processor (410) may stop outputting the execution result based on whether the user's (501) gaze is located outside the second reference distance from the AI ​​object (530). The processor (410) may stop outputting the execution result based on whether the user's (501) gaze is identified as being outside the second reference distance from the AI ​​object (530).

[0155] The processor (410) can identify the remaining time required to complete output of the execution result based on whether the user's (501) gaze is located outside the second reference distance from the AI ​​object (530). The processor (410) can identify the required time required to complete output of the execution result based on whether the user's (501) gaze is located outside the second reference distance from the AI ​​object (530).

[0156] The processor (410) may continue to output the execution result in response to the remaining time (or required time) being less than or equal to the first reference time. The processor (410) may continue to output the execution result based on the user's (501) gaze being located outside the second reference distance from the AI ​​object (530) and the remaining time (or required time) being less than or equal to the first reference time. However, the present invention is not limited thereto.

[0157] The processor (410) may stop outputting the execution result in response to the remaining time (or required time) exceeding the first reference time. The processor (410) may stop outputting the execution result based on the user's (501) gaze being located outside the second reference distance from the AI ​​object (530) and the remaining time (or required time) exceeding the first reference time.

[0158] The processor (410) can determine whether to resume output of the execution result while the output of the execution result has been suspended. The processor (410) can determine whether to resume output of the execution result based on the gaze of the user (501). The processor (410) can determine whether to resume output of the execution result based on whether the gaze of the user (501) is located within a first reference distance from the AI ​​object (530).

[0159] The processor (410) may resume outputting the execution result after stopping outputting the execution result, based on the identification that the user's (501) gaze is located within a first reference distance from the AI ​​object (530). Here, resuming outputting the execution result may mean resuming outputting the execution result from the point where the output was stopped. Resuming outputting the execution result may mean newly starting outputting the execution result.

[0160] The processor (410) may resume outputting the execution result based on the elapsed time from the point in time when the output of the execution result was stopped. In response to the elapsed time from the point in time when the output of the execution result was stopped being within a second reference time, the processor (410) may resume outputting the execution result from the point where the output was stopped. In response to the elapsed time from the point in time when the output of the execution result was stopped being within a second reference time, the processor (410) may output the execution result based on the point where the execution result was stopped being. In response to the elapsed time from the point in time when the output of the execution result was stopped being exceeding the second reference time, the processor (410) may output the execution result from the beginning. However, the present invention is not limited thereto. In response to the elapsed time from the point in time when the output of the execution result was stopped being within a second reference time, the processor (410) may generate data summarizing a previously outputted portion and output the generated data. After outputting the generated data, the processor (410) may resume outputting the execution result from the point where the output was stopped being. Here, the length of time required to output data summarizing previously outputted portions may be increased based on the elapsed time since the interruption. In response to the fact that the elapsed time since the interruption of outputting the execution result is within the second reference time, the processor (410) may resume outputting the interrupted portion from a predetermined time prior to the interruption. Here, the predetermined time may be increased based on the elapsed time since the interruption.

[0161] As described above, the wearable device (401) can provide the execution result at a desired time to the user (501) by stopping the output of the execution result while the user (501) has stopped interacting with the AI ​​object (530) (e.g., while the user (501) is not looking at the AI ​​object (530)).

[0162] As described above, the wearable device (401) can prevent a situation in which information required by the user (501) is not delivered to the user (501) by providing the output result based on the urgency of the output of the execution result while the user (501) has stopped interacting with the AI ​​object (530) (e.g., while the user (501) is not looking at the AI ​​object (530).

[0163] As described above, the wearable device (401) can provide an AI assistant (485) with increased usability to the user (501) by providing an execution result to the user (501) through an interaction based on the gaze (505) between the user (501) and the AI ​​object (530).

[0164] FIG. 6c illustrates an example of a screen displayed by a wearable device (401) in one embodiment.

[0165] The wearable device (401) of FIG. 6C may correspond to the electronic device (101) of FIG. 1. The wearable device (401) of FIG. 6C may correspond to the wearable device (200) of FIGS. 2A and 2B. The wearable device (401) of FIG. 6C may correspond to the wearable device (300) of FIGS. 3A and 3B. The wearable device (401) of FIG. 6C may correspond to the wearable device (401) of FIG. 4. FIG. 6C may be described with reference to FIGS. 1, 2A, 2B, 3A, 3B, and 4.

[0166] Situation (605) of FIG. 6c may illustrate situations subsequent to situations (601, 602) of FIG. 6a. For example, situation (605) of FIG. 6c may illustrate situations subsequent to the processor (410) obtaining input (e.g., voice input) from the user (501) through the AI ​​assistant (485).

[0167] The processor (410) can identify that an execution result for the input of the user (501) is generated. For example, the processor (410) can identify that an execution result for the input of the user (501) is generated through the AI ​​assistant (485). For example, the processor (410) can identify that data for outputting an execution result for the input of the user (501) is acquired through the AI ​​assistant (485). The processor (410) can identify that the execution result for the input of the user (501) is acquired after the user's (501) gaze (505) moves away from the AI ​​object (530). The processor (410) can identify that the acquisition of the execution result for the input of the user (501) is completed after the user's (501) gaze (505) moves away from the AI ​​object (530).

[0168] The processor (410) can identify the gaze (505) of the user (501). When an execution result for the input of the user (501) is generated, the processor (410) can identify the gaze (505) of the user (501). When an execution result for the input of the user (501) is generated, the processor (410) can notify the user (501) that the generation of the execution result is complete based on the gaze (505) of the user (501).

[0169] If the processor (410) identifies that the user's (501) gaze (505) is located outside the first reference distance from the AI ​​object (530), the processor (410) may change the display location of the AI ​​object (530). For example, the processor (410) may move the display location of the AI ​​object (530) toward the user's (501) gaze (505). For example, the processor (410) may move the display location of the AI ​​object (530) to a location adjacent to the content (520) toward which the user's (501) gaze (505) is directed. For example, referring to the situation (605) of FIG. 6c, the processor (410) may move the display location of the AI ​​object (530) to the outskirts of the content (520) toward which the user's (501) gaze (505) is directed. For example, the processor (410) can move the display position of the AI ​​object (530) in the direction of the user's (501) gaze (505) and change the display of the AI ​​object (530).

[0170] The processor (410) may output an execution result when it is identified that the user's (501) gaze (505) is located within a first reference distance from the AI ​​object (530). The processor (410) may output an execution result when it is identified that the user's (501) gaze (505) is located within a first reference distance from the moved AI object (530).

[0171] As described above, the wearable device (401) can provide the user (501) with enhanced responsiveness based on the user's (501) gaze (505) by moving the AI ​​object (530) in the direction of the user's (501) gaze (505).

[0172] FIG. 6d illustrates an example of a screen displayed by a wearable device (401) in one embodiment.

[0173] The wearable device (401) of FIG. 6d may correspond to the electronic device (101) of FIG. 1. The wearable device (401) of FIG. 6d may correspond to the wearable device (200) of FIGS. 2a and 2b. The wearable device (401) of FIG. 6d may correspond to the wearable device (300) of FIGS. 3a and 3b. The wearable device (401) of FIG. 6d may correspond to the wearable device (401) of FIG. 4. FIG. 6d may be described with reference to FIGS. 1, 2a, 2b, 3a, 3b, and 4.

[0174] Situation (606) of FIG. 6d may illustrate situations subsequent to situations (601, 602) of FIG. 6a. For example, situation (606) of FIG. 6d may illustrate situations subsequent to the processor (410) obtaining input (e.g., voice input) from the user (501) through the AI ​​assistant (485).

[0175] For example, the processor (410) can identify that an execution result for the input of the user (501) is generated. For example, the processor (410) can identify that an execution result for the input of the user (501) is generated through the AI ​​assistant (485). For example, the processor (410) can identify that data for outputting an execution result for the input of the user (501) is acquired through the AI ​​assistant (485). The processor (410) can identify that the execution result for the input of the user (501) is acquired after the user's (501) gaze (505) moves away from the AI ​​object (530). The processor (410) can identify that the acquisition of the execution result for the input of the user (501) is completed after the user's (501) gaze (505) moves away from the AI ​​object (530).

[0176] Referring to situation (606) of FIG. 6D, the processor (410) can determine whether to output the execution result to the user (501). In response to identifying that acquisition of the execution result is complete, the processor (410) can determine whether to output the execution result. The processor (410) can determine whether to output the execution result based on whether the output of the execution result is urgent. The processor (410) can determine whether to output the execution result based on whether the output of the execution result must be completed within a specified time.

[0177] The processor (410) may suspend the output of the execution result in response to identifying that the output of the execution result does not need to be completed within a specified time. For example, the processor (410) may suspend the output of the execution result until the time at which the output of the execution result is requested. Here, the time at which the output of the execution result is requested may vary depending on the intention of the voice input of the user (501). For example, if the user (501) has set a time at which the output of the execution result is requested, the processor (410) may suspend the output of the execution result until the time at which the user (501) requests the output.

[0178] The processor (410) may output the execution result if it is identified that the output of the execution result must be completed within a specified time. If it is identified that the output of the execution result must be completed within a specified time, the processor (410) may output the execution result regardless of the user's (501) gaze (505). For example, the processor (410) may output the execution result when the time at which the output of the execution result is required has arrived. For example, if it is identified that the output of the execution result must be completed within a specified time, the processor (410) may output the execution result even if the user's (501) gaze (505) is not directed toward the AI ​​object (530). For example, if a user (501) inquires about the arrival time of a specific bus and the specific bus is arriving soon, the processor (410) can output an audio signal indicating the execution result (e.g., “The bus you inquired about will arrive soon”) even if the user’s (501) gaze (505) is not directed toward the AI ​​object (530).

[0179] As described above, the wearable device (401) may provide execution results regardless of the user's (501) gaze (505) in a situation where the user (501) is focused on content, so that the user experience of the wearable device (501) of the user (501) may not be degraded.

[0180] FIG. 6e illustrates an example of a screen displayed by a wearable device (401) in one embodiment.

[0181] The wearable device (401) of FIG. 6E may correspond to the electronic device (101) of FIG. 1. The wearable device (401) of FIG. 6E may correspond to the wearable device (200) of FIGS. 2A and 2B. The wearable device (401) of FIG. 6E may correspond to the wearable device (300) of FIGS. 3A and 3B. The wearable device (401) of FIG. 6E may correspond to the wearable device (401) of FIG. 4. FIG. 6E may be described with reference to FIGS. 1, 2A, 2B, 3A, 3B, and 4.

[0182] The processor (410) can obtain input from the user (501). The processor (120) can obtain input from the user (501) that includes a trigger. Here, the trigger may include a keyword for calling the AI ​​assistant (485). For example, the trigger may be "Hi Bixby." However, the present invention is not limited thereto.

[0183] Referring to situation (607) of FIG. 6e, the processor (410) may obtain an input, “Tell me the bus arrival time,” through the input module (150). For example, the processor (410) may obtain an input including a trigger and the utterance, “Tell me the bus arrival time,” through the input module (150).

[0184] The processor (410) may activate the AI ​​assistant (485) based on acquiring the input of the user (501). Here, activating the AI ​​assistant (485) may mean executing the AI ​​assistant (485). Activating the AI ​​assistant (485) may mean transmitting the input (e.g., voice input) of the user (501) to the AI ​​assistant (485).

[0185] The processor (410) can identify the intention of the user (501) through the AI ​​assistant (485) based on the input of the user (501). The processor (410) can establish a plan corresponding to the intention of the user (501) through the AI ​​assistant (485). The processor (410) can execute the established plan through the AI ​​assistant (485). The processor (410) can obtain an execution result of the established plan through the AI ​​assistant (485). Here, the execution result can include feedback to be provided to the user (501). The feedback can include at least one of voice feedback and visual feedback. Here, the visual feedback can be based on at least one of an image, a video, or text.

[0186] The processor (410) can obtain an execution result based on the input of the user (501) (e.g., voice input) through the AI ​​assistant (485). For example, the processor (410) can identify that an execution result is generated for the input of the user (501). For example, the processor (410) can identify that an execution result is generated for the input of the user (501) through the AI ​​assistant (485). For example, the processor (410) can identify the acquisition of data for outputting an execution result for the input of the user (501) through the AI ​​assistant (485).

[0187] The processor (410) may notify the user (501) that the generation of the execution result is complete. For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that the task is complete based on the completion of the generation of the execution result. For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that the task is complete through the display (420) based on the generation of the execution result. For example, the processor (410) may output a notification related to the AI ​​assistant (485) indicating that the task is complete based on the generation of the execution result. Referring to the situation (608) of FIG. 6D , the processor (410) may output a notification related to the AI ​​assistant (485) indicating that the task is complete through the audio output module (155) (e.g., a speaker) based on the generation of the execution result.

[0188] The processor (410) can determine whether the execution result is output. In response to identifying that acquisition of the execution result is complete, the processor (410) can determine whether the execution result is output. The processor (410) can determine whether the execution result is output based on whether the output of the execution result is urgent. The processor (410) can determine whether the execution result is output based on whether the output of the execution result must be completed within a specified time.

[0189] The processor (410) may suspend the output of the execution result in response to identifying that the output of the execution result does not need to be completed within a specified time. For example, the processor (410) may suspend the output of the execution result until the time at which the output of the execution result is requested. Here, the time at which the output of the execution result is requested may vary depending on the intention of the voice input of the user (501). For example, if the user (501) has set a time at which the output of the execution result is requested, the processor (410) may suspend the output of the execution result until the time at which the user (501) requests the output.

[0190] The processor (410) may output the execution result if it is identified that the output of the execution result must be completed within a specified time. If it is identified that the output of the execution result must be completed within a specified time, the processor (410) may output the execution result regardless of the user's (501) gaze (505). For example, the processor (410) may output the execution result when the time at which the output of the execution result is required has arrived. For example, if it is identified that the output of the execution result must be completed within a specified time, the processor (410) may output the execution result even if the user's (501) gaze (505) is not directed toward the AI ​​object (530). For example, if a user (501) inquires about the arrival time of a specific bus and the specific bus is arriving soon, the processor (410) can output the execution result (e.g., “The bus you inquired about will arrive soon”) even if the user’s (501) gaze (505) is not directed toward the AI ​​object (530).

[0191] FIG. 7 illustrates an example of a screen displayed by a wearable device (401) in one embodiment.

[0192] The wearable device (401) of FIG. 7 may correspond to the electronic device (101) of FIG. 1. The wearable device (401) of FIG. 7 may correspond to the wearable device (200) of FIGS. 2A and 2B. The wearable device (401) of FIG. 7 may correspond to the wearable device (300) of FIGS. 3A and 3B. The wearable device (401) of FIG. 7 may correspond to the wearable device (401) of FIG. 4. FIG. 7 may be described with reference to FIGS. 1, 2A, 2B, 3A, 3B, and 4.

[0193] The wearable device (401) may include at least one microphone buffer (730) in the memory (415).

[0194] Referring to situation (701) of FIG. 7, the processor (410) can identify that the gaze (505) of the user (501) has moved. The processor (410) can identify that the gaze (505) of the user (501) has moved based on the execution of the gaze tracker (474). The processor (410) can identify that the gaze (505) of the user (501) is directed toward the AI ​​object (530).

[0195] The processor (410) can store the user's (501) speech. The processor (410) can store the user's (501) speech obtained through the input module (150). The processor (410) can store the user's (501) speech in the microphone buffer (730).

[0196] The processor (410) may store the user's (501) speech in the microphone buffer (730) while the user's (501) gaze (505) moves. For example, the processor (410) may store the user's (501) speech in the microphone buffer (730) while the user's (501) gaze (505) moves from the content (520) to the AI ​​object (530).

[0197] The processor (410) can obtain user input based on at least one utterance of the user (501) stored in the microphone buffer (730) based on the movement of the user's (501) gaze (505) toward the AI ​​object (530).

[0198] When the user's (501) gaze (505) is positioned on the AI ​​object (530), the processor (410) may utilize one or more of the user's (501) utterances stored in the microphone buffer (730) as user input. For example, the processor (410) may utilize the first utterance stored in the microphone buffer (730) while the user's (501) gaze (505) moves to the AI ​​object (530), the second utterance stored in the microphone buffer (730) while the user's (501) gaze (505) is positioned on the AI ​​object (530), and the third utterance stored in the microphone buffer (730) while the user's (501) gaze (505) moves away from the AI ​​object (530) as user input. However, the present invention is not limited thereto. For example, the processor (410) may select different numbers of utterances depending on the words included in the first utterance, the second utterance, the third utterance, or the fourth utterance, and / or the intention of the user (501). For example, the processor (410) may utilize the first utterance and the second utterance as user input. For example, the processor (410) may utilize the second utterance and the third utterance as user input. For example, the processor (410) may utilize the second utterance, the third utterance, and the fourth utterance as user input.

[0199] As described above, the wearable device (401) can reduce latency in the user's (501) voice input by using not only the utterance after the AI ​​assistant (485) is triggered by the gaze (505) but also the utterance before that as voice input.

[0200] FIG. 8 illustrates an example of a screen displayed by a wearable device (401) in one embodiment.

[0201] The wearable device (401) of FIG. 8 may correspond to the electronic device (101) of FIG. 1. The wearable device (401) of FIG. 8 may correspond to the wearable device (200) of FIGS. 2A and 2B. The wearable device (401) of FIG. 8 may correspond to the wearable device (300) of FIGS. 3A and 3B. The wearable device (401) of FIG. 8 may correspond to the wearable device (401) of FIG. 4. FIG. 8 may be described with reference to FIGS. 1, 2A, 2B, 3A, 3B, and 4.

[0202] The processor (410) may provide a virtual space service based on the execution of the virtual space manager (451). For example, the processor (410) may display a screen (820) representing an actual space acquired through the camera (425) on at least a portion of the display (420) based on the execution of the pass-through manager (453). For example, the processor (410) may display a screen (820) representing an actual space acquired through the camera (425) on the entire area of ​​the FOV (510) based on the execution of the pass-through manager (453). Hereinafter, the screen (820) representing an actual space may be referred to as a pass-through screen (820).

[0203] The processor (410) can identify the gaze (505) of the user (501). For example, the processor (410) can identify the gaze (505) of the user (501) in a pass-through screen (820).

[0204] The processor (410) can identify that the user's (501) gaze (505) is positioned on the object (810). For example, the processor (410) can identify that the user's (501) gaze (505) is positioned on the object (810) for a specified period of time or longer. Here, the object (810) may be an object included in a pass-through screen (820). The object (810) on which the user's (501) gaze (505) is positioned for a specified period of time or longer may be referred to as an object of interest.

[0205] The processor (410) can display the AI ​​object (530) on the display (420) based on the gaze (505) of the user (501). For example, the processor (410) can display the AI ​​object (530) on the display (420) based on the gaze (505) of the user (501) being positioned on the object (810) for a specified period of time or longer. For example, the processor (410) can display the AI ​​object (530) on the pass-through screen (820) based on the gaze (505) of the user (501) being positioned on the object (810) for a specified period of time or longer. For example, the processor (410) can display the AI ​​object (530) around the object (810) on the pass-through screen (820). For example, referring to situation (801) of FIG. 8, the processor (410) can display an AI object (530) by overlapping it on a pass-through screen (820).

[0206] The processor (410) can obtain input from the user (501) while the AI ​​object (530) is displayed around the object (810).

[0207] The processor (410) can identify the intention of the user (501) through the AI ​​assistant (485) based on the input of the user (501). For example, the processor (410) can identify the intention of the user (501) through the AI ​​assistant (485) based on the input of the user (501) and the object (810) on which the gaze (505) is located for a specified period of time. For example, the processor (410) can identify the intention of the user (501) through the AI ​​assistant (485) based on information about the object (810) and the input of the user (501). For example, if the object (810) is a “coffee shop” and the user’s (501) input is “What menu is there?”, the processor (410) can obtain information about the menu sold at the “coffee shop” related to the object (810) through the AI ​​assistant (485). The processor (410) can provide the obtained information related to the object (810) to the user (501).

[0208] Referring to situation (801) of FIG. 8, the processor (410) can identify that the user's (501) gaze (505) has moved. The processor (410) can identify that the user's (501) gaze (505) has moved away from the object (810). The processor (410) can identify that the user's (501) gaze (505) has moved away from the object (810).

[0209] The processor (410) can change the display of the AI ​​object (530) based on the user's (501) gaze (505). For example, the processor (410) can change the display location of the AI ​​object (530) based on whether the user's (501) gaze (505) moves away from the object (810). For example, the processor (410) can display the AI ​​object (530) outside the pass-through screen (820) based on whether the user's (501) gaze (505) moves away from the object (810). For example, the processor (410) can move the AI ​​object (530) outside the pass-through screen (820) based on whether the user's (501) gaze (505) moves away from the object (810).

[0210] The processor (410) can determine whether to stop outputting information related to the object (810) while outputting the information. The processor (410) can determine whether to stop outputting the information based on the user's (501) gaze (505). The processor (410) can stop outputting the information based on the user's (501) gaze (505) moving away from the object (810).

[0211] The processor (410) may resume information output after stopping information output, based on the identification that the user's (501) gaze is positioned within a first reference distance from the object (810). Here, resuming information output may mean resuming information output from the point where the information output was stopped. Resuming information output may mean newly initiating information output.

[0212] FIG. 9 illustrates a flowchart of operations performed by a wearable device (401) in one embodiment.

[0213] FIG. 9 can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, FIG. 3a, FIG. 3b, FIG. 4, and FIG. 5a.

[0214] Referring to FIG. 9, in operation 910, the processor (410) may obtain a voice input. The processor (410) may obtain a voice input from a user (501) through the input module (150). However, the present invention is not limited thereto. The processor (410) may obtain at least one of a text input or a voice input through the input module (150).

[0215] In operation 920, the processor (410) may execute a function related to voice input. The processor (410) may execute a function related to voice input through the AI ​​assistant (485) of the user (501). The processor (410) may obtain an execution result of the function related to voice input through the AI ​​assistant (485) of the user (501). For example, the processor (410) may identify the intention of the user (501) through the AI ​​assistant (485) based on the voice input of the user (501). The processor (410) may establish a plan corresponding to the intention of the user (501) through the AI ​​assistant (485). The processor (410) may execute the established plan through the AI ​​assistant (485). The processor (410) may obtain an execution result of the established plan through the AI ​​assistant (485).

[0216] In operation 930, the processor (410) may determine whether the user's (501) gaze (505) is within a reference distance from the executable object (530). While obtaining an execution result of a function related to the user's (501) voice input through the AI ​​assistant (485), the processor (410) may determine whether the user's (501) gaze (505) is within a first reference distance from the executable object (530). Here, the first reference distance may mean a distance at which the user (501) is recognized as interacting with the AI ​​object (530). The first reference distance may be set according to the size of the AI ​​object (530).

[0217] In operation 930, if the user's (501) gaze (505) is within a reference distance from the executable object (530) ('Yes'), the processor (410) may perform operation 940. In operation 930, if the user's (501) gaze (505) is outside the reference distance from the executable object (530) ('No'), the processor (410) may perform operation 930 again.

[0218] In operation 940, the processor (410) may output an execution result. The processor (410) may output the execution result based on data for outputting the execution result. The processor (410) may output the execution result through the audio output module (155). The processor (410) may output the execution result through the display (420). The processor (410) may output the execution result through the audio output module (155) and the display (420). For example, the processor (410) may output a screen indicating the execution result through the display (420).

[0219] FIG. 10 illustrates a flowchart of operations performed by a wearable device (401) in one embodiment.

[0220] FIG. 10 may be described with reference to FIGS. 1, 2a, 2b, 3a, 3b, 4, and 5a. Operations 910, 920, 930, and 940 of FIG. 10 may correspond to operations 910, 920, 930, and 940 of FIG. 9, respectively.

[0221] Referring to FIG. 10, in operation 910, the processor (410) may obtain a voice input. The processor (410) may obtain a voice input from a user (501) through the input module (150). However, the present invention is not limited thereto. The processor (410) may obtain at least one of a text input or a voice input through the input module (150).

[0222] In operation 920, the processor (410) may execute a function related to voice input. The processor (410) may execute a function related to voice input through the AI ​​assistant (485) of the user (501). The processor (410) may obtain an execution result of the function related to voice input through the AI ​​assistant (485) of the user (501). For example, the processor (410) may identify the intention of the user (501) through the AI ​​assistant (485) based on the voice input of the user (501). The processor (410) may establish a plan corresponding to the intention of the user (501) through the AI ​​assistant (485). The processor (410) may execute the established plan through the AI ​​assistant (485). The processor (410) may obtain an execution result of the established plan through the AI ​​assistant (485).

[0223] In operation 930, the processor (410) may determine whether the user's gaze is within a reference distance from the executable object. While obtaining the execution result of a function related to the user's (501) voice input through the AI ​​assistant (485), the processor (410) may determine whether the user's gaze (505) is within a second reference distance from the executable object (530). Here, the second reference distance may refer to a distance at which the user (501) is recognized as releasing the interaction with the AI ​​object (530). The second reference distance may be longer than the first reference distance, but is not limited thereto. For example, the second reference distance may be the same as the first reference distance.

[0224] In operation 930, if the user's (501) gaze (505) is within a reference distance from the executable object (530) ('Yes'), the processor (410) may perform operation 940. In operation 930, if the user's (501) gaze (505) is outside the reference distance from the executable object (530) ('No'), the processor (410) may perform operation 1010.

[0225] In operation 1010, the processor (410) may determine whether immediate output is required. The processor (410) may determine whether immediate output of the execution result is required based on whether the output of the execution result is required (or urgent). The processor (410) may determine whether output of the execution result is required based on whether output of the execution result must be completed within a specified time. For example, the processor (410) may determine whether a time has arrived when output of the execution result is required. Here, the time when output of the execution result is required may vary depending on the intention of the voice input of the user (501). For example, if the user (501) has set a time at which output of the execution result is requested, the processor (410) may postpone output of the execution result until the time requested by the user (501).

[0226] In operation 1010, if immediate output of the execution result is required ('Yes'), the processor (410) may perform operation 940. In operation 1010, if immediate output is not required ('No'), the processor (410) may perform operation 930 again.

[0227] In operation 940, the processor (410) may output an execution result. The processor (410) may output the execution result based on data for outputting the execution result. The processor (410) may output the execution result through the audio output module (155). The processor (410) may output the execution result through the display (420). The processor (410) may output the execution result through the audio output module (155) and the display (420). For example, the processor (410) may output a screen indicating the execution result through the display (420).

[0228] Figure 11 illustrates a flowchart of operations performed by a wearable device in one embodiment.

[0229] FIG. 11 can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, FIG. 3a, FIG. 3b, FIG. 4, and FIG. 5a.

[0230] Referring to FIG. 11, in operation 1110, the processor (410) may initiate output of an execution result. The processor (410) may output the execution result based on data for outputting the execution result. The processor (410) may output the execution result through the audio output module (155). The processor (410) may output the execution result through the display (420). The processor (410) may output the execution result through the audio output module (155) and the display (420). For example, the processor (410) may output a screen indicating the execution result through the display (420).

[0231] The processor (410) may output an execution result when the user's (501) gaze (505) is identified as being located within a first reference distance from the AI ​​object (530). In response to the user's (501) gaze (505) being identified as being located within a first reference distance from the AI ​​object (530), the processor (410) may output an execution result based on data for outputting the execution result.

[0232] In operation 1120, the processor (410) may determine whether the user's (501) gaze (505) is within a reference distance from an executable object. The processor (410) may determine whether the user's (501) gaze (505) is located within a second reference distance from an AI object (530). In order to determine whether to stop outputting the execution result, the processor (410) may determine whether the user's (501) gaze (505) is located within a second reference distance from the AI ​​object (530). Here, the second reference distance may mean a distance at which the user (501) is recognized as releasing interaction with the AI ​​object (530). The second reference distance may be longer than the first reference distance, but is not limited thereto. For example, the second reference distance may be the same as the first reference distance.

[0233] In operation 1120, if the user's (501) gaze (505) is within a reference distance from the executable object ('Yes'), the processor (410) may perform operation 1130. In operation 1120, if the user's (501) gaze (505) is outside the reference distance from the executable object ('No'), the processor (410) may perform operation 1140.

[0234] In operation 1130, the processor (410) may continue to output the execution result. The processor (410) may continue to output the execution result based on whether the user's (501) gaze is positioned within a second reference distance from the AI ​​object (530).

[0235] In operation 1140, the processor (410) may stop outputting the execution result. The processor (410) may stop outputting the execution result based on whether the user's (501) gaze is located outside the reference distance from the AI ​​object (530). The processor (410) may stop outputting the execution result based on whether the user's (501) gaze is identified as being outside the reference distance from the AI ​​object (530).

[0236] Depending on the embodiment, the processor (410) may continue outputting the execution result in a different manner. For example, the processor (410) may continue outputting the execution result in a different manner based on whether the user's (501) gaze (505) deviates from a reference distance from the executable object. For example, the processor (120) may output the execution result, which is output as audio, in a text-based manner based on whether the user's (501) gaze (505) deviates from a reference distance from the executable object.

[0237] FIG. 12 illustrates a flowchart of operations performed by a wearable device (401) in one embodiment.

[0238] FIG. 12 can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, FIG. 3a, FIG. 3b, FIG. 4, and FIG. 5a.

[0239] Referring to FIG. 12, in operation 1210, the processor (410) may stop outputting the execution result. The processor (410) may stop outputting the execution result based on whether the user's (501) gaze is located outside the reference distance from the AI ​​object (530). The processor (410) may stop outputting the execution result based on whether the user's (501) gaze is identified as being outside the reference distance from the AI ​​object (530).

[0240] In operation 1220, the processor (410) may determine whether the user's gaze is within a reference distance from the executable object. To determine whether to resume output of the execution result, the processor (410) may determine whether the user's gaze is located within a first reference distance from the AI ​​object (530). Here, the first reference distance may refer to a distance at which the user (501) is recognized as interacting with the AI ​​object (530). The first reference distance may be set according to the size of the AI ​​object (530).

[0241] In operation 1220, if the user's (501) gaze (505) is within a reference distance from the executable object ('Yes'), the processor (410) may perform operation 1230. In operation 1220, if the user's (501) gaze (505) is outside the reference distance from the executable object ('No'), the processor (410) may perform operation 1210 again.

[0242] In operation 1230, the processor (410) may determine whether to resume output of a previous execution result. The processor (410) may determine whether to resume output of a previous execution result based on an elapsed time from the time at which output of the execution result was stopped. The processor (410) may determine whether the elapsed time from the time at which output of the execution result was stopped is within a reference time. If the elapsed time from the time at which output of the execution result was stopped is within the reference time, the processor (410) may determine to resume output of the previous execution result. If the elapsed time from the time at which output of the execution result was stopped exceeds the reference time, the processor (410) may determine to newly output the previous execution result.

[0243] In operation 1230, if it is determined that the output of the previous execution result is to be resumed ('Yes'), the processor (410) may perform operation 1240. In operation 1230, if it is determined that the output of the previous execution result is not to be resumed ('No'), the processor (410) may perform operation 1250 again.

[0244] At operation 1240, the processor (410) may resume outputting the execution result. Here, resuming outputting the execution result may mean resuming outputting the execution result from where it was interrupted. The processor (410) may output the execution result based on the point where the execution result was interrupted.

[0245] The processor (410) may generate data summarizing a previously output portion in response to the time elapsed from the point in time when the output of the execution result was stopped exceeding a reference time, and output the generated data. After outputting the generated data, the processor (410) may resume outputting the execution result from the point in time where the output was stopped. Here, the length of time required to output the data summarizing the previously output portion may be increased based on the time elapsed from the point in time when the output was stopped. The processor (410) may resume outputting the execution result from a predetermined time before the point in time when the output was stopped in response to the time elapsed from the point in time when the output of the execution result was stopped being within a reference time. Here, the predetermined time may be increased based on the time elapsed from the point in time when the output was stopped.

[0246] In operation 1250, the processor (410) may output the execution result anew. Here, outputting the execution result anew may mean outputting the execution result from the beginning.

[0247] As described above, the wearable device (401) may include at least one sensor (430) for obtaining data for identifying a gaze (505) of a user (501) wearing the wearable device (401). The wearable device (401) may include a display (420) capable of displaying a stereoscopic image. The wearable device (401) may include at least one microphone. The wearable device (401) may include at least one processor (410). The wearable device (401) may include at least one memory (415) for storing instructions. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to obtain a voice input of the user (501) through the at least one microphone. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to execute a function indicated by the voice input based on an artificial intelligence (AI) assistant. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify the gaze (505) of the user (501) based on the data acquired through the at least one sensor (430). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify, in response to generation of data for outputting a result of execution of the function indicated by the voice input, whether the gaze (505) is located within a first reference distance from a visual object (530) associated with the AI ​​assistant (485) displayed on the display (420).The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to output the execution result based on the data for outputting the execution result in response to identifying that the gaze (505) is located within the first reference distance from the visual object (530). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to postpone outputting the execution result in response to identifying that the gaze (505) is located outside the first reference distance from the visual object (530).

[0248] The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify whether the gaze (505) is located within a second reference distance from the visual object (530) while outputting the execution result. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to stop outputting the execution result based on identifying that the gaze (505) is located outside the second reference distance from the visual object (530).

[0249] The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify a time required to complete the output of the execution result in response to identifying that the gaze (505) is located outside the second reference distance from the visual object (530). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to continue the output of the execution result in response to the identified time required being less than or equal to a first reference time. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to stop the output of the execution result in response to the identified time required exceeding the first reference time.

[0250] The above instructions, when executed by the at least one processor (410), may cause the wearable device (401) to resume outputting the execution result after stopping the outputting of the execution result, based on the identification that the gaze (505) is located within the first reference distance from the visual object (530).

[0251] The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to output the execution result from the beginning in response to a time elapsed from the time at which the output of the execution result was stopped exceeding a second reference time. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to resume outputting the execution result based on a portion at which the output of the execution result was stopped in response to a time elapsed from the time at which the output of the execution result was stopped being within a second reference time.

[0252] The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify whether the gaze (505) is located within a third reference distance from the visual object (530). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify, in response to identifying that the gaze (505) is located within a third reference distance from the visual object (530), the voice input from utterance of the user (501) obtained through the at least one microphone.

[0253] The above instructions, when executed by the at least one processor (410), may cause the wearable device (401) to change the display of the visual object (530) in response to the data being generated for outputting the execution result.

[0254] As described above, the wearable device (401) may further include at least one speaker. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to output the execution result through the display (420) and / or the at least one speaker in response to identifying that the line of sight (505) is located within the first reference distance from the visual object (530). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to display the execution result as text on the display (420) based on identifying that the line of sight (505) is located beyond the second reference distance from the visual object (530) while outputting the execution result.

[0255] The above text can be displayed within a specified distance from the visual object (530).

[0256] The above instructions, when executed by the at least one processor (410), may cause the wearable device (401) to display the visual object (530) within a fourth reference distance from the designated object in response to the user's (501) gaze (505) being maintained on the designated object for a third reference time.

[0257] As described above, the wearable device (401) may further include at least one camera (425) configured to capture an external environment of the wearable device (401). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify a designated object in an image acquired through the at least one camera (425). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to display the visual object (530) within a fourth reference distance from the designated object on the display (420), in response to the designated object being identified in the image.

[0258] The above instructions, when executed by the at least one processor (410), may cause the wearable device (401) to execute the function based on information related to the specified object and the voice input.

[0259] The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify whether the execution result needs to be output within a fourth reference time. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to output the execution result regardless of the line of sight (505) in response to identifying that the execution result needs to be output within the fourth reference time.

[0260] The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to change the display of the visual object (530) in response to identifying that the gaze (505) is located within the first reference distance from the visual object (530). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to obtain, through the at least one microphone, the voice input in response to changing the display of the visual object (530).

[0261] As described above, the method may be executed in a wearable device (401) including at least one sensor (430) for obtaining data for identifying a gaze (505) of a user (501) wearing the wearable device (401), a display (420) capable of displaying a stereoscopic image, and at least one microphone. The method may include an operation of obtaining a voice input of the user (501) through the at least one microphone. The method may include an operation of executing a function indicated by the voice input based on an artificial intelligence (AI) assistant. The method may include an operation of identifying the gaze (505) of the user (501) based on the data obtained through the at least one sensor (430). The method may include, in response to data being generated for outputting an execution result of the function indicated by the voice input, an operation of identifying whether the gaze (505) is located within a first reference distance from a visual object (530) associated with the AI ​​assistant (485) displayed on the display (420). The method may include, in response to identifying that the gaze (505) is located within the first reference distance from the visual object (530), an operation of outputting the execution result based on the data for outputting the execution result. The method may include, in response to identifying that the gaze (505) is outside the first reference distance from the visual object (530), an operation of postponing the outputting of the execution result.

[0262] The method may include an operation of identifying whether the line of sight (505) is located within a second reference distance from the visual object (530) while outputting the execution result. The method may include an operation of stopping the output of the execution result based on identifying that the line of sight (505) is outside the second reference distance from the visual object (530).

[0263] The method may include an operation of identifying a time required to complete the output of the execution result in response to identifying that the gaze (505) is outside the second reference distance from the visual object (530). The method may include an operation of continuing the output of the execution result in response to the identified time required being less than or equal to the first reference time. The method may include an operation of stopping the output of the execution result in response to the identified time required exceeding the first reference time.

[0264] The method may include an operation of resuming the output of the execution result based on the identification that the gaze (505) is located within the first reference distance from the visual object (530) after stopping the output of the execution result.

[0265] The method may include an operation of outputting the execution result from the beginning in response to a time elapsed since the time at which the output of the execution result was stopped exceeding a second reference time. The method may include an operation of resuming the output of the execution result based on a portion where the output of the execution result was stopped in response to a time elapsed since the time at which the output of the execution result was stopped being within a second reference time.

[0266] The method may include an operation of identifying whether the gaze (505) is located within a third reference distance from the visual object (530). The method may include an operation of identifying the voice input from the user's (501) speech acquired through the at least one microphone in response to identifying the gaze (505) as being located within the third reference distance from the visual object (530).

[0267] As described above, a non-transitory computer readable storage medium may include a program including instructions. The instructions, when executed by at least one processor (410) of a wearable device (401), including at least one sensor (430) for obtaining data for identifying a gaze (505) of a user (501) wearing the wearable device (401), a display (420) capable of displaying a stereoscopic image, and at least one microphone, may cause the wearable device (401) to obtain a voice input of the user (501) through the at least one microphone. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to perform a function indicated by the voice input based on an artificial intelligence (AI) assistant. The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify the gaze (505) of the user (501) based on the data acquired through the at least one sensor (430). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to identify, in response to the generation of data for outputting an execution result of the function indicated by the voice input, whether the gaze (505) is located within a first reference distance from a visual object (530) associated with the AI ​​assistant (485) displayed on the display (420).The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to output the execution result based on the data for outputting the execution result in response to identifying that the gaze (505) is located within the first reference distance from the visual object (530). The instructions, when executed by the at least one processor (410), may cause the wearable device (401) to postpone outputting the execution result in response to identifying that the gaze (505) is located outside the first reference distance from the visual object (530).

[0268] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0269] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0270] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0271] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0272] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0273] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (401), At least one sensor (430) for acquiring data for identifying the gaze (505) of a user (501) wearing the wearable device (401); A display (420) capable of displaying stereoscopic images; At least one microphone, at least one processor (410), and At least one memory (415) for storing instructions, wherein the instructions, when executed by the at least one processor (410), cause the wearable device (401) to: Obtaining a voice input of the user (501) through at least one microphone, Executes the function indicated by the above voice input based on the AI ​​(artificial intelligence) voice recognition model (485), Based on the data acquired through at least one sensor (430), the gaze (505) of the user (501) is identified, In response to the generation of data for outputting the execution result of the function indicated by the voice input, it is identified whether the gaze (505) is located within a first reference distance from a visual object (530) related to the AI ​​voice recognition model (485) displayed on the display (420). In response to identifying that the above gaze (505) is located within the first reference distance from the above visual object (530), based on the data for outputting the execution result, outputting the execution result, and In response to identifying that the above gaze (505) is located outside the first reference distance from the above visual object (530), causing the output of the execution result to be postponed. Wearable devices.

2. In claim 1, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: While outputting the above execution result, it is identified whether the gaze (505) is located within the second reference distance from the visual object (530), Causing the output of the execution result to be stopped based on the identification that the above gaze (505) is located outside the second reference distance from the above visual object (530). Wearable devices.

3. In claim 1 or claim 2, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: In response to identifying that the above gaze (505) is located outside the second reference distance from the above visual object (530), identifying the time required to complete the output of the execution result, In response to the identified time required being less than or equal to the first reference time, the output of the execution result is continued, In response to the identified time required exceeding the first reference time, causing the output of the execution result to be stopped, Wearable devices.

4. In any one of claims 1 to 3, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: After stopping the output of the execution result, based on the identification that the gaze (505) is located within the first reference distance from the visual object (530), causing the output of the execution result to be resumed. Wearable devices.

5. In any one of claims 1 to 4, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: In response to the time elapsed from the point in time at which the output of the above execution result was stopped exceeding the second reference time, the execution result is output from the beginning, In response to the fact that the time elapsed from the point in time at which the output of the execution result was interrupted is within the second reference time, causing the output of the execution result to be resumed based on the portion at which the output of the execution result was interrupted. Wearable devices.

6. In any one of claims 1 to 5, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: Identifying whether the above gaze (505) is located within a third reference distance from the above visual object (530), In response to identifying that the above gaze (505) is located within a third reference distance from the above visual object (530), causing the voice input to be identified from the user's (501) speech acquired through the at least one microphone. Wearable devices.

7. In any one of claims 1 to 6, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: In response to the data being generated for outputting the execution result, causing the display of the visual object (530) to be changed, Wearable devices.

8. In any one of claims 1 to 7, Including at least one more speaker, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: In response to identifying that the above gaze (505) is located within the first reference distance from the above visual object (530), outputting the execution result through the display (420) and / or the at least one speaker, While outputting the execution result, based on the identification that the line of sight (505) deviates from the second reference distance from the visual object (530), causing the execution result to be displayed as text on the display (420). Wearable devices.

9. In any one of claims 1 to 8, The above text is displayed within a specified distance from the visual object (530). Wearable devices.

10. In any one of claims 1 to 9, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: In response to the user (501) maintaining the gaze (505) on the designated object for a third reference time, causing the visual object (530) to be displayed within a fourth reference distance from the designated object. Wearable devices.

11. In any one of claims 1 to 10, Further comprising at least one camera (425) configured to capture the external environment of the wearable device (401), The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: Identifying a specified object in an image acquired through at least one camera (425), In response to the designated object being identified in the image, causing the visual object (530) to be displayed within a fourth reference distance from the designated object on the display (420). Wearable devices.

12. In any one of claims 1 to 11, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: Causing the execution of the function based on information related to the above-mentioned specified object and the above-mentioned voice input, Wearable devices.

13. In any one of claims 1 to 12, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: Identify whether the above execution result needs to be output within the 4th reference time, In response to identifying that the execution result needs to be output within the fourth reference time, causing the execution result to be output regardless of the line of sight (505). Wearable devices.

14. In any one of claims 1 to 13, The above instructions, when executed by at least one processor (410), cause the wearable device (401) to: In response to identifying that the above gaze (505) is located within the first reference distance from the above visual object (530), changing the display of the above visual object (530), In response to changing the display of the above visual object (530), causing the acquisition of the voice input through the at least one microphone, Wearable devices.

15. A method executed in a wearable device (401) including at least one sensor (430) for obtaining data for identifying a gaze (505) of a user (501) wearing the wearable device (401), a display (420) capable of displaying a stereoscopic image, and at least one microphone, An operation of obtaining a voice input of the user (501) through at least one microphone; An action to execute a function indicated by the above voice input based on an AI (artificial intelligence) voice recognition model (485). An operation of identifying the gaze (505) of the user (501) based on the data acquired through at least one sensor (430). In response to the generation of data for outputting the execution result of the function indicated by the voice input, an operation of identifying whether the gaze (505) is located within a first reference distance from a visual object (530) related to the AI ​​voice recognition model (485) displayed on the display (420). In response to identifying that the above gaze (505) is located within the first reference distance from the above visual object (530), an operation of outputting the execution result based on the data for outputting the execution result, and In response to identifying that the above gaze (505) deviates from the first reference distance from the above visual object (530), an action is included to postpone the output of the execution result. method.

Images