Wearable device and method for calibrating gaze information of user, and computer-readable storage medium

Abstract

This wearable device may comprise: a display; one or more cameras for tracking the eyeballs of a user; at least one processor; and a memory including one or more storage media that store instructions. While a plurality of visual objects are displayed on the display, the wearable device may: determine, on the basis of the one or more cameras, whether to calibrate gaze-related gaze information; reduce, on the basis of the determination of the calibration of the gaze information, the sizes of gaze input areas, each formed on the visual objects; generate calibration information to be used for calibrating the gaze information, by using the reduced sizes so as to use the gaze information detected while providing visual feedback; acquire calibrated gaze information by applying the generated calibration information to the gaze information; and expand the sizes of the gaze input areas by using the calibration information.

Description

Wearable device, method, and computer-readable storage medium for correcting user's gaze information

[0001] The present disclosure relates to a wearable device, a method, and a computer-readable storage medium for correcting a user's gaze information.

[0002] To provide an enhanced user experience, wearable devices can provide various services. Wearable devices can be worn on a part of the user's body and operate. For example, wearable devices can be AR glasses and / or head-mounted devices (HMDs). To provide various services to the user, wearable devices can identify gaze information indicating the gaze direction of the user's eyes using a camera to track the user's eyes. To enhance user convenience, the wearable device may be required to perform calibration of the gaze information.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] A wearable device is disclosed. In one embodiment, the wearable device may include a display system including a first display and a second display, and a camera system including a plurality of cameras. The camera system may be configured to capture images of a user's eyes while the user wears the wearable device. The wearable device may include at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a visual object through the display system. The visual object may be associated with a gaze input area having a first size. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to provide visual feedback for the visual object when identified gaze information falls within the gaze input area. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to, when attempting to perform correction related to the gaze information while the visual object is displayed, reduce the size of the gaze input area related to the visual object from the first size to the second size, generate correction information based on the gaze information of the user gazing at the visual object while the size of the gaze input area is reduced, and increase the size of the gaze input area related to the visual object from the second size after the correction information is generated.

[0005] A wearable device is disclosed. In one embodiment, the wearable device may include a display, one or more cameras configured to track the eyes of a user wearing the wearable device, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a plurality of visual objects on the display. The plurality of visual objects may each include gaze input areas formed on the plurality of visual objects to provide visual feedback for the user's gaze directed toward the plurality of visual objects. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine, while displaying the plurality of visual objects, whether to correct gaze information related to the gaze determined based on the one or more cameras. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to reduce the sizes of each of the gaze input areas formed on the visual objects, respectively, based on determining a correction of the gaze information. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate correction information to be used to correct the gaze information using the gaze information detected while providing the visual feedback using the reduced sizes.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to apply the generated correction information to the gaze information to obtain the corrected gaze information. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to expand the sizes of the gaze input areas to provide visual feedback for the gaze through the gaze information corrected using the correction information.

[0006] A method for a wearable device is disclosed. In one embodiment, the method may include displaying a plurality of visual objects on a display of the wearable device. The plurality of visual objects may include gaze input areas formed on each of the plurality of visual objects to provide visual feedback regarding a user's gaze directed toward the plurality of visual objects. The method may include determining, while displaying the plurality of visual objects, whether to correct gaze information related to the gaze, determined based on one or more cameras of the wearable device configured to track eyes of a user wearing the wearable device. The method may include reducing the sizes of each of the gaze input areas formed on each of the visual objects based on the determination of correction of the gaze information. The method may include generating correction information to be used to correct the gaze information using the gaze information detected while providing the visual feedback using the reduced sizes. The method may include applying the generated correction information to the gaze information to obtain the corrected gaze information. The method may include an operation of expanding the sizes of the gaze input areas to provide visual feedback for the gaze through the gaze information corrected using the correction information.

[0007] A non-transitory computer-readable storage medium is disclosed. In one embodiment, a non-transitory computer-readable storage medium storing one or more programs may include instructions that, when executed by at least one processor of a wearable device having a display and one or more cameras configured to track the eyes of a user wearing the wearable device, cause the display to display a plurality of visual objects on the display. The plurality of visual objects may each include gaze input areas formed on the plurality of visual objects to provide visual feedback for the user's gaze directed toward the plurality of visual objects. The one or more programs may include instructions that, when executed by the at least one processor, cause the display of the plurality of visual objects to determine whether to correct gaze information related to the gaze, the gaze information being determined based on the one or more cameras. The one or more programs may include instructions that, when executed by the at least one processor, cause the sizes of each of the gaze input areas formed on the visual objects to be reduced based on determining the correction of the gaze information. The one or more programs may include instructions that, when executed by the at least one processor, cause the gaze information detected while providing the visual feedback using the reduced sizes to generate correction information to be used for correcting the gaze information.The one or more programs, when executed by the at least one processor, may include instructions that cause the one or more programs, when executed by the at least one processor, to expand the sizes of the gaze input areas using the corrected gaze information to provide visual feedback for the gaze using the corrected gaze information.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment, according to one embodiment.

[0009] FIGS. 2A and 2B illustrate an example of a perspective view of a wearable device, according to one embodiment.

[0010] FIGS. 3A and 3B illustrate an example of an appearance of a wearable device according to one embodiment.

[0011] FIG. 4 illustrates an example block diagram of a wearable device according to one embodiment.

[0012] FIG. 5 illustrates a portion of a screen displayed through a display of an exemplary wearable device.

[0013] Figures 6a, 6b, 6c, 6d, and 6e illustrate the operation of an exemplary wearable device for correction of gaze information.

[0014] Figure 7 is a flow chart showing the operation of an exemplary wearable device for correcting gaze information.

[0015] Figure 8 is a flowchart showing the operation of an exemplary wearable device for correcting gaze information.

[0016] Figures 9a, 9b, and 9c illustrate the operation of an exemplary wearable device for providing visual feedback.

[0017] Figure 10 is a graph showing the positions of multiple predicted points of a user's gaze on the display of an exemplary wearable device over time.

[0018] Figure 11 is a graph showing the relationship between error values ​​corresponding to correction information of an exemplary wearable device and padding values ​​of gaze input areas.

[0019] FIG. 12 illustrates a portion of a screen displayed through a display of an exemplary wearable device.

[0020] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0021] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0022] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0023] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0024] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0025] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0026] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0027] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0028] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0029] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0030] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0031] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0032] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0033] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0034] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0035] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0036] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0037] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0038] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0039] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0040] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0041] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0042] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0043] FIGS. 2A and 2B illustrate an example of a perspective view of a wearable device, according to one embodiment.

[0044] According to one embodiment, the wearable device (200) may have the form of glasses that are wearable on a body part of a user (e.g., head). The wearable device (200) of FIGS. 2A and 2B may be an example of the electronic device (101) of FIG. 1. The wearable device (200) may include a head-mounted display (HMD). For example, the housing of the wearable device (200) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (200) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0045] Referring to FIG. 2A, according to one embodiment, a wearable device (200) may include at least one display (250) and a frame (295) supporting at least one display (250).

[0046] According to one embodiment, the wearable device (200) can be worn on a part of a user's body. The wearable device (200) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (200). For example, the wearable device (200) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0047] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0048] Referring to FIG. 2B, at least one display (250) can provide the user with visual information transmitted from external light and other visual information distinct from the visual information through a lens included in the at least one display (250). The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When the user wears the wearable device (200), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). According to one embodiment, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0049] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (200) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0050] The wearable device (200) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (200) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (200) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (200) can view an image displayed on at least one display (250).

[0051] According to one embodiment, the frame (295) may be formed as a physical structure that allows the wearable device (200) to be worn on the user's body. According to one embodiment, the frame (295) may be configured so that, when the user wears the wearable device (200), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (295) may support at least one display (250). For example, the frame (295) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0052] Referring to FIG. 2A, the frame (295) may include a region (220) that at least partially contacts a part of the user's body when the user wears the wearable device (200). For example, the region (220) of the frame (295) that contacts a part of the user's body may include a region that contacts a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (200) makes contact with. According to one embodiment, the frame (295) may include a nose pad (210) that contacts a part of the user's body. When the wearable device (200) is worn by the user, the nose pad (210) may contact a part of the user's nose. The frame (295) may include a first temple (204) and a second temple (205) that contact a part of the user's body that is distinct from the part of the user's body.

[0053] For example, the frame (295) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of an ear of the wearer, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of an ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (200) can identify an external object (e.g., a user's fingertip) touching the frame (295) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (295).

[0054] According to one embodiment, the wearable device (200) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within a frame (295).

[0055] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (200) may be disposed on at least a portion of the frame (295) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B , but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B . When the number of microphones (265) included in the wearable device (200) is two or more, the wearable device (200) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (295).

[0056] In one embodiment, at least one optical device (282, 284) can project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) can be a projector. At least one optical device (282, 284) can be disposed adjacent to at least one display (250) or can be included within at least one display (250) as a part of at least one display (250). In one embodiment, the wearable device (200) can include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0057] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or motion recognition cameras (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (295) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (200). For example, the wearable device (200) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (200) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (200) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (200) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (200) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other. The wearable device (200) can obtain an image having visual quality of a first area matching the user's gaze and visual quality of a second area using foveated rendering.For example, if the wearable device (200) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1). An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0058] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (260-4) is superimposed on a virtual image provided through at least one optical device (282, 284). The wearable device (200) can compensate for depth information (e.g., the distance between the wearable device (200) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (260-4). The wearable device (200) can perform object recognition through an image acquired using a photographing camera (260-4). The wearable device (200) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the photographing camera (260-4). The wearable device (200) can perform a pass-through function to display an image acquired through the photographing camera (260-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the photographing camera (260-4) can be disposed on a bridge (203) disposed between a first rim (201) and a second rim (202).

[0059] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (200) and matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (200) looks straight ahead, the wearable device (200) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (200) is positioned.

[0060] The gesture recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The gesture recognition camera (260-2, 260-3) can recognize the user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (260-2, 260-3) can be used to perform simultaneous localization and mapping (SLAM) for 6 degrees of freedom pose (6 dof pose) and / or a spatial recognition function using a depth map. The processor can perform a gesture recognition function and / or an object tracking function using the gesture recognition camera (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) may be positioned on the first rim (201) and / or the second rim (202).

[0061] The camera (260) included in the wearable device (200) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (200) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (200) can identify an external object based on a sensor for identifying the distance between the wearable device (200) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (200) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (200).

[0062] Although not shown, in one embodiment, the wearable device (200) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (295) and the hinge units (206, 207).

[0063] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (200). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0064] The antenna module (275) can transmit signals or power to the outside of the wearable device (200), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0065] The speaker (255) can output an audio signal to the outside of the wearable device (200). The audio output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (200). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus adjacent to the user's right ear.

[0066] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state, in order to visually provide information regarding a specific state of the wearable device (200) to the user. For example, when the wearable device (200) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0067] Referring to FIG. 2B, according to one embodiment, a wearable device (200) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (200) (e.g., hardwares illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (200) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0068] According to one embodiment, a wearable device (200) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (200) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (200) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (200) based on the IMU.

[0069] FIGS. 3A and 3B illustrate an example of an appearance of a wearable device according to one embodiment.

[0070] The wearable device (300) of FIGS. 3A and 3B may be an example of the electronic device (101) of FIG. 1 and the wearable device (200) of FIGS. 2A and 2B. According to one embodiment, an example of the appearance of a first side (310) of a housing of the wearable device (200) is illustrated in FIG. 3A, and an example of the appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B.

[0071] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (200) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (200) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (200) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0072] According to one embodiment, the wearable device (200) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (200) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (200) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (200) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar representing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (200).

[0073] Referring to FIG. 3B, a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor 330) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A to obtain information related to the external environment of the wearable device (200). For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0074] For example, using cameras (260-11, 260-12), the wearable device (200) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be placed on the second face (320) of the wearable device (200) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be placed on the second face (320) of the wearable device (200) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0075] According to one embodiment, the wearable device (200) may include a depth sensor (330) disposed on the second surface (320) to identify a distance between the wearable device (200) and an external object. Using the depth sensor (330), the wearable device (200) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (200). According to one embodiment, the sensor (330) may be configured to detect an angle (a) between a reference axis (x1) and an orientation axis (x2) of the wearable device (200). Based on the angle (a), the sensor (330) may detect an inclination of the wearable device (200). Although not shown, a microphone may be placed on the second side (320) of the wearable device (200) to acquire sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0076] FIG. 4 illustrates an example block diagram of a wearable device according to one embodiment.

[0077] Referring to FIG. 4, a wearable device (200) according to one embodiment may include at least one of a processor (410), a memory (415), a display (420), a camera (425), a sensor (430), or a communication circuit (435). The processor (410), the memory (415), the display (420), the camera (425), the sensor (430), and the communication circuit (435) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (402). The type and / or number of hardware components included in the wearable device (200) is not limited to those illustrated in FIG. 4. For example, the wearable device (200) may include only some of the hardware components illustrated in FIG. 4. The elements within the memory described below (e.g., layers and / or modules) may be logically distinct, but are not limited thereto.

[0078] According to one embodiment, the processor (410) of the wearable device (200) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (410) may be one or more. For example, the processor (410) may have a multi-core processor structure, such as a dual core, a quad core, or a hexa core. For example, the processor (410) may be referred to as at least one processor.

[0079] Any function or operation described herein may be processed by at least one processor (410). According to one embodiment, at least one processor (410) may include a processing circuit. At least one processor (410) may include, but is not limited to, an application processor (AP, e.g., a central processing unit (CPU)) and / or a communication processor (CP, e.g., a modem). At least one processor (410) may include a graphics processing unit (e.g., a GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a wireless-fidelity (Wi-Fi) chip, a Bluetooth chip®, a global positioning system (GPS) chip, a near field communication (NFC) chip, connectivity chips, a sensor controller, a touch controller, a finger-print sensor controller, a display drive integrated circuit (DDI), an audio CODEC chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on chip (SoC), an integrated circuit (IC), or a similar circuit.

[0080] The memory (415) of the wearable device (200) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (410). The memory (415) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi media card (eMMC).

[0081] In one embodiment, the display (420) of the wearable device (200) can output visualized information to a user of the wearable device (200). For example, the display (420) can be controlled by a processor (410) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (420) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0082] In one embodiment, the camera (425) of the wearable device (200) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (425) may be arranged in the form of a two-dimensional array. The camera (425) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (425) may mean one (a) two-dimensional frame data acquired from the camera (425). For example, video data captured using the camera (425) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (425) according to a frame rate. The camera (425) may further include a flash light that is positioned toward the direction in which the camera (425) receives light and outputs light toward the direction.

[0083] According to one embodiment, the wearable device (200) may include a plurality of cameras, for example, cameras (425), arranged in different directions. A first camera among the plurality of cameras may be referred to as a motion recognition camera (e.g., motion recognition cameras 260-2 and 260-3 of FIG. 2B ), and a second camera may be referred to as a gaze tracking camera (e.g., gaze tracking camera 260-1 of FIG. 2B ). The wearable device (200) may identify a position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (200) may identify a direction of a gaze of a user wearing the wearable device (200) using an image acquired using the second camera. For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0084] According to one embodiment, a sensor (430) of a wearable device (200) may generate electrical information that may be processed by a processor (410) and / or a memory (415) of the wearable device (200) from non-electronic information related to the wearable device (200). The information may be referred to as sensor data. The sensor (430) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (200), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (200).

[0085] In one embodiment, the communication circuit (435) of the wearable device (200) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (200) and an external electronic device. The communication circuit (435) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (435) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0086] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by the processor (410) of the wearable device (200) may be stored in the memory (415) of the wearable device (200). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (200) and / or the processor (410) may perform at least one of the operations according to the embodiments described below when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (200) may mean that one or more instructions provided in the form of an application are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (200)). As an example, an application may include a program and / or a library related to a service provided to a user.

[0087] Referring to FIG. 4, programs installed in the wearable device (200) may be classified into one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (200) (e.g., the display (420), the camera (425), and / or the sensor (430)) may be classified within the hardware abstraction layer (480). The framework layer (450) may be referred to as an XR framework layer in that it includes one or more programs for providing an XR (extended reality) service. For example, FIG. 4 illustrates layers being divided within the memory (415), the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (415).

[0088] For example, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a position tracker (471), a space recognizer (472), a gesture tracker (473), and / or an eye tracker (474), a face tracker (475)) may be classified. Programs classified within the framework layer (450) may provide an executable API (application programming interface) based on other programs.

[0089] For example, within the application layer (440), programs designed to target users controlling wearable devices (200) may be classified. Examples of programs classified into the application layer (440) include, but are not limited to, an XR (extended reality) system UI (user interface) and / or an XR application (442). For example, programs (e.g., software applications) classified into the application layer (440) may call an API (application programming interface) to cause execution of functions supported by programs classified into the framework layer (450).

[0090] For example, the wearable device (200) may display one or more visual objects on the display (420) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (441). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (200) may provide a service for controlling functions available within a virtual space to the user based on the execution of the XR system UI (441).

[0091] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plug-in (444) are illustrated as being included within the XR system UI (441), but are not limited thereto. For example, the XR system UI (441) may cause execution of functions supported by the lightweight renderer (443) and / or the XR plug-in (444) included within the framework layer (450).

[0092] For example, the wearable device (200) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (200) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (444). The XR plugin (444) may be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0093] For example, the wearable device (200) may display a screen representing at least a portion of a virtual space on the display (420) based on the execution of the XR application (442). The XR plug-in (444-1) included in the XR application (442) may be referenced by the XR plug-in (444) of the XR system UI (441). Descriptions of the XR plug-in (444-1) that overlap with the description of the XR plug-in (444) may be omitted. The wearable device (200) may cause the execution of the virtual space manager (451) based on the execution of the XR application (442).

[0094] According to one embodiment, the wearable device (200) may provide a virtual space service based on the execution of the virtual space manager (451). For example, the virtual space manager (451) may include a platform (e.g., an Android platform) for supporting the virtual space service. The wearable device (200) may display the posture of a virtual object representing the user's posture rendered using data acquired through the sensor (430) based on the execution of the virtual space manager (451) on the display. The virtual space manager (451) may be referred to as a composition presentation manager (CPM).

[0095] For example, the virtual space manager (451) may include a runtime service (452). As an example, the runtime service (452) may be referred to as an OpenXR runtime module. The wearable device (200) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (200) based on the execution of the runtime service (452). As an example, the wearable device (200) may be used to perform rendering for a virtual space service to a user based on the execution of the runtime service (452). For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the execution of the runtime service (452).

[0096] For example, the virtual space manager (451) may include a pass-through manager (453). Based on the execution of the pass-through manager (453), the wearable device (200) may display a screen representing a virtual space on the display (420), while another screen representing an actual space acquired through the camera (425) may be superimposed on at least a portion of the screen.

[0097] For example, the virtual space manager (451) may include an input manager (454). Based on the execution of the input manager (454), the wearable device (200) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (470). The wearable device (200) may initiate execution of at least one of the functions of the wearable device (200) using the acquired data.

[0098] For example, the perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). From the perspective of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referenced as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0099] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data acquired from a sensor (430) (or a camera (425)). The one or more programs may include at least one of a position tracker (471), a space recognizer (472), a gesture tracker (473), an eye tracker (474), and / or a face tracker (475). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those illustrated in FIG. 4.

[0100] For example, the wearable device (200) can identify the pose of the wearable device (200) using the sensor (430) based on the execution of the position tracker (471). The wearable device (200) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (200) using data acquired using the camera (425) and the IMU based on the execution of the position tracker (471). The position tracker (471) can be referred to as a head tracking (HeT) module.

[0101] For example, the wearable device (200) may be used to construct a three-dimensional virtual space around the wearable device (200) (or a user of the wearable device (200)) based on the execution of the space recognizer (472). The wearable device (200) may reconstruct the three-dimensional surroundings of the wearable device (200) using data acquired using the camera (425) based on the execution of the space recognizer (472). The wearable device (200) may identify at least one of a plane, a slope, and stairs based on the three-dimensionally reconstructed surroundings of the wearable device (200) based on the execution of the space recognizer (472). The space recognizer (472) may be referred to as a scene understanding (SU) module.

[0102] For example, the wearable device (200) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (200) based on the execution of the gesture tracker (473). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (430) based on the execution of the gesture tracker (473). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using a camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0103] For example, the wearable device (200) may identify (or track) eye movements of a user of the wearable device (200) based on the execution of the gaze tracker (474). As an example, the wearable device (200) may identify eye movements of the user using data acquired from at least one sensor based on the execution of the gaze tracker (474). As an example, the wearable device (200) may identify eye movements of the user based on data acquired using a camera (e.g., the gaze tracking camera (260-1) of FIGS. 2A and 2B) and / or an infrared light emitting diode (IR LED) based on the execution of the gaze tracker (474). The gaze tracker (474) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0104] For example, the recognition service layer (470) of the wearable device (200) may further include a face tracker (475) for tracking the user's face. For example, the wearable device (200) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (475). The wearable device (200) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (475). As an example, the wearable device (200) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (475).

[0105] For the embodiments described below, the wearable device (200) of FIG. 4 may be referred to. For example, the embodiments described below may be performed by the processor (410) of the wearable device (200) of FIG. 4.

[0106] FIG. 5 illustrates a portion of a screen displayed through a display of an exemplary wearable device.

[0107] Referring to FIG. 5, the wearable device (200) may include a camera (e.g., camera (425) of FIG. 4) positioned in front of the user (50) while the device is being worn by the user (50). The front of the user (50) may include the direction in which the user's (50's) head and / or the user's (50's) gaze is directed. According to one embodiment, the wearable device (200) may include a sensor (e.g., sensor (430) of FIG. 4) for identifying the motion of the user's (50's) head and / or the wearable device (200) while the device is being worn by the user (50).

[0108] A processor of a wearable device (200) (e.g., processor (410) of FIG. 4) may identify an angle of the wearable device (200) based on data of a sensor (430). For example, referring also to FIG. 3B, the processor (410) may identify an angle (a) between an orientation axis (x2) representing an orientation of the wearable device (200) and a reference axis (x1) using information acquired through a camera (425) and / or a sensor (430) of the wearable device (200). In order to provide a user interface (UI) based on virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) to a user (50) wearing the wearable device (200), the processor (410) may control the camera (425) and / or the sensor (430). The UI may be related to a metaverse service provided by a wearable device (200), and / or a server connected to the wearable device (200), and / or a notification service.

[0109] According to one embodiment, the processor (410) may execute functions related to augmented reality (AR) and / or mixed reality (MR). While the user (50) is wearing the wearable device (200), the processor (410) may include at least one lens positioned adjacent to the eyes of the user (50). Ambient light passing through the lens may be combined (or mixed) with light emitted from a display of the wearable device (200) (e.g., display (420) of FIG. 4). A display area of ​​the display (420) may be formed within the lens through which the ambient light passes. Because the processor (410) combines the ambient light and the light emitted from the display (420), the user (50) can see an image that is a mixture of a real object recognized by the ambient light and a virtual object (e.g., a plurality of visual objects (520)) formed by the light emitted from the display (420).

[0110] According to one embodiment, a wearable device (200) may perform functions related to video see-through (VST) and / or virtual reality (VR). When a user (50) wears the wearable device (200), the wearable device (200) may include a housing that covers the eyes of the user (50). When the user (50) wears the wearable device (200), the wearable device (200) may include a display (420) disposed on a first side facing the eyes (e.g., the first side (310) of FIG. 3A). The wearable device (200) may include a camera (425) (or cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12) of FIG. 3B) disposed on a second surface opposite to the first surface (e.g., the second surface (320) of FIG. 3A). Using the camera (425), the processor (410) may acquire frame images including ambient light. The processor (410) may output the frame images to a display (420) disposed on the first surface, thereby allowing the user (50) to recognize the ambient light through the display (420). The display area of ​​the display (420) disposed on the first surface may be formed by one or more pixels included in the display (420). The processor (410) can synthesize a virtual object within frame images output through the display (420) to enable the user (50) to recognize the virtual object together with a real object recognized by ambient light.

[0111] According to one embodiment, a wearable device (200) may provide a user (50) experience based on mixed reality (MR) by utilizing a virtual space. A processor (410) of the wearable device (200) may recognize an external space (e.g., a real space) in which the wearable device (200) is included, and generate a virtual space mapped to the external space. The spatial recognition performed by the processor (410) may include simultaneous localization and mapping (SLAM) and / or spatial mapping (e.g., scene understanding).

[0112] According to one embodiment, the processor (410) may be configured to determine the positions of a plurality of prediction points (510) for predicting the direction of the gaze of the user (50) on the display (420) using gaze information related to the gaze of the user (50). For example, the processor (410) may be configured to obtain gaze information related to the gaze of the user (50) wearing the wearable device (200) based on image information acquired through the camera (425) of the wearable device (200). For example, the processor (410) may be configured to obtain gaze information related to the gaze of a user (50) wearing the wearable device (200) through a sensor (430) configured to detect an angle between an orientation axis of the wearable device (200) (e.g., the orientation axis (x2) of FIG. 3B) and a reference axis (x1) within the wearable device (200). However, the above-described embodiments are exemplary and are not limited thereto. According to one embodiment, the gaze information related to the gaze may include information based on a position of an eye, a position of a pupil, an angle of a gaze, a direction of a gaze, a gaze vector, and / or a gaze point.

[0113] For example, the processor (410) may be configured to determine the positions of a plurality of predicted points (510) on the display (420) for predicting the gaze of the user (50) based on gaze information related to the gaze of the user (50). For example, the processor (410) may change the positions of the plurality of predicted points (510) on the screen (501) provided through the display (420) through gaze information acquired using a camera (425) and / or a sensor (430). The processor (410) can change the positions of the plurality of predicted points (510) on the display (420) by using the gaze information to determine the positions of the plurality of predicted points (510) on the display (420), according to the movement of the user (50) (e.g., movement of the gaze, rotation of the head wearing the wearable device (200)) for gaze input to the plurality of visual objects (520) displayed on the display (420).

[0114] According to one embodiment, the processor (410) may be configured to display a plurality of visual objects (520) on the display (420) of the wearable device (200). The plurality of visual objects (520) may include gaze input regions (530) formed on each of the plurality of visual objects to provide visual feedback for the gaze of the user (50) directed toward the plurality of visual objects. For example, the processor (410) may control the display (420) to display the plurality of visual objects (520) on the screen (501) based on receiving a user input for displaying the plurality of visual objects (520). For example, at least some of the visual objects (520) displayed through the display (420) may correspond to images of actual objects acquired through the camera (425). For example, at least some of the visual objects (520) obtained through the display (420) may be virtual objects for receiving user input. However, the above-mentioned embodiments are exemplary and are not limited thereto.

[0115] For example, a plurality of visual objects (520) may form gaze input areas (530) to provide visual feedback to the user (50) based on the gaze of the user (50) toward the plurality of visual objects, respectively. The gaze input areas (530) may be set or formed on the plurality of visual objects (520), respectively. The gaze input areas (530) may have an area within a designated range surrounding each of the plurality of visual objects (520) in order to receive user input to the plurality of visual objects (520) based on the gaze of the user (50). The size of each of the gaze input areas (530) may be determined by, but is not limited to, a padding value between a boundary of the plurality of visual objects (520) and a boundary of the gaze input areas (530). According to one embodiment, the padding value of the gaze input areas (530) may mean a distance from a minimum gaze input area (or reference input area) of the gaze input areas (530). For example, the minimum gaze input area (or reference gaze input area) of the gaze input areas (530) may be larger or smaller than an area of ​​a plurality of visual objects (520).

[0116] For example, the gaze input areas (530) may be areas for providing visual feedback to the user (50) wearing the wearable device (200) based on the positions of a plurality of predicted points (510) on the display (420) for predicting the direction of the gaze of the user (50). For example, the processor (410) may control the display (420) to display visual objects (520) for receiving user input based on the positions of the plurality of predicted points (510). The processor (410) may be configured to identify user input to the plurality of visual objects (520) based on the positions of the plurality of predicted points (510) for predicting the direction of the gaze of the user (50) with respect to the gaze input areas (530) respectively formed on the plurality of visual objects (520).

[0117] For example, the processor (410) may, based on the gaze information determined by the camera (425), position a plurality of predicted points (510) around the visual object (521a) and the gaze input area (531a) of the visual object (521a) among the plurality of visual objects (520) for user input to the visual object (521a). For example, the processor (410) may identify a plurality of predicted points (510) positioned to at least partially overlap with the gaze input area (531a). Based on the identification of the plurality of predicted points (510) positioned to at least partially overlap with the gaze input area (531a), the processor (410) may receive a user input to the visual object (521a) corresponding to the gaze input area (531a). The processor (410) may control the display (420) to display visual information provided through the visual object (521a) on the display (420) based on receiving the user input to the visual object (521a). For example, the processor (410) may switch the screen (501) of the display (420) to display contents provided through the visual object (521a) corresponding to the gaze input area (531a) on the display (420) based on a plurality of predicted points (510) positioned to at least partially overlap the gaze input area (531a). However, the above-mentioned embodiment is exemplary and is not limited thereto.

[0118] The processor (410) may receive a user input to each of a plurality of visual objects (520) corresponding to each of the gaze input areas (530) based on a plurality of predicted points (510) positioned to overlap each of the gaze input areas (530) on the display (420). The processor (410) may control the display (420) to display a screen related to each of the plurality of visual objects (520) based on the user input received to each of the plurality of visual objects (520). For example, as illustrated, the processor (410) can control the display (420) to display visual objects (521a, 521b, 522a, 522b, 523a, 523b, 523c, 523d, 524a, 524b, 524c, 524d) on the screen (501) of the display (420). On the above display, gaze input areas (531a, 531b, 532a, 532b, 533a, 533b, 533c, 533d, 534a, 534b, 534c, 534d) corresponding to each of the visual objects (521a, 521b, 522a, 522b, 523a, 523b, 523c, 523d, 524a, 524b, 524c, 524d) can be set.

[0119] According to one embodiment, the processor (410) may determine the position of a gaze prediction point (not shown) on the display (420) corresponding to the gaze of the user (50) using the positions of the plurality of prediction points (510). For example, the processor (410) may identify the position of the gaze prediction point using position information of each of the plurality of prediction points (510). The position of the gaze prediction point may be calculated as an average value of the positions of the plurality of prediction points (510), but is not limited thereto. For example, reception of a user input through the plurality of visual objects (520) to the processor (410) may be determined according to the positions of the gaze prediction points determined by the plurality of prediction points (510). For example, among the plurality of prediction points (510), the prediction points (511, 512) may be positioned to overlap with the gaze input area (531a) of the visual object (521a). Among the plurality of predicted points (510), the predicted points (513, 514, 515) may be located outside the gaze input area (531a). The position of the gaze predicted point of the user (50), determined by the average value of the positions of the predicted points (511, 512) and the positions of the predicted points (513, 514, 515), may be located outside the gaze input area (531a). Since the gaze predicted point is located outside the gaze input area (531a), a user input through the visual object (521a) to the processor (410) may not be received. However, the above-mentioned embodiments are exemplary and are not limited thereto.

[0120] According to one embodiment, the processor (410) may identify whether to correct gaze information related to the gaze of the user (50), which is determined based on the camera (425), while displaying a plurality of visual objects (520) on the display (420). For example, due to an external factor of the wearable device (200) (e.g., slippage due to an external impact applied to the wearable device (200), the actual gaze direction (541) of the user (50) and the gaze direction (542) predicted by the wearable device (200) may be significantly different. For example, the position of the gaze prediction point corresponding to the actual gaze direction (541) of the user (50) may be positioned to overlap with the gaze input area (531a) for user input to the visual object (521a). The position of the gaze prediction point corresponding to the direction (542) of the gaze of the user (50) predicted by the wearable device (200) may be located outside the gaze input area (531a) according to the gaze information determined by the camera (425). The wearable device (200) may be required to perform an operation for calibrating the gaze information in order to calibrate the position of the gaze prediction point predicted by the wearable device (200) to the position of the gaze prediction point corresponding to the actual gaze of the user (50).

[0121] According to one embodiment, the processor (410) may obtain an error value between the position of the gaze prediction point corresponding to the actual gaze direction (541) of the user (50) and the position of the gaze prediction point corresponding to the gaze direction (542) of the user (50) predicted by the wearable device (200) based on a change in the position of the plurality of prediction points (510) (or the gaze prediction point determined by the plurality of prediction points (510). The error value may be obtained, for example, from the direction (543) between the direction (541) and the direction (542). The processor (410) may be configured to correct gaze information to be used for determining the position of the gaze prediction point, for example, by using the error value. For example, the processor (410) may execute a function for correcting the gaze information through a session for storing the gaze information, based on determining the correction of the gaze information. When a function for correcting the gaze information is executed through a session for storing the gaze information, the correction of the gaze information may take a considerable amount of time. In order to reduce the time required for correcting the gaze information, the wearable device (200) may be required to correct the gaze information using the sizes of each of the plurality of divided areas (550) and / or the gaze input areas (530) on the display (420) where the gaze prediction point is located. An operation for correcting the gaze information using the sizes of each of the plurality of divided areas (550) and / or the gaze input areas (530) is described below with reference to FIGS. 6A to 6D.

[0122] According to one embodiment, the processor (410) may set a plurality of partitioned areas (550) on a display (420) that displays a plurality of visual objects (520). The processor (410) may use the plurality of partitioned areas (550) to identify an area among the plurality of partitioned areas (550) where a plurality of predicted points (510) (or predicted gaze points) determined through gaze information are located.

[0123] For example, the plurality of partition areas (550) can control the display (420) to set a first partition area (551), a second partition area (552) surrounding the first partition area (551), and a third partition area (553) surrounding the second partition area (552) on the screen (501) while the plurality of visual objects (520) are displayed on the screen (501) through the display (420). For example, the processor (410) can set the partition areas (551, 552, 553) based on the FoV of the user (50). The first partition area (551) can be set based on the FoV (e.g., 20 degrees) within the first reference range of the user (50). The second partition area (552) surrounding the first partition area (551) may be set based on a FoV (e.g., 40 degrees) within a second reference range that is larger than the first reference range. The third partition area (553) surrounding the second partition area (552) may be set as the remaining area of ​​the first partition area (551) and the second partition area (552). However, the above-mentioned embodiments are exemplary and are not limited thereto.

[0124] For example, the processor (410) may identify a first set of visual objects (521) located within the first partitioned area (551) and a first set of gaze input areas (531) formed on the first set of visual objects (521) based on identifying a plurality of predicted points (510) within the first partitioned area (551). For example, the processor (410) may identify a second set of visual objects (522) located within the second partitioned area (552) and a second set of gaze input areas (532) formed on the second set of visual objects (522).

[0125] According to one embodiment, the processor (410) may determine an area where visual objects (523, 524) and / or gaze input areas (533, 534) that at least partially overlap with boundaries (b1, b2) between the plurality of segmented areas (550) are located. For example, the processor (410) may set a center point on the display (420) of each of the plurality of visual objects (520) as a location of each of the plurality of visual objects (520). Based on the location of the center point, an area where the plurality of visual objects (520) are located may be determined.

[0126] For example, the center points of the third set of visual objects (523) that at least partially overlap with the boundary (b1) of the first partition area (551) and the second partition area (552) may each be located within the second partition area (552). The processor (410) may determine, based on the center points of the third set of visual objects (523) located within the second partition area (552), that the third set of visual objects (523) and the third set of gaze input areas (533) formed on the third set of visual objects (523) are located within the second partition area (552). For example, the center points of the fourth set of visual objects (524) that at least partially overlap with the boundary (b2) of the second partition area (552) and the third partition area (553) may each be located within the second partition area (552). The processor (410) may determine, based on the center points of the fourth set of visual objects (524) located within the second divided area (552), the fourth set of visual objects (524) and the fourth set of gaze input areas (534) formed on the fourth set of visual objects (524), respectively, to be located within the second divided area (552). However, the above-described embodiments are exemplary and are not limited thereto. The processor (410) may perform gaze information correction using the plurality of divided areas (550) and the gaze input areas (530) of the plurality of visual objects (520) located within the plurality of divided areas (550). The operation of the processor (410) for correcting the gaze information through the plurality of divided areas (550) and the gaze input areas (530) is illustrated and described in FIGS. 6A to 6D.

[0127] Figures 6a, 6b, 6c, 6d, and 6e illustrate the operation of an exemplary wearable device for correction of gaze information.

[0128] Referring to FIGS. 6A, 6B, 6C, and 6D, a processor (e.g., processor (410) of FIG. 4) may be configured to display a plurality of visual objects (520) on a display (e.g., display (420) of FIG. 4). The plurality of visual objects (520) may include gaze input areas (530) formed on each of the plurality of visual objects (520) to provide visual feedback for a gaze of a user (e.g., user (50) of FIG. 5) directed toward the plurality of visual objects (520). The processor (410) may be configured to acquire gaze information related to the gaze, which is determined based on a camera (e.g., camera (425) of FIG. 4) of the wearable device (200), while displaying the plurality of visual objects (520). The processor (410) may be configured to use the gaze information to determine a plurality of prediction points (510) for predicting the gaze of the user and / or a location on the display (420) of a gaze prediction point determined by the plurality of prediction points (510). The processor (410) may be configured to set a plurality of partition areas (550) on the display (420) that display the plurality of visual objects (520).

[0129] In the following, redundant descriptions of the configuration and / or operation described in Fig. 5 are omitted.

[0130] Referring to FIG. 6A, the processor (410) may be configured to identify whether to correct gaze information related to the user's gaze within a state (601) in which a plurality of visual objects (520) are displayed through the display (420).

[0131] For example, the processor (410) may be configured to change the positions of the plurality of predicted points (510) based on a first user input for visual feedback to the gaze input regions (530). The processor (410) may be configured to measure a time interval at which the first user input is received based on identifying at least some of the plurality of predicted points (510) located within one of the gaze input regions (530). The processor (410) may determine a correction of gaze information related to the user's gaze based on the time interval exceeding a set reference time interval.

[0132] For example, the processor (410) may change the positions of a plurality of predicted points (510) on the display (420) based on receiving a first user input for visual feedback through a visual object (521a). While the first user input is being received, the processor (410) may measure a time interval until the input for visual feedback through the visual object (521a) is received. The processor (410) may be configured to determine a correction of gaze information related to the user's gaze based on identifying the time interval exceeding a set reference time interval.

[0133] For example, the processor (410) may measure a failure rate of visual feedback through a plurality of predicted points (510) and gaze input areas (530) for a first user input. The processor (410) may be configured to determine correction of gaze information related to the user's gaze based on whether the failure rate exceeds a set reference failure rate.

[0134] For example, the processor (410) may include a sensor (e.g., sensor (430) of FIG. 4) for detecting a position of the wearable device (200). The processor (410) may be configured to detect an angle (e.g., angle (a) of FIG. 3b) between a reference axis (e.g., reference axis (x1) of FIG. 3b) of the wearable device (200) and an orientation axis (e.g., orientation axis (x2) of FIG. 3b) of the wearable device (200) through the at least one sensor.

[0135] The orientation axis (x2) of the wearable device (200) may be an axis indicating the orientation (or direction) of the wearable device (200). For example, the orientation axis (x2) may rotate along the wearable device (200) due to the rotation of the wearable device (200). The reference axis (x1) of the wearable device (200) may be an axis that is fixed with respect to the orientation axis (x2) of the wearable device (200). The reference axis (x1) may be, for example, an axis that is independently fixed with respect to the movement and / or rotation of the wearable device (200). The orientation axis (x2) of the wearable device (200) can provide information on the inclination of the wearable device (200) by indicating the orientation of the wearable device (200) according to the rotation of the wearable device (200) through the angle (a) between the reference axis (x1) fixed to the orientation axis (x2) and the orientation axis (x2), thereby using at least one sensor.

[0136] The processor (410) may be configured to determine correction of gaze information related to the user's gaze in response to identifying that the angle (a) exceeds a set reference angle. However, the above-described embodiments are exemplary and not limited thereto. The processor (410) may be configured to detect a deterioration in the eye tracking function based on the user's gaze of the wearable device (200) in various ways other than the above-described embodiments. The processor (410) may be configured to determine correction of gaze information related to the user's gaze based on detecting a deterioration in the eye tracking function.

[0137] According to one embodiment, the processor (410) may be configured to reduce the sizes of each of the gaze input areas (530) based on determining the correction of the gaze information. For example, the processor (410) may minimize the sizes of the plurality of gaze input areas (530) formed on the plurality of visual objects (520) based on determining the correction of the gaze information. According to one embodiment, the processor (410) may be configured to set a padding value of each of the plurality of visual objects (520) for the plurality of gaze input areas (530) to 0 or to a minimum value based on determining the correction of the gaze information.

[0138] For example, a state in which the padding value of each of the plurality of visual objects (520) is 0 may be a state in which the plurality of visual objects (520) and the plurality of gaze input areas (530) overlap each other on the plurality of visual objects (520), so that no space is provided in the display area between the plurality of visual objects (520) and the plurality of gaze input areas (530). For example, a state in which the padding value of each of the plurality of visual objects (520) is minimum may be a state in which, among the plurality of sizes of the gaze input areas (530) corresponding to each of the plurality of visual objects (520) that can be set within the wearable device (200), the size of each of the gaze input areas (530) is a size at which the size of the display area between the gaze input areas (530) and the plurality of visual objects (520) becomes the minimum size.

[0139] However, the above-described embodiments are exemplary, and the padding value reset for correction of gaze information may vary depending on the wearable device (200) and / or settings. For example, in response to the direction of the user's gaze, the position and / or direction of the display area having the minimum size between the gaze input areas (530) and the plurality of visual objects (520) may be changed depending on the position of the prediction points (510) for predicting the gaze on the display (420).

[0140] According to one embodiment, the size of each of the gaze input areas (530) corresponding to each of the plurality of visual objects (520) may be set differently depending on the head mounted display (HMD), auxiliary lens, and glasses that provide a screen for displaying the plurality of visual objects (520) of the wearable device (200).

[0141] Referring sequentially to FIGS. 6A and 6B, the processor (410) may be configured to generate correction information to be used to correct the gaze information using the detected gaze information while providing visual feedback using the sizes of the reduced gaze input areas (530).

[0142] For example, the processor (410) may receive a second user input for visual feedback from the gaze input areas (530) while changing from a state (601) to a state (602). Based on the second user input, the processor (410) may change the positions of the plurality of predicted points (510) on the display (420). The processor (410) may be configured to obtain correction information for correcting the gaze information used to determine the positions of the predicted points (510) through the changed positions of the plurality of predicted points (510).

[0143] For example, the processor (410) may induce a user input for visual feedback through the gaze input regions (530) (and / or the plurality of visual objects (520)) by reducing the sizes of the gaze input regions (530) based on determining a correction of the gaze information. For example, the processor (410) may be configured to induce a user input for input to a visual object (521a) by minimizing the sizes of the gaze input regions (530) based on determining a correction of the gaze information. The processor (410) may induce an input to the visual object (521a) by changing the positions of the plurality of predicted points (510) based on a second user input for input to the visual object (521a).

[0144] According to one embodiment, the processor (410) may be configured to identify whether an error value corresponding to correction information to be used to correct gaze information exceeds a specified error value. The processor (410) may be configured to execute a function for correcting the gaze information through a session for storing the gaze information based on identifying that the error value exceeds the specified error value. The processor (410) may execute a function for correcting the gaze information by adjusting the sizes of a plurality of gaze input areas (530) based on the error value being less than the specified error value. The processor (410) may be configured to correct an error value exceeding the specified error value through the session, thereby increasing the accuracy of correction for relatively large errors.

[0145] According to one embodiment, the processor (410) may be configured to expand the sizes of the gaze input areas (530) using correction information to be used to correct the gaze information. The processor (410) may be configured to determine the gaze input area to be expanded in size based on an area in which a plurality of prediction points (510) (and / or a gaze prediction point determined by the plurality of prediction points (510)) are located.

[0146] For example, the processor (410) may be configured to use gaze information to identify a sub-region (550) among a plurality of set sub-regions (550) in which a plurality of predicted points (510) on the display (420) are located. The processor (410) may be configured to identify one or more visual objects located within the sub-region and one or more gaze input areas formed on each of the one or more visual objects. The processor (410) may be configured to adjust the size of the one or more gaze input areas based on the identification of the one or more visual objects within the sub-region.

[0147] For example, the processor (410) may identify a plurality of predicted points (510) located within a first partition area (551) among a plurality of partition areas (550) based on determining correction of gaze information. The processor (410) may identify a first set of visual objects (521) located within the first partition area (551) based on the plurality of predicted points (510) located within the first partition area (551). The processor (410) may expand the sizes of the first set of gaze input areas (531) formed on each of the first set of visual objects (521) from a size (s1) to a size (s2) based on identifying the first set of visual objects (521). The processor (410) can induce user input to the first set of visual objects (521) within the first divided area (551) by expanding the sizes of the first set of gaze input areas (531) within the first divided area (551) based on identifying the plurality of gaze prediction points (510) within the first divided area (551).

[0148] For example, referring sequentially to FIGS. 6B and 6C, the processor (410) may receive a user input for moving the positions of the plurality of predicted points (510) based on determining the correction of the gaze information. Based on the user input, the processor (410) may move the positions of the plurality of predicted points (510) from the first partition area (551) to the second partition area (552).

[0149] For example, referring sequentially to FIGS. 6C and 6D , the processor (410) may identify a plurality of predicted points (510) located within a second partition area (552) among the plurality of partition areas (550) while performing correction of gaze information. For example, while changing from a state (603) to a state (604), the processor (410) may identify a second set of visual objects (522), a third set of visual objects (523), and a fourth set of visual objects (524) located within the second partition area (552) based on the plurality of predicted points (510) located within the second partition area (552). The processor (410) can expand the sizes of the second set of gaze input areas (532) formed on the second set of visual objects (522), the third set of gaze input areas (533) formed on the third set of visual objects (523), and the fourth set of gaze input areas (534) formed on the fourth set of visual objects (524) from the first size (s1) to the third size (s3). The processor (410) can induce user input to the visual objects (521, 522, 523) in the second divided area (552) by expanding the sizes of the gaze input areas (531, 532, 533) in the second divided area (552) based on identifying the plurality of gaze prediction points (510) in the second divided area (552). However, the above-mentioned embodiments are illustrative and not limiting.

[0150] For example, the processor (410) may be configured to obtain correction information for correcting the gaze information by using the position information of the plurality of predicted points (510) that have changed while changing from the state (601) to the state (602) and the position information of the plurality of predicted points (510) that have changed while changing from the state (603) to the state (604). For example, the processor (410) may generate correction information by using the position information of the plurality of predicted points (510) that have changed for input to the visual object (521a) while changing from the state (601) to the state (602). The processor (410) may update the correction information by using the position information of the plurality of predicted points (510) that have changed for input to the visual object (523d) while changing from the state (603) to the state (604). However, the present invention is not limited thereto.

[0151] According to one embodiment, the processor (410) may identify one of a plurality of divided regions (550) in which a plurality of prediction points (510) are located. The processor (410) may be configured to identify whether correction of the gaze information has been performed within the identified one divided region based on determining correction of the gaze information. The processor (410) may reduce the size of each of one or more gaze input regions formed on one or more visual objects displayed within the one divided region among the plurality of visual objects (520) based on identifying that correction of the gaze information has not been performed within the one divided region. The processor (410) may obtain correction information using the gaze information based on identifying that correction of the gaze information has been performed within the one divided region.

[0152] For example, referring to FIG. 6A, the processor (410) may identify whether a correction has been made within the first partition area (551) based on identifying a first partition area (551) in which a plurality of prediction points (510) are located. The processor (410) may reduce the sizes of the first set of gaze input areas (531) within the first partition area (551) based on identifying that a correction has not been made within the first partition area (551).

[0153] For example, referring to FIGS. 6B and 6C, the processor (410) may be configured to maintain the sizes of the gaze input regions (530) adjusted based on the correction while the correction is being performed. For example, in FIG. 6C, the sizes of the first set of gaze input regions (531) within the first partitioned area (551) where the correction was performed may be maintained at the second size (s2). Although not shown, when the plurality of predicted points (510) are again located within the first partitioned area (551) from the state (604) of FIG. 6D, the processor (410) may be configured to adjust the sizes of the gaze input regions (531) based on the second size (s2) while the correction is being performed.

[0154] According to one embodiment, the processor (410) may be configured to, based on identifying that correction of gaze information has been performed within one of the plurality of partitioned areas (550), compare the correction information with previously acquired correction information to identify whether the correction information converges to an error value to be used to correct the gaze information. The processor (410) may be configured to correct the gaze information using the error value based on the correction information converging to the error value.

[0155] For example, the processor (410) may receive a plurality of user inputs corresponding to a plurality of attempts to input a visual object (532a) based on positional information of a plurality of predicted points (510) while changing from a state (601) to a state (602). The processor (410) may compare error values ​​obtained based on the plurality of user inputs to identify whether the error values ​​converge. While identifying the error values ​​that do not converge, the processor (410) may execute a function for correcting gaze information. Based on identifying the error values ​​that converge, the processor (410) may determine the error values ​​as correction information for correcting the gaze information. However, the present invention is not limited thereto. For example, if the error value obtained while changing from state (601) to state (602) does not converge, the processor (410) can identify whether the error value obtained while changing from state (603) to state (604), including the error value obtained while changing from state (601) to state (602), converges. Based on identifying the error value that converges, the processor (410) can determine the error value as correction information for correcting the line of sight information.

[0156] Referring to FIG. 6E, unlike the screens (601, 602, 603, 604) illustrated in FIGS. 6A to 6D, a plurality of visual objects (520) of various shapes and gaze input areas (530) corresponding to each of the plurality of visual objects (520) may be displayed through the screen (605) of the display (420). The processor (410) may be configured to set a plurality of segmented areas (550) of various shapes for adjusting the size of each of the gaze input areas (530). For example, as illustrated, the gaze adjustment areas corresponding to each of the plurality of visual objects (520) may have a polygonal shape, but are not limited thereto. For example, the processor (410) may be configured to display a visual object (621) displayed within a first segmented area (551) and a gaze input area (631) formed on the visual object (621). The processor (410) may include a fifth set of visual objects (622) that are displayed to surround the visual object (621) and are displayed within the second segmented area (552), and a fifth set of gaze input areas (632) formed on each of the fifth set of visual objects (622). Each of the visual objects (622a, 622b, 622c, 622d, 622e, 622f) included in the fifth set of visual objects (622) may include gaze input areas (632a, 632b, 632c, 632d, 632e, 632f) formed on each of the visual objects (622a, 622b, 622c, 622d, 622e, 622f). However, the above-mentioned embodiments are exemplary and are not limited thereto.

[0157] According to the above-described embodiment, the processor (410) of the wearable device (200) can perform an operation to correct gaze information related to the user's gaze by adjusting the sizes of gaze input areas (530) formed on a plurality of segmented areas (550) and a plurality of visual objects (520). By performing the operation to correct the gaze information, the processor (410) can provide an improved user experience through gaze tracking.

[0158] Figure 7 is a flow chart showing the operation of an exemplary wearable device for correcting gaze information.

[0159] The operations of FIG. 7 may be executed by the electronic device (101) and / or the processor (120) of FIG. 1, the wearable device (200) of FIG. 2A, the wearable device (200) of FIG. 3A, the wearable device (200) of FIG. 4 and / or the processor (410).

[0160] Referring to FIG. 7, in operation (701), the processor (410) may be configured to display a plurality of visual objects (e.g., a plurality of visual objects (520) of FIG. 5) on a display (e.g., a display (420) of FIG. 4). For example, the processor (410) may control the display (420) to display a plurality of visual objects (520) including gaze input areas (e.g., gaze input areas (530) of FIG. 5) for providing visual feedback for a user's gaze on the display (420).

[0161] In operation (703), the processor (410) may identify whether correction of the gaze information is necessary. For example, the processor (410) may determine correction of the gaze information based on detecting a degradation in the gaze tracking function of the wearable device (200). For example, the processor (410) may determine correction of the gaze information based on a time period during which user input for visual feedback by a plurality of visual objects (520) was performed and / or a failure rate of the visual feedback for the user input. For example, the processor (410) may determine correction of the gaze information based on an angle (a) between a reference axis (e.g., the reference axis (x1) of FIG. 3B) and an orientation axis (e.g., the orientation axis (x2) of FIG. 3B) of the wearable device (200), which is detected through a sensor (e.g., the sensor (430) of FIG. 4). The processor (410) may perform operation (701) if it does not detect a deterioration in the gaze tracking function of the wearable device (200) (e.g., 703-No). For example, the processor (410) may be configured to obtain voice information from the user through the sensor (430). The processor (410) may be configured to obtain or analyze natural language (e.g., a natural language sentence such as “Why can’t I make a good choice?”) spoken by the user using the voice information obtained through the sensor (430). The processor (410) may be configured to determine correction of gaze information based on the voice information obtained from the user. For example, the wearable device (200) may be linked to a controller that is operable by a user wearing the wearable device (200).The controller may be connected to the wearable device (200) via, for example, a communication circuit of the wearable device (200) (e.g., a communication circuit (435) of FIG. 4) and may be used for user input to a plurality of visual objects (520) displayed on the display (420) of the wearable device (200). For example, the controller may be provided in a hand held form so that it can be operated by the user's hand, unlike the wearable device (200) that is worn on the user's head. The controller may provide a tracking function for user input to a plurality of visual objects (520) displayed on the display (420) of the wearable device (200) based on the motion of the user's hand operating the controller, separately from the eye tracking function of the wearable device (200). For example, the controller may be connected to the wearable device (200) to cause the processor (410) of the wearable device (200) to display a pointer on the display (420) for user input to a plurality of visual objects (520) displayed on the display (420) of the wearable device (200). The processor (410) may be configured to determine correction of the gaze information by comparing information obtained from the controller and / or the pointer provided by the controller with gaze information obtained through the wearable device (200) (e.g., through a camera (425) or a sensor (430) of the wearable device (200).

[0162] In operation (705), the processor (410) may reduce the sizes of each of the gaze input areas formed on the plurality of visual objects (520) based on determining the correction of gaze information. For example, the processor (410) may minimize padding values ​​of the plurality of visual objects (520) within the plurality of partitioned areas (550) of FIG. 5) on the display (420) based on identifying that no correction has been made within the plurality of partitioned areas (550).

[0163] In operation (707), the processor (410) may generate correction information to be used to correct the gaze information by using the gaze information detected while providing visual feedback using the sizes of each of the reduced gaze input areas (530). For example, the processor (410) may detect the gaze information through a plurality of predicted points (e.g., the plurality of predicted points (510) of FIG. 5) whose positions have been changed through user input while the correction is being performed. The processor (410) may generate correction information for correcting the gaze information by using the position information of the changed plurality of predicted points (510).

[0164] In operation (707), the processor (711) can identify whether the error value corresponding to the acquired correction information exceeds a specified error value. If the processor (711) does not identify the error value exceeding the specified error value (e.g., 709 - No), the processor (711) can perform operation (713).

[0165] In operation (709), the processor (410) may be configured to execute a function for correcting the gaze information through a session for storing the gaze information based on identifying an error value corresponding to the correction information that exceeds a specified error value. The processor (410) may be configured to correct the error value that exceeds the specified error value through the session, thereby increasing the accuracy of correction for relatively large errors.

[0166] In operation (713), the processor (410) may expand the size of the gaze input areas (530) using the correction information. For example, the processor (410) may adjust the padding value of each of the plurality of visual objects (520) using the acquired correction information. For example, the processor (410) may be configured to adjust the sizes of the gaze input areas (530) from an error value corresponding to the correction information based on a set function (e.g., a function illustrated in the graph (1100) of FIG. 11).

[0167] According to the above-described embodiment, the processor (410) of the wearable device (200) can perform an operation to correct gaze information related to the user's gaze by adjusting the sizes of gaze input areas (530) formed on a plurality of visual objects (520). By performing the operation to correct the gaze information, the processor (410) can provide an improved user experience through gaze tracking.

[0168] Figure 8 is a flowchart showing the operation of an exemplary wearable device for correcting gaze information.

[0169] The operations of FIG. 8 may be executed by the electronic device (101) and / or the processor (120) of FIG. 1, the wearable device (200) of FIG. 2A, the wearable device (200) of FIG. 3A, the wearable device (200) of FIG. 4, and / or the processor (410). The operation (805) of FIG. 8 may correspond to the operation (703) of FIG. 7.

[0170] In operation (801), the processor (410) may be configured to detect that the wearable device (200) is worn on a part of the user's body (e.g., head). The processor (410) may be configured to obtain information related to the location of the wearable device (200) using a sensor (e.g., sensor (430) of FIG. 4) while the wearable device (200) is worn by the user, for example.

[0171] In operation (803), the processor (410) may set a plurality of segmented areas (e.g., a plurality of segmented areas (550) of FIG. 5) on a display (e.g., a display (420) of FIG. 4) on which a plurality of visual objects (e.g., a plurality of visual objects (520) of FIG. 5) are displayed. For example, the processor (410) may set the plurality of segmented areas (550) based on a FoV of a user wearing the wearable device (200), but is not limited thereto.

[0172] In operation (805), the processor (410) can identify whether correction of gaze information is required. If the processor (410) does not detect a deterioration in the gaze tracking function of the wearable device (200) for correction of the gaze information (e.g., 803 - No), the processor (410) can perform operation (801).

[0173] In operation (807), the processor (410) may identify a segmented area in which prediction points (e.g., the plurality of prediction points (510) of FIG. 5) for predicting a direction of gaze are located on the display (420) among the plurality of segmented areas (550). For example, the processor (410) may identify a segmented area in which the plurality of prediction points (510) are located based on position information of the plurality of prediction points (510) on the display (420).

[0174] In operation (809), the processor (410) may identify whether correction of gaze information has been performed within the identified single-partition area. If the processor (410) does not identify that correction of the gaze information has been performed within the single-partition area (e.g., 809-No), the processor (410) may perform operation (811).

[0175] In operation (811), the processor (410) may obtain correction information using the gaze information based on identifying that correction of the gaze information has been performed within the identified single-partitioned area. For example, the processor (410) may identify that a function for correcting the gaze information is being executed based on identifying that correction of the gaze information has been performed within the identified single-partitioned area. Based on identifying that the function has been executed, the processor (410) may obtain correction information using positional information of a plurality of predicted points (510) within the identified single-partitioned area.

[0176] In operation (813), the processor (410) may reduce the size of each of one or more gaze input areas formed on each of one or more visual objects displayed within the identified partitioned area. For example, referring also to FIG. 6A, the processor (410) may reduce the size of a first set of gaze input areas (531) formed on each of a first set of visual objects (521) within the first partitioned area (551), based on the first partitioned area (551) on which no correction has been performed.

[0177] In operation (815), the processor (410) can identify whether the acquired correction information converges to an error value to be used for correcting the gaze information. If the processor (410) fails to identify the correction information that converges to the error value (e.g., 815-No), the processor (410) can update the correction information for correcting the gaze information by performing operation (809).

[0178] In operation (817), the processor (410) may correct the gaze information using the error value based on identifying correction information that converges to the error value to be used for utilizing the gaze information. For example, the processor (410) may apply the converged error value to the gaze information based on identifying correction information having the converged error value. By correcting the gaze information using the error value, the wearable device (200) may provide an enhanced user experience through gaze tracking.

[0179] In operation (819), the processor (410) may be configured to adjust the sizes of the gaze input areas (530) to provide visual feedback through the corrected gaze information using the converged error value. For example, the processor (410) may be configured to determine the error value as correction information for correcting the gaze information based on the identification of the converged error value. The processor (410) may be configured to adjust the sizes of the gaze input areas (530) using the determined correction information through one or more functions (e.g., the function of FIG. 11) set in the wearable device (200). The processor (410) may be configured to perform operation (805) while the visual feedback is provided after adjusting the sizes of the gaze input areas (530).

[0180] Figures 9a, 9b, and 9c illustrate the operation of an exemplary wearable device for providing visual feedback.

[0181] Referring to FIGS. 9A, 9B, and 9C, states (901, 902, 903) of a visual object (920) displayed through a display (e.g., display (420) of FIG. 4) are illustrated. The operations described through FIGS. 9A, 9B, and 9C may be executed by the electronic device (101) and / or processor (120) of FIG. 1, the wearable device (200) of FIG. 2A, the wearable device (200) of FIG. 3A, the wearable device (200) of FIG. 4, and / or the processor (410).

[0182] Referring to FIG. 9A, in a state (901), the processor (410) may include a visual object (920) including a gaze input area (930) for providing visual feedback. In a state (901), the processor (410) may determine the positions of a plurality of prediction points (510) for predicting the direction of the gaze based on gaze information related to the user's gaze. For example, the processor (410) may be configured to change the color and / or size of visual objects that are close to the prediction points (510) for predicting the user's gaze among the plurality of visual objects (520). For example, the processor (410) may be configured to detect the user's movement and / or motion through the sensor (430) while providing visual feedback through the display (420). The processor (410) may be configured to determine one of the visual objects near the predicted points (510) based on detecting the motion of the user and using a direction and / or location corresponding to the motion.

[0183] Referring to FIGS. 9A and 9B, the processor (410) may determine correction of gaze information while changing from state (901) to state (902). Based on determining correction of the gaze information, the processor (410) may reduce the size of the gaze input area (930).

[0184] According to one embodiment, the processor (410) may be configured to display one or more guide objects (910) corresponding to the user's gaze, predicted based on a camera (e.g., camera (425) of FIG. 4) on the display (420), based on the size of the reduced gaze input area (930). For example, the processor (410) may be configured to display a first guide object (911) at the locations of a plurality of predicted points (510) for predicting the direction of the user's gaze, in response to the reduced gaze input area (930). The processor (410) may guide and / or induce a change in the user's gaze direction for input via the visual object (920) by displaying the first guide object (911) on the display (420).

[0185] For example, the processor (410) may be configured to display a second visual object (912) on the display (420) in response to a reduced gaze input area (930) to guide a movement direction for input to a visual object (920) around the first guide object (911). The processor (410) may guide and / or induce a change in the user's gaze direction for input through the visual object (920) by displaying the second guide object (912) on the display (420).

[0186] According to one embodiment, the processor (410) may be configured to determine the position of one or more guide objects (910) using the positions of a plurality of predicted points on the display (420). For example, the position of a first guide object (911) may be determined as an average value of the positions of a plurality of predicted points (510) on the display (420). For example, the position of a second guide object (912) may be determined based on the position of the first guide object (911).

[0187] According to one embodiment, the processor (410) may be configured to display a graphic area (950) that guides the location of the predicted points (510) on the visual object (920) based on the predicted points (510) as visual feedback. For example, the processor (410) may be configured to display at least a portion of the graphic area (950) configured to provide a gradation effect based on the predicted points (510) for predicting the user's gaze, on the visual object (920). The graphic area (950) may guide the movement of the user's gaze to the visual object (920) by being displayed on the visual object (920). The graphic area (950) may be referenced as one or more guide objects (910) in that it guides the movement of the predicted points (510) for predicting the user's gaze to the visual object (920).

[0188] Referring to FIGS. 9B and 9C, the processor (410) may generate correction information for correcting gaze information by using a change in the position information of the first guide object (911) for predicting the user's gaze while changing from the state (902) to the state (903). The processor (410) may expand the gaze input area (930) by using the correction information. For example, the position of the first guide object (911) for predicting the gaze may be changed by a user input guided by one or more guide objects (910). Based on the changed position information of the first guide object (911), the processor (410) may generate correction information for correcting the gaze information. By expanding the size of the gaze input area (930) by using the correction information, the first guide object (911) may be positioned within the gaze input area (930). The processor (410) may receive a user input to a visual object (920) based on the first guide object (911) located within the gaze input area (930). The processor (410) may be configured to display a screen related to the visual object (920) through the display (420) based on receiving the user input to the visual object (920).

[0189] For example, the predicted points (510) may be located on a visual object (920). A graphic area (950) for guiding the location of the predicted points (510) may be displayed entirely within the visual object (920). The graphic area (950) may be configured to guide the location of the predicted points (510) for predicting the user's gaze within the visual object (920).

[0190] According to the above-described embodiment, the processor (410) of the wearable device (200) can provide an improved user experience through gaze tracking by providing one or more visual objects (920) for guiding the position of the user's gaze while correction of gaze information is performed.

[0191] Figure 10 is a graph showing the positions of multiple predicted points of a user's gaze on the display of an exemplary wearable device over time.

[0192] Referring to FIG. 10, the horizontal axis of the graph (1000) represents time. The vertical axis of the graph (1000) represents the positions of objects displayed on a display (e.g., display (420) of FIG. 4).

[0193] Referring to the graph (1000), the positions of the plurality of prediction points (510) for predicting the user's gaze on the display (420) may change according to the movement of the user's gaze and / or the movement of the camera (e.g., the camera (425) of FIG. 4). For example, the positions of the plurality of prediction points (510) may correspond to an average value of the positions of each of the plurality of prediction points (510). For example, the positions of the plurality of prediction points (510) may change from a first position (l1) to a second position (l2) during a change from timing (t1) to timing (t2). The processor (410) may use the first position (l1) and the second position (l2) to generate an error value (e) corresponding to correction information for the time period between timing (t1) and timing (t2). The processor (410) may be configured to adjust the size of a gaze input area (1010) of a visual object (e.g., a visual object (920) of FIG. 9A) displayed on a display (420) using the error value (e). An explanation of the adjustment of the gaze input area (1010) using the error value (e) will be described later in FIG. 11.

[0194] Figure 11 is a graph showing the relationship between error values ​​corresponding to correction information of an exemplary wearable device and padding values ​​of gaze input areas.

[0195] Referring to FIG. 11, the horizontal axis of the graph (1100) represents an error value (e) corresponding to correction information for correcting gaze information. The vertical axis of the graph (1100) represents a padding value of a visual object (e.g., a visual object (920) of FIG. 9a) displayed on a display (e.g., a display (420) of FIG. 4).

[0196] Referring to the graph (1100), the processor (410) of the wearable device (200) may be configured to adjust the padding value of the visual object using the acquired error value based on the set function. For example, the processor (410) may set the padding value of the visual object to the second padding value (p2) using the function shown in the graph (1100) through the first error value (e1) corresponding to the acquired correction information. For example, the processor (410) may set the padding value of the visual object to the first padding value (p1) smaller than the second padding value (p2) using the function shown in the graph (1100) through the second error value (e2) larger than the first error value (e1). The processor (410) may adjust the size of the gaze input area (e.g., the gaze input area (930) of FIG. 9A) of the visual object by adjusting the padding value of the visual object using the function.

[0197] According to the above-described embodiment, the processor (410) of the wearable device (200) can perform an operation to correct gaze information related to the user's gaze by adjusting the sizes of the gaze input area of ​​the visual object. By performing the operation to correct the gaze information, the processor (410) can provide an enhanced user experience through gaze tracking.

[0198] FIG. 12 illustrates a portion of a screen displayed through a display of an exemplary wearable device.

[0199] Referring to FIG. 12, an exemplary screen (1201) provided through a display (e.g., a display (420) of FIG. 4) of a wearable device (e.g., a wearable device (200) of FIG. 2A) is illustrated. Referring to FIG. 12, a processor (e.g., a processor (410) of FIG. 4) may be configured to display a plurality of visual objects (1220) and gaze input areas (e.g., gaze input areas (530) of FIG. 5) formed on each of the plurality of visual objects (1220). For example, the processor (410) may be configured to set a partition area (1251) for selecting visual objects (1221a, 1221b, 1221c, 1221d), a partition area (1252) for selecting visual objects (1222) for video playback, and a partition area (1253) for visual objects (1223a, 1223b) for playback of a video other than the video. The processor (410) may be configured to adjust the size of one or more gaze input areas on one or more visual objects displayed within one of the partition areas (1251, 1252, 1253) based on determining correction of gaze information related to the user's gaze. However, the above-mentioned embodiments are exemplary, and the processor (410) may be configured to perform corrections to provide visual feedback from visual objects displayed within a screen displayed on a display (420) having various interfaces, or to adjust the size of gaze input areas formed on the visual objects.

[0200] According to the above-described embodiment, a wearable device (e.g., a wearable device (200) of FIG. 2A) may include a display system (e.g., a display module (160) of FIG. 1) including a first display (e.g., a first display (250-1) of FIG. 3A, a display (420) of FIG. 4) and a second display (e.g., a second display (250-2) of FIG. 3A, a display (420) of FIG. 4), and a camera system (e.g., a camera module (180) of FIG. 1, a camera (425) of FIG. 4) including a plurality of cameras (e.g., cameras (260) of FIG. 2B). The camera system may be configured to capture images of a user's eyes while the user wears the wearable device. The wearable device may include at least one processor including a processing circuit (e.g., processor (120) of FIG. 1, processor (410) of FIG. 4), and a memory including one or more storage media storing instructions (e.g., memory (130) of FIG. 1, memory (415) of FIG. 4). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a visual object (e.g., visual object (521a) of FIG. 5) through the display system. The visual object may be associated with a gaze input area (e.g., gaze input area (531a) of FIG. 5) having a first size. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to provide visual feedback for the visual object when identified gaze information falls within the gaze input area.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to, when attempting to perform correction related to the gaze information while the visual object is displayed, reduce the size of the gaze input area related to the visual object from the first size to the second size, generate correction information based on the gaze information of the user gazing at the visual object while the size of the gaze input area is reduced, and increase the size of the gaze input area related to the visual object from the second size after the correction information is generated.

[0201] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether an error value corresponding to the correction information exceeds a specified error value. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to execute a function for correcting the gaze information through a session for storing the gaze information based on identifying the error value as exceeding the specified error value.

[0202] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a plurality of visual objects including the visual object (e.g., the plurality of visual objects (520) of FIG. 5) on the first display and the second display through the display system. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to set a plurality of partition areas (e.g., the plurality of partition areas (550) of FIG. 5) on the first display and the second display to display the plurality of visual objects. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify one sub-region (e.g., the first sub-region (551) of FIG. 5) in which a plurality of prediction points (e.g., the plurality of prediction points (510) of FIG. 5) on the first display and the second display for predicting a direction of the user's gaze among the plurality of sub-regions are located using the gaze information. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to, when attempting to perform a correction related to the gaze information, identify whether a correction of the gaze information has been performed within the identified one sub-region, and based on identifying that a correction of the gaze information has not been performed within the one sub-region, reduce the size of the gaze input area associated with the visual object displayed within the one sub-region among the plurality of visual objects from the first size to the second size.

[0203] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain correction information using the gaze information based on identifying that correction of the gaze information has been performed within the single partition. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to compare the correction information with previously acquired correction information to identify whether the correction information converges to an error value to be used to correct the gaze information. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to correct the gaze information using the error value based on the correction information converging to the error value.

[0204] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to increase the size of the gaze input area having the second size by the acquired correction information to a third size by using a designated function configured to determine the size of the input area according to the error value using the converged error value.

[0205] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive a first user input for the visual feedback into the gaze input area. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to change positions of a plurality of prediction points on the first display and the second display for predicting a direction of the user's gaze based on the first user input. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to measure a time interval during which the first user input was received based on identifying at least some of the plurality of prediction points located within the gaze input area. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a correction of the gaze information associated with the gaze based on the time interval exceeding a set reference time interval.

[0206] For example, the wearable device may further include at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to detect an angle (e.g., angle (a) in FIG. 3B) between an axis representing an orientation of the wearable device (e.g., orientation axis (x2) in FIG. 3B) and a reference axis (e.g., reference axis (x1) in FIG. 3B) via the at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a correction of the gaze information related to the gaze based on identifying that the angle exceeds a set reference angle.

[0207] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive a second user input for the visual feedback toward the gaze input area while performing the correction related to the gaze information, change positions of a plurality of prediction points on the first display and the second display for predicting a direction of the user's gaze based on the second user input, increase the size of the gaze input area using the correction information obtained through the changed positions of the plurality of prediction points, and provide the visual feedback based on identifying at least some of the plurality of prediction points located within the gaze input area having the increased size.

[0208] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display one or more guide objects (e.g., one or more guide objects (910) of FIG. 9B) corresponding to the user's gaze predicted based on the plurality of cameras on the first display and the second display, based on the size of the gaze input area reduced from the first size to the second size.

[0209] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a position of the one or more guide objects using positions of a plurality of predicted points on the first display and the second display for predicting a direction of the user's gaze.

[0210] According to one embodiment, a wearable device (e.g., electronic device (101) of FIG. 1, wearable device (200) of FIG. 2A) may include a display (e.g., display (420) of FIG. 4), one or more cameras configured to track eyes of a user wearing the wearable device (e.g., camera (425) of FIG. 4), at least one processor including processing circuitry (e.g., processor (410) of FIG. 4), and a memory including one or more storage media storing instructions (e.g., memory (415) of FIG. 4). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a plurality of visual objects (e.g., a plurality of visual objects (520) of FIG. 5) on the display. The plurality of visual objects may include gaze input areas (e.g., gaze input areas (530)) formed on each of the plurality of visual objects to provide visual feedback for the user's gaze directed toward the plurality of visual objects. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether to correct gaze information related to the gaze, determined based on the one or more cameras, while displaying the plurality of visual objects. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to reduce the sizes of each of the gaze input areas formed on each of the visual objects based on determining the correction of the gaze information.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate correction information to be used to correct the gaze information using the gaze information detected while providing the visual feedback using the reduced sizes. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to expand the sizes of the gaze input areas using the correction information.

[0211] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether an error value corresponding to the correction information exceeds a specified error value. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to execute a function for correcting the gaze information through a session for storing the gaze information based on identifying the error value as exceeding the specified error value.

[0212] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to set a plurality of partitions (e.g., the plurality of partitions (550) of FIG. 5) on the display that display the plurality of visual objects. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, using the gaze information, one partition of the plurality of partitions in which a plurality of prediction points on the display for predicting the direction of the gaze are located. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, based on determining a correction of the gaze information, whether a correction of the gaze information has been performed within the identified one partition. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to reduce the size of each of one or more gaze input areas formed on one or more visual objects displayed within the one-part area among the plurality of visual objects, based on identifying that no correction of the gaze information has been performed within the one-part area.

[0213] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain correction information using the gaze information based on identifying that correction of the gaze information has been performed within the single partition. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to compare the correction information with previously acquired correction information to identify whether the correction information converges to an error value to be used to correct the gaze information. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to correct the gaze information using the error value based on the correction information converging to the error value.

[0214] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to change the size of each of the one or more gaze input areas having a first size by the acquired correction information to a second size (e.g., the second size (s2) of FIG. 6b) different from the first size (e.g., the first size (s1) of FIG. 6a) using the converged error value.

[0215] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive a first user input for the visual feedback from the gaze input regions. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to change the positions of a plurality of prediction points on the display for predicting the direction of the gaze based on the first user input. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to measure a time interval during which the first user input was received based on identifying at least some of the plurality of prediction points located within one of the gaze input regions. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a correction of the gaze information associated with the gaze based on the time interval exceeding a set reference time interval.

[0216] For example, the electronic device may further include at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to detect an angle (e.g., angle (a) in FIG. 3B) between an axis representing an orientation of the wearable device (e.g., orientation axis (x2) in FIG. 3B) and a reference axis (e.g., reference axis (x1) in FIG. 3B) via the at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a correction of the gaze information related to the gaze based on identifying that the angle exceeds a set reference angle.

[0217] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive a second user input for the visual feedback from the gaze input areas based on determining a correction of the gaze information. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to change positions of a plurality of prediction points on the display for predicting a direction of the gaze based on the second user input. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to expand the sizes of the gaze input areas using the correction information obtained through the changed positions of the plurality of prediction points. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to provide the visual feedback based on identifying at least some of the plurality of predicted points located within one of the gaze input areas having the extended sizes.

[0218] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, on the display, one or more guide objects (e.g., one or more guide objects (910) of FIG. 9B) corresponding to the gaze predicted based on the one or more cameras, based on the sizes of each of the reduced gaze input regions.

[0219] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine the positions of the one or more guide objects using the positions of a plurality of predicted points on the display for predicting the direction of the gaze.

[0220] In one embodiment, a method of a wearable device may include an operation of displaying a plurality of visual objects on a display of the wearable device. The plurality of visual objects may include gaze input areas formed on each of the plurality of visual objects to provide visual feedback for a gaze of the user directed toward the plurality of visual objects. The method may include an operation of identifying whether to correct gaze information related to the gaze, determined based on one or more cameras of the wearable device configured to track eyes of a user wearing the wearable device while the plurality of visual objects are displayed. The method may include an operation of reducing the sizes of each of the gaze input areas formed on each of the visual objects based on determining the correction of the gaze information. The method may include an operation of generating correction information to be used to correct the gaze information using the gaze information detected while providing the visual feedback using the reduced sizes. The method may include an operation of expanding the sizes of the gaze input areas using the correction information.

[0221] For example, the method may further include an operation of identifying whether an error value corresponding to the correction information exceeds a specified error value. The method may further include an operation of executing a function for correcting the gaze information through a session for storing the gaze information based on identifying the error value exceeding the specified error value. For example, the method may further include an operation of executing a function for correcting the gaze information through a session for storing the gaze information based on identifying that an error value less than the specified error value is detected for a specified number of times and / or for a specified period of time or more.

[0222] For example, the method may include an operation of identifying whether corrected gaze information is acquired a specified number of times or more based on the correction information being detected a specified number of times or more. The method may include an operation of determining, based on the corrected gaze information acquired a specified number of times or more, a value corresponding to the corrected gaze information as a value corresponding to the corrected gaze information to be provided through a session for storing the gaze information. In this case, the wearable device (or the processor of the wearable device) may acquire a value corresponding to the corrected gaze information to be provided through the session without using the session.

[0223] For example, the method may further include an operation of setting a plurality of segmented areas on the display that displays the plurality of visual objects. The method may further include an operation of identifying, using the gaze information, a segmented area in which a plurality of prediction points on the display for predicting the direction of the gaze are located among the plurality of segmented areas. The method may further include an operation of identifying, based on determining a correction of the gaze information, whether correction of the gaze information has been performed within the identified segmented area. The method may further include an operation of reducing the size of each of one or more gaze input areas formed on one or more visual objects displayed within the segmented area among the plurality of visual objects, based on identifying that correction of the gaze information has not been performed within the segmented area.

[0224] For example, the method may further include an operation of obtaining correction information using the gaze information based on identifying that correction of the gaze information has been performed within the single-partitioned region. The method may further include an operation of comparing the correction information with previously obtained correction information to identify whether the correction information converges to an error value to be used to correct the gaze information. The method may further include an operation of correcting the gaze information using the error value based on the correction information converging to the error value.

[0225] For example, the method may further include an operation of changing the size of each of the one or more gaze input areas having a first size by the acquired correction information to a second size different from the first size by using the converged error value.

[0226] For example, the method may further include an operation of receiving a first user input for the visual feedback from the gaze input areas. The method may further include an operation of changing positions of a plurality of prediction points on the display for predicting a direction of the gaze based on the first user input. The method may further include an operation of measuring a time interval at which the first user input is received based on identifying at least some of the plurality of prediction points located within one of the gaze input areas. The method may further include an operation of determining a correction of the gaze information related to the gaze based on the time interval exceeding a set reference time interval.

[0227] For example, the method may further include an operation of detecting an angle between an axis representing the orientation of the wearable device and a reference axis through at least one sensor of the wearable device. The method may further include an operation of determining a correction of the gaze information related to the gaze based on identifying that the angle exceeds a set reference angle.

[0228] For example, the method may further include an operation of receiving a second user input for the visual feedback from the gaze input areas based on determining a correction of the gaze information. The method may further include an operation of changing positions of a plurality of prediction points on the display for predicting a direction of the gaze based on the second user input. The method may further include an operation of expanding the sizes of the gaze input areas using the correction information obtained through the changed positions of the plurality of prediction points. The method may further include an operation of providing the visual feedback based on identifying at least some of the plurality of prediction points located within one of the gaze input areas having the expanded sizes.

[0229] For example, the method may further include an operation of displaying one or more guide objects corresponding to the gaze predicted based on the one or more cameras on the display based on the sizes of each of the reduced gaze input areas.

[0230] In one embodiment, a non-transitory computer readable storage medium storing one or more programs may include instructions that, when executed by at least one processor of a wearable device having a display and one or more cameras configured to track eyes of a user wearing the wearable device, cause the display to display a plurality of visual objects on the display. The plurality of visual objects may each include gaze input areas formed on the plurality of visual objects to provide visual feedback for a gaze of the user directed toward the plurality of visual objects. The one or more programs may include instructions that, when executed by the at least one processor, cause the display of the plurality of visual objects to identify whether to correct gaze information related to the gaze, the gaze information being determined based on the one or more cameras. The one or more programs may include instructions that, when executed by the at least one processor, cause the sizes of each of the gaze input areas formed on the visual objects to be reduced based on determining the correction of the gaze information. The one or more programs may include instructions that, when executed by the at least one processor, cause the at least one processor to generate correction information to be used for correcting the gaze information using the gaze information detected while providing the visual feedback using the reduced sizes. The one or more programs may include instructions that, when executed by the at least one processor, cause the sizes of the gaze input areas to be expanded using the correction information.

[0231] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0232] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0233] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0234] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0235] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0236] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In wearable devices, A display system comprising a first display and a second display; A camera system comprising a plurality of cameras, the camera system configured to capture images of the eyes of a user while the user is wearing the wearable device; At least one processor comprising a processing circuit; and A memory comprising one or more storage media storing instructions, said instructions, when individually or collectively executed by said at least one processor, causing said wearable device to: Displaying a visual object through the display system, wherein the visual object is associated with a gaze input area having a first size; If the identified gaze information falls within the gaze input area, visual feedback for the visual object is provided; If you want to perform a correction related to the above gaze information while the above visual object is displayed: Reducing the size of the gaze input area related to the visual object from the first size to the second size; While the size of the above gaze input area is reduced, correction information is generated based on the gaze information of the user gazing at the visual object; and causing the size of the gaze input area related to the visual object to be increased from the second size after the correction information is generated; Wearable devices.

2. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Identifying whether the error value corresponding to the above correction information exceeds a specified error value; and Based on identifying the error value exceeding the specified error value, causing a function for correction of the gaze information to be executed through a session for storing the gaze information. Wearable devices.

3. In paragraph 1 or 2, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Displaying a plurality of visual objects including the visual object on the first display and the second display through the display system; Setting a plurality of split areas on the first display and the second display that display the plurality of visual objects; Using the above gaze information, one of the plurality of divided regions is identified, in which a plurality of prediction points on the first display and the second display are located for predicting the direction of the user's gaze; If you want to perform corrections related to the above gaze information: Identifying whether correction of the gaze information has been performed within the identified said one-part area; and Based on identifying that correction of the gaze information has not been performed within the above-described single-partition area, causing the size of the gaze input area related to the visual object displayed within the above-described single-partition area among the plurality of visual objects to be reduced from the first size to the second size. Wearable devices.

4. In paragraph 3, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on identifying that correction of the gaze information has been performed within the above-described single-division area, the correction information is obtained using the gaze information; Comparing the above correction information with the previously acquired correction information to identify whether the correction information converges to an error value to be used to correct the gaze information; and Based on the correction information that converges to the error value, causing the line of sight information to be corrected using the error value, Wearable devices.

5. In paragraph 4, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing the size of the gaze input area having the second size to be increased to a third size by using the converged error value through a designated function configured to determine the size of the input area according to the error value by using the correction information obtained above. Wearable devices.

6. In any one of paragraphs 1 to 5, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Receiving a first user input for the visual feedback to the above gaze input area; Based on the first user input, changing the positions of a plurality of predicted points on the first display and the second display for predicting the direction of the user's gaze; Measuring a time interval at which the first user input is received based on identifying at least some of the plurality of predicted points located within the gaze input area; and Causing to determine correction of said gaze information related to said gaze based on said time interval exceeding a set reference time interval, Wearable devices.

7. In any one of paragraphs 1 to 6, comprising at least one more sensor, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Detecting an angle between an axis representing the orientation of the wearable device and a reference axis through at least one sensor; and Based on identifying that the above angle exceeds a set reference angle, causing a correction of the above gaze information related to the above gaze to be determined, Wearable devices.

8. In any one of paragraphs 1 to 7, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: While performing the above corrections related to the above gaze information: Receive a second user input for the visual feedback to the above gaze input area; Based on the second user input, changing the positions of a plurality of predicted points on the first display and the second display for predicting the direction of the user's gaze; increasing the size of the gaze input area by using the correction information obtained through the changed positions of the plurality of prediction points; and Providing the visual feedback based on identifying at least some of the plurality of predicted points located within the gaze input area having the increased size, Wearable devices.

9. In any one of paragraphs 1 to 8, The instructions, when individually or collectively executed by the at least one processor, cause the wearable device to display one or more guide objects corresponding to the user's gaze predicted based on the plurality of cameras on the first display and the second display based on the size of the gaze input area reduced from the first size to the second size. Wearable devices.

10. In paragraph 9, The instructions, when individually or collectively executed by the at least one processor, cause the wearable device to determine a position of the one or more guide objects using positions of a plurality of predicted points on the first display and the second display for predicting a direction of the user's gaze. Wearable devices.

11. In the method of a wearable device, An action of displaying a plurality of visual objects on a display of the wearable device, the plurality of visual objects including gaze input areas formed on each of the plurality of visual objects to provide visual feedback for the user's gaze directed toward the plurality of visual objects; An action of determining whether to correct gaze information related to the gaze, determined based on one or more cameras of the wearable device configured to track the eyes of a user wearing the wearable device while displaying the plurality of visual objects; An operation of reducing the sizes of each of the gaze input areas formed on each of the visual objects based on determining the correction of the above gaze information; An operation of generating correction information to be used to correct the gaze information by using the gaze information detected while providing the visual feedback using the reduced sizes; An operation of applying the generated correction information to the gaze information to obtain the corrected gaze information; and In order to provide visual feedback for the gaze through the gaze information corrected using the above correction information, including an operation of expanding the sizes of the gaze input areas, method.

12. In paragraph 11, An operation for identifying whether an error value corresponding to the above correction information exceeds a specified error value; and Further comprising an action of executing a function for correcting the gaze information through a session for storing the gaze information based on identifying the error value exceeding the specified error value. method.

13. In paragraph 11 or 12, An action of setting a plurality of partition areas on the display that displays the plurality of visual objects; An operation of identifying a single partition area in which a plurality of prediction points on the display are located for predicting the direction of the gaze among the plurality of partition areas using the above gaze information; An operation of identifying whether correction of the gaze information has been performed within the identified single-partition area based on determining correction of the gaze information; and An operation of reducing the size of each of one or more gaze input areas formed on one or more visual objects displayed within the one-partition area among the plurality of visual objects based on identifying that correction of the gaze information has not been performed within the one-partition area. method.

14. In paragraph 13, An operation of obtaining the correction information using the gaze information based on identifying that correction of the gaze information has been performed within the above-described single-division area; An operation of comparing the above correction information with previously acquired correction information to identify whether the correction information converges to an error value to be used to correct the gaze information; and Further comprising an operation of correcting the line of sight information using the error value based on the correction information converging to the error value. method.

15. A non-transitory computer readable storage medium storing one or more programs, wherein the one or more programs are executed by at least one processor of a wearable device having a display and one or more cameras configured to track the eyes of a user wearing the wearable device. Displaying a plurality of visual objects on the display, wherein the plurality of visual objects include gaze input areas formed on each of the plurality of visual objects to provide visual feedback for the user's gaze directed toward the plurality of visual objects; In a state where the plurality of visual objects are displayed, identifying whether to correct the gaze information related to the gaze determined based on one or more cameras; Based on determining the correction of the above gaze information, the sizes of each of the gaze input areas formed on each of the visual objects are reduced; Using the gaze information detected while providing the visual feedback using the reduced sizes, correction information to be used to correct the gaze information is generated; Applying the generated correction information to the gaze information to obtain the corrected gaze information; and In order to provide visual feedback for the gaze through the gaze information corrected using the above correction information, instructions causing the sizes of the gaze input areas to be expanded, Computer readable storage medium.

Images