Wearable device, method, and non-transitory computer-readable recording medium for gaze tracking on basis of personal characteristics

Abstract

A wearable device is disclosed. The wearable device may comprise a display arranged to face the eyes of a user wearing the wearable device. The wearable device may comprise a camera configured to acquire an image of at least a portion of the eyes of the user wearing the wearable device. The wearable device may, on the basis of the image acquired by the camera, identify whether slippage of the wearable device has occurred while tracking the gaze of the user viewing a screen displayed through the display. The wearable device may correct the user's gaze by using a correction value identified on the basis of user characteristics associated with the user's head shape, in response to the slippage being identified.

Description

Wearable device, method, and non-transitory computer-readable recording medium for eye tracking based on personal characteristics

[0001] The following descriptions relate to a wearable device, a method, and a non-transitory computer-readable recording medium for gaze tracking based on personal characteristics.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] The wearable device can track the movement of the user's eye pupils and / or the user's gaze based on images of the user's eyes. Through gaze tracking, the wearable device can determine what the user is looking at or focusing on.

[0004] A wearable device is disclosed. The wearable device may include a display arranged to face the eye of a user wearing the wearable device. The wearable device may include a camera configured to acquire an image of at least a portion of the eye of the user wearing the wearable device. The wearable device may include at least one processor including a processing circuit. The wearable device may include a memory storing instructions and including one or more storage media. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether slippage of the wearable device has occurred while tracking the gaze of the user looking at a screen displayed through the display based on the image acquired by the camera. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to correct the gaze using a correction value identified based on a user characteristic associated with the user's head, based on the identification of the slippage.

[0005] A method is disclosed. The method can be executed by a wearable device including a display arranged to face an eye of a user wearing the wearable device, and a camera configured to acquire an image of at least a portion of the eye of the user wearing the wearable device. The method can include an operation of identifying whether slippage of the wearable device has occurred while tracking the user's gaze on a screen displayed through the display based on the image acquired by the camera. The method can include an operation of correcting the gaze using a correction value identified based on a user characteristic of the user based on the identification that slippage has occurred.

[0006] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium may store one or more programs including instructions. The instructions, when executed individually or collectively by at least one processor of a wearable device, the wearable device including a display arranged to face an eye of a user wearing the wearable device, and a camera configured to acquire an image of at least a portion of the eye of the user wearing the wearable device, may cause the wearable device to identify whether slippage of the wearable device has occurred while tracking the user's gaze on a screen displayed through the display based on the image acquired by the camera. The instructions, when executed individually or collectively by the at least one processor, may cause the wearable device to correct the gaze based on the identification that slippage has occurred using a correction value identified based on a user characteristic of the user.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2A illustrates an example of an appearance of a wearable device according to one embodiment.

[0009] FIG. 2b illustrates an example of an appearance of a wearable device according to one embodiment.

[0010] Figure 3a shows the positional relationship between the displays and the user's two eyes determined through fit adjustment.

[0011] Figure 3b shows an example of an image of the user's left eye.

[0012] Figure 3c shows the anatomical structure of the user's left eye.

[0013] Figure 3d shows an example of slippage of the user's left eye.

[0014] Figure 4a shows a synthetic image generated according to the slippage state.

[0015] Figure 4b shows a synthetic image generated according to the slippage state.

[0016] Figure 5 shows an example of user calibration for obtaining reference images.

[0017] Figure 6 is a block diagram of an electronic device according to one embodiment.

[0018] FIG. 7 is a diagram illustrating an example of an operation of an electronic device to correct a gaze error according to one embodiment.

[0019] FIG. 8 is a diagram illustrating an example of an operation of an electronic device to correct a gaze error according to one embodiment.

[0020] FIG. 9 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0021] FIG. 10 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0022] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0023] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0024] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0025] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0026] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0027] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0028] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0029] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0030] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0031] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0032] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0033] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0034] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0036] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0037] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0038] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0040] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, a loss coverage (e.g., 664 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip) for realizing URLLC.

[0041] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0042] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0043] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0044] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0045] FIG. 2A illustrates an example of the appearance of a wearable device according to one embodiment. FIG. 2B illustrates an example of the appearance of a wearable device according to one embodiment.

[0046] The wearable device (200) of FIGS. 2A and 2B may correspond to the electronic device (101) of FIG. 1.

[0047] Referring to FIG. 2A, according to one embodiment, a first surface (210) of a wearable device (200) may have a form that is attachable to a body part of a user (e.g., the face of the user). According to one embodiment, the wearable device (200) may have a form factor for being worn on a head of a user. In one embodiment, the wearable device (200) may be worn on a body part of a user (e.g., the head). In one embodiment, the wearable device (200) may be referred to as a wearable device in the sense that it is worn on a body part of a user (e.g., the head). Although not shown, the wearable device (200) may further include a strap and / or one or more temples for being fixed on a body part of a user.

[0048] According to one embodiment, the wearable device (200) may include a first display (250-1) and a second display (250-2). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the left and right eyes of the user, respectively. According to an embodiment, the wearable device (200) may further include a rubber or silicone packing formed on the first surface (210) to prevent or reduce interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0049] In one embodiment, the wearable device (200) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (200) through displays (250-1, 250-2). For example, the wearable device (200) may provide a user experience (e.g., video see-through (VST)) in which real objects and reference objects are mixed by combining reference objects within a frame that includes real objects and is displayed through the first display (250-1) and the second display (250-2).

[0050] According to one embodiment, the wearable device (200) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0051] According to one embodiment, a wearable device (200) may include cameras (240-1, 240-2) for capturing and / or recognizing a user's face. The cameras (240-1, 240-2) may be referred to as FT (face tracking) cameras.

[0052] According to one embodiment, a wearable device (200) may include cameras (240-3, 240-4) for photographing and / or tracking both eyes of a user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (240-3, 240-4) may be referred to as ET (eye tracking) cameras.

[0053] Cameras (240-3, 240-4) can output data (or images) representing the gaze of a user wearing the wearable device (200). For example, the wearable device (200) can detect the gaze of the user from images containing the user's pupils obtained through the cameras (240-3, 240-4). In one embodiment, the data (or images) representing the gaze may include images of the user's eyes. In one embodiment, the data (or images) representing the gaze may include images of the pupil and iris of the user's eyes.

[0054] The wearable device (200) can identify the position of the user's pupil and / or iris based on an image representing light reflected from the user's iris acquired through the gaze cameras (240-3, 240-4). The wearable device (200) can identify the user's gaze and / or movement of the gaze based on the position and / or position change of the user's pupil and / or iris. Although not shown, in one embodiment, the wearable device (200) may further include a light source (e.g., a light emitting diode (LED)) that emits light toward a subject (e.g., the user's eyes, face, and / or an external object within the FoV) being captured using the gaze cameras (240-3, 240-4). The light source may emit light of an infrared wavelength.

[0055] The displays (250-1, 250-2) described with reference to FIG. 2A may correspond to the display module (160) of FIG. 1. The cameras (240-1, 240-2, 240-3, 240-4) described with reference to FIG. 2A may correspond to the camera module (180) of FIG. 1. However, the present invention is not limited thereto. Some of the cameras (240-1, 240-2, 240-3, 240-4) described with reference to FIG. 2A (e.g., cameras (240-3, 240-4)) may correspond to the sensor module (176) of FIG. 1.

[0056] Referring to FIG. 2B, cameras (240-5, 240-6, 240-7, 240-8, 240-9, 240-10) and / or a depth sensor (230) for obtaining information related to the external environment of the wearable device (200) may be placed on a second surface (220) opposite to the first surface (210) of FIG. 2A. For example, the cameras (240-5, 240-6, 240-7, 240-8, 240-9, 240-10) may be placed on the second surface (220) to recognize external objects different from the wearable device (200). The cameras (240-5, 240-6, 240-7, 240-8, 240-9, 240-10) described with reference to FIG. 2b may correspond to the camera module (180) of FIG. 1. The depth sensor (230) described with reference to FIG. 2b may correspond to the sensor module (176) of FIG. 1.

[0057] For example, using cameras (240-9, 240-10), the wearable device (200) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (240-9) can be placed on the second face (220) of the wearable device (200) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (240-10) can be placed on the second face (220) of the wearable device (200) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes.

[0058] According to one embodiment, the wearable device (200) may include a depth sensor (230) disposed on the second surface (220) to identify a distance between the wearable device (200) and an external object. Using the depth sensor (230), the wearable device (200) may obtain spatial information (e.g., a depth map) for at least a portion of a field of view (FoV) of a user wearing the wearable device (200).

[0059] According to one embodiment, a wearable device (200) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (200) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (200) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (200) based on the IMU. In one embodiment, the IMU may correspond to the sensor module (176) of FIG. 1.

[0060] Fig. 3a shows the positional relationship between displays (250-1, 250-2) and the user's two eyes (300-1, 300-2) determined through fit adjustment. Fig. 3b shows an example of an image of the left eye (300-1) of the user (300). Fig. 3c shows the anatomical structure of the left eye (300-1) of the user (300). Fig. 3d shows an example of slippage of the left eye (300-1) of the user (300). Fig. 4a shows a synthetic image generated according to a slippage state. Fig. 4b shows a synthetic image generated according to a slippage state.

[0061] FIGS. 3A, 3B, 3C, 3D, 4A, and 4B may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIGS. 2A and 2B. The operations described with reference to FIGS. 3A, 3B, 3C, 3D, 4A, and 4B may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0062] In one embodiment, the electronic device (101) can obtain characteristic information of the user (300). In one embodiment, the electronic device (101) can obtain characteristic information of the user (300) based on images obtained from cameras (240-1, 240-2, 240-3, 240-4).

[0063] In one embodiment, the feature information may include information on feature parts of the user's (300) head. In one embodiment, the feature parts of the head may include parts of the electronic device (101) worn by the user (300). For example, the feature parts of the head may include the two eyes (300-1, 300-2), and / or the surrounding areas of the two eyes (300-1, 300-2) (e.g., nose, and / or cheeks). However, the present invention is not limited thereto. The feature parts of the head may include other parts of the user's (300) head that affect the slippage of the electronic device (101) (e.g., forehead, temples, or cheekbones).

[0064] In one embodiment, the feature information may include a positional relationship between the displays (250-1, 250-2) and the two eyes (300-1, 300-2) of the user (300). Referring to FIG. 3A, the positional relationship may include information about a first distance (310) between the displays (250-1, 250-2), information about a second distance (321) between the first display (250-1) and the left eye (300-1), information about a third distance (322) between the second display (250-2) and the right eye (300-2), and information about a fourth distance (330) between the two eyes (300-1, 300-2). Hereinafter, the second distance (321) and the third distance (322) may be referred to as eye relief. Information about the second distance (321) and information about the third distance (322) may be referred to as eye relief information. The fourth distance (330) may be referred to as interpupillary distance (IPD). Information about the fourth distance (330) may be referred to as IPD information.

[0065] In one embodiment, the feature information may include location information of feature parts of the two eyes (300-1, 300-2) of the user (300). In one embodiment, referring to FIG. 3B, the feature parts may include the sclera (361), the iris (363), the pupil (365), and / or the glint (367) where a reflection by a light source occurs. In FIG. 3B, only the left eye (300-1) is illustrated, but this is merely an example. The feature parts for the right eye (300-2) may also include the sclera, the iris, the pupil, and / or the glint where a reflection by a light source occurs.

[0066] In one embodiment, the electronic device (101) can obtain eye relief information and IPD information based on images obtained from cameras (240-1, 240-2, 240-3, 240-4). In one embodiment, the electronic device (101) can obtain position information of displays (250-1, 250-2) based on a sensor (e.g., IMU).

[0067] In one embodiment, the feature information may include location information of the surrounding areas (e.g., nose and / or cheeks) of the user's (300) two eyes (300-1, 300-2). In one embodiment, the location information of the nose and / or cheeks may include information about the size and / or height of the nose and / or cheeks.

[0068] In one embodiment, the electronic device (101) can obtain location information of peripheral areas (e.g., nose and / or cheeks) of the two eyes (300-1, 300-2) based on images obtained from the cameras (240-1, 240-2, 240-3, 240-4).

[0069] In one embodiment, the electronic device (101) may obtain characteristic information of the user (300) during fit adjustment. Here, the fit adjustment may include adjusting the positions of the displays (250-1, 250-2), adjusting the distance between the displays (250-1, 250-2), adjusting the distance between the displays (250-1, 250-2) and the two eyes (300-1, 300-2), and / or fixing the electronic device (101) to the head.

[0070] In one embodiment, the electronic device (101) may generate an eye model based on feature information. In one embodiment, the electronic device (101) may generate an eye model including one or more parameters. Referring to FIG. 3C, the one or more parameters may represent a radius of curvature of the cornea (381) of each of the two eyes (300-1, 300-2), an angle (or kappa angle) difference between the visual axis (371) and the optical axis (372). The visual axis (371) may be a straight line passing through the center of the lens (383) from the fovea (382). The optical axis (372) may be a straight line passing through the pupil (365) from the center of rotation of each of the two eyes (300-1, 300-2). The visual axis (371) is inclined about 5 degrees inwardly in the horizontal direction and about 1 degree downwardly in the vertical direction from the optical axis (372) in the direction of the center of the pupil (365). One or more parameters may represent the position of each of the two eyes (300-1, 300-2), the radius of each of the two eyes (300-1, 300-2), the size of the iris (363), the radius of the iris (363), the radius of the pupil (365), the center position of the pupil (365), and the position of at least one glint (367). In one embodiment, the gaze direction (390) of the user (300) may correspond to the direction of the visual axis (371). In one embodiment, tracking the gaze of the user (300) may include, but is not limited to, obtaining the gaze direction (390) of the user (300).

[0071] In one embodiment, the electronic device (101) may identify a head model corresponding to the user's (300) characteristic information and / or the user's (300) gender among a plurality of head models. In one embodiment, the plurality of head models may have different appearances depending on the shape of the characteristic parts of the head and / or the user's gender. In one embodiment, the electronic device (101) may identify a head model corresponding to the information of the user's (300) two eyes (300-1, 300-2), nose, cheeks, and / or forehead.

[0072] In one embodiment, the electronic device (101) may generate one or more synthetic images of a head model corresponding to a user (300). In one embodiment, the one or more synthetic images may represent virtual images of the two eyes (300-1, 300-2) of the user (300) obtained from cameras (240-3, 240-4) in different sleep states. In one embodiment, one or more composite images may represent virtual images of the user's (300) two eyes (300-1, 300-2) obtained from the cameras (240-3, 240-4) based on the structure of the electronic device (101), the location of the light source, the location of the cameras (240-1, 240-2, 240-3, 240-4), and / or the location of the displays (250-1, 250-2).

[0073] For example, slippage may indicate a situation in which the wearing state of the electronic device (101) is changed according to the relative movement between the electronic device (101) and the user (300) through fit adjustment. For example, the slippage state (301) of FIG. 3D may indicate a state in which the left eye (300-1) is moved relatively lower than the first display (250-1) according to the relative movement of the left eye (300-1) of the electronic device (101) and the user (300). In the slippage state (301), the eye relief may be changed from the second distance (321) to another distance (323). In the slippage state (301), the image of the left eye (300-1) captured by the camera (240-3) (hereinafter, the slippage image) may be different from the image of the left eye (300-1) captured in the wearing state according to the fit adjustment (hereinafter, the normal wearing image). For example, the slippage image may represent a state in which the left eye (300-1) is captured from above compared to the normal wearing image. For example, the slippage state (302) of FIG. 3D may represent a state in which the first display (250-1) and the left eye (300-1) are relatively close to each other according to the relative movement of the electronic device (101) and the left eye (300-1) of the user (300). In the slippage state (302), the eye relief may be changed from the second distance (321) to another distance (324). In the slippage state (302), the slippage image for the left eye (300-1) captured through the camera (240-3) may be different from the normal wearing image for the left eye (300-1) captured in the wearing state according to the fit adjustment. For example, the slippage image may represent a state in which the left eye (300-1) is captured in a more enlarged state compared to the normal wearing image. According to an embodiment, the slippage image may represent a state in which the left eye (300-1) is captured from below compared to the normal wearing image. The slippage image may represent a state in which the left eye (300-1) is captured in a reduced state compared to the normal wearing image.

[0074] In one embodiment, the one or more composite images may represent virtual images of the two eyes (300-1, 300-2) acquired from the cameras (240-3, 240-4) in each of a specified slippage range. For example, the one or more composite images may represent virtual images of the two eyes (300-1, 300-2) when each of the two eyes (300-1, 300-2) is in a downward direction, an upward direction, an approaching direction, an away direction, or a combination thereof relative to the electronic device (101). For example, the one or more composite images may represent virtual images in each of the slippage states according to the degree of movement in the vertical direction (e.g., downward direction and upward direction) and the degree of movement in the horizontal direction (e.g., approaching direction and away direction). The degree of movement in the vertical direction and the degree of movement in the horizontal direction may be defined within the specified slippage range. In one embodiment, the specified slippage range may be determined by a head model corresponding to the user (300) among a plurality of head models, but is not limited thereto. For example, the slippage range may be determined by the head model of the user (300) and / or the adjustment of the portion of the electronic device (101) worn on the user (300). For example, the portion worn on the user (300) may include a forehead pad of the electronic device (101), a light seal for preventing (or reducing) external light from reaching the eyes (300-1, 300-2) of the user (300), glasses arms (e.g., temples), and / or a headband.

[0075] In one embodiment, one or more composite images may represent a virtual image for the two eyes (300-1, 300-2) when the user (300) views an object displayed through the displays (250-1, 250-2) through the two eyes (300-1, 300-2), each within a specified slippage range.

[0076] For example, referring to FIG. 4A, the composite image (401) may represent a state in which the left eye (300-1) is photographed from below compared to the normal wearing image. For example, the composite image (402) may represent a state in which the left eye (300-1) is photographed from a lower angle compared to the composite image (401).

[0077] Referring to the composite images (401, 402), it can be confirmed that the iris regions (411, 412) change depending on the slippage state. Therefore, an update of the eye model depending on the slippage state may be required.

[0078] In one embodiment, the electronic device (101) can determine a correction value of the eye model according to the slippage. In one embodiment, the electronic device (101) can determine a correction value of one or more parameters of the eye model according to the slippage. For example, the electronic device (101) can determine a correction value of one or more parameters of the eye model based on the positions of each of the two eyes (300-1, 300-2), the radius of each of the two eyes (300-1, 300-2), the size of the iris (363), the radius of the iris (363), the radius of the pupil (365), the center position of the pupil (365), and the position of at least one glint (367), which are identified in each of the composite images. For example, the electronic device (101) can determine a correction value for the visual axis (371) of the eye model based on the positions of each of the two eyes (300-1, 300-2), the radii of each of the two eyes (300-1, 300-2), the size of the iris (363), the radius of the iris (363), the radius of the pupil (365), the center position of the pupil (365), and the position of at least one glint (367), which are identified in each of the composite images.

[0079] According to an embodiment, the electronic device (101) may determine a correction value for the visual axis (371) obtained through an eye model according to slippage. Instead of correcting the eye model, the electronic device (101) may determine a correction value for the visual axis (371) obtained through an uncorrected eye model. Hereinafter, the correction value for the eye model may be referred to as an eye model change value. Hereinafter, the correction value for the visual axis (371) obtained through an uncorrected eye model may be referred to as a gaze error correction value.

[0080] In one embodiment, the electronic device (101) may obtain eye model change values ​​and / or gaze error correction values ​​for each of the states in the sleep state based on the synthetic images.

[0081] According to an embodiment, the electronic device (101) may obtain real images to be used as one or more composite images. For example, the electronic device (101) may guide the user (300) on a sleep state (e.g., guidance through a graphical user interface (GUI) or an auditory user interface (AUI)). For example, the electronic device (101) may obtain real images through the cameras (240-3, 240-4) in each of the sleep states according to the relative vertical movement and horizontal movement between the electronic device (101) and the user (300) by the user's (300) operation according to the guide. In one embodiment, the relative vertical and horizontal movement between the electronic device (101) and the user (300) due to the user's (300) manipulation according to the guide can be identified based on images acquired from cameras (240-1, 240-2, 240-3, 240-4) and / or sensors (e.g., IMU).

[0082] In one embodiment, one or more synthetic images, correction values ​​(e.g., eye model change values ​​and / or gaze error correction values), feature information, and a head model for each of the synthetic images may be included in user characteristic information. In one embodiment, the user characteristic information may be used to correct gaze errors of the user (300) in the electronic device (101). Correction of gaze errors of the user (300) may be described with reference to FIGS. 6 to 8.

[0083] According to an embodiment, the electronic device (101) may obtain user characteristic information (e.g., correction values) based on an artificial intelligence (AI) model (e.g., a machine learning model). According to an embodiment, the electronic device (101) may obtain correction values ​​(e.g., eye model change values ​​and / or gaze error correction values) for each of the slippage degrees within the slippage range through the AI ​​model. For example, the AI ​​model may be trained to output user characteristic information based on actual images through the cameras (240-3, 240-4) in each of the slippage states according to the relative vertical movement degree and horizontal movement degree. In one embodiment, the AI ​​model may include a pre-trained AI model. In one embodiment, the AI ​​model may include a plurality of parameters related to a neural network having a structure based on an encoder and a decoder, such as a transformer, but is not limited thereto. In one embodiment, the AI ​​model may include parameters for driving a neural network, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a feedforward neural network (FNN), and / or a long short-term memory (LSTM).

[0084] Figure 5 shows an example of user calibration for obtaining reference images.

[0085] FIG. 5 may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIG. 2A and FIG. 2B. FIG. 5 may be described with reference to FIG. 3A, FIG. 3B, FIG. 3C, FIG. 3D, FIG. 4A, and FIG. 4B. The operations described with reference to FIG. 5 may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0086] In one embodiment, the electronic device (101) may perform calibration to correct data representing the gaze of the user (300) by reflecting the characteristic information of the user (300). Referring to FIG. 5, the electronic device (101) may display virtual reference objects (511, 513, 515, 517, 519) whose locations (coordinates) are known on the screen (510) while the user (300) is wearing the electronic device (101). The electronic device (101) can calibrate the angle (or kappa angle) difference between the visual axis (371) and the optical axis (372) of the user (300) while the user (300) looks at the reference objects (511, 513, 515, 517, 519), the positions of the two eyes (300-1, 300-2), and the direction of gaze (or visual axis (371)) through the two eyes (300-1, 300-2). In one embodiment, the electronic device (101) can sequentially display the reference objects (511, 513, 515, 517, 519) on the screen (510) for calibration.

[0087] In one embodiment, the electronic device (101) may acquire reference images while performing calibration. In one embodiment, the reference images may be images of the two eyes (300-1, 300-2) looking at each of the reference objects (511, 513, 515, 517, 519) acquired through the cameras (240-1, 240-2, 240-3, 240-4).

[0088] According to an embodiment, the electronic device (101) may generate, as a reference image, an image of the two eyes (300-1, 300-2) when looking at a location on the screen (510) other than reference objects (511, 513, 515, 517, 519) based on reference images acquired while performing calibration.

[0089] In one embodiment, reference images may be used to correct errors in the user's (300) gaze in the electronic device (101). Correction of errors in the user's (300) gaze may be described with reference to FIGS. 6 to 8.

[0090] FIG. 6 is a block diagram of an electronic device (101) according to one embodiment.

[0091] FIG. 6 may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIG. 2A and FIG. 2B. The operations described with reference to FIG. 6 may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0092] Referring to FIG. 6, the electronic device (101) may include a processor (120), a memory (130), a display (660), a sensor (676), and a camera (680). In one embodiment, the processor (120) of FIG. 6 may correspond to the processor (120) of FIG. 1. In one embodiment, the memory (130) of FIG. 6 may correspond to the memory (130) of FIG. 1. In one embodiment, the display (660) of FIG. 6 may correspond to the display module (160) of FIG. 1. In one embodiment, the display (660) of FIG. 6 may correspond to the displays (250-1, 250-2) of FIGS. 2A, 2B, and 3A. In one embodiment, the sensor (676) of FIG. 6 may correspond to the sensor module (176) of FIG. 1. In one embodiment, the sensor (676) of FIG. 6 may be an IMU. In one embodiment, the camera (680) of FIG. 6 may correspond to the camera module (180) of FIG. 1. In one embodiment, the camera (680) of FIG. 6 may correspond to the cameras (240-1, 240-2, 240-3, 240-4) of FIG. 2a, FIG. 2b, and FIG. 3a.

[0093] In one embodiment, the memory (130) may include user characteristic information (611), calibration information (613), reference images (615), and a gaze error correction unit (620). In one embodiment, the user characteristic information (611) may include one or more synthetic images, feature information, and information about a head model. In one embodiment, the calibration information (613) may indicate a value for correcting an angular difference (or kappa angle) between the visual axis (371) and the optical axis (372) of the user (300), the positions of the two eyes (300-1, 300-2), and a gaze direction (or visual axis (371)) through the two eyes (300-1, 300-2).

[0094] In one embodiment, the gaze error correction unit (620) can identify the gaze of the user (300) (or the gaze based on the visual axis (371)) based on images of the two eyes (300-1, 300-2) acquired through the camera (680).

[0095] In one embodiment, the gaze error correction unit (620) can identify the occurrence of slippage of the electronic device (101) through the sensor (676) and / or the camera (680). For example, the gaze error correction unit (620) can identify that the electronic device (101) is relatively moved from the user (300) through the sensor (676) and / or the camera (680). For example, the gaze error correction unit (620) can identify the occurrence of slippage of the electronic device (101) based on a change in the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) looking at an object, as identified through the camera (680). For example, the gaze error correction unit (620) can identify the occurrence of slippage based on two gazes of the user (300) looking at an object displayed at a designated location on the screen (510), which are acquired at different times. For example, slippage can be identified as occurring based on the fact that the first gaze (or visual axis (371)) (or a vector representing the visual axis (371)) at a first time point of the user (300) looking at an object displayed at the same location and the second gaze (or visual axis (371)) (or a vector representing the visual axis (371)) at a second time point of the user (300) looking at an object displayed at the same location are different from each other. In one embodiment, the first gaze (or visual axis (371)) (or a vector representing the visual axis (371)) at the first time point can be acquired based on displaying objects during calibration. In one embodiment, the second gaze (or visual axis (371)) (or a vector representing the visual axis (371)) at the second point in time can be acquired while the electronic device (101) provides augmented reality (AR), virtual reality (VR), or mixed reality (MR) to the user (300).

[0096] In one embodiment, the gaze error correction unit (620) may correct the gaze of the user (300) (or the gaze based on the visual axis (371)) based on the occurrence of slippage of the electronic device (101). In one embodiment, the gaze error correction unit (620) may correct the gaze of the user (300) (or the gaze based on the visual axis (371)) using the user characteristic information (611) and reference images (615) based on the occurrence of slippage of the electronic device (101). Correcting the gaze of the user (300) (or the gaze based on the visual axis (371)) may include correcting the parameters of the eye model based on the eye model change value. Correcting the gaze of the user (300) (or the gaze based on the visual axis (371)) may include correcting the visual axis (371) obtained through the uncorrected eye model based on a gaze error correction value.

[0097] According to an embodiment, the gaze error correction unit (620) may include an AI model. In one embodiment, the gaze error correction unit (620) may correct the gaze (or the gaze based on the visual axis (371)) of the user (300) based on the AI ​​model. For example, the AI ​​model of the gaze error correction unit (620) may be trained to output information about an eye model change value, a gaze error correction value, or a corrected gaze based on actual images acquired in each of the slippage states according to the relative vertical and horizontal movement degrees. In one embodiment, the AI ​​model may include a pre-trained AI model. In one embodiment, the AI ​​model may include a plurality of parameters related to a neural network having a structure based on an encoder and a decoder, such as a transformer, but is not limited thereto. In one embodiment, the AI ​​model may include parameters for driving a neural network such as a CNN, an RNN, an FNN, and / or an LSTM.

[0098] In one embodiment, the gaze error correction unit (620) can obtain an eye model change value or a gaze error correction value by inputting an image into the AI ​​model. In one embodiment, the gaze error correction unit (620) can correct the gaze based on the obtained model change value or the gaze error correction value. In one embodiment, the gaze error correction unit (620) can obtain information about the corrected gaze by inputting an image into the AI ​​model.

[0099] Hereinafter, with reference to FIGS. 7 and 8, an operation of the electronic device (101) to correct the gaze (or gaze based on the visual axis (371)) of the user (300) will be described.

[0100] As described above, the electronic device (101) can correct gaze errors according to the degree of slippage that occurs based on the head shape of the user (300). Accordingly, the electronic device (101) can provide a more sophisticated gaze tracking function to the user (300).

[0101] FIG. 7 is a diagram illustrating an example of an operation of an electronic device (101) to correct a gaze error according to one embodiment.

[0102] FIG. 7 may be described with reference to the electronic device (101) of FIG. 1 and FIG. 6, and the wearable device (200) of FIG. 2A and FIG. 2B. The operations described with reference to FIG. 7 may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0103] Referring to FIG. 7, the electronic device (101) can obtain images (710) of two eyes (300-1, 300-2) obtained through a camera (680).

[0104] In one embodiment, the electronic device (101) may acquire object information (730) about an object on a screen (510) displayed through a display (660) at the time of acquisition (or time interval) of an image (710). In one embodiment, the object information (730) may indicate a location of the object in a three-dimensional (3D) space. However, the present invention is not limited thereto. In one embodiment, the object information (730) may indicate a location in a two-dimensional (2D) space displayed through the display (660) for gaze tracking.

[0105] In one embodiment, the electronic device (101) may identify at least one reference image (740) among the reference images (615) through the reference image selection unit (720). For example, the electronic device (101) may identify a reference image (740) corresponding to object information (730) among the reference images (615). In one embodiment, the reference image (740) corresponding to the object information (730) may include a correspondence between a location of an object displayed when the reference image (740) is acquired and a location of an object on a screen (510) displayed through the display (660) when the image (710) is acquired.

[0106] In one embodiment, the reference image selection unit (720) may be included as a part of the gaze error correction unit (620), but is not limited thereto. The reference image selection unit (720) may be stored in the memory (130) as a separate module from the gaze error correction unit (620).

[0107] In one embodiment, the electronic device (101) can determine whether slippage has occurred by comparing the image (710) with the reference image (740). In one embodiment, the electronic device (101) can determine whether slippage has occurred by comparing the line of sight (or visual axis (371)) (or a vector representing the visual axis (371)) identified from the image (710) with the reference line of sight (or reference visual axis) (or a reference vector representing the reference visual axis) of the reference image (740). For example, the electronic device (101) can determine that slippage has occurred based on the difference between the line of sight identified from the image (710) and the reference line of sight being greater than or equal to a reference difference.

[0108] In one embodiment, the electronic device (101) may obtain gaze information (750) from the image (710) through the gaze error correction unit (620). For example, the gaze information (750) may represent the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) or a corrected gaze (or visual axis (371)) (or a vector representing the visual axis (371)).

[0109] In one embodiment, the electronic device (101) may obtain gaze information (750) without performing correction on the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) through the gaze error correction unit (620) based on the determination that no slippage has occurred.

[0110] In one embodiment, the electronic device (101) may obtain gaze information (750) in a state in which correction has been performed on the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) based on the determination that slippage has occurred.

[0111] In one embodiment, the electronic device (101) can identify a synthetic image corresponding to the image (710) based on the user characteristic information (611) based on the determination that slippage has occurred. In one embodiment, the electronic device (101) can identify a synthetic image corresponding to the degree of slippage appearing in the image (710). In one embodiment, the electronic device (101) can identify eye model variation values ​​for parameters of the eye model and / or gaze error correction values ​​for the visual axis (371) corresponding to the identified synthetic image.

[0112] In one embodiment, the electronic device (101) can correct the eye model based on the eye model change value. In one embodiment, the electronic device (101) can obtain the corrected gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) as gaze information (750) based on the corrected eye model.

[0113] In one embodiment, the electronic device (101) can identify the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) obtained through an uncorrected eye model. In one embodiment, the electronic device (101) can correct the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) based on a gaze error correction value, thereby obtaining the corrected gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) as gaze information (750).

[0114] According to an embodiment, the gaze error correction unit (620) may correct the gaze (or the gaze based on the visual axis (371)) of the user (300) based on the AI ​​model. In one embodiment, the gaze error correction unit (620) may obtain an eye model change value or a gaze error correction value by inputting an image into the AI ​​model. In one embodiment, the gaze error correction unit (620) may obtain a corrected gaze as gaze information (750) based on the obtained model change value or gaze error correction value. In one embodiment, the gaze error correction unit (620) may obtain gaze information (750) for the corrected gaze by inputting an image into the AI ​​model.

[0115] According to an embodiment, the AI ​​model of the gaze error correction unit (620) may be trained to determine whether slippage has occurred. If it is determined that slippage has occurred, the AI ​​model of the gaze error correction unit (620) may be trained to output an eye model change value, a gaze error correction value, and gaze information (750) for the corrected gaze.

[0116] FIG. 8 is a diagram illustrating an example of an operation of an electronic device (101) to correct a gaze error according to one embodiment.

[0117] FIG. 8 may be described with reference to the electronic device (101) of FIGS. 1, 6, and 7, and the wearable device (200) of FIGS. 2A and 2B. The operations described with reference to FIG. 8 may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0118] FIG. 8 can illustrate an operation of an electronic device (101) to obtain gaze information (750) based on image features (820, 830) obtained from images (710, 740), compared to FIG. 7.

[0119] Referring to FIG. 8, the electronic device (101) can obtain images (710) of two eyes (300-1, 300-2) obtained through a camera (680).

[0120] In one embodiment, the electronic device (101) can identify at least one reference image (740) among the reference images (615) through the reference image selection unit (720). For example, the electronic device (101) can identify a reference image (740) corresponding to object information (730) among the reference images (615).

[0121] In one embodiment, the electronic device (101) may extract image features (820, 830) of each of the image (710) and the reference image (740) through the image feature extraction unit (810). The image features (820, 830) may represent feature points of feature portions of the two eyes (300-1, 300-2). In one embodiment, the feature points may be feature points extracted based on edges of the feature portions. For example, the feature points may be obtained by performing ellipse fitting on the edges. In one embodiment, the image feature extraction unit (810) may be included as a part of the gaze error correction unit (620). However, the present invention is not limited thereto. The image feature extraction unit (810) may be stored in the memory (130) as a separate module from the gaze error correction unit (620).

[0122] In one embodiment, the electronic device (101) can determine whether slippage has occurred by comparing the image features (820) of the image (710) with the reference image features (830) of the reference image (740). In one embodiment, the electronic device (101) can determine whether slippage has occurred by comparing the positions of feature points included in the image features (820) with the reference positions of reference feature points included in the reference image features (820). For example, the electronic device (101) can determine that slippage has occurred based on the difference between the positions of feature points and the reference positions of the reference feature points being identified as being greater than or equal to a reference difference.

[0123] In one embodiment, the electronic device (101) can obtain gaze information (750) from image features (820) of an image (710) through a gaze error correction unit (620).

[0124] In one embodiment, the electronic device (101) may obtain gaze information (750) without performing correction on the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) through the gaze error correction unit (620) based on the determination that no slippage has occurred.

[0125] In one embodiment, the electronic device (101) may obtain gaze information (750) in a state in which correction has been performed on the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) based on the determination that slippage has occurred.

[0126] In one embodiment, the electronic device (101) may identify a synthetic image corresponding to an image feature (820) of an image (710) based on user characteristic information (611) based on the determination that slippage has occurred. In one embodiment, the electronic device (101) may identify an eye model variation value for parameters of an eye model and / or a gaze error correction value for a visual axis (371) corresponding to the identified synthetic image.

[0127] In one embodiment, the electronic device (101) can correct the eye model based on the eye model change value. In one embodiment, the electronic device (101) can obtain the corrected gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) as gaze information (750) based on the corrected eye model.

[0128] In one embodiment, the electronic device (101) can identify the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) obtained through an uncorrected eye model. In one embodiment, the electronic device (101) can correct the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) based on a gaze error correction value, thereby obtaining the corrected gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) as gaze information (750).

[0129] According to an embodiment, the gaze error correction unit (620) may correct the gaze (or the gaze based on the visual axis (371)) of the user (300) based on the AI ​​model. In one embodiment, the gaze error correction unit (620) may obtain an eye model change value or a gaze error correction value by inputting an image into the AI ​​model. In one embodiment, the gaze error correction unit (620) may obtain a corrected gaze as gaze information (750) based on the obtained model change value or gaze error correction value. In one embodiment, the gaze error correction unit (620) may obtain gaze information (750) for the corrected gaze by inputting an image into the AI ​​model.

[0130] According to an embodiment, the AI ​​model of the gaze error correction unit (620) may be trained to determine whether slippage has occurred. If it is determined that slippage has occurred, the AI ​​model of the gaze error correction unit (620) may be trained to output an eye model change value, a gaze error correction value, and gaze information (750) for the corrected gaze.

[0131] According to an embodiment, the image feature extraction unit (810) may include an AI model. The AI ​​model of the image feature extraction unit (810) may be trained to extract image features from images. The image feature extraction unit (810) may obtain image features (820, 830) from images (710, 740) through the AI ​​model.

[0132] FIG. 9 is a flowchart showing the operation of an electronic device (101) according to one embodiment.

[0133] FIG. 9 may be described with reference to the electronic device (101) of FIGS. 1, 6, 7, and 8, and the wearable device (200) of FIGS. 2A and 2B. The operations described with reference to FIG. 9 may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0134] Referring to FIG. 9, in operation 910, the electronic device (101) may identify user characteristics (e.g., one or more synthetic images, correction values ​​for each of the synthetic images (e.g., eye model variation values ​​and / or gaze error correction values), feature information, and a head model). In one embodiment, the electronic device (101) may acquire user characteristics of the user (300) during fit adjustment.

[0135] In operation 920, the electronic device (101) may acquire a reference image. In one embodiment, the electronic device (101) may acquire a plurality of reference images while performing calibration. In one embodiment, the reference images may be actual images of the two eyes (300-1, 300-2) when looking at each of the reference objects (511, 513, 515, 517, 519) acquired through the cameras (240-1, 240-2, 240-3, 240-4).

[0136] In operation 930, the electronic device (101) may perform gaze tracking based on user characteristics and reference images. In one embodiment, gaze tracking may be described with reference to FIG. 10.

[0137] FIG. 10 is a flowchart showing the operation of an electronic device (101) according to one embodiment.

[0138] FIG. 10 may be described with reference to the electronic device (101) of FIGS. 1, 6, 7, and 8, and the wearable device (200) of FIGS. 2A and 2B. The operations described with reference to FIG. 10 may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101). Operations 1010, 1020, and 1030 of FIG. 10 may be included in operation 930 of FIG. 9.

[0139] Referring to FIG. 10, in operation 1010, the electronic device (101) can identify a gaze. The electronic device (101) can identify a gaze (or a gaze based on a visual axis (371)) of a user (300) based on images (710) of two eyes (300-1, 300-2) acquired through a camera (680).

[0140] In operation 1020, the electronic device (101) can determine whether slippage has occurred. In one embodiment, the electronic device (101) can identify the occurrence of slippage of the electronic device (101) through the sensor (676) and / or the camera (680). For example, the electronic device (101) can identify that the electronic device (101) has moved relative to the user (300) through the sensor (676) and / or the camera (680). For example, the electronic device (101) can identify the occurrence of slippage of the electronic device (101) based on a change in the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) looking at an object, as identified through the camera (680). For example, the electronic device (101) can identify the occurrence of slippage based on two gazes of the user (300) looking at an object displayed at a designated location on the screen (510), which are acquired at different points in time. For example, slippage can be identified as occurring based on the fact that a first gaze (or visual axis (371)) (or a vector representing the visual axis (371)) at a first point in time of the user (300) looking at an object displayed at the same location and a second gaze (or visual axis (371)) (or a vector representing the visual axis (371)) at a second point in time are different from each other. In one embodiment, the first gaze (or visual axis (371)) (or a vector representing the visual axis (371)) at the first point in time can be acquired based on displaying objects during calibration. In one embodiment, the second gaze (or visual axis (371)) (or a vector representing the visual axis (371)) at the second point in time can be acquired while the electronic device (101) provides augmented reality (AR), virtual reality (VR), or mixed reality (MR) to the user (300).

[0141] In operation 1020, based on determining that slippage has occurred, the electronic device (101) may perform operation 1030. In operation 1020, based on determining that slippage has not occurred, the electronic device (101) may perform operation 1010.

[0142] In operation 1030, the electronic device (101) can identify an adjusted gaze based on user characteristics and a reference image.

[0143] In one embodiment, the electronic device (101) can identify a synthetic image corresponding to the image (710) based on the user characteristic information (611) based on the determination that slippage has occurred. In one embodiment, the electronic device (101) can identify a synthetic image corresponding to the degree of slippage appearing in the image (710). In one embodiment, the electronic device (101) can identify eye model variation values ​​for parameters of the eye model and / or gaze error correction values ​​for the visual axis (371) corresponding to the identified synthetic image.

[0144] In one embodiment, the electronic device (101) can correct the eye model based on the eye model change value. In one embodiment, the electronic device (101) can obtain the corrected gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) as gaze information (750) based on the corrected eye model.

[0145] In one embodiment, the electronic device (101) can identify the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) obtained through an uncorrected eye model. In one embodiment, the electronic device (101) can correct the gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) based on a gaze error correction value, thereby obtaining the corrected gaze (or visual axis (371)) (or a vector representing the visual axis (371)) of the user (300) as gaze information (750).

[0146] As described above, the wearable device (101) may include a display (250-1, 250-2, 660) arranged to face the eye (300-1, 300-2) of a user (300) wearing the wearable device (101). The wearable device (101) may include a camera (240-1, 240-2, 240-3, 240-4, 680) configured to acquire an image (710) of at least a portion of the eye (300-1, 300-2) of the user (300) wearing the wearable device (101). The wearable device (101) may include at least one processor (120) including a processing circuit. The wearable device (101) may include a memory (130) storing instructions and including one or more storage media. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify whether slippage of the wearable device (101) occurs while tracking the gaze of a user (300) looking at a screen (510) displayed through the display (250-1, 250-2, 660) based on the image (710) acquired by the camera (240-1, 240-2, 240-3, 240-4, 680). The above instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to correct the gaze using a correction value identified based on a user (300) characteristic associated with the head of the user (300), based on the identification of the slippage.

[0147] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify, based on the identified slippage, the correction value for a composite image corresponding to eye relief of the image (710) among a plurality of composite images generated at each of the slippage degrees within the designated slippage range of the user (300).

[0148] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify, among a plurality of head models, a head model corresponding to positional information of feature portions of a head of the user (300), based on another image (710) of at least a portion of a face of the user (300) wearing the wearable device (101) obtained during a fit adjustment of the wearable device (101). The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to determine the slippage range corresponding to the identified head model.

[0149] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify the eye relief between the eye (300-1, 300-2) of the user (300) wearing the wearable device (101) and the display (250-1, 250-2, 660) based on the image (710) acquired during a fit adjustment of the wearable device (101). The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to determine the slippage range corresponding to the identified eye relief.

[0150] The plurality of composite images may be virtual images of at least a portion of the eye (300-1, 300-2) of the user (300) that is expected to be acquired through the camera (240-1, 240-2, 240-3, 240-4, 680) for each of the slippage degrees. For each of the plurality of composite images for the slippage degrees, a correction value may be set for correcting one or more parameters of the eye model of the user (300) or a vector representing the gaze identified through the eye model of the user (300).

[0151] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to correct one or more parameters of the eye model of the user (300) based on the correction value. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify the gaze based on the eye model in which the one or more parameters have been corrected.

[0152] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to correct a vector representing the gaze identified through the eye model of the user (300) based on the correction value. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify the gaze based on the corrected vector.

[0153] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify whether slippage has occurred based on comparing the reference gaze identified through a reference image (740) of at least a portion of the eye (300-1, 300-2) of the user (300) obtained while displaying a reference object (511, 513, 515, 517, 519) at the same location as the object on the screen (510) toward which the gaze of the user (300) is directed.

[0154] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify whether slippage has occurred based on comparing reference feature points of a reference image (740) for at least a portion of the eye (300-1, 300-2) of the user (300) and feature points of the image (710) obtained while displaying a reference object (511, 513, 515, 517, 519) at the same location as the object on the screen (510) toward which the gaze of the user (300) is directed.

[0155] As described above, the method can be executed by the wearable device (101) including a display (250-1, 250-2, 660) arranged to face the eye (300-1, 300-2) of a user (300) wearing the wearable device (101), and a camera (240-1, 240-2, 240-3, 240-4, 680) configured to acquire an image (710) of at least a part of the eye (300-1, 300-2) of the user (300) wearing the wearable device (101). The method may include an operation of identifying whether slippage of the wearable device (101) has occurred while tracking the gaze of the user (300) on the screen (510) displayed through the display (250-1, 250-2, 660) based on the image (710) acquired by the camera (240-1, 240-2, 240-3, 240-4, 680). The method may include an operation of correcting the gaze using a correction value identified based on the user (300) characteristics of the user (300) based on the identification that the slippage has occurred.

[0156] The above-described operation of correcting the gaze may include an operation of identifying the correction value for a composite image corresponding to the eye relief of the image (710) among a plurality of composite images generated at each of the slippage degrees within the specified slippage range of the user (300), based on the identification that the slippage has occurred.

[0157] The method may include an operation of identifying a head model corresponding to position information of feature portions of a head of a user (300) among a plurality of head models based on another image (710) of at least a portion of a face of the user (300) wearing the wearable device (101) obtained during a fit adjustment of the wearable device (101). The method may include an operation of determining the slippage range corresponding to the identified head model.

[0158] The method may include an operation of identifying the eye relief between the eyes (300-1, 300-2) of the user (300) wearing the wearable device (101) and the display (250-1, 250-2, 660) based on the image (710) acquired during the fit adjustment of the wearable device (101). The method may include an operation of determining the slippage range corresponding to the identified eye relief.

[0159] The plurality of composite images may be virtual images of at least a portion of the eye (300-1, 300-2) of the user (300) that is expected to be acquired through the camera (240-1, 240-2, 240-3, 240-4, 680) for each of the slippage degrees. For each of the plurality of composite images for the slippage degrees, a correction value may be set for correcting one or more parameters of the eye model of the user (300) or a vector representing the gaze identified through the eye model of the user (300).

[0160] The method may include an operation of correcting one or more parameters of the eye model of the user (300) based on the correction value. The method may include an operation of identifying the gaze based on the eye model in which the one or more parameters have been corrected.

[0161] The method may include an operation of correcting a vector representing the gaze identified through the eye model of the user (300) based on the correction value. The method may include an operation of identifying the gaze based on the corrected vector.

[0162] The method may include an operation of identifying whether slippage of the wearable device (101) has occurred based on comparing the reference gaze identified through a reference image (740) of at least a portion of the eyes (300-1, 300-2) of the user (300) obtained while displaying a reference object (511, 513, 515, 517, 519) at the same location as the object on the screen (510) toward which the gaze of the user (300) is directed, with the gaze.

[0163] The method may include an operation of identifying whether slippage of the wearable device (101) has occurred based on comparing feature points of a reference image (740) for at least a portion of the eye (300-1, 300-2) of the user (300) and feature points of the image (710) obtained while displaying a reference object (511, 513, 515, 517, 519) at the same location as the object on the screen (510) toward which the gaze of the user (300) is directed.

[0164] As described above, a non-transitory computer readable storage medium can store one or more programs including instructions. The instructions, when executed individually or collectively by at least one processor (120) of the wearable device (101), which includes a display (250-1, 250-2, 660) arranged to face the eye (300-1, 300-2) of a user (300) wearing the wearable device (101), and a camera (240-1, 240-2, 240-3, 240-4, 680) configured to acquire an image (710) of at least a portion of the eye (300-1, 300-2) of the user (300) wearing the wearable device (101), cause the wearable device (101) to acquire the image (710) obtained by the camera (240-1, 240-2, 240-3, 240-4, 680). Based on the image (710), while tracking the gaze of the user (300) on the screen (510) displayed through the display (250-1, 250-2, 660), the wearable device (101) may be caused to identify whether slippage has occurred. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to correct the gaze using a correction value identified based on the user (300) characteristics of the user (300) based on the identification that the slippage has occurred.

[0165] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101) to identify the correction value for a composite image corresponding to eye relief of the image (710) among a plurality of composite images generated at each of the slippage degrees within the specified slippage range of the user (300) based on the identification that the slippage has occurred.

[0166] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0167] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0168] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0169] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0170] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0171] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (101), A display (250-1, 250-2, 660) arranged to face the eyes (300-1, 300-2) of a user (300) wearing the wearable device (101); A camera (240-1, 240-2, 240-3, 240-4, 680) configured to acquire an image (710) of at least a portion of the eye (300-1, 300-2) of the user (300) wearing the wearable device (101); At least one processor (120) comprising a processing circuit; and A wearable device (101) comprises a memory (130) storing instructions and including one or more storage media, wherein the instructions, when individually or collectively executed by the at least one processor (120), While tracking the gaze of a user (300) looking at a screen (510) displayed through the display (250-1, 250-2, 660) based on the image (710) acquired by the camera (240-1, 240-2, 240-3, 240-4, 680), identify whether slippage of the wearable device (101) occurs, Based on the above slippage being identified, the gaze is corrected using a correction value identified based on the user (300) characteristics related to the head of the user (300). Wearable devices.

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101) to: Based on the above slippage being identified, causing the correction value for the synthetic image corresponding to the eye relief of the image (710) among the plurality of synthetic images generated at each of the slippage degrees within the specified slippage range of the user (300) to be identified. Wearable devices.

3. In claim 1 or claim 2, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101) to: Based on another image (710) of at least a part of the face of the user (300) wearing the wearable device (101) obtained during the fit adjustment of the wearable device (101), a head model corresponding to the location information of the characteristic parts of the head of the user (300) is identified among a plurality of head models, causing the slippage range corresponding to the above identified head model to be determined, Wearable devices.

4. In any one of claims 1 to 3, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101) to: Based on the image (710) acquired during the fit adjustment of the wearable device (101), the eye relief between the eye (300-1, 300-2) of the user (300) wearing the wearable device (101) and the display (250-1, 250-2, 660) is identified, causing the slippage range corresponding to the above identified eye relief to be determined, Wearable devices.

5. In any one of claims 1 to 4, The above plurality of composite images are virtual images of at least a part of the eye (300-1, 300-2) of the user (300) that is expected to be acquired through the camera (240-1, 240-2, 240-3, 240-4, 680) at each of the above slippage degrees, For each of the plurality of synthetic images for the above slippage degrees, the correction value is set to correct one or more parameters of the eye model of the user (300) or the vector representing the gaze identified through the eye model of the user (300). Wearable devices.

6. In any one of claims 1 to 5, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101) to: Based on the above correction value, one or more parameters of the eye model of the user (300) are corrected, causing said gaze to be identified based on said eye model in which said one or more parameters are corrected; Wearable devices.

7. In any one of claims 1 to 6, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101) to: Based on the above correction value, the vector representing the gaze identified through the eye model of the user (300) is corrected, Based on the above corrected vector, causing the above gaze to be identified, Wearable devices.

8. In any one of claims 1 to 7, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101) to: Based on comparing the reference gaze identified through a reference image (740) of at least a part of the eyes (300-1, 300-2) of the user (300) obtained while displaying a reference object (511, 513, 515, 517, 519) at the same location as the object on the screen (510) to which the gaze of the user (300) is directed, whether or not slippage of the wearable device (101) occurs is identified. Wearable devices.

9. In any one of claims 1 to 8, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101) to: Based on comparing the reference feature points of the reference image (740) for at least a part of the eye (300-1, 300-2) of the user (300) and the feature points of the image (710) obtained while displaying the reference object (511, 513, 515, 517, 519) at the same location as the object on the screen (510) to which the gaze of the user (300) is directed, causing the occurrence of slippage of the wearable device (101) to be identified. Wearable devices.

10. A method executed by a wearable device (101), comprising a display (250-1, 250-2, 660) arranged to face the eye (300-1, 300-2) of a user (300) wearing the wearable device (101), and a camera (240-1, 240-2, 240-3, 240-4, 680) configured to acquire an image (710) of at least a part of the eye (300-1, 300-2) of the user (300) wearing the wearable device (101), An operation of identifying whether slippage of the wearable device (101) occurs while tracking the gaze of the user (300) on the screen (510) displayed through the display (250-1, 250-2, 660) based on the image (710) acquired by the camera (240-1, 240-2, 240-3, 240-4, 680), and Based on the identification that the above slippage has occurred, an operation of correcting the gaze using a correction value identified based on the user (300) characteristics of the user (300) is included. method.

11. In claim 10, the action of correcting the gaze is: An operation of identifying the correction value for a synthetic image corresponding to the eye relief of the image (710) among a plurality of synthetic images generated at each of the slippage degrees within the specified slippage range of the user (300) based on the identification that the slippage has occurred. method.

12. In claim 10 or claim 11, An operation of identifying a head model corresponding to position information of feature parts of the head of the user (300) among a plurality of head models based on another image (710) of at least a part of the face of the user (300) wearing the wearable device (101) obtained during the fit adjustment of the wearable device (101), and An operation for determining the slippage range corresponding to the identified head model is included. method.

13. In any one of claims 10 to 12, An operation of identifying the eye relief between the eye (300-1, 300-2) of the user (300) wearing the wearable device (101) and the display (250-1, 250-2, 660) based on the image (710) acquired during the fit adjustment of the wearable device (101). An operation comprising determining the slippage range corresponding to the identified eye relief. method.

14. In any one of claims 10 to 13, The above plurality of composite images are virtual images of at least a part of the eye (300-1, 300-2) of the user (300) that is expected to be acquired through the camera (240-1, 240-2, 240-3, 240-4, 680) at each of the above slippage degrees, For each of the plurality of synthetic images for the above slippage degrees, the correction value is set to correct one or more parameters of the eye model of the user (300) or the vector representing the gaze identified through the eye model of the user (300). method.

15. In a non-transitory computer readable storage medium, Store one or more programs containing instructions, The instructions, when executed individually or collectively by at least one processor (120) of the wearable device (101), which includes a display (250-1, 250-2, 660) arranged to face the eye (300-1, 300-2) of a user (300) wearing the wearable device (101), and a camera (240-1, 240-2, 240-3, 240-4, 680) configured to acquire an image (710) of at least a portion of the eye (300-1, 300-2) of the user (300) wearing the wearable device (101), cause the wearable device (101) to: While tracking the gaze of the user (300) on the screen (510) displayed through the display (250-1, 250-2, 660) based on the image (710) acquired by the camera (240-1, 240-2, 240-3, 240-4, 680), identify whether slippage of the wearable device (101) occurs, Based on the identification that the above slippage has occurred, causing the gaze to be corrected using a correction value identified based on the user (300) characteristics of the user (300). Non-transitory computer-readable recording medium.

Images