Wearable device, method, and non-transitory computer readable recording medium for eye calibration

Abstract

This method is executed in a wearable device comprising: a display system including a first display and a second display which face eyes of a user when worn; and a plurality of cameras arranged to acquire an image including the eyes of the user when worn, and the method may comprise the operations of: displaying objects at least at different viewpoints on a screen displayed through the display system; identifying gazes directed to the objects; identifying errors related to the gazes on the basis of the identified gazes, wherein the errors indicate differences between display positions of the objects and focal positions of the gazes corresponding one-to-one to the objects; displaying, on a background screen of the display system, a visual object moving via some display positions selected on the basis of the errors from among the display positions; and correcting the errors.

Description

Wearable device, method, and non-transitory computer-readable recording medium for eye calibration

[0001] The present disclosure relates to a wearable device, method, and non-transitory computer-readable recording medium for eye calibration.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, displaying computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these devices may be AR glasses or head-mounted devices (HMDs).

[0003] Wearable devices can track the movement of the user's eye pupils or the user's gaze based on images of the eyes. Through gaze tracking, the wearable device can determine what the user is looking at or focusing on.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0005] A wearable device is disclosed. According to an aspect of the disclosure, the wearable device comprises a display system including a first display and a second display facing the eyes of a user when worn, a plurality of cameras arranged to acquire images including the eyes of the user when worn, at least one processor including a processing circuit; and a memory storing instructions and including one or more storage media, wherein the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to display objects at least from different viewpoints on a three-dimensional (3D) screen displayed through the display system, identify gazes looking at the objects based on the images, identify errors associated with the identified gazes, the errors representing differences between display positions of the objects and focal positions of the gazes corresponding one-to-one to the objects, display a visual object moving via some of the display positions selected based on the errors, on a background screen on the display system, and correct the errors based on the different gazes looking at the visual object.

[0006] According to an aspect of the disclosure, a method executed in a wearable device, comprising a display system including a first display and a second display facing the eyes of a user when worn, and a plurality of cameras arranged to acquire an image including the eyes of the user when worn, may include the following actions: displaying objects at least at different viewpoints on a three-dimensional (3D) screen displayed through the display system; identifying gazes looking at the objects based on the images; identifying errors associated with the identified gazes, the errors representing differences between display positions of the objects and focus positions of the gazes corresponding one-to-one to the objects; displaying a visual object moving via some of the display positions selected based on the errors among the display positions, on a background screen on the display system; and correcting the errors based on the different gazes looking at the visual object.

[0007] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0008] FIG. 1 illustrates an electronic device within a network environment according to one or more embodiments.

[0009] FIG. 2A illustrates an example of an appearance of a wearable device according to one embodiment.

[0010] FIG. 2b illustrates an example of an appearance of a wearable device according to one embodiment.

[0011] Figure 3a shows the positional relationship between the displays and the user's two eyes.

[0012] Figure 3b shows an example of an image of the user's left eye.

[0013] Figure 3c shows the anatomical structure of the user's left eye.

[0014] FIG. 4A illustrates an example of a screen for calibrating a user's gaze, according to one embodiment.

[0015] FIG. 4b illustrates an example of an error between an object and a line of sight, according to one embodiment.

[0016] FIG. 5 illustrates an example of a screen displayed by a wearable device according to one embodiment.

[0017] FIG. 6 illustrates a block diagram of a wearable device according to one embodiment.

[0018] FIG. 7A illustrates an example of a movement path of a visual object for user calibration of a wearable device, according to one embodiment.

[0019] FIG. 7b illustrates an example of a movement path of a visual object for user calibration of a wearable device according to one embodiment.

[0020] FIG. 8 illustrates an operation performed by a wearable device according to one embodiment.

[0021] FIG. 9 illustrates an operation performed by a wearable device according to one embodiment.

[0022] FIG. 10 illustrates an operation performed by a wearable device according to one embodiment.

[0023] FIG. 11 illustrates an operation performed by a wearable device according to one embodiment.

[0024] FIG. 12 illustrates an operation performed by a wearable device according to one embodiment.

[0025] FIG. 1 illustrates an electronic device within a network environment according to one or more embodiments.

[0026] In FIG. 1, in a network environment (100), an electronic device (101) can communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or can communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) can communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0027] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0028] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0029] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0030] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0034] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0035] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, an illuminance sensor, an IMU (inertial measurement unit) sensor, or a touch sensor. For example, when the electronic device (101) detects a user movement through an IMU sensor or the like, the processor (120) of the electronic device (101) can correct the rendering data received from the external electronic device (102) based on the movement information and output it to the display module (160). Alternatively, the processor (120) can transmit the movement information to the external electronic device (102) and request rendering so that the screen data is updated accordingly.

[0036] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0039] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0041] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, a loss coverage (e.g., 664 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip) for realizing URLLC.

[0044] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0045] According to one or more embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0046] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technologies. In one embodiment, the external electronic device (102) can be various types of devices, such as a smartphone or a case device capable of storing and charging the electronic device (101).

[0048] FIG. 2A illustrates an example of the appearance of a wearable device according to one embodiment. FIG. 2B illustrates an example of the appearance of a wearable device according to one embodiment.

[0049] The wearable device (200) of FIGS. 2A and 2B may correspond to the electronic device (101) of FIG. 1. According to one embodiment, the wearable device (200) may be substantially the same as the electronic device (101) of FIG. 1 and may be implemented to be wearable on a user's body. In one embodiment, each of the external electronic devices (102, 104) of FIG. 1 may be the same or a different type of device as the electronic device (101) or the wearable device (200). According to one embodiment, all or part of the operations executed in the electronic device (101) or the wearable device (200) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when an electronic device (101) or a wearable device (200) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) or the wearable device (200) may, instead of or in addition to executing the function or service on its own, request one or more external electronic devices to execute at least a part of the function or service. The one or more external electronic devices that receive the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101) or the wearable device (200). The electronic device (101) or the wearable device (200) may process the result as is or additionally and provide it as at least a part of a response to the request.

[0050] In FIG. 2A, according to one embodiment, the first side (210) of the wearable device (200) may have a form that is attachable to a body part of a user (e.g., the face of the user). According to one embodiment, the wearable device (200) may have a form factor for being worn on the head of a user. In one embodiment, the wearable device (200) may be worn on a body part of a user (e.g., the head). In one embodiment, the wearable device (200) may be referred to as a wearable device in terms of being worn on a body part of a user (e.g., the head). In one embodiment, the wearable device (200) may further include a strap and / or one or more temples for being fixed on a body part of a user.

[0051] According to one embodiment, the wearable device (200) may include a first display (250-1) and a second display (250-2). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the left and right eyes of the user, respectively. According to an embodiment, the wearable device (200) may further include a rubber or silicone packing formed on the first surface (210) to prevent or reduce interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0052] In one embodiment, the wearable device (200) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (200) through displays (250-1, 250-2). For example, the wearable device (200) may provide a user experience (e.g., video see-through (VST)) in which real objects and reference objects are mixed by combining reference objects within a frame that includes real objects and is displayed through the first display (250-1) and the second display (250-2).

[0053] According to one embodiment, the wearable device (200) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0054] According to one embodiment, a wearable device (200) may include cameras (240-1, 240-2) for capturing and / or recognizing a user's face. The cameras (240-1, 240-2) may be referred to as FT (face tracking) cameras.

[0055] According to one embodiment, a wearable device (200) may include cameras (240-3, 240-4) for photographing or tracking both eyes of a user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (240-3, 240-4) may be referred to as ET (eye tracking) cameras.

[0056] Cameras (240-3, 240-4) can output data (or images) representing the gaze of a user wearing the wearable device (200). For example, the wearable device (200) can detect the gaze of the user from images containing the user's pupils obtained through the cameras (240-3, 240-4). In one embodiment, the data (or images) representing the gaze may include images of the user's eyes. In one embodiment, the data (or images) representing the gaze may include images of the pupil and iris of the user's eyes.

[0057] The wearable device (200) can identify the position of the user's pupil and / or iris based on an image representing light reflected from the user's iris acquired through the gaze cameras (240-3, 240-4). The wearable device (200) can identify the user's gaze and / or movement of the gaze based on the position and / or position change of the user's pupil and / or iris. In one embodiment, the wearable device (200) can further include a light source (e.g., a light emitting diode (LED)) that emits light toward a subject (e.g., the user's eyes, face, and / or an external object within the field of view (FoV)) being captured using the gaze cameras (240-3, 240-4). The light source can emit light of an infrared wavelength.

[0058] The displays (250-1, 250-2) described with reference to FIG. 2A may correspond to the display module (160) of FIG. 1. The cameras (240-1, 240-2, 240-3, 240-4) described with reference to FIG. 2A may correspond to the camera module (180) of FIG. 1. However, the present disclosure is not limited to the above-described embodiment. Some of the cameras (240-1, 240-2, 240-3, 240-4) described with reference to FIG. 2A (e.g., cameras (240-3, 240-4)) may correspond to the sensor module (176) of FIG. 1.

[0059] In FIG. 2B, cameras (240-5, 240-6, 240-7, 240-8, 240-9, 240-10) and / or a depth sensor (230) may be placed on a second surface (220) opposite to the first surface (210) of FIG. 2A to obtain information related to the external environment of the wearable device (200). For example, the cameras (240-5, 240-6, 240-7, 240-8, 240-9, 240-10) may be placed on the second surface (220) to recognize external objects different from the wearable device (200). The cameras (240-5, 240-6, 240-7, 240-8, 240-9, 240-10) described with reference to FIG. 2b may correspond to the camera module (180) of FIG. 1. The depth sensor (230) described with reference to FIG. 2b may correspond to the sensor module (176) of FIG. 1.

[0060] For example, using cameras (240-9, 240-10), the wearable device (200) can obtain images or videos to be transmitted to (at least one or) each of the user's two eyes. The camera (240-9) can be placed on the second face (220) of the wearable device (200) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (240-10) can be placed on the second face (220) of the wearable device (200) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes.

[0061] According to one embodiment, the wearable device (200) may include a depth sensor (230) disposed on the second surface (220) to identify a distance between the wearable device (200) and an external object. Using the depth sensor (230), the wearable device (200) may obtain spatial information (e.g., a depth map) for at least a portion of a field of view (FoV) of a user wearing the wearable device (200).

[0062] According to one embodiment, a wearable device (200) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (200) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (200) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (200) based on the IMU. In one embodiment, the IMU may correspond to the sensor module (176) of FIG. 1.

[0063] Fig. 3a shows the positional relationship between the displays (250-1, 250-2) and the two eyes (300-1, 300-2) of the user (300). Fig. 3b shows an example of an image of the left eye (300-1) of the user (300). Fig. 3c shows the anatomical structure of the left eye (300-1) of the user (300).

[0064] FIGS. 3A, 3B, and 3C may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIGS. 2A and 2B. The operations described with reference to FIGS. 3A, 3B, and 3C may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0065] In one embodiment, the cameras (240-3, 240-4) (e.g., cameras for eye tracking) may be located inside the housing of the electronic device (101) such that the camera lenses of the cameras (240-3, 240-4) face the eyes (300-1, 300-2) of the user (300) when the user (300) wears the electronic device (101). In one embodiment, the cameras (240-3, 240-4) may be installed for the left eye (300-1) and the right eye (300-2), respectively, and the performance and specifications of each camera may be the same.

[0066] In one embodiment, the cameras (240-3, 240-4) may be in fixed barrels around the displays (250-1, 250-2). In one embodiment, the cameras (240-3, 240-4) may be arranged on the electronic device (101) so as to face the user (300). In one embodiment, the cameras (240-3, 240-4) may be in the space between the displays (250-1, 250-2). For example, the camera (240-3) may be on the right side of the display (250-1) when looking at the first side (e.g., the first side (210) of FIGS. 2A and 2B) of the electronic device (101) (e.g., the side attached to the face of the user (300). For example, the camera (240-4) may be located on the left side of the display (250-2) when looking at the first side (e.g., the first side (210) of FIGS. 2A and 2B) of the electronic device (101) (e.g., the side attached to the face of the user (300). For example, the distance between the cameras (240-3, 240-4) may be shorter than the distance between the displays (250-1, 250-2). However, the present disclosure is not limited to the above embodiment. In one embodiment, the cameras (240-3, 240-4) may be located in a space other than the space between the displays (250-1, 250-2). For example, the camera (240-3) may be located on the left side of the display (250-1) when looking at the first side (e.g., the first side (210) of FIGS. 2A and 2B) of the electronic device (101) (e.g., the side attached to the face of the user (300)). For example, the camera (240-4) may be located on the right side of the display (250-2) when looking at the first side (e.g., the first side (210) of FIGS. 2A and 2B) of the electronic device (101) (e.g., the side attached to the face of the user (300)). For example, the distance between the cameras (240-3, 240-4) may be longer than the distance between the displays (250-1, 250-2).

[0067] In one embodiment, the electronic device (101) may generate (or update) an eye model of a user (300). In one embodiment, the electronic device (101) may generate (or update) the eye model of the user (300) upon the initial operation of the electronic device (101) (e.g., out of box experience (OOBE)) and / or upon the occurrence of a specified event. In one embodiment, the specified event may be registration (or change) of a user account of the user (300) on the electronic device (101). In one embodiment, the specified event may include a request from the user (300). For example, a request from a user (300) may be identified by the electronic device (101) based on selection of a user interface (UI) element (or graphical UI (GUI)) for requesting creation (or update) of an eye model of the user (300) on a three-dimensional (3D) screen displayed through displays (250-1, 250-2) of the electronic device (101). However, the present disclosure is not limited to the above-described embodiment. For example, a request from a user (300) may be identified based on recognition of a command from the user (300) through voice recognition.

[0068] In one embodiment, the electronic device (101) may acquire images through cameras (240-1, 240-2, 240-3, 240-4) to create (or update) an eye model of a user (300). In one embodiment, the images for creating (or updating) an eye model of a user (300) may include at least a portion of a face of the user (300). In one embodiment, the images for creating (or updating) an eye model of a user (300) may include at least a portion of eyes (300-1, 300-2) of the user (300).

[0069] In one embodiment, the electronic device (101) may acquire images through cameras (240-1, 240-2, 240-3, 240-4) while the user (300) looks at a 3D screen displayed through displays (250-1, 250-2) to create (or update) an eye model of the user (300). The 3D screen displayed to create (or update) the eye model of the user (300) may be a screen displayed according to the execution of an application for eye calibration. Hereinafter, the screen displayed according to the execution of the application for eye calibration may be described with reference to FIGS. 4A and 4B.

[0070] In one embodiment, the electronic device (101) may acquire feature information of the user (300) obtained through images in order to generate (or update) an eye model of the user (300). In one embodiment, the electronic device (101) may acquire feature information of the user (300) based on images acquired from cameras (240-1, 240-2, 240-3, 240-4). In one embodiment, the electronic device (101) may acquire feature information of the user (300) based on images including at least a part of the face of the user (300) acquired from cameras (240-1, 240-2). In one embodiment, the electronic device (101) may obtain feature information of the user (300) based on an image including at least a portion of the user's (300's) eyes (300-1, 300-2) obtained from cameras (240-3, 240-4).

[0071] In one embodiment, the feature information may include a positional relationship between the displays (250-1, 250-2) and the two eyes (300-1, 300-2) of the user (300). In FIG. 3A, the positional relationship may include information about a first distance (310) between the displays (250-1, 250-2), information about a second distance (321) between the first display (250-1) and the left eye (300-1), information about a third distance (322) between the second display (250-2) and the right eye (300-2), and information about a fourth distance (330) between the two eyes (300-1, 300-2). Hereinafter, the second distance (321) and the third distance (322) may be referred to as 'eye relief' in the present disclosure. Information about the second distance (321) and information about the third distance (322) may be referred to as "eye relief information" in the present disclosure. The fourth distance (330) may be referred to as "interpupillary distance (IPD)" in the present disclosure. Information about the fourth distance (330) may be referred to as "IPD information" in the present disclosure.

[0072] In one embodiment, the feature information may include location information of feature parts of the two eyes (300-1, 300-2) of the user (300). In one embodiment, in FIG. 3B, the feature parts may include the sclera (361), the iris (363), the pupil (365), and / or glints (367) where reflections by a light source occur. In FIG. 3B, only the left eye (300-1) is illustrated, but this is merely an example. The feature parts for the right eye (300-2) may also include the sclera, the iris, the pupil, and / or the glints where reflections by a light source occur.

[0073] In one embodiment, the electronic device (101) may generate an eye model based on the feature information. In one embodiment, the eye model may include one or more parameters. In FIG. 3C, the one or more parameters may represent a radius of curvature of the cornea (381) of each (or at least one) of the two eyes (300-1, 300-2), an angle (or “kappa angle”) difference between the visual axis (371) and the optical axis (372). The visual axis (371) may be a straight line passing through the center of the lens (383) from the fovea (382). The optical axis (372) may be a straight line passing through the pupil (365) from the center of rotation of each (or at least one) of the two eyes (300-1, 300-2). For example, the visual axis (371) is inclined about 5 degrees inwardly horizontally and about 1 degree downwardly vertically from the optical axis (372) in the direction of the center of the pupil (365). The one or more parameters may represent the position of each (or at least one) of the two eyes (300-1, 300-2), the radius of each (or at least one) of the two eyes (300-1, 300-2), the size of the iris (363), the radius of the iris (363), the radius of the pupil (365), the center position of the pupil (365), and the position of at least one glint (367). In one embodiment, the gaze direction (390) of the user (300) may correspond to the direction of the visual axis (371). In one embodiment, tracking the gaze of the user (300) may include acquiring the gaze direction (390) of the user (300). However, the present disclosure is not limited to the above embodiments.

[0074] FIG. 4A illustrates an example of a screen for calibrating a user's gaze according to one embodiment. FIG. 4B illustrates an example of an error between an object and gaze according to one embodiment.

[0075] FIGS. 4A and 4B may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIGS. 2A and 2B. FIGS. 4A and 4B may be described with reference to FIGS. 3A, 3B, and 3C. The operations described with reference to FIGS. 4A and 4B may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0076] In one embodiment, the electronic device (101) can perform eye calibration to correct data representing a gaze identified through an eye model of the user (300). In FIG. 4A, the electronic device (101) can display virtual reference objects (420, 430, 440, 450, 460) whose positions (coordinates) are known on a 3D (three-dimensional) screen (410) while the user (300) is wearing the electronic device (101). For example, the electronic device (101) can display virtual reference objects (420, 430, 440, 450, 460) on the 3D screen (410) by executing an application for eye calibration. In one embodiment, the electronic device (101) may sequentially display reference objects (420, 430, 440, 450, 460) on a 3D screen (410) for eye calibration.

[0077] For example, the electronic device (101) can acquire images of the two eyes (300-1, 300-2) through the cameras (240-3, 240-4) while reference objects (420, 430, 440, 450, 460) are sequentially displayed on the 3D screen (410). For example, the electronic device (101) can acquire images of the two eyes (300-1, 300-2) through the cameras (240-3, 240-4) when the reference object (420) is displayed at a first point in time on the 3D screen (410). For example, the electronic device (101) can acquire images of the two eyes (300-1, 300-2) through the cameras (240-3, 240-4) when a reference object (430) is displayed at a second viewpoint that does not overlap with the first viewpoint on the 3D screen (410). For example, the electronic device (101) can acquire images of the two eyes (300-1, 300-2) through the cameras (240-3, 240-4) when a reference object (440) is displayed at a third viewpoint that does not overlap with the first viewpoint and the second viewpoint on the 3D screen (410).

[0078] For example, the electronic device (101) can obtain images of the two eyes (300-1, 300-2) through the cameras (240-3, 240-4) when a reference object (450) is displayed at a fourth viewpoint that does not overlap with the first viewpoint, the second viewpoint, and the third viewpoint on the 3D screen (410). For example, the electronic device (101) can obtain images of the two eyes (300-1, 300-2) through the cameras (240-3, 240-4) when a reference object (460) is displayed at a fifth viewpoint that does not overlap with the first viewpoint, the second viewpoint, the third viewpoint, and the fourth viewpoint on the 3D screen (410).

[0079] For example, the electronic device (101) can identify the gazes (423, 433, 443, 453, 463) based on the images of the two eyes (300-1, 300-2) acquired when reference objects (420, 430, 440, 450, 460) are sequentially displayed on the 3D screen (410). For example, the electronic device (101) can identify the gazes (423, 433, 443, 453, 463) based on the feature information (e.g., the position of the glint (367)) of the images of the two eyes (300-1, 300-2) and the eye model. For example, the electronic device (101) can identify, based on the eye model, the lines of sight (423, 433, 443, 453, 463) (or visual axes (371)) corresponding to the optical axes (372) according to the images of the two eyes (300-1, 300-2).

[0080] For example, the electronic device (101) can acquire images of two eyes (300-1, 300-2) whose gazes (423, 433, 443, 453, 463) are located within selection areas (421, 431, 441, 451, 461) of the reference objects (420, 430, 440, 450, 460) while reference objects (420, 430, 440, 450, 460) are sequentially displayed on the 3D screen (410). For example, the selection areas (421, 431, 441, 451, 461) can represent areas where the user (300) is judged to have looked at the reference objects.

[0081] For example, when the gaze of the user (300) is located within the selection area of ​​a specific object, the electronic device (101) can determine that the user (300) is looking at the specific object. In FIG. 4A, the selection areas (421, 431, 441, 451, 461) are illustrated as being rectangular, but this is merely an example. For example, the selection areas (421, 431, 441, 451, 461) may have various shapes (e.g., circular, oval, polygonal). In FIG. 4A, the selection areas (421, 431, 441, 451, 461) are illustrated as being larger than the reference objects (420, 430, 440, 450, 460), but this is merely an example. For example, the sizes of the selection areas (421, 431, 441, 451, 461) may be substantially the same as the sizes of the reference objects (420, 430, 440, 450, 460). However, the present disclosure is not limited to the above embodiments.

[0082] For example, the electronic device (101) can identify errors (425, 435, 445, 455, 465) between the center positions of reference objects (420, 430, 440, 450, 460) and the lines of sight (423, 433, 443, 453, 463). In one embodiment, the errors (425, 435, 445, 455, 465) can be three-dimensional errors. For example, the errors (425, 435, 445, 455, 465) can include errors in the x-axis direction, errors in the y-axis direction, and errors in the z-axis direction. For example, in FIG. 4B, the error (445) may include an error in the x-axis direction (471), an error in the y-axis direction (473), and an error in the z-axis direction. For example, the errors (425, 435, 445, 455, 465) may represent a distance difference between the center positions of the reference objects (420, 430, 440, 450, 460) and the lines of sight (423, 433, 443, 453, 463). However, the present disclosure is not limited to the above-described embodiment.

[0083] In one embodiment, the electronic device (101) can compensate for the lines of sight (423, 433, 443, 453, 463) based on the errors (425, 435, 445, 455, 465). For example, the electronic device (101) can compensate for the angle (or kappa angle) difference between the visual axis (371) and the optical axis (372) based on the errors (425, 435, 445, 455, 465), thereby compensating for the lines of sight (423, 433, 443, 453, 463).

[0084] For example, the electronic device (101) can identify parameters for correcting the lines of sight (423, 433, 443, 453, 463) based on the errors (425, 435, 445, 455, 465). For example, the electronic device (101) can correct the lines of sight (423, 433, 443, 453, 463) based on the identified parameters. For example, the electronic device (101) can set (or determine) parameters for each (or at least one) of the positions of the 3D screen (410). For example, the electronic device (101) can set parameters for each (or at least one) of the locations where reference objects (420, 430, 440, 450, 460) are displayed on the 3D screen (410). Hereinafter, the parameters for correcting the gaze identified through the eye model may be referred to as error correction parameters.

[0085] FIG. 5 illustrates an example of a screen displayed by a wearable device according to one embodiment.

[0086] FIG. 5 may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIG. 2A and FIG. 2B. FIG. 5 may be described with reference to FIG. 3A, FIG. 3B, FIG. 3C, FIG. 4A and FIG. 4B. The operations described with reference to FIG. 5 may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0087] In one embodiment, the electronic device (101) can display a 3D screen. For example, the electronic device (101) can display a 3D screen after performing eye calibration. In one embodiment, the 3D screen can be configured to provide media content to enhance immersion in a virtual world based on augmented reality (AR), virtual reality environment (VR), mixed reality (MR), and / or extended reality (XR).

[0088] In one embodiment, the 3D screen may include a background area (510) (or, wallpaper). In one embodiment, the background area (510) may also be referred to as a wallpaper. In one embodiment, the background area (510) may be distinguished from the application's execution screen (520). In one embodiment, the background area (510) may be an area where a wallpaper is displayed. In one embodiment, the wallpaper may be an image determined according to default settings or an image determined according to the user's (300) settings.

[0089] In one embodiment, the application's execution screen (520) may be displayed in a portion of the 3D screen. However, the present disclosure is not limited to the above embodiment. In one embodiment, the application's execution screen (520) may be displayed in the entire 3D screen. In one embodiment, the 3D screen may include at least one visual object (531, 532, 533, 534, 535, 536).

[0090] In one embodiment, the visual objects (531, 532, 533) may represent a background area (510) or a real environment (e.g., current weather). For example, the visual objects (531, 532, 533) may be virtual objects representing the background area (510) and / or the real environment (e.g., current weather), such as snow, rain, flower petals, and / or leaves. However, the present disclosure is not limited to the above embodiments.

[0091] In one embodiment, the visual object (534) may be a panel. For example, the visual object (534) may include icons of applications. For example, the visual object (534) may include icons for controlling the electronic device (101). For example, the icons for controlling the electronic device (101) may include icons for changing the mode of the electronic device (101) (e.g., sound mode or vibration mode). However, the present disclosure is not limited to the above embodiment.

[0092] In one embodiment, the visual object (535, 536) may be a widget of the application. For example, the widget may be a graphical user interface (GUI) for controlling some function of the application associated with the widget and / or displaying information.

[0093] In one embodiment, while providing VR, AR, and / or MR-based media content to a user (300) wearing an electronic device (101), a gaze error may occur. For example, a gaze error may occur after eye calibration of the user (300).

[0094] For example, a gaze error may be caused by slippage. In one embodiment, slippage may be caused by a movement of the user (300) (e.g., a head movement of the user (300)). In one embodiment, slippage may refer to or correspond to a change in the distances (321, 322) (or eye relief) between the displays (250-1, 250-2) and the two eyes (300-1, 300-2) measured when the user (300) wears the electronic device (101). In one embodiment, slippage may refer to a change in the eye reliefs compared to the time when the user (300) wears the electronic device (101).

[0095] However, the present disclosure is not limited to the above-described embodiments. For example, slippage may be caused by factors (or environments) other than the movement of the user (300) (e.g., head movement of the user (300)). For example, slippage may be caused by factors (or environments) that change (or reduce) the contact state (or frictional force) between the user (300) and the electronic device (101). For example, slippage may more easily occur in a situation where the contact state (or frictional force) between the user (300) and the electronic device (101) worsens (or reduces) due to sweat of the user (300) or environmental changes (e.g., humidity changes and / or temperature changes).

[0096] For example, a gaze error may occur based on the user (300) re-wearing the electronic device (101). Here, the user (300) re-wearing the electronic device (101) may mean that the user (300) takes off the electronic device (101) for a moment and then immediately puts it on. For example, when the user (300) re-wears the electronic device (101), the eye reliefs after re-wearing the electronic device (101) and the eye reliefs before re-wearing the electronic device (101) may be different from each other.

[0097] For example, a gaze error may occur due to a change in the state of the user (300) after eye calibration. For example, a change in the state of the user (300) may change the characteristic information of the user (300). For example, a change in the state of the user (300) may include wearing lenses and / or glasses. For example, wearing lenses and / or glasses may include being worn by the user (300) (e.g., glasses being placed on the nose of the user (300), lenses being placed on the eyes (300-1, 300-2) of the user (300). For example, wearing lenses or glasses may include being attached to the electronic device (101) (e.g., lenses being attached to the displays (250-1, 250-2)).

[0098] For example, a change in the state of the user (300) may include or correspond to a change that changes eye reliefs (e.g., a wound around the eye, a change in thickness around the eye).

[0099] For example, when a gaze error occurs, the time it takes for the electronic device (101) to determine that the user's (300) gaze is directed toward a visual object (531, 532, 533, 534, 535, 536) may increase. For example, when a gaze error occurs, the electronic device (101) may determine that the user's (300) gaze is directed toward an object different from the visual object (531, 532, 533, 534, 535, 536) that the user (300) is actually looking at.

[0100] Accordingly, the electronic device (101) can perform a session (e.g., eye calibration) to reduce gaze errors when they occur. However, eye calibration may require a considerable amount of time. Furthermore, as the electronic device (101) displays the screen of an application for eye calibration to perform eye calibration, the provision of media content being provided to the user (300) may be interrupted.

[0101] Therefore, a separate method may be required to reduce gaze error without disturbing the reality (e.g., VR, AR, and / or XR) in which the user (300) is immersed.

[0102] Hereinafter, with reference to FIGS. 6 to 7b, a method for reducing gaze error without disturbing the reality (e.g., VR, AR, and / or XR) in which a user (300) is immersed may be described.

[0103] FIG. 6 illustrates a block diagram of a wearable device according to one embodiment. FIG. 7A illustrates an example of a movement path of a visual object for user calibration in a wearable device according to one embodiment. FIG. 7B illustrates an example of a movement path of a visual object for user calibration in a wearable device according to one embodiment.

[0104] FIGS. 6, 7A, and 7B may be described with reference to the electronic device (101) of FIG. 1 and / or the wearable device (200) of FIGS. 2A and 2B. FIGS. 6, 7A, and 7B may be described based on the eye model described with reference to FIGS. 3A, 3B, and 3C. FIGS. 6, 7A, and 7B may be described based on error correction parameters obtained through user calibration described with reference to FIGS. 4A and 4B.

[0105] The operations described with reference to FIG. 6 may be executed by the electronic device (101) and / or the processor (120) of the electronic device (101).

[0106] In FIG. 6, the electronic device (101) may include a processor (120), a memory (130), a display system (660), a sensor (676), and a camera system (680). In one embodiment, the processor (120) of FIG. 6 may correspond to the processor (120) of FIG. 1. In one embodiment, the memory (130) of FIG. 6 may correspond to the memory (130) of FIG. 1. In one embodiment, the display system (660) of FIG. 6 may correspond to the display module (160) of FIG. 1. In one embodiment, the display system (660) of FIG. 6 may correspond to the displays (250-1, 250-2) of FIGS. 2A, 2B, and 3A. In one embodiment, the sensor (676) of FIG. 6 may correspond to the sensor module (176) of FIG. 1. In one embodiment, the sensor (676) of FIG. 6 may be an inertial measurement unit (IMU). In one embodiment, the camera system (680) of FIG. 6 may correspond to the camera module (180) of FIG. 1. In one embodiment, the camera system (680) of FIG. 6 may correspond to the cameras (240-1, 240-2, 240-3, 240-4) of FIG. 2a, FIG. 2b, and FIG. 3a.

[0107] In one embodiment, the memory (130) may include a gaze determination unit (611), a gaze accuracy detection unit (613), a background area determination unit (615), a virtual object generation unit (617), and an auto-calibration unit (619).

[0108] In one embodiment, the gaze determination unit (611) can identify the gaze through images of the two eyes (300-1, 300-2) of the user (300) obtained through the camera system (680). In one embodiment, the gaze determination unit (611) can obtain feature information (e.g., the position of the glint (367)) through images of the two eyes (300-1, 300-2) of the user (300) obtained through the camera system (680). In one embodiment, the gaze determination unit (611) can identify the gaze corresponding to the feature information (e.g., the position of the glint (367)) based on an eye model. In one embodiment, the gaze determination unit (611) can adjust the identified gaze based on an error correction parameter. Hereinafter, the identification of the gaze by the gaze determination unit (611) may include identification of the adjusted gaze based on an error correction parameter.

[0109] In one embodiment, the gaze determination unit (611) can identify a gaze on a 3D screen displayed through the display system (660). In one embodiment, the gaze determination unit (611) can identify a gaze on a background area (510). In one embodiment, the gaze determination unit (611) can identify a gaze toward a visual object (531, 532, 533, 534, 535, 536) displayed on the 3D screen. However, the present disclosure is not limited to the above embodiment. In one embodiment, the gaze determination unit (611) can identify a gaze on a 3D screen of an application displayed as a full screen on the display system (660). For example, the fact that the 3D screen of the application is displayed as a full screen may include that the background area (510) is not included in the 3D screen displayed through the display system (660). For example, displaying the 3D screen of an application in full screen may include displaying only the execution screen of the application on the 3D screen displayed through the display system (660).

[0110] In one embodiment, the gaze determination unit (611) can identify the gaze when an interaction is identified. In one embodiment, the gaze determination unit (611) can identify the gaze when an interaction is identified for one of the visual objects (531, 532, 533, 534, 535, 536). For example, the interaction may include a gesture (e.g., hand-pinch) for selecting one visual object. However, the present disclosure is not limited to the above embodiment.

[0111] In one embodiment, the gaze determination unit (611) can identify a visual object toward which the identified gaze is directed. In one embodiment, the gaze determination unit (611) can identify an object having a selection area in which the identified gaze is located. In one embodiment, the gaze determination unit (611) can identify a visual object toward which the identified gaze is directed when an interaction (e.g., a gesture for selecting an object) is identified. In one embodiment, the gaze determination unit (611) can identify an object having a selection area in which the identified gaze is located when the interaction is identified.

[0112] In one embodiment, the gaze accuracy detection unit (613) can identify a gaze error. In one embodiment, the gaze accuracy detection unit (613) can identify a gaze error based on the gaze and visual object identified by the gaze determination unit (611). For example, the gaze accuracy detection unit (613) can identify the difference between the gaze (or the focus position of the gaze) and the center position of the visual object as a gaze error. For example, the gaze error can be a three-dimensional error. For example, the gaze error can include an error in the x-axis direction, an error in the y-axis direction, and an error in the z-axis direction.

[0113] In one embodiment, the gaze accuracy detection unit (613) can identify a gaze error of a gaze directed toward a visual object that satisfies a specified condition. For example, the specified condition may be related to the size of the visual object. For example, the gaze accuracy detection unit (613) can identify a gaze error of a gaze directed toward a visual object that is smaller than a specified size. For example, the specified condition may be related to the display time of the visual object. For example, the gaze accuracy detection unit (613) can identify a gaze error of a gaze directed toward a visual object displayed within a specified display time. However, the present disclosure is not limited to the above-described embodiment.

[0114] In one embodiment, the gaze accuracy detection unit (613) may store information about gaze error. For example, the information about gaze error may include the position of a visual object, the focus position of the gaze, the gaze error, and / or the interaction time. In one embodiment, the interaction time may be the time taken for an interaction with a visual object (e.g., a gesture to select the visual object). In one embodiment, the interaction time may be the time taken until both the gaze directed toward the visual object and the interaction to select the visual object are identified after the visual object is displayed. In one embodiment, if the gaze is located within the selection area of ​​the visual object, the gaze may be evaluated as directed toward the visual object.

[0115] In one embodiment, the gaze accuracy detection unit (613) may store information about gaze errors for each designated area of ​​a 3D screen displayed through the display system (660). In one embodiment, the designated areas of the 3D screen may be areas that do not overlap each other. The information about the gaze errors for each (or at least one) of the designated areas of the 3D screen may include information about gaze errors identified through visual objects displayed in each of the designated areas (e.g., average gaze error (or representative gaze error), average interaction time (or representative interaction time)). However, the present disclosure is not limited to the above embodiment. In one embodiment, the representative gaze error may represent a frequency value of gaze errors identified through visual objects displayed in each (or at least one) of the designated areas. In one embodiment, the representative interaction time may represent a frequency value of interaction times identified through visual objects displayed in each (or at least one) of the designated areas.

[0116] In one embodiment, the gaze accuracy detection unit (613) can determine whether eye calibration is necessary.

[0117] For example, the gaze accuracy detection unit (613) can determine whether eye calibration is necessary based on the gaze error. For example, the gaze accuracy detection unit (613) can determine that eye calibration is necessary based on at least one gaze error exceeding a reference gaze error. For example, the gaze accuracy detection unit (613) can determine that eye calibration is necessary based on the gaze error (or average gaze error) (or representative gaze error) of at least one region among a plurality of regions of the 3D screen exceeding the reference gaze error.

[0118] For example, the gaze accuracy detection unit (613) can determine whether eye calibration is necessary based on the interaction time. For example, the gaze accuracy detection unit (613) can determine that eye calibration is necessary based on at least one interaction time exceeding a reference interaction time. For example, the gaze accuracy detection unit (613) can determine that eye calibration is necessary based on the interaction time (or average interaction time) (or representative interaction time) of at least one region among a plurality of regions of the 3D screen exceeding the reference interaction time.

[0119] For example, the gaze accuracy detection unit (613) may determine whether eye calibration is necessary based on slippage (and / or re-wearing). In one embodiment, the gaze accuracy detection unit (613) may identify the occurrence of slippage (and / or re-wearing) of the electronic device (101) via the sensor (676) and / or the camera system (680). For example, the gaze accuracy detection unit (613) may identify the occurrence of slippage based on the relative movement of the electronic device (101) from the user (300) via the sensor (676) and / or the camera system (680). For example, the gaze accuracy detection unit (613) may identify, via the sensor (676) and / or the camera system (680), that the electronic device (101) is put back on the user (300) within a predetermined time after being taken off from the user (300). However, the present disclosure is not limited to the above embodiments.

[0120] For example, the gaze accuracy detection unit (613) may determine that eye calibration is necessary based on the identification of slippage (and / or re-wearing).

[0121] In one embodiment, the gaze accuracy detection unit (613) may determine whether to perform eye calibration based on whether eye calibration is determined to be necessary.

[0122] For example, the gaze accuracy detection unit (613) may determine to postpone performing eye calibration when only the execution screen of the application is displayed on the 3D screen displayed through the display system (660) (or, when the 3D screen of the application is displayed in full screen). For example, the gaze accuracy detection unit (613) may determine to postpone performing eye calibration when the gaze of the user (300) is on the execution screen of the application displayed through the display system (660) (or, when an interaction with the application is identified). However, the present disclosure is not limited to the above embodiment. For example, the gaze accuracy detection unit (613) may determine to perform eye calibration even when only the execution screen of the application is displayed on the 3D screen displayed through the display system (660) (or, when the 3D screen of the application is displayed in full screen).

[0123] For example, the gaze accuracy detection unit (613) may determine that eye calibration is performed when at least a portion of the background area (510) is included in the 3D screen displayed through the display system (660). For example, the gaze accuracy detection unit (613) may determine that eye calibration is performed when the gaze of the user (300) is located in an area other than the execution screen of the application displayed through the display system (660). However, the present disclosure is not limited to the above-described embodiment.

[0124] For example, the gaze accuracy detection unit (613) may determine that eye calibration is to be performed when at least one gaze error exceeds a threshold gaze error. For example, the gaze accuracy detection unit (613) may determine that eye calibration is to be performed when the gaze error (or average gaze error) (or representative gaze error) of at least one area among a plurality of areas of the 3D screen exceeds a threshold gaze error.

[0125] For example, the gaze accuracy detection unit (613) may determine that eye calibration is to be performed if at least one interaction time exceeds a threshold interaction time. For example, the gaze accuracy detection unit (613) may determine that eye calibration is to be performed if the interaction time (or average interaction time) (or representative interaction time) of at least one area among a plurality of areas of the 3D screen exceeds the threshold interaction time.

[0126] For example, the gaze accuracy detection unit (613) may determine that eye calibration is to be performed based on the identification of slippage (and / or re-wearing).

[0127] In one embodiment, the background area determination unit (615) can identify the background area (510). In one embodiment, the background area determination unit (615) can identify the background area (510) based on the determination that eye calibration is performed.

[0128] In one embodiment, the background area determination unit (615) can identify an area other than the application execution screen (520) on the 3D screen as the background area (510).

[0129] In one embodiment, the background area determination unit (615) can identify locations where a virtual object is to be displayed in the background area (510). In one embodiment, the background area determination unit (615) can identify (or select) locations where a virtual object is to be displayed based on line-of-sight errors. In one embodiment, the background area determination unit (615) can identify (or select) locations where a line-of-sight error is greater than a specified line-of-sight error among locations where a line-of-sight error is identified. In one embodiment, the background area determination unit (615) can identify (or select) locations where a virtual object is to be displayed in descending order of line-of-sight errors in the background area (510). However, the present disclosure is not limited to the above embodiment. In one embodiment, when only the execution screen of the application is displayed on the 3D screen (or, when the 3D screen of the application is displayed in full screen), the background area determination unit (615) can identify (or select) locations where virtual objects are to be displayed in the order of highest line-of-sight errors on the execution screen of the application (e.g., execution screen (520)).

[0130] According to an embodiment, the background area determination unit (615) may identify areas where virtual objects are to be displayed. In one embodiment, the background area determination unit (615) may identify (or select) areas where virtual objects are to be displayed based on gaze errors. In one embodiment, the background area determination unit (615) may identify (or select) areas where virtual objects are to be displayed based on locations where gaze errors are identified and have a gaze error greater than or equal to a specified gaze error. For example, the areas where virtual objects are to be displayed may be some of a plurality of areas of a 3D screen. For example, the areas where virtual objects are to be displayed may be some of a plurality of areas of a 3D screen that are pre-partitioned. For example, the areas where virtual objects are to be displayed may be areas where the gaze errors of the plurality of areas of the 3D screen are greater than or equal to a specified gaze error. For example, the gaze error of the plurality of areas may be an average gaze error (or representative gaze error) of the gaze errors of locations included in the area. However, the present disclosure is not limited to the above embodiment. For example, the areas in which virtual objects are to be displayed may be areas of clusters of locations having a gaze error greater than or equal to the gaze error (or clusters of locations within a specified distance from the center positions of the clusters).

[0131] In one embodiment, the virtual object generation unit (617) can generate a virtual object. In one embodiment, the virtual object generation unit (617) can generate a virtual object based on the determination that eye calibration is performed.

[0132] In one embodiment, the virtual object generation unit (617) may generate a virtual object based on the background area (510) (and / or wallpaper displayed in the background area (510)).

[0133] In one embodiment, the virtual object generation unit (617) may generate a virtual object based on at least one word representing the background area (510) (and / or wallpaper displayed in the background area (510). For example, the at least one word may be obtained through a prompt generator. In one embodiment, the prompt generator may be an artificial intelligence (AI) model (e.g., a stable diffusion model) capable of converting an input image into text. However, the present disclosure is not limited to the above embodiment.

[0134] In one embodiment, the virtual object generation unit (617) may obtain a prompt for generating a virtual object based on the background area (510) (and / or wallpaper displayed in the background area (510). In one embodiment, the prompt may include data for guiding the generation of a virtual object based on an input image. In one embodiment, the prompt may be a work instruction for a generative AI model. In one embodiment, the prompt may be a set of words (or a sentence including words) for generating an image for a virtual object from an input image through the generative AI model.

[0135] In one embodiment, the virtual object generation unit (617) may identify a prompt for generating a virtual object based on a situation depicted by the input image. For example, the situation depicted by the input image may be classified based on a method of depicting objects included in the background area (510) (or wallpaper of the background area (510)) (e.g., landscape, portrait, still life), a design pattern of the background area (510) (or wallpaper of the background area (510)), a type of objects included in the background area (510) (e.g., people, animals, plants, objects), a relationship between objects (e.g., friends, family), and / or a time depicted by the background area (510) (or wallpaper of the background area (510)) (e.g., morning, am, afternoon, evening, dawn).

[0136] In one embodiment, the virtual object generation unit (617) can obtain a virtual object by inputting a prompt corresponding to the background area (510) (or wallpaper of the background area (510)) into the generative AI model. In one embodiment, the generative AI model can include a plurality of parameters related to a neural network having a structure based on an encoder and a decoder, such as a transformer.

[0137] In one embodiment, the virtual object generation unit (617) may identify a prompt for generating a virtual object based on some of the visual objects included in the background area (510) (or the wallpaper of the background area (510)). In one embodiment, some of the visual objects may be visual objects having a selection area where the identified gaze is located.

[0138] According to an embodiment, when only the execution screen of the application is displayed on the 3D screen (or, when the 3D screen of the application is displayed in full screen), the virtual object generation unit (617) may generate a virtual object based on the execution screen of the application (e.g., the execution screen (520)). In one embodiment, when only the execution screen of the application is displayed on the 3D screen (or, when the 3D screen of the application is displayed in full screen), the virtual object generation unit (617) may generate a virtual object based on at least one word representing the execution screen of the application. For example, the at least one word may be obtained through a prompt generator.

[0139] In one embodiment, the auto calibration unit (619) can display the virtual object generated by the virtual object generation unit (617) at locations identified by the background area determination unit (615).

[0140] In one embodiment, the auto-calibration unit (619) may display a virtual object so that the virtual object moves through the identified locations. For example, the auto-calibration unit (619) may display a virtual object so that the virtual object moves through the identified locations in order of highest line-of-sight error. For example, in FIG. 7A, the auto-calibration unit (619) may display a virtual object (710) along a path (711, 713) connecting some visual objects (532, 533, 531) in order of highest line-of-sight error among the visual objects (531, 532, 533, 534, 535, 536). However, the present disclosure is not limited to the above embodiment. For example, the auto-calibration unit (619) may display a virtual object so that the virtual object moves along the shortest path (or path) between identified locations. For example, the auto-calibration unit (619) may display a virtual object (710) so that the virtual object moves along a path that does not cross the application's execution screen (520).

[0141] In one embodiment, the auto-calibration unit (619) may display a virtual object so that the virtual object is displayed in an identified area (or an area identified by the background area determination unit (615)) among a plurality of areas of the 3D screen. For example, the auto-calibration unit (619) may display the virtual object in an area with a high line-of-sight error among the plurality of areas of the 3D screen. For example, in FIG. 7B, the auto-calibration unit (619) may display the virtual object (710) in an area with a high line-of-sight error (e.g., area (721)) among areas (721, 723, 725, 727). For example, the auto-calibration unit (619) can display a virtual object (710) along a path (731) connecting visual objects (532, 534) in the order of highest line-of-sight error among visual objects (532, 534) within a region (e.g., region (721)) with high line-of-sight error.

[0142] For example, the auto-calibration unit (619) can display virtual objects in multiple areas of the 3D screen in descending order of the highest line-of-sight error. For example, in FIG. 7B, the auto-calibration unit (619) can display a virtual object (710) in an area with the highest line-of-sight error (e.g., area (721)), and then display the virtual object (710) in an area with the next highest line-of-sight error (e.g., area (723)).

[0143] In one embodiment, the auto-calibration unit (619) can identify a gaze error based on the gaze of the user (300) looking at the virtual object (710).

[0144] In one embodiment, the auto-calibration unit (619) can identify a gaze error based on the gaze of the user (300) looking at the virtual object (710) and the location where the virtual object (710) is displayed. For example, the auto-calibration unit (619) can identify the difference between the gaze (or the focus location of the gaze) and the center location of the virtual object (710) as the gaze error. For example, the auto-calibration unit (619) can identify the difference between the center location of a virtual object (710) moving along a path (711, 713) connecting some visual objects (532, 533, 531) and the gaze (or the focus location of the gaze) as the gaze error. For example, the auto-calibration unit (619) can identify the difference between the line of sight (or the focus position of the line of sight) and the center position of the virtual object (710) when the virtual object (710) moving along the path (711, 713) is located at the position of some visual objects (532, 533, 531) as a line of sight error.

[0145] In one embodiment, the auto-calibration unit (619) may identify the difference between the gaze (or the focus position of the gaze) and the center position of the virtual object (710) as the gaze error based on the movement of the gaze of the user (300) in response to the direction and / or speed of the movement of the virtual object (710). In one embodiment, the auto-calibration unit (619) may identify the difference between the gaze (or the focus position of the gaze) and the center position of the virtual object (710) as the gaze error while the gaze of the user (300) is directed toward the virtual object (710). The gaze of the user (300) directed toward the virtual object (710) may mean that the gaze of the user (300) is located within a selection area of ​​the virtual object (710). However, the present disclosure is not limited to the above embodiment.

[0146] In one embodiment, the auto-calibration unit (619) may store information about a gaze error based on the difference between the gaze (or the focus position of the gaze) and the center position of the virtual object (710). For example, the information about the gaze error may include the position of the virtual object (710), the focus position of the gaze, and / or the gaze error. For example, the information about the gaze error may be set for each (or at least one) of the designated areas of the 3D screen. For example, the information about the gaze error for each (or at least one) of the designated areas of the 3D screen may include information about gaze errors identified through the virtual object (710) displayed in each (or at least one) of the designated areas (e.g., an average gaze error (or representative gaze error).

[0147] In one embodiment, the auto-calibration unit (619) may identify and store error correction parameters for reducing the gaze error of each (or at least one) of the locations where the gaze error is identified. In one embodiment, the auto-calibration unit (619) may identify and store error correction parameters for each (or at least one) of the designated areas of the 3D screen based on the gaze error for each (or at least one) of the designated areas of the 3D screen.

[0148] Thereafter, the gaze determination unit (611) can identify the gaze on the 3D screen displayed through the display system (660) using the error correction parameter obtained based on the virtual object (710).

[0149] For example, the gaze determination unit (611) can identify the gaze through images of the two eyes (300-1, 300-2) of the user (300) obtained through the camera system (680). For example, the gaze determination unit (611) can obtain feature information (e.g., the position of the glint (367)) through images of the two eyes (300-1, 300-2) of the user (300) obtained through the camera system (680). For example, the gaze determination unit (611) can identify the gaze corresponding to the feature information (e.g., the position of the glint (367)) based on an eye model. For example, the gaze determination unit (611) can adjust the identified gaze based on an error correction parameter. For example, the gaze determination unit (611) can correct (or adjust) the gaze error of the identified gaze based on the error correction parameter. For example, the gaze determination unit (611) can correct (or adjust) the gaze error of the identified gaze by adjusting the value of the determined parameter of the identified gaze by the value for error correction.

[0150] For example, the error correction parameters for adjusting the identified gaze may include error correction parameters obtained through eye calibration based on FIG. 4A and error correction parameters obtained based on a virtual object (710). However, the present disclosure is not limited to the above-described embodiment.

[0151] According to an embodiment, the electronic device (101) may determine whether to maintain the error correction parameter obtained based on the virtual object (710). For example, the electronic device (101) may determine whether to maintain the error correction parameter obtained based on the virtual object (710) based on the electronic device (101) being removed from the user (300).

[0152] For example, the electronic device (101) may determine to maintain the error correction parameters acquired based on the virtual object (710) based on the electronic device (101) being temporarily removed from the user (300) (or based on the electronic device (101) being re-worn on the user (300).

[0153] For example, the electronic device (101) may decide to discard (or remove) (or delete) the error correction parameter obtained based on the virtual object (710) based on the state in which the electronic device (101) is removed from the user (300) for a reference time or longer. Discarding (or removing) (or deleting) the error correction parameter obtained based on the virtual object (710) may indicate that when the user (300) later wears the electronic device (101), the identification of the user's (300) gaze is adjusted based on the error correction parameter obtained through eye calibration based on FIG. 4A.

[0154] According to an embodiment, the electronic device (101) may merge an error correction parameter obtained based on the virtual object (710) with an error correction parameter obtained through eye calibration based on FIG. 4A. For example, merging an error correction parameter obtained based on the virtual object (710) with an error correction parameter obtained through eye calibration based on FIG. 4A may include updating the eye calibration based on FIG. 4A based on the error correction parameter obtained based on the virtual object (710).

[0155] For example, if the gaze error identified based on the virtual object (710) is less than or equal to a reference error, the electronic device (101) may merge the error correction parameter obtained based on the virtual object (710) with the error correction parameter obtained through eye calibration based on FIG. 4A. However, the present disclosure is not limited to the above-described embodiment.

[0156] As described above, when a gaze error occurs, the electronic device (101) can acquire error correction parameters for reducing the gaze error without switching to the screen of an application for eye calibration. Accordingly, the electronic device (101) can improve the user experience related to gaze without interrupting the provision of media content to the user (300).

[0157] FIG. 8 illustrates an operation performed by a wearable device according to one embodiment.

[0158] FIG. 8 can be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIG. 2a and FIG. 2b.

[0159] In FIG. 8, at operation 810, the electronic device (101) can identify a gaze error. In one embodiment, the electronic device (101) can identify a gaze error based on the gaze of the user (300) and a visual object. For example, the electronic device (101) can identify a difference between the gaze (or the focus position of the gaze) and the center position of the visual object toward the gaze as a gaze error. For example, the gaze error can be a three-dimensional error. For example, the gaze error can include an error in the x-axis direction, an error in the y-axis direction, and an error in the z-axis direction.

[0160] In operation 820, the electronic device (101) may store the gaze error. In one embodiment, the electronic device (101) may store each (or at least one) of the gaze errors for designated areas of the corresponding 3D screen. In one embodiment, the electronic device (101) may map and store each (or at least one) of the gaze errors to corresponding areas. In one embodiment, the designated areas of the 3D screen may be areas that do not overlap each other. In one embodiment, the electronic device (101) may store the gaze error after the eye calibration according to FIG. 4A. In one embodiment, the electronic device (101) may discard (or remove) the gaze error obtained before the eye calibration according to FIG. 4A. In one embodiment, the electronic device (101) may store the gaze error obtained after the current wearing of the electronic device (101). In one embodiment, the electronic device (101) may discard (or remove) gaze errors acquired prior to wearing the current electronic device (101).

[0161] FIG. 9 illustrates an operation performed by a wearable device according to one embodiment.

[0162] FIG. 9 may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIG. 2A and FIG. 2B. The operations of FIG. 9 may be performed after the operations of FIG. 8. However, the present disclosure is not limited to the above-described embodiments.

[0163] In FIG. 9, at operation 910, the electronic device (101) can identify an event for correction of gaze error.

[0164] For example, the electronic device (101) may identify an event for correcting the gaze error based on the gaze error. For example, the electronic device (101) may identify an event for correcting the gaze error based on the gaze error exceeding a reference gaze error.

[0165] For example, the electronic device (101) may identify an event for correcting a gaze error based on an interaction time. For example, the electronic device (101) may identify an event for correcting a gaze error based on an interaction time exceeding a reference interaction time.

[0166] For example, the electronic device (101) can identify an event for correction of gaze error based on the occurrence of slippage.

[0167] In operation 920, the electronic device (101) may display a visual object for correction based on gaze error.

[0168] In one embodiment, the electronic device (101) may display visual objects corresponding to the background area (510) (and / or wallpaper displayed in the background area (510)) for correction based on gaze error. However, the present disclosure is not limited to the above embodiment. In one embodiment, the electronic device (101) may display visual objects corresponding to some of the visual objects included in the background area (510) (or wallpaper of the background area (510). In one embodiment, some of the visual objects may be visual objects having a selection area where the identified gaze is located.

[0169] In one embodiment, the electronic device (101) may display a visual object in a background area (510) of a 3D screen for correction based on gaze error. In one embodiment, the electronic device (101) may display a visual object in locations (or areas) in the background area (510) that have a gaze error greater than a specified gaze error for correction based on gaze error.

[0170] In operation 930, the electronic device (101) can correct a gaze error based on a gaze toward a visual object. In one embodiment, the electronic device (101) can identify a gaze error based on the gaze of the user (300) looking at the visual object and the location at which the visual object is displayed. In one embodiment, the electronic device (101) can identify and store an error correction parameter for correcting (or reducing) the gaze error of each (or at least one) of the locations at which the gaze error is identified. In one embodiment, the electronic device (101) can correct the gaze of the user (300) based on the error correction parameter.

[0171] FIG. 10 illustrates an operation performed by a wearable device according to one embodiment.

[0172] FIG. 10 may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIG. 2A and FIG. 2B. The operations of FIG. 10 may be included in operation 920 of FIG. 9. However, the present disclosure is not limited to the above embodiments.

[0173] In FIG. 10, at operation 1010, the electronic device (101) may identify locations at which visual objects are to be displayed. In one embodiment, the electronic device (101) may identify a background area (510) as an area including locations at which visual objects are to be displayed.

[0174] In one embodiment, the electronic device (101) can identify locations at which to display a visual object based on gaze errors. In one embodiment, the electronic device (101) can identify locations with a gaze error greater than a specified gaze error among the locations at which gaze errors are identified. In one embodiment, the electronic device (101) can identify (or select) locations at which to display a visual object in order of highest gaze errors in the background area (510).

[0175] In operation 1020, the electronic device (101) may display a visual object to move through locations.

[0176] For example, the electronic device (101) may display a visual object so that the visual object moves through the identified locations in the order of highest line-of-sight error. For example, the electronic device (101) may display the visual object along a path (711, 713) that connects some display locations selected in the order of highest line-of-sight error. However, the present disclosure is not limited to the above embodiment. For example, the electronic device (101) may display the visual object so that the visual object moves through the path that takes the shortest time (or path) to move through the display locations. For example, the electronic device (101) may display the visual object so that the visual object moves through a path that does not cross the execution screen (520) of the application.

[0177] In one embodiment, the electronic device (101) may display a virtual object so that the virtual object moves within an identified region (or a region identified by the background region determination unit (615)) among a plurality of regions of a 3D screen. For example, the electronic device (101) may display a virtual object so that the virtual object moves through identified locations within a region (e.g., region (721)) with a high line-of-sight error among the regions (721, 723, 725, 727) in the order of high line-of-sight error.

[0178] FIG. 11 illustrates an operation performed by a wearable device according to one embodiment.

[0179] FIG. 11 may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIG. 2A and FIG. 2B. The operations of FIG. 11 may be included in operation 920 of FIG. 9. However, the present disclosure is not limited to the above embodiments.

[0180] In FIG. 11, at operation 1110, the electronic device (101) may identify a prompt for generating a visual object.

[0181] In one embodiment, the electronic device (101) may obtain a prompt to create a visual object based on the background area (510) (and / or wallpaper displayed in the background area (510)).

[0182] In one embodiment, the electronic device (101) may identify a prompt for generating a visual object based on a situation depicted by the input image. For example, the situation depicted by the input image may be distinguished based on the way objects included in the background area (510) (or the wallpaper of the background area (510)) are depicted (e.g., landscape, portrait, still life), the design pattern of the background area (510) (or the wallpaper of the background area (510)), the types of objects included in the background area (510) (e.g., people, animals, plants, objects), the relationships between objects (e.g., friends, family), and / or the time of day depicted by the background area (510) (or the wallpaper of the background area (510)) (e.g., morning, am, afternoon, evening, dawn).

[0183] In one embodiment, the electronic device (101) may identify a prompt for generating a visual object based on some of the visual objects included in the background area (510) (or the wallpaper of the background area (510)). In one embodiment, some of the visual objects may be visual objects having a selection area where the identified gaze is located.

[0184] In operation 1120, the electronic device (101) may generate a visual object based on the prompt.

[0185] In one embodiment, the electronic device (101) can obtain a virtual object by inputting a prompt into a generative AI model. In one embodiment, the electronic device (101) can obtain a virtual object by inputting a prompt corresponding to a background area (510) (or wallpaper of the background area (510)) into the generative AI model. In one embodiment, the electronic device (101) can obtain a virtual object by inputting a prompt corresponding to a visual object having a selection area where an identified gaze is located into the generative AI model.

[0186] In operation 1130, the electronic device (101) can display the generated visual object.

[0187] In one embodiment, the electronic device (101) may display a visual object generated in a background area (510) of a 3D screen for correction based on a gaze error. In one embodiment, the electronic device (101) may display a visual object generated in locations (or areas) having a gaze error greater than a specified gaze error in the background area (510) for correction based on a gaze error. However, the present disclosure is not limited to the above embodiment.

[0188] FIG. 12 illustrates an operation performed by a wearable device according to one embodiment.

[0189] FIG. 12 may be described with reference to the electronic device (101) of FIG. 1 and the wearable device (200) of FIG. 2A and FIG. 2B. The operations of FIG. 12 may be performed after operation 930 of FIG. 9. However, the present disclosure is not limited to the above-described embodiment.

[0190] In FIG. 12, at operation 1210, the electronic device (101) can identify the removal of the device.

[0191] At operation 1220, the electronic device (101) can determine whether removal of a parameter for line-of-sight error correction is necessary.

[0192] For example, the electronic device (101) can determine whether the removal of a parameter for correcting a gaze error is necessary based on a state in which the electronic device (101) is taken off from the user (300). For example, the electronic device (101) can determine that the removal of a parameter for correcting a gaze error is necessary based on a state in which the electronic device (101) is taken off from the user (300) lasting for a reference time or longer. For example, the electronic device (101) can determine that the removal of a parameter for correcting a gaze error is not necessary based on a state in which the electronic device (101) is taken off from the user (300) lasting for less than a reference time (or the electronic device (101) is temporarily taken off from the user (300) (or the electronic device (101) is put back on the user (300).

[0193] For example, the electronic device (101) can determine whether the removal of the parameter for the gaze error correction is necessary based on the fact that the gaze error used to obtain the parameter for the gaze error correction is less than or equal to a reference gaze error. For example, the electronic device (101) can determine that the removal of the parameter for the gaze error correction is necessary based on the fact that the gaze error used to obtain the parameter for the gaze error correction exceeds the reference gaze error. For example, the electronic device (101) can determine that the removal of the parameter for the gaze error correction is not necessary based on the fact that the gaze error used to obtain the parameter for the gaze error correction is less than or equal to a reference gaze error.

[0194] In operation 1220, based on determining that removal of a parameter for gaze error correction is necessary, the electronic device (101) may perform operation 1230. In operation 1220, based on determining that removal of a parameter for gaze error correction is not necessary, the electronic device (101) may perform operation 1240.

[0195] In operation 1230, the electronic device (101) may remove a parameter for gaze error correction. Removing the parameter for gaze error correction may indicate that when the user (300) later wears the electronic device (101), the identification of the user's (300) gaze is adjusted based on the error correction parameter obtained through eye calibration based on FIG. 4A.

[0196] In operation 1240, the electronic device (101) may maintain parameters for gaze error correction. Maintaining the parameters for gaze error correction may indicate that when the user (300) later wears the electronic device (101), the identification of the user's (300) gaze is adjusted based on the error correction parameters obtained through eye calibration based on FIG. 4A and the parameters for gaze error correction.

[0197] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0198] As described above, the wearable device (101, 200) may include a display system (660) including a first display (250-1) and a second display (250-2) arranged to face the user's eyes (300-1, 300-2) when worn, a camera system (680) including a plurality of cameras (240-1, 240-2) arranged to acquire an image including the user's eyes (300-1, 300-2) when worn, at least one processor (120) including a processing circuit; and a memory (130) storing instructions and including one or more storage media. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to display objects (531, 532, 533, 534, 535, 536) at at least different viewpoints on a three-dimensional (3D) screen displayed through the display system (660). The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to identify errors associated with gazes of the eyes (300-1, 300-2) that look at the objects (531, 532, 533, 534, 535, 536) based on the gazes of the eyes (300-1, 300-2) identified based on the image. The above errors may represent differences between the display positions of the objects (531, 532, 533, 534, 535, 536) and the focal positions of the gazes of the eyes (300-1, 300-2) corresponding one-to-one to the objects (531, 532, 533, 534, 535, 536).The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to display a visual object on a background screen (510) on the display system (660) to move through some of the display locations selected based on the errors among the display locations. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to correct the errors based on different gazes of the eyes (300-1, 300-2) looking at the visual object.

[0199] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to select the some display locations in descending order of the errors identified at the display locations.

[0200] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to display the visual object by moving through the partial display locations in descending order of errors of the partial display locations.

[0201] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to identify at least one word associated with the background screen (510). The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to generate the visual object by inputting the at least one word as a prompt into a generative AI (artificial intelligence) model. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to display the generated visual object on the background screen (510).

[0202] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to identify an object on which the user's gaze is located among the objects (531, 532, 533, 534, 535, 536) displayed on the background screen (510). The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to display a visual object having the same shape as the shape of the object on the background screen (510).

[0203] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to identify a time taken for a first gaze of the eye (300-1, 300-2) to be positioned within an interaction area of ​​a first object among the objects (531, 532, 533, 534, 535, 536) after the first object is displayed. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to: display the visual object on the background screen (510) based on the time exceeding a reference time. The above instructions, when executed individually or collectively by the at least one processor (120), may cause the wearable device (101, 200) to correct the errors based on different gazes of the eyes (300-1, 300-2) looking at the visual object.

[0204] The wearable device (101, 200) may include an inertial sensor. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to identify, using the inertial sensor, whether slippage of the wearable device (101, 200) has occurred. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to: display the visual object on the background screen (510) based on the identification that the slippage has occurred. The above instructions, when executed individually or collectively by the at least one processor (120), may cause the wearable device (101, 200) to correct the errors based on different gazes of the eyes (300-1, 300-2) looking at the visual object.

[0205] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to identify that the wearable device (101, 200) is worn. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to display a calibration screen for gaze calibration on the display system (660) based on identifying that the wearable device (101, 200) is worn. The above instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to determine a parameter for identifying the gaze of the eye (300-1, 300-2) based on the positions of the eye (300-1, 300-2) identified through the image while the objects (531, 532, 533, 534, 535, 536) are displayed at different times on the calibration screen. The errors may be corrected by an operation of adjusting the value of the determined parameter by a value for error correction.

[0206] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to identify that the wearable device (101, 200) is taken off. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to remove the value for the error correction based on identifying that the wearable device (101, 200) is taken off.

[0207] The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to display the visual object based on the fact that the 3D screen displayed through the display system (660) includes the background screen (510). The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to postpone displaying the visual object based on the fact that the 3D screen displayed through the display system (660) includes only screens of an application other than the background screen (510).

[0208] Some of the above display locations may be located on the background screen (510) excluding the screen area of ​​the application among the 3D screens displayed through the display system (660).

[0209] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to display the visual object on the background screen (510) on the display system (660) so as to move along a path that does not cross the screen area of ​​the application among the 3D screens displayed through the display system (660).

[0210] As described above, the method can be executed in a wearable device (101, 200) including a display system (660) including a first display (250-1) and a second display (250-2) arranged to face the eyes (300-1, 300-2) of a user when worn, and a camera system (680) including a plurality of cameras (240-1, 240-2) arranged to acquire an image including the eyes (300-1, 300-2) of the user when worn. The method can include an operation of displaying objects (531, 532, 533, 534, 535, 536) at at least different viewpoints on a three-dimensional (3D) screen displayed through the display system (660). The method may include an operation of identifying errors associated with the gazes of the eyes (300-1, 300-2) looking at the objects (531, 532, 533, 534, 535, 536) based on the gazes of the eyes (300-1, 300-2) identified based on the image. The errors may represent differences between the display positions of the objects (531, 532, 533, 534, 535, 536) and the focal positions of the gazes of the eyes (300-1, 300-2) corresponding to the objects (531, 532, 533, 534, 535, 536) in a one-to-one correspondence. The method may include an operation of displaying a visual object on a background screen (510) on the display system (660) so as to move through some of the display positions selected based on the errors among the display positions. The above method may include an operation of correcting the errors based on different gazes of the eyes (300-1, 300-2) looking at the visual object.

[0211] The method may include an operation of selecting some of the display locations in order of the highest number of errors identified at the display locations.

[0212] The method may include an action of identifying at least one word associated with the background screen (510). The method may include an action of generating the visual object by inputting the at least one word as a prompt into a generative artificial intelligence (AI) model. The method may include an action of displaying the generated visual object on the background screen (510).

[0213] The method may include an operation of identifying an object at which the user's gaze is located among the objects (531, 532, 533, 534, 535, 536) displayed on the background screen (510). The method may include an operation of displaying a visual object having the same shape as the shape of the object on the background screen (510).

[0214] The method may include an operation of identifying a time taken for a first gaze of the eye (300-1, 300-2) to be positioned within an interaction area of ​​a first object among the objects (531, 532, 533, 534, 535, 536) after the first object is displayed. The method may include an operation of displaying the visual object on the background screen (510) based on whether the time exceeds a reference time. The method may include an operation of correcting the errors based on different gazes of the eyes (300-1, 300-2) looking at the visual object.

[0215] The method may include an operation of identifying whether slippage of the wearable device (101, 200) has occurred based on an inertial sensor. The method may include an operation of displaying the visual object on the background screen (510) based on the identification that the slippage has occurred. The method may include an operation of correcting the errors based on different gazes of the eyes (300-1, 300-2) looking at the visual object.

[0216] The method may include an operation of identifying that the wearable device (101, 200) is worn. The method may include an operation of displaying a calibration screen for gaze calibration on the display system (660) based on identifying that the wearable device (101, 200) is worn. The method may include an operation of determining a parameter for identifying the gaze of the eyes (300-1, 300-2) based on positions of the eyes (300-1, 300-2) identified through the images while objects (531, 532, 533, 534, 535, 536) are displayed at different times on the calibration screen. The operation of correcting the errors may include an operation of adjusting the value of the determined parameter by a value for error correction.

[0217] The method may include an operation of displaying the visual object based on the fact that the 3D screen displayed through the display system (660) includes the background screen (510). The method may include an operation of postponing the display of the visual object based on the fact that the 3D screen displayed through the display system (660) includes only the screen of an application other than the background screen (510).

[0218] As described above, a non-transitory computer readable storage medium can store a program including instructions. The instructions, when executed individually or collectively by at least one processor (120) of a wearable device (101, 200), comprising a display system (660) including a first display (250-1) and a second display (250-2) arranged to face the eyes (300-1, 300-2) of a user when worn, and a camera system (680) including a plurality of cameras (240-1, 240-2) arranged to acquire an image including the eyes (300-1, 300-2) of the user when worn, may cause the wearable device (101, 200) to display objects (531, 532, 533, 534, 535, 536) at at least different viewpoints on a three-dimensional (3D) screen displayed through the display system (660). The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to identify errors associated with the gazes of the eyes (300-1, 300-2) looking at the objects (531, 532, 533, 534, 535, 536) based on the gazes of the eyes (300-1, 300-2) identified based on the image. The errors may represent differences between the display positions of the objects (531, 532, 533, 534, 535, 536) and the focal positions of the gazes of the eyes (300-1, 300-2) that correspond one-to-one with respect to the objects (531, 532, 533, 534, 535, 536).The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to display a visual object on a background screen (510) on the display system (660) to move through some of the display locations selected based on the errors among the display locations. The instructions, when individually or collectively executed by the at least one processor (120), may cause the wearable device (101, 200) to correct the errors based on different gazes of the eyes (300-1, 300-2) looking at the visual object.

[0219] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0220] Electronic devices according to one or more embodiments disclosed herein may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments disclosed herein are not limited to the aforementioned devices.

[0221] The description of one or more embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but rather to encompass various modifications, equivalents, or alternatives of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly dictates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" each include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components, and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0222] The term "module" as used in one or more embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or portion of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0223] One or more embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0224] According to one embodiment, a method according to one or more embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0225] According to one or more embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. In some embodiments, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one or more embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (101, 200), A display system (660) including a first display (250-1) and a second display (250-2) facing the user's eyes (300-1, 300-2) when worn, A plurality of cameras (240-1, 240-2) arranged to acquire images including the user's eyes (300-1, 300-2) when worn; At least one processor (120) comprising a processing circuit; and A wearable device (101, 200) comprises a memory (130) storing instructions and including one or more storage media, wherein the instructions, when individually or collectively executed by the at least one processor (120), Displaying objects (531, 532, 533, 534, 535, 536) at least at different points in time on a 3D (three dimensional) screen displayed through the above display system (660), Based on the above image, identify the gazes looking at the objects (531, 532, 533, 534, 535, 536), Identify errors associated with the identified gazes, wherein the errors represent differences between the display positions of the objects (531, 532, 533, 534, 535, 536) and the focal positions of the gazes corresponding one-to-one to the objects (531, 532, 533, 534, 535, 536), Displaying a visual object moving through some of the display positions selected based on the errors among the above display positions on the background screen (510) of the display system (660), Based on the different gazes looking at the visual object, causing the errors to be corrected, Wearable devices.

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: The errors identified at the above display locations cause some of the display locations to be selected in the order of high, Wearable devices.

3. In claim 2, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: The errors of the above some display positions are in the order of increasing, causing the visual object to be displayed moving through the above some display positions. Wearable devices.

4. In any one of claims 1 to 3, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: Identify at least one word associated with the above background image (510), By inputting at least one word as a prompt into a generative AI (artificial intelligence) model, the visual object is generated, Causing the above-mentioned generated visual object to be displayed on the background screen (510), Wearable devices.

5. In any one of claims 1 to 4, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: Identify the object where the user's gaze is located among the objects (531, 532, 533, 534, 535, 536) displayed on the above background screen (510), Causing the visual object having the same shape as the shape of the object to be displayed on the background screen (510). Wearable devices.

6. In any one of claims 1 to 5, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: After the first object among the above objects (531, 532, 533, 534, 535, 536) is displayed, the time taken for the first gaze to be positioned within the interaction area of ​​the first object is identified, Based on the above time exceeding the reference time: Display the above visual object on the background screen (510), Based on the different gazes looking at the visual object, causing the errors to be corrected. Wearable devices.

7. In any one of claims 1 to 6, Includes an inertial sensor, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: Using the above inertial sensor, it is identified whether slippage of the wearable device (101, 200) occurs, Based on the above slippage being identified as occurring: Display the above visual object on the background screen (510), Based on the different gazes looking at the visual object, causing the errors to be corrected. Wearable devices.

8. In any one of claims 1 to 7, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: Identifying that the above wearable device (101, 200) is worn, Based on identifying that the wearable device (101, 200) is worn, a calibration screen for gaze calibration is displayed on the display system (660), While objects (531, 532, 533, 534, 535, 536) are displayed at different times on the calibration screen, based on the positions of the eyes (300-1, 300-2) identified through the images, a parameter for identifying the gaze is determined, The above errors are corrected by an operation of adjusting the value of the determined parameter by the value for error correction. Wearable devices.

9. In claim 8, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: Identifying that the above wearable device (101, 200) is taken off, Based on identifying that the wearable device (101, 200) is taken off, causing the value for the error correction to be removed. Wearable devices.

10. In any one of claims 1 to 9, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: Based on the fact that the 3D screen displayed through the above display system (660) includes the background screen (510), the visual object is displayed, Based on the fact that the 3D screen displayed through the above display system (660) includes only the screen of the application other than the background screen (510), causing the display of the visual object to be delayed, Wearable devices.

11. In claim 10, The above-mentioned partial display locations are located on the background screen (510) excluding the screen area of ​​the application among the 3D screens displayed through the display system (660). Wearable devices.

12. In claim 11, The above instructions, when individually or collectively executed by the at least one processor (120), cause the wearable device (101, 200) to: Causing the visual object moving through a path that does not cross the screen area of ​​the application among the 3D screens displayed through the display system (660) to be displayed on the background screen (510) on the display system (660). Wearable devices.

13. A method executed in a wearable device (101, 200), comprising a display system (660) including a first display (250-1) and a second display (250-2) facing the user's eyes (300-1, 300-2) when worn, and a plurality of cameras (240-1, 240-2) arranged to acquire an image including the user's eyes (300-1, 300-2) when worn, An operation of displaying objects (531, 532, 533, 534, 535, 536) at least at different points in time on a 3D (three dimensional) screen displayed through the above display system (660). An action of identifying gazes looking at the objects (531, 532, 533, 534, 535, 536) based on the above image, An operation of identifying errors associated with the identified gazes, wherein the errors represent differences between the display positions of the objects (531, 532, 533, 534, 535, 536) and the focus positions of the gazes corresponding one-to-one to the objects (531, 532, 533, 534, 535, 536), An operation of displaying a visual object moving through some of the display positions selected based on the errors among the above display positions on a background screen (510) on the display system (660), and Based on the different gazes looking at the visual object, including an operation to correct the errors, method.

14. In claim 13, An operation of selecting some of the display positions in the order of the errors identified at the above display positions. method.

15. In claim 13 or claim 14, An action of identifying at least one word associated with the above background screen (510), An action of generating the visual object by inputting at least one word as a prompt into a generative AI (artificial intelligence) model, and An operation including displaying the generated visual object on the background screen (510) method.

Images