Electronic device for identifying gaze direction and method for operating the same

Abstract

An electronic device may include a camera and at least one processor. The at least one processor may be configured to: obtain an eyeball image of a user through the camera; obtaining pupil data and a retina image from the obtained eyeball image; obtaining an eyeball model including the retinal image based on the obtained pupil data and the obtained retinal image; and identifying at least one of an eyeball position or a gaze direction based on the obtained eyeball model.

Description

Technical Field

[0001] The present disclosure relates to a wearable electronic device for identifying a gaze direction of a user and a method for operating the electronic device. Background Art

[0002] With the development of gaze tracking technology using a camera, a wearable electronic device that provides information according to a person's interest (for example, tracking a person's gaze and zooming in or out on an object focused on a screen or displaying relevant information around the object) is provided. As an example, the wearable electronic device may be a head mounted device (HMD), an augmented reality (AR) device, or a virtual reality (VR) device, which obtains and provides an image of the real world in a direction viewed by a user through a camera.

[0003] Wearable electronic devices can be used to optimize the operation and configuration of a display for a user by tracking the user's gaze direction, such as zooming in or out on a particular portion of a display screen when the user focuses on that portion. Summary of the invention

[0004]

Problem Solution

[0005] According to an embodiment, an electronic device may include a camera and at least one processor. The at least one processor may be configured to obtain an eye image of a user's eye from the camera. The at least one processor may be configured to obtain pupil data and a retinal image from the eye image. The at least one processor may be configured to obtain an eye model including a retinal image based on the obtained pupil data and the obtained retinal image. The at least one processor may be configured to identify at least one of a position or a gaze direction of the eye based on the obtained eye model.

[0006] According to an embodiment, a method for operating an electronic device may include obtaining an eye image of a user's eye from a camera. According to an embodiment, the method for operating an electronic device may include obtaining pupil data and a retinal image from the eye image. According to an embodiment, the method for operating an electronic device may include obtaining an eye model including a retinal image based on the pupil data and the retinal image. According to an embodiment, the method for operating an electronic device may include identifying at least one of a position or a gaze direction of the eye based on the eye model.

[0007] According to an embodiment, in a non-transitory computer-readable storage medium storing one or more programs, the one or more programs may include obtaining an eye image of a user's eye from a camera. The one or more programs may include obtaining pupil data and a retinal image from the eye image. The one or more programs may include obtaining an eye model including a retinal image based on the pupil data and the retinal image. The one or more programs may include identifying at least one of a position or a gaze direction of the eye based on the eye model. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0009] Figure 1 is a block diagram illustrating an electronic device in a network environment according to an embodiment of the present disclosure.

[0010] Figure 2 is a block diagram showing an electronic device according to an embodiment of the present disclosure.

[0011] Figure 3 is a perspective view showing an electronic device according to an embodiment of the present disclosure.

[0012] Figure 4a is a diagram illustrating a state of obtaining an eye image of a user's eye according to an embodiment of the present disclosure.

[0013] Figure 4b An image forming surface according to an embodiment of the present disclosure is shown.

[0014] Figure 4c A pupil image obtained from an eye image according to an embodiment of the present disclosure is shown.

[0015] Figure 5a A state in which the circular shape of the pupil is restored according to an embodiment of the present disclosure is shown.

[0016] Figure 5b According to the embodiment of the present disclosure, Figure 5a The rotational position of the eye in the pupil image.

[0017] Figure 6 The sequence of obtaining an eye model according to an embodiment of the present disclosure is shown.

[0018] Figure 7 The figure shows the change of the eye model according to the movement of the eye according to the embodiment of the present disclosure.

[0019] Figure 8 The sequence of obtaining an eye model according to an embodiment of the present disclosure is shown.

[0020] Fig. 9The process of identifying the gaze direction based on the eye image and the eye model according to an embodiment of the present disclosure is shown.

[0021] Fig.10 A process of detecting changes in eye positions according to an embodiment of the present disclosure is shown.

[0022] Fig.11 A process of detecting a changed eye position according to an embodiment of the present disclosure is shown.

[0023] Fig.12 An overall retinal image of an eye model according to an embodiment of the present disclosure is shown.

[0024] Fig.13 is a flowchart illustrating a method for operating an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to an embodiment of the present disclosure.

[0026] refer to Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In an embodiment, at least one of the components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added in the electronic device 101. According to an embodiment, some of the components (e.g., the sensor module 176, the camera module 180, or the antenna module 197) may be integrated into a single component (e.g., the display module 160). The processor 120 may run, for example, software (e.g., the program 140) to control at least one other component (e.g., a hardware component or a software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to an embodiment, as at least a part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., the sensor module 176 or the communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may be configured to use lower power than the main processor 121 or be designated for a designated function. The auxiliary processor 123 may be implemented to be separate from the main processor 121, or to be implemented as part of the main processor 121.

[0027] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 180 or a communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated via machine learning. For example, such learning may be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0028] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). For example, the various data may include input data or output data of software (e.g., the program 140) and commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134. The nonvolatile memory 134 may include an internal memory 136 and an external memory 138.

[0029] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0030] The input module 150 may receive commands or data to be used by other components (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0031] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as part of the speaker.

[0032] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display 160 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the strength of a force generated by a touch.

[0033] The audio module 170 converts sound into an electrical signal and vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150 or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0034] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0035] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly with an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0036] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or motion) or electric stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0038] The camera module 180 may capture still images or moving images. According to embodiments, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0039] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0040] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0041] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network (LAN) or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify or authenticate the electronic device 101 in a communication network (e.g., the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0042] The wireless communication module 192 may support 5G networks after 4G networks and next generation communication technologies (e.g., new radio (NR) access technologies). NR access technologies may support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable low latency communications (URLLC). The wireless communication module 192 may support high frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 may support various technologies for ensuring performance on high frequency bands, such as, for example, beamforming, massive multiple input multiple output (massive MIMO), full dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0043] The antenna module 197 may transmit a signal or power to the outside (e.g., an external electronic device), or receive a signal or power from the outside (e.g., an external electronic device). According to an embodiment, the antenna module may include an antenna including a radiator formed by a conductor or a conductive pattern formed on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an antenna array). In this case, for example, the communication module 190 may select at least one antenna suitable for a communication scheme used in a communication network (e.g., the first network 198 or the second network 199) from the plurality of antennas. A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, other components other than the radiator (e.g., a radio frequency integrated circuit (RFIC)) may be further formed as part of the antenna module 197.

[0044] According to an embodiment, the antenna module 197 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high frequency band.

[0045] At least some of the above components may be coupled to each other and transmit signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

[0046] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be a device of the same or different type as the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute the requested at least part of the function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or health care) based on 5G communication technology or IoT-related technologies.

[0047] The electronic device according to an embodiment of the present disclosure may be one of various types of electronic devices. For example, the electronic device may include a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to the above-mentioned devices.

[0048] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalents or alternatives of the corresponding embodiments. With respect to the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It should be understood that the nouns in the singular form corresponding to the term may include one or more things unless otherwise clearly indicated by the relevant context. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include all possible combinations of items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply separate the corresponding components from other components, and do not limit the components in other aspects (e.g., importance or order). It should be understood that if an element (e.g., a first element) is referred to as being “coupled”, “coupled to”, “connected to” or “connected to” another element (e.g., a second element) with or without the terms “operably” or “communicatively”, it means that the element can be coupled to the other element directly (e.g., wired), wirelessly or via a third element.

[0049] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "component," or "circuit"). A module may be a single integrated component or its smallest unit or a portion thereof suitable for performing one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0050] According to an embodiment, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. Some of the multiple entities may be arranged in different components, respectively. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components performing one or more functions before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

[0051] Figure 2 is a block diagram showing an electronic device 200 according to an embodiment of the present disclosure. Figure 3 is a perspective view showing an electronic device 200 according to an embodiment of the present disclosure.

[0052] Further references Figure 2 and Figure 3 According to an embodiment, the electronic device 200 may include a camera 210 (eg, Figure 1 camera module 180), light source 240, display 250, processor 220 (eg, Figure 1 processor 120) and / or memory 230 (e.g., Figure 1 Here, the electronic device 200 is assumed to be a glasses-type AR device, but is not limited thereto, and the electronic device 200 may include or have an interface with any of the same features as described above for the electronic device 101.

[0053] In an embodiment, the camera 210 may be fixed to the wearable electronic device 200 worn near an eyeball of a user, and may obtain an eyeball image of an eyeball of the user wearing the wearable electronic device 200 .

[0054] In an embodiment, the light source 240 may radiate light of a preset wavelength band corresponding to the camera 210 to the eye. Since both the pupil formed on the surface of the user's eyeball and the retina inside the eyeball are composed of parts of the spherical eyeball and reflect the light incident from the light source 240 substantially as it is, the incident position and the reflected position may be substantially the same. In an embodiment, the light source 240 and the camera 210 may be disposed adjacent to each other as a first pair (with the first of the light source 240 and the camera 210 on the left side for the left eye L) and as a second pair (with the second of the light source 240 and the camera 210 on the right side for the right eye R).

[0055] In an embodiment, the camera 210 and the light source 240 may be disposed to correspond to the left eye L and the right eye R of the glasses-type wearable electronic device 200, respectively. The camera 210 and the light source 240 of the glasses-type wearable electronic device 200 may be disposed at positions spaced apart from the left eye L and the right eye R of the user, and as an example, are fixed to the lowermost ends of the edges of the lenses located in front of the left eye L and the right eye R, respectively.

[0056] In an embodiment, the light source 240 may be an infrared (IR) light source that emits IR light, and the camera 210 may be an IR camera. Therefore, the camera 210 may be robust to interference of visible light introduced from the outside. In an embodiment, the light source 240 may be a visible light source that emits visible light, and the camera 210 may also be a red-green-blue (RGB) camera.

[0057] For example, the display 250 may be a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), or an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0058] In an embodiment, the display 250 may be a transparent display 250 (e.g., a transparent touch screen or a transparent window) located on an extension line of the user's gaze direction, and the display 250 may display a transparent (including semi-transparent) picture. As an example, the display 250 may be a transparent lens located in front of the left eye and / or right eye of the user, and may output a screen through light emitted by itself or light emitted from the outside of the display 250. In an embodiment, the display 250 may provide a picture to the user's eyes through a light waveguide formed of glass, plastic, or polymer.

[0059] In an embodiment, the processor 220 may be operated by a program previously stored in the memory 230, and the operations in the processor 220 are described below. The memory 230 may store data obtained or generated by the processor 220. For example, an eye model obtained based on pupil data and / or an eye model obtained by combining a retinal image with the eye model may be stored in the memory 230.

[0060] Figure 4a is a diagram showing a state of obtaining an eye image of a user's eye E according to an embodiment of the present disclosure. Figure 4b An image forming surface is shown. Figure 4c A pupil image obtained from an eye image according to an embodiment of the present disclosure is shown.

[0061] refer to Figure 4a , Figure 4b and Figure 4c , the camera 210 may be located in front of the user's eye E and capture an eye image for the user's eye E. Here, the point representing the camera 210 may represent a lens center point of the camera 210, and the range indicated by two dotted lines from the camera 210 may represent a field of view (FOV) of the camera 210. As an embodiment, an electronic device (e.g., Figure 2 The processor 220 can extract the area of ​​the pupil P and the retinal image I from the eyeball image captured by the camera 210.

[0062] For ease of understanding, Figure 4b The configuration is schematically shown in which the user's eye E is captured by a camera 210 located in front of the user's eye E (just as the image is formed on the imaging plane VP). However, in actual physical phenomena, it is preferred that Figure 4b As shown, the actual imaging plane Plane is located in the opposite direction of the eye E as the subject relative to the camera 210, and the captured image of the user's eye E is located on the actual imaging plane Plane. Figure 4a The subsequent figures are shown in a similar manner, but preferably, it should be understood that the actual physical phenomena are Figure 4b occurs in the manner shown.

[0063] In an embodiment, the electronic device 200 may identify an area of ​​the pupil P from the captured eye image, and obtain the position of the pupil P, the shape of the pupil P, and / or the gaze direction G of the eye E based on the identified area of ​​the pupil P.

[0064] In the embodiment, it is assumed that the position of the eye E is fixed, and according to the rotation of the eye E due to various changes in the gaze direction G of the eye E, the area of ​​the pupil P may be as follows: Figure 4b Specifically, when the gaze direction G of the eye E is directly toward the camera 210, the area of ​​the pupil P can be identified as a circle, and when the gaze direction G of the eye E deviates from the capturing direction in the up / down or left / right direction, the area of ​​the pupil P can be identified as an ellipse. In other words, when the pupil P rotates from the front of the camera 210, the area of ​​the pupil P can be identified as having an elliptical shape, in which the major axis is fixed and the minor axis changes according to the rotation angle of the eye E.

[0065] In an embodiment, the electronic device may identify an area of ​​the pupil P from an eye image, and obtain the shape of the pupil P and the position of the pupil P based on the identified area of ​​the pupil P. Specifically, the electronic device may obtain position information about the pupil P on a plane (e.g., x-coordinate and y-coordinate) and a circular shape P' or an elliptical shape of the pupil (the length of the major axis and the length of the minor axis).

[0066] Figure 5a A state in which the circular shape P′ of the pupil P is restored according to an embodiment of the present disclosure is shown. Figure 5b According to the embodiment of the present disclosure, Figure 5a Pupil image of the rotated position of eye E in .

[0067] refer to Figure 5a and Figure 5b In an embodiment, the electronic device 200 may obtain the gaze direction G of the eye E and the circular shape P' of the pupil P based on the obtained shape of the pupil P. In an embodiment, the electronic device 200 may calculate the ratio of the length of the major axis to the length of the minor axis according to the elliptical shape of the pupil P by the following equation, and calculate the angle θ of the pupil P rotating from the gaze direction G of the eye E looking straight toward the camera 210.

[0068] [Equation 1]

[0069]

[0070] Here, θ is an angle at which the pupil P rotates from the gaze direction G of the eyeball E looking straight toward the camera 210 .

[0071] In an embodiment, the electronic device 200 can restore the circular shape P' of the pupil P by the shape of the pupil P and the angle θ of the pupil P rotation. For example, since the shape of the pupil P is a vertically / horizontally symmetrical shape, the rotation direction of the eye E can be estimated as two symmetrical directions (e.g., up or down / left or right). The electronic device 200 can determine the rotation direction of the eye E based on the identified position of the pupil P and the two estimated rotation directions of the eye E.

[0072] In an embodiment, the electronic device 200 may identify the area of ​​the pupil P from the eye image captured by the camera 210, and estimate the gaze direction G of the eye E based on the pupil data obtained for the identified area of ​​the pupil P. In addition, the electronic device 200 may store the obtained pupil data in a memory (e.g., Figure 1 Memory 130 or Figure 2In an embodiment, the pupil data may include the position of the pupil P, the shape of the pupil P, the rotation direction of the eye E, and / or the rotation angle of the eye E.

[0073] Figure 6 It shows that according to an embodiment of the present disclosure, for example Figure 8 , Fig. 9 and Fig.10 The order of the eye model M is shown.

[0074] refer to Figure 6 , the electronic device 200 may receive a plurality of eye images according to the rotation of the eye E from the camera 210 , obtain a plurality of pupil data from the plurality of eye images, and obtain an eye model M based on the plurality of pupil data.

[0075] In an embodiment, the electronic device 200 may obtain the center line of the pupil P based on pupil data obtained from an eye image. Since the three-dimensional coordinates of the center of the pupil P cannot be obtained, the center line of the pupil P (e.g., the center line CL) may be a straight line extending in 3D from the pupil P identified in 2D. The electronic device 200 may obtain the center line of the eye E by using the radius R of the eye E, the rotation direction of the eye E, and the center line of the pupil P, by spacing the radius R of the eye E in a direction perpendicular to the circular shape P' of the pupil P restored from the center line of the pupil P as shown in the figure. Here, a value generally preset as the radius of the eye E may be used as the radius R of the eye E, or a value estimated based on the eye model M may be used.

[0076] In an embodiment, the electronic device 200 may collect a plurality of pupil data according to the rotation of the eye E from a plurality of eye images, thereby obtaining a plurality of eye center lines CL. The electronic device 200 may estimate the position where the obtained plurality of eye center lines CL converge as the eye center point CP. In other words, the electronic device 200 may obtain the eye model M based on a plurality of eye images or a plurality of pupil data.

[0077] In an embodiment, the eye model M may include the position of the eye E and / or the shape of the eye E. Here, the position of the eye E may represent the 3D position of the eye center point CP, and the shape of the eye E may represent the spherical shape of the eye E with the eye center point CP as the center and a radius R.

[0078] In an embodiment, the electronic device 200 may identify a region of the pupil P from an eye image received by the camera 210 by using the obtained eye model M, and estimate a gaze direction G of the eye E based on pupil data obtained for the identified region of the pupil P.

[0079] Figure 7FIG. 4 shows the change of the eye model M according to the movement of the eye E according to an embodiment of the present disclosure.

[0080] Further references Figure 7 In an embodiment, the position of the user's eye E may change in the operating state of the electronic device 200. For example, due to the sliding or shaking of the electronic device 200 worn by the user, a repositioning may occur between the electronic device 200 and the eye E. Here, the repositioning of the eye E may refer to a relative repositioning between the electronic device 200 and the eye E, or a repositioning between the camera 210 and the eye E.

[0081] In an embodiment, the electronic device 200 may not be able to detect the repositioning of the eye E and thus update the position of the eye E according to the repositioning of the eye E. Therefore, the electronic device 200 may mistakenly recognize the change in the area of ​​the pupil P due to the repositioning of the eye E as the change in the area of ​​the pupil P due to the rotation of the eye E, so that an error may occur when estimating the gaze direction G of the eye E based on the eye model M. The electronic device 200 may continuously update the eye model M based on the newly received eye image to detect the repositioning of the eye E, but sufficient time is required to update the eye model M with the repositioning of the eye E.

[0082] Figure 8 The sequence of obtaining the eye model M according to an embodiment of the present disclosure is shown.

[0083] refer to Figure 8 In an embodiment, the electronic device 200 may receive a plurality of eye images according to the rotation of the eye E, and simultaneously sequentially display objects 251 for guiding the user to gaze at different positions on the display 250 .

[0084] In an embodiment, the electronic device 200 may obtain a plurality of pupil data from a plurality of eye images, and obtain the eye model M from the plurality of pupil data, as described above.

[0085] In an embodiment, the electronic device 200 may obtain a plurality of retinal images I from a plurality of eye images, and map the plurality of retinal images I to correspond to a plurality of pupil data, respectively. The retinal image I may be an image of light emitted from a light source and incident on a retina inside the eye E through the pupil P, obtained by the camera 210 located outside the eye E through the pupil P. When the pupil P moves according to the rotation of the eye E, the plurality of retinal images I may be obtained in different retinal areas inside the eye E, respectively.

[0086] In an embodiment, the electronic device 200 may obtain a plurality of pupil data (including the position of the pupil P, the shape of the pupil P, the rotation direction of the eye E, and / or the rotation angle of the eye E) from the plurality of eye images and store them in a memory, and may further store retinal images I respectively corresponding to the pupil data in the memory. The electronic device may create a 3D eye model M based on the plurality of pupil data, and map the plurality of retinal images I to the created eye model M at positions respectively corresponding to the pupil data.

[0087] In an embodiment, the electronic device 200 may sequentially display objects 251 to guide the user to gaze at the objects 251 at a plurality of different positions of the display 250 to guide the rotation of the eye E. The gaze direction G of the user may be preset according to each position of the object 251, and the electronic device 200 may obtain the user's eye image according to the preset gaze direction G.

[0088] In an embodiment, the electronic device 200 may sequentially display objects 251 at preset positions of the display 250. In an embodiment, the electronic device 200 may display the object 251 on the display 250 for guiding the user's eye E to rotate to a rotational position of the eye E corresponding to a position where the retinal image I does not exist in the eye model M, while mapping a plurality of retinal images I obtained from a plurality of eye images to the eye model M.

[0089] Fig. 9 The process of identifying the gaze direction G based on the eye image and the eye model M according to an embodiment of the present disclosure is shown.

[0090] refer to Fig. 9 In an embodiment, the electronic device 200 may identify the gaze direction G based on the eye image and the eye model M.

[0091] In an embodiment, the electronic device 200 may identify the gaze direction G and the position of the eye E based on the eye model M stored in the memory. Under the assumption that the position of the eye E of the eye model M is stored (indicated by the dotted line), the electronic device 200 may identify the gaze direction G from the eye image captured by the camera 210. In an embodiment, the electronic device 200 may identify the area of ​​the pupil P from the eye image, restore the circular shape P' of the pupil using the shape of the identified pupil P, and obtain the gaze direction G according to the rotation direction and rotation angle of the eye E based on the restored circular shape P' of the pupil. The electronic device 200 may update the eye model M corresponding to the gaze direction G obtained at the position of the eye E of the stored eye model M based on the eye image.

[0092] Fig.10 The process of detecting the position change of the eye E according to an embodiment of the present disclosure is shown.

[0093] refer to Fig.10 In an embodiment, the electronic device 200 can detect the repositioning of the eye E by comparing the retinal image I of the eye model M updated corresponding to the gaze direction G obtained based on the area of ​​the pupil P identified from the eye image with the retinal image I obtained from the eye image.

[0094] As an example, when the retinal image I of the eye model M is the same as the retinal image I obtained from the eye image, the electronic device 200 can detect that the position of the eye E has not changed, and when the retinal image I of the eye model M is different from the retinal image I obtained from the eye image, the electronic device 200 can detect that the position of the eye E has changed.

[0095] In an embodiment, when it is detected that the position of the eye E has not changed, the electronic device 200 may obtain the gaze direction G and / or position of the eye E based on the eye model M.

[0096] Fig.11 A process of detecting a changed eye (E) position according to an embodiment of the present disclosure is shown.

[0097] refer to Fig.11 In an embodiment, the electronic device 200 may obtain the position of the eye E based on the retinal image I and the eye model M. In an embodiment, the electronic device 200 may obtain the retinal image I from the eye image. The retinal image I may be a retinal image I of the inside of the eye E captured through the area of ​​the pupil P according to the rotational position of the eye E.

[0098] In an embodiment, the electronic device 200 may estimate the pupil centerline PL based on the region of the pupil P identified from the eye image. For example, the electronic device 200 may estimate the pupil centerline PL as a straight line extending from the 2D position of the region of the pupil P identified from the eye image.

[0099] In an embodiment, the electronic device 200 may estimate the pupil centerline PL using the eye model M and the retinal image I. Specifically, the electronic device 200 may estimate the pupil centerline PL where the area of ​​the pupil P of the eye model M is located by estimating the eye model M, so that the retinal image mapped to the eye model M corresponds to the retinal image I obtained from the eye image. In addition, the electronic device 200 may estimate the eye centerline CL where the eye center point CP is located at a position spaced apart from the circular shape P' of the pupil restored from the estimated pupil centerline PL by the shape of the pupil P (e.g., radius R).

[0100] In an embodiment, the electronic device 200 may estimate the center line at which the eye center point CP of the eye E is located, and estimate the eye center point CP based on a ratio (proportion) of the retinal image I obtained from the eye image and the retinal image mapped to the eye model M. The electronic device 200 may update the eye model M according to the newly estimated eye center point CP.

[0101] Fig.12 An overall retinal image of the eye model M according to an embodiment of the present disclosure is shown.

[0102] refer to Fig.12 , the electronic device 200 may obtain the eye model M by mapping the plurality of retinal images I obtained according to the rotation of the eye E, and the obtained eye model M may include an overall retinal image generated by synthesizing the plurality of retinal images I. The electronic device 200 may store the eye model M including the overall retinal image in a memory.

[0103] In an embodiment, the electronic device 200 may determine whether the user is a user corresponding to the stored eye model M based on the obtained retinal image I. The electronic device 200 may determine whether the obtained image corresponds to a pre-stored eye image of the user by identifying whether the retinal image I obtained from the eye image obtained by the camera 210 corresponds to the retinal image I stored in the eye model M previously stored in the memory. Specifically, the electronic device 200 may obtain an eye image through the camera 210, obtain the retinal image I from the obtained eye image, and compare the obtained retinal image I with the retinal image mapped to the eye model M to determine whether the user is a user corresponding to the eye model M.

[0104] In an embodiment, when a user newly wears or re-wears the electronic device 200, the electronic device 200 may determine whether the user is a user whose eye model M is stored. As an example, the electronic device 200 may obtain a preset number of retinal images I according to the rotation of the eye E, and identify whether the overall retinal image mapped to the eye model M has areas respectively corresponding to the retinal images I. When it is identified that there are a preset number or more of retinal images I in the overall retinal image mapped to the eye model M, the electronic device 200 may determine that the user is a user of the eye model M.

[0105] In an embodiment, when it is determined that the user is a user whose eye model M was previously stored based on the obtained retinal image I, the electronic device can identify at least one of the gaze direction G or position of the eye E based on the previously stored eye model M, and when it is determined that the user is not a user whose eye model M was previously stored, the electronic device can obtain a new eye model M based on the obtained eye image.

[0106] Fig.13 is a flowchart 1300 illustrating a method for operating the electronic device 200 according to an embodiment of the present disclosure.

[0107] Further references Fig.13 According to an embodiment, the electronic device 200 may obtain an eye image of a user through a camera in operation 1310. In an embodiment, the electronic device 200 may obtain a plurality of eye images while sequentially displaying objects for guiding the user to gaze at different positions on a display.

[0108] According to an embodiment, in operation 1330, the electronic device 200 may obtain pupil data and a retinal image from the obtained eye image. In an embodiment, the electronic device 200 may obtain a plurality of respectively corresponding pupil data and retinal images from a plurality of eye images. In an embodiment, the electronic device 200 may identify the area of ​​the pupil from the eye image, and obtain at least one of the position of the pupil, the shape of the pupil, or the gaze direction based on the identified pupil area. For example, the electronic device 200 may obtain the position coordinates of the pupil, the length of the major axis and the minor axis of the pupil shape, and the rotation information about the pupil, and store them in a memory, while storing the retinal images respectively corresponding to the pupil data.

[0109] According to an embodiment, in operation 1350, the electronic device 200 may determine whether the obtained eye image is an eye image of a user corresponding to a pre-stored eye model based on the obtained retinal image. In an embodiment, when the obtained eye image corresponds to a pre-stored eye model of the user, the electronic device 200 may recognize at least one of the gaze direction or position of the eye based on the pre-stored eye model. Here, there may be one or more previously stored user eye models.

[0110] In an embodiment, when the obtained eye image does not correspond to a pre-stored user eye model, the electronic device 200 may obtain an eye model based on pupil data and a retinal image obtained from the obtained eye image in operation 1370. In an embodiment, the electronic device 200 may obtain a plurality of eye images while sequentially displaying objects for guiding the user to gaze at different positions on a display to obtain an eye model. In an embodiment, the electronic device 200 may display an object for guiding gaze on a display (at a position corresponding to an area where a retinal image is not obtained when creating an eye model based on the obtained eye image).

[0111] In an embodiment, when the obtained eye image does not correspond to a pre-stored user eye model, the electronic device 200 may identify at least one of a gaze direction or an eye position corresponding to the eye image based on the obtained eye model in operation 1390. Here, the obtained eye model may be an eye model obtained based on an eye image obtained in real time, or an eye model obtained in advance and stored in a memory.

[0112] In an embodiment, in operation 1391, the electronic device 200 may identify an area of ​​the pupil corresponding to the pupil from the eye image, and obtain a gaze direction based on the identified area of ​​the pupil in operation 1393. In an embodiment, the electronic device 200 may identify an area of ​​the pupil from the eye image, restore a circular shape of the pupil using the position and shape of the area of ​​the pupil, and obtain a gaze direction of the user.

[0113] In an embodiment, the electronic device 200 may detect whether the position of the eye has changed based on the obtained retinal image and the eye model corresponding to the obtained gaze direction in operation 1395. In an embodiment, when it is determined that the position of the eye has not changed, the electronic device 200 may identify the position of the eye from the obtained eye model and update the eye model with the obtained gaze direction.

[0114] In an embodiment, upon detecting eye repositioning, the electronic device 200 may obtain the position of the eye based on the retinal image and the eye model in operation 1397. In an embodiment, the electronic device 200 may estimate the pupil centerline and the eye centerline spaced apart from the pupil centerline by the eye radius based on the pupil data, and estimate the eye center point on the eye centerline based on the ratio (proportion) of the sizes of the retinal image and the eye model. In an embodiment, the electronic device 200 may update the eye model based on the newly estimated eye position.

[0115] According to an embodiment, the electronic device 101 or 200 may include a camera and at least one processor 120 or 220. The at least one processor 120 or 220 may be configured to obtain an eye image of the user through the camera 210. The at least one processor 120 or 220 may be configured to obtain pupil data and a retinal image I from the obtained eye image. The at least one processor 120 or 220 may be configured to obtain an eye model M including a retinal image I based on the obtained pupil data and the obtained retinal image I. The at least one processor 120 or 220 may be configured to recognize at least one of a position of an eye E or a gaze direction G based on the obtained eye model M.

[0116] According to an embodiment, the electronic device 101 or 200 may further include a light source 240 that emits light of a preset wavelength band corresponding to the camera 210 toward the eye E.

[0117] According to an embodiment, at least one processor 120 or 220 may be configured to receive a plurality of eye images according to the rotation of the eye E from the camera 210. At least one processor 120 or 220 may be configured to obtain a plurality of pupil data and a retinal image I from the plurality of eye images. At least one processor 120 or 220 may be configured to obtain an eye model M based on the plurality of pupil data and the plurality of retinal images I.

[0118] According to an embodiment, the electronic device 101 or 200 may further include a display 250. The at least one processor 120 or 220 may be configured to sequentially display on the display 250 objects for guiding the user to gaze at different positions.

[0119] According to an embodiment, in the electronic device 101 or 200, at least one processor 120 or 220 may be configured to identify an area of ​​the pupil P from the eye image. At least one processor 120 or 220 may be configured to obtain at least one of the position of the pupil P, the shape of the pupil P, or the gaze direction G based on the identified area of ​​the pupil P.

[0120] According to an embodiment, in the electronic device 101 or 200, at least one processor 120 or 220 may be configured to obtain the position of the eye E or the shape of the eye E based on at least one of the obtained position of the pupil P, the shape of the pupil P, or the gaze direction G.

[0121] According to an embodiment, in the electronic device 101 or 200, at least one processor 120 or 220 may be configured to obtain an eye model M by mapping a retinal image I corresponding to pupil data to an eye E to correspond to the pupil data.

[0122] According to an embodiment, in the electronic device 101 or 200, at least one processor 120 or 220 may be configured to recognize at least one of a position or a gaze direction G of an eye E corresponding to an eye image based on the obtained eye model M.

[0123] According to an embodiment, in the electronic device 101 or 200, at least one processor 120 or 220 may be configured to determine whether the eye image corresponds to a pre-stored eye image of a user corresponding to a pre-stored eye model M based on the obtained retinal image I.

[0124] According to an embodiment, in the electronic device 101 or 200, at least one processor 120 or 220 may be configured to identify a region of the pupil P corresponding to the pupil P from the eye image. At least one processor 120 or 220 may be configured to obtain a gaze direction G based on the identified region of the pupil P.

[0125] According to an embodiment, in the electronic device 101 or 200, at least one processor 120 or 220 may be configured to detect whether the position of the eye E is changed based on the obtained retinal image I and the eye model M corresponding to the obtained gaze direction G.

[0126] According to an embodiment, in the electronic device 101 or 200, at least one processor 120 or 220 may be configured to obtain the position of the eye E based on the retinal image I and the eye model M when a change in the position of the eye E is detected.

[0127] According to an embodiment, the method 1300 for operating the electronic device 101 or 200 may include: obtaining (1310) an eye image of a user through the camera 210. The method 1300 for operating the electronic device 101 or 200 may include: obtaining (1330) pupil data and a retinal image I from the obtained eye image. The method 1300 for operating the electronic device 101 or 200 may include: obtaining (1370) an eye model M including the retinal image I based on the obtained pupil data and the obtained retinal image I. The method 1300 for operating the electronic device 101 or 200 may include: identifying at least one of a position of an eye E or a gaze direction G based on the obtained eye model M.

[0128] According to an embodiment, in the method 1300 for operating the electronic device 101 or 200, obtaining (1310) an eye image may include: receiving a plurality of eye images according to the rotation of the eye E from the camera 210. Obtaining (1330) pupil data and a retinal image I may include: obtaining the plurality of pupil data and the retinal image I from the plurality of eye images. Obtaining (1370) an eye model M may include: obtaining the eye model M based on the plurality of pupil data and the plurality of retinal images I.

[0129] According to an embodiment, in the method 1300 for operating the electronic device 101 or 200 , obtaining ( 1310 ) an eye image may include obtaining a plurality of eye images while sequentially displaying objects for guiding the eye E to gaze at different positions on the display 250 .

[0130] According to an embodiment, in the method 1300 for operating the electronic device 101 or 200, obtaining (1330) pupil data and a retinal image I may include: identifying an area of ​​the pupil P from the eye image, and obtaining at least one of a position of the pupil P, a shape of the pupil P, or a gaze direction G based on the area of ​​the pupil P. Obtaining (1370) an eye model M may include: obtaining a position of the eye E or a shape of the eye E based on at least one of the obtained position of the pupil P, the shape of the pupil P, or the gaze direction G.

[0131] According to an embodiment, in the method 1300 for operating the electronic device 101 or 200, identifying (1390) at least one of the position of the eye E or the gaze direction G may include: identifying at least one of the position of the eye E or the gaze direction G corresponding to the eye image based on the obtained eye model M.

[0132] According to an embodiment, the method 1300 for operating the electronic device 101 or 200 may further include determining ( 1350 ) whether the eye image corresponds to a pre-stored eye image of a user corresponding to a pre-stored eye model M based on the obtained retinal image I.

[0133] According to an embodiment, in the method 1300 for operating the electronic device 101 or 200, identifying (1390) at least one of the position of the eye E or the gaze direction G may include: identifying (1391) an area of ​​the pupil P corresponding to the pupil P from the eye image. Identifying (1390) at least one of the position of the eye E or the gaze direction G may include: obtaining (1393) the gaze direction G based on the identified area of ​​the pupil P.

[0134] According to an embodiment, in the method 1300 for operating the electronic device 101 or 200, identifying (1390) at least one of the position of the eye E or the gaze direction G may include: detecting (1395) whether the position of the eye E has changed based on the obtained retinal image I and the eye model M corresponding to the obtained gaze direction G. Identifying (1390) at least one of the position of the eye E or the gaze direction G may include: obtaining (1397) the position of the eye E based on the retinal image I and the eye model M when a change in the position of the eye E is detected.

[0135] According to an embodiment, in a non-transitory computer-readable storage medium storing one or more programs, the one or more programs may include obtaining (1310) an eye image of a user through the camera 210. The one or more programs may include obtaining (1330) pupil data and a retinal image I from the obtained eye image. The one or more programs may include obtaining (1370) an eye model M including the retinal image I based on the obtained pupil data and the obtained retinal image I. The one or more programs may include identifying (1390) at least one of a position of an eye E or a gaze direction G based on the obtained eye model M.

Claims

1. An electronic device (101; 200), comprising: A camera (210), and At least one processor (120; 220) configured to: obtaining an eye image of a user's eye from the camera (210), obtaining pupil data and a retinal image (I) from said eye image, obtaining an eye model (M) including the retinal image (I) based on the pupil data and the retinal image (I), and At least one of a position or a gaze direction (G) of the eye is identified based on the eye model (M).

2. The electronic device (101; 200) according to claim 1, further comprising a light source (240), the light source (240) being configured to emit light of a preset wavelength band toward the eye (E).

3. The electronic device (101; 200) according to any one of claims 1 to 2, in, The at least one processor (120; 220) is further configured to: receiving a plurality of eye images according to a rotation of the eye (E) from the camera (210), the plurality of eye images including the eye image, obtaining a plurality of pupil data including the pupil data and a plurality of retinal images including the retinal images from the plurality of eye images (I), and The eye model (M) is obtained based on the plurality of pupil data and the plurality of retinal images (I).

4. The electronic device (101; 200) according to claim 3, further comprising a display (250), in, The at least one processor (120; 220) is also configured to sequentially display objects on the display (250) for guiding the user to gaze at different positions.

5. The electronic device (101; 200) according to any one of claims 1 to 4, wherein: The at least one processor (120; 220) is further configured to identify an area of ​​a pupil (P) from the eye image, and to obtain at least one of a position of the pupil (P), a shape of the pupil (P), or the gaze direction (G) based on the area of ​​the pupil (P).

6. The electronic device (101; 200) according to claim 5, wherein: The at least one processor (120; 220) is further configured to obtain any one of the position of the eye (E) or the shape of the eye (E) based on at least one of the position of the pupil (P), the shape of the pupil (P), or the gaze direction (G).

7. The electronic device (101; 200) according to any one of claims 1 to 6, wherein: The at least one processor (120; 220) is further configured to obtain the eye model (M) by mapping the retinal image (I) to correspond to the pupil data.

8. The electronic device (101; 200) according to any one of claims 1 to 7, wherein: The at least one processor (120; 220) is further configured to identify at least one of a gaze direction (G) corresponding to the eye image or a position of the eye (E) based on the eye model (M).

9. The electronic device (101; 200) according to claim 8, wherein: The at least one processor (120; 220) is further configured to determine, based on the retinal image (I), whether the eye image corresponds to a pre-stored eye image of the user corresponding to a pre-stored eye model (M).

10. The electronic device (101; 200) according to claim 8 or 9, wherein: The at least one processor (120; 220) is further configured to: identify a region of a pupil (P) corresponding to the pupil (P) from the eye image, and obtain the gaze direction (G) based on the region of the pupil (P).

11. The electronic device (101; 200) according to claim 10, wherein: The at least one processor (120; 220) is further configured to detect whether the position of the eye (E) changes based on the retinal image (I) and an eye model (M) corresponding to the gaze direction (G).

12. The electronic device (101; 200) according to claim 11, wherein: The at least one processor (120; 220) is further configured to obtain the position of the eye (E) based on the retinal image (I) and the eye model (M) when a change in the position of the eye (E) is detected.

13. A method (1300) for operating an electronic device (101; 200), the method comprising: obtaining (1310) an eye image of a user's eye (E) from a camera (210), obtaining (1330) pupil data and a retinal image (I) from the eye image, obtaining (1370) an eye model (M) comprising the retinal image (I) based on the pupil data and the retinal image (I), and At least one of a position or a gaze direction (G) of the eye (E) is identified (1390) based on the eye model (M).

14. The method (1300) according to claim 13, in, Obtaining (1310) the eye image comprises: receiving from the camera (210) a plurality of eye images according to a rotation (E) of the eye, the plurality of eye images comprising the eye image, wherein obtaining (1330) the pupil data and the retinal image (I) comprises: obtaining a plurality of pupil data including the pupil data and a plurality of retinal images including the retinal image (I) from the plurality of eye images, and Wherein, obtaining (1370) the eye model (M) comprises: obtaining the eye model (M) based on the plurality of pupil data and the plurality of retinal images (I).

15. The method (1300) according to any one of claims 13 to 14, in, Obtaining (1330) the pupil data and the retinal image (I) comprises: identifying an area of ​​the pupil (P) from the eye image, and obtaining at least one of a position of the pupil (P), a shape of the pupil (P), or the gaze direction (G) based on the area of ​​the pupil (P), and Wherein, obtaining (1370) the eye model (M) comprises: obtaining any one of the position of the eye (E) or the shape of the eye (E) based on at least one of the position of the pupil (P), the shape of the pupil (P) or the gaze direction (G).

Images