Eye tracking method and eye tracking device

Abstract

An eye tracking method and an eye tracking device are disclosed. The eye tracking method comprises determining a difference between an input image and a reconstructed image, selecting one of the input image, the reconstructed image and a replacement image based on the determined difference, and performing eye tracking for the one of the input image, the reconstructed image and the replacement image.

Description

[0001] This application claims priority to Korean Patent Application No. 10-2019-0163642, filed on December 10, 2019, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0002] The methods and apparatus consistent with the exemplary embodiments relate to a method and apparatus for tracking the eye based on eye reconstruction. Background Technology

[0003] Head-up display (HUD) devices provide various driving information to aid the driver by displaying virtual images in front of them. Recently, three-dimensional (3D) HUD devices have been under development. For example, 3D HUD devices utilize augmented reality (AR). In this example, driving information is displayed as an overlay on real objects, allowing the driver to more intuitively recognize the information. Various types of 3D displays exist. Among these, eye-tracking 3D displays with relatively high resolution and relatively high degrees of freedom can be applied to 3D HUD devices. Summary of the Invention

[0004] One or more exemplary embodiments may at least solve the above-described problems and / or disadvantages, as well as other disadvantages not described above. Furthermore, exemplary embodiments are not required to overcome the above-described disadvantages, and exemplary embodiments may not overcome any of the above-described problems.

[0005] According to one aspect of an exemplary embodiment, an eye-tracking method is provided, the eye-tracking method comprising: generating a reconstructed image by performing eye reconstruction on an input image; determining a difference between the input image and the reconstructed image; determining a target image by selecting one of the input image, the reconstructed image, and a replacement image based on the determined difference; and performing eye tracking based on the target image.

[0006] Eye reconstruction may include reducing noise components in the input image. The generation steps may include generating a reconstructed image using high-priority portions of the principal component vectors corresponding to the input image, where each principal component vector may correspond to a predetermined eigenface based on principal component analysis for various facial images.

[0007] The selection steps may include: if the determined difference is less than a first threshold, selecting the input image as the target image; if the determined difference is greater than the first threshold and less than a second threshold, selecting the reconstructed image as the target image; and if the determined difference is greater than the second threshold, selecting the substitute image as the target image.

[0008] The replacement image may differ from both the input image and the reconstructed image. The eye-tracking method may further include selecting, from a database of sample images, the sample image with the highest similarity to the input image as the replacement image. The similarity may be determined based on a comparison between feature points of the input image and feature points of each of the sample images. The feature points of the input image and the feature points of each of the sample images may each be extracted from regions other than the eyes. The sample images may correspond to images from which eye tracking has been successfully performed previously. The eye-tracking method may further include storing the input image as a sample image in the database if eye tracking is successful based on the input image or the reconstructed image.

[0009] If eye detection is successful for the input image, the generated steps can be performed. If a substitute image is selected as the target image, the steps performed may include: performing eye tracking based on eye position information mapped to the substitute image.

[0010] According to one aspect of an exemplary embodiment, an electronic device is provided, the electronic device comprising: a processor; a memory configured to store instructions executable by the processor; and a camera configured to generate an input image by photographing a user, wherein, when the instructions are executed by the processor, the processor may be configured to: generate a reconstructed image by performing eye reconstruction on the input image, determine a difference between the input image and the reconstructed image, determine a target image by selecting one of the input image, the reconstructed image, and a replacement image based on the determined difference, and perform eye tracking based on the target image.

[0011] According to one aspect of an exemplary embodiment, an eye-tracking device is provided, the eye-tracking device comprising: a processor; and a memory configured to store instructions executable by the processor, wherein, when the instructions are executed by the processor, the processor is configured to: generate a reconstructed image by performing eye reconstruction on an input image, determine a difference between the input image and the reconstructed image, determine a target image by selecting one of the input image, the reconstructed image, and a replacement image based on the determined difference, and perform eye tracking based on the target image. Attached Figure Description

[0012] The above and / or other aspects will become clearer by describing specific exemplary embodiments with reference to the accompanying drawings, in which:

[0013] Figure 1 This illustrates the operation of an eye-tracking device using an input image, a reconstructed image, and a replacement image according to an exemplary embodiment.

[0014] Figure 2 This is a flowchart illustrating eye detection processing and eye tracking processing according to an exemplary embodiment;

[0015] Figure 3 This is a flowchart illustrating a process for determining a target image and performing eye tracking according to an exemplary embodiment;

[0016] Figure 4 An enhanced appearance space is shown according to an exemplary embodiment;

[0017] Figure 5 This illustrates the eye reconstruction process according to an exemplary embodiment;

[0018] Figure 6 This illustrates the operation of selecting a target image if the difference is less than a second threshold, according to an exemplary embodiment.

[0019] Figure 7 This illustrates the operation of selecting a target image if the difference is greater than a second threshold, according to an exemplary embodiment.

[0020] Figure 8 This illustrates an operation of matching features of an input image with features of a sample image according to an exemplary embodiment;

[0021] Figure 9 The illustration shows sample images stored in a database according to an exemplary embodiment;

[0022] Figure 10 This is a flowchart illustrating an eye-reconstruction-based eye-tracking method according to an exemplary embodiment;

[0023] Figure 11 This is a block diagram illustrating an eye-tracking device based on eye reconstruction according to an exemplary embodiment; and

[0024] Figure 12 This is a block diagram illustrating an electronic device including an eye-tracking device according to an exemplary embodiment. Detailed Implementation

[0025] The detailed structural or functional descriptions below are provided by way of example only, and various changes and modifications may be made to the exemplary embodiments. Therefore, the exemplary embodiments are not to be construed as limited to the disclosure and should be understood to include all changes, equivalents, and alternatives within the scope of the disclosed technology.

[0026] Components may be described here using terms such as first, second, etc. Each of these terms is not used to define the nature, order, or sequence of the corresponding component, but only to distinguish the corresponding component from one or more other components. For example, the first component may be referred to as the second component, and similarly, the second component may be referred to as the first component.

[0027] Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It will also be understood that when the terms “comprising” and / or “including” are used herein, they specify the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof.

[0028] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0029] In the following description, exemplary embodiments will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used for the same elements.

[0030] Figure 1 This illustrates the operation of an eye-tracking device using an input image, a reconstructed image, and a replacement image, according to an exemplary embodiment. (Refer to...) Figure 1 The eye-tracking device 100 can perform eye tracking on an input image 110 and output eye position information as a result of the eye tracking. The input image 110 may include the face of a user (e.g., a viewer or driver). The eye position information can be used in various applications (such as automated stereoscopic 3D displays and driver status monitoring). Automated stereoscopic 3D displays may include various types of displays (e.g., 3D head-up displays (HUDs), 3D TVs, and 3D mobile devices).

[0031] An autostereoscopic 3D display can show different images to a user's two eyes. To achieve this, information about the user's eye positions is required. For example, a left image for the left eye and a right image for the right eye can be prepared to provide a 3D image. In this example, the 3D display device determines the user's eye positions based on the eye position information and provides the left image to the left eye and the right image to the right eye. In this way, the user can view the 3D image.

[0032] If the input image 110 is generated by photographing the user when the lighting is suitable and the user's eyes are fully exposed, it is relatively easy to track the user's eyes. Conversely, if the input image 110 is generated by photographing the user when the lighting is insufficient, or when the user's eyes are covered by sunglasses, or when there is a light reflection component around the user's eyes, it may be difficult to track the user's eyes. The eye-tracking device 100 can selectively use the input image 110, the reconstructed image 121, and the alternative image 131 as appropriate, thereby successfully performing eye tracking even in such unsuitable environments.

[0033] Input image 110 can be a video comprising multiple frames. Input image 111 can correspond to a single frame of input image 110. The eye-tracking process described below can be performed sequentially for each frame of input image 110.

[0034] Upon receiving an input image 111, the eye-tracking device 100 can generate a reconstructed image 121 by performing eye reconstruction on the input image 111. Eye reconstruction can include various techniques for reducing noise components in the input image 111. For example, noise components can include high-frequency components (e.g., light reflected from glasses or halos) and occlusion components (e.g., sunglasses, thick glasses, or hair). Dimensionality reduction via principal component analysis (PCA) can be used for eye reconstruction. Detailed examples of eye reconstruction will be described later.

[0035] When generating the reconstructed image 121, the eye-tracking device 100 can determine the difference between the input image 111 and the reconstructed image 121 by comparing the input image 111 with the reconstructed image 121. Hereinafter, the determined difference will be referred to as the difference value. For example, the eye-tracking device 100 can determine the difference between the input image 111 and the reconstructed image 121 by comparing corresponding pixels of the input image 111 with those of the reconstructed image 121. Corresponding pixels can be pixels located at the same position in the various images.

[0036] The eye-tracking device 100 can determine a target image by selecting one of an input image 111, a reconstructed image 121, and a substitute image 131 based on the difference. The target image can be an image that serves as the target for eye tracking and corresponds to one of the input image 111, the reconstructed image 121, and the substitute image 131. For example, if the difference is less than a first threshold, the input image 111 can be selected as the target image. If the difference is greater than the first threshold and less than a second threshold, the reconstructed image 121 can be selected as the target image. If the difference is greater than the second threshold, the substitute image 131 can be selected as the target image. It can be assumed that the second threshold is greater than the first threshold.

[0037] The alternative image 131 may differ from the input image 111 and the reconstructed image 121, and is selected from the sample images 130. The sample images 130 may include various images in a state suitable for eye tracking. The sample image with the highest similarity to the input image 111 can be selected from the sample images 130 as the alternative image 131. For example, the similarity between each sample image and the input image 111 can be determined based on a comparison between feature points of the input image 111 and feature points of each sample image in the sample images 130.

[0038] The eye-tracking device 100 can select the most suitable image for eye tracking from the input image 111, the reconstructed image 121, and the alternative image 131 by comparing similarity and a threshold. For example, if the input image 111 is suitable for eye tracking, then the input image 111 can be selected as the target image because there is a small difference (less than a first threshold) between the input image 111 and the reconstructed image 121.

[0039] If the input image 111 includes noise components (such as light reflections), the reconstructed image 121 may be more suitable for eye tracking than the input image 111. In this example, noise components can be removed through eye reconstruction, and due to the presence or absence of noise components, there may be a difference between the input image 111 and the reconstructed image 121 (e.g., a difference greater than a first threshold and less than a second threshold). Therefore, the reconstructed image 121 can be selected as the target image.

[0040] If, as in the example where the user is wearing sunglasses, there are many noise components in the input image 111, then the alternative image 131 may be more suitable for eye tracking than both the input image 111 and the reconstructed image 121. In this example, during eye reconstruction processing, there may be a large difference between the input image 111 and the reconstructed image 121 (e.g., a difference greater than a second threshold), therefore, the alternative image 131 may be selected as the target image.

[0041] Components that interfere with eye tracking can be removed from the components constituting the input image 111 through eye reconstruction processing. For example, through eye reconstruction processing, components corresponding to a general face can be preserved, while components not corresponding to a general face can be removed. One of the input image 111, the reconstructed image 121, and the replacement image 131 can be selected based on the proportion of components removed from the input image 111 by the eye reconstruction processing. For example, in the case of high-frequency components, the proportion of components removed can be relatively low, so the reconstructed image 121 can be selected as the target image. In the case of occluded components, the proportion of components removed can be relatively large, so the replacement image 131 can be selected as the target image.

[0042] When a target image is selected, the eye-tracking device 100 can generate eye position information by performing eye tracking on the target image. The eye-tracking device 100 can sequentially receive multiple frames and track the eyes in each frame. The eye position information can include the position of the eyes appearing in each frame. For example, the position of the eyes can be represented as the coordinates of the eyes in the image.

[0043] Figure 2 This is a flowchart illustrating eye detection processing and eye tracking processing according to an exemplary embodiment. (Refer to...) Figure 2 If the first frame of the input image is received, in operation 210, the eye-tracking device may perform eye detection on the first frame of the input image. The eye-tracking device may determine the eye detection region, including the user's eyes, through eye detection processing. In operation 220, the eye-tracking device determines whether the eye detection was successful. If the eye detection fails, operation 210 may be performed on the second frame of the input image; if the eye detection is successful, operation 230 may be performed. That is, operations 210 and 220 may be performed iteratively for each frame until the eye detection is successful.

[0044] In operation 230, the eye-tracking device performs eye reconstruction on the first frame of the input image. After eye reconstruction, a reconstructed image and a replacement image corresponding to the first frame of the input image are generated, and one of the first frame of the input image, the reconstructed image, and the replacement image can be selected as the target image. In operation 240, the eye-tracking device performs eye tracking on the target image. The eye-tracking device determines the eye-tracking region for eye tracking based on the eye detection region. The eye-tracking device extracts feature points in the eye-tracking region and performs eye tracking by aligning the extracted feature points.

[0045] In operation 250, the eye-tracking device can determine whether eye tracking is successful. If the eye-tracking area includes the user's eyes, eye tracking is considered successful. If the eye-tracking area does not include the user's eyes, eye tracking is considered unsuccessful. If eye tracking is successful, the eye-tracking device can update the eye-tracking area and perform operation 230 for the second frame of the input image. For example, the eye-tracking device can adjust the position of the eye-tracking area based on the position of the eyes. Specifically, the eye-tracking device can adjust the position of the eye-tracking area such that the midpoint between the eyes can be placed at the center of the eye-tracking area. The eye-tracking device can determine a target image corresponding to the second frame of the input image and use the target image and the updated eye-tracking area to continue eye tracking.

[0046] Figure 3 This is a flowchart illustrating a process for determining a target image and performing eye tracking according to an exemplary embodiment. Figure 3 Operations 310 to 390 can correspond to Figure 2 Operations 230 and 240 are marked with dashed boxes, and when in Figure 2 If the eye detection is successful in operation 220, operation 310 can be executed.

[0047] Reference Figure 3 In operation 310, the eye-tracking device can generate a reconstructed image by performing eye reconstruction. Eye reconstruction can include various techniques for reducing noise components in the input image. For example, dimensionality reduction via PCA can be applied to eye reconstruction, which will refer to... Figure 4 and Figure 5 Further description.

[0048] Figure 4 An augmented appearance space according to an exemplary embodiment is illustrated. Multiple principal component vectors can be obtained by performing PCA on various reference face images. The dimension of the principal component vectors can be equal to the dimension of the data (e.g., pixels) of each image. In this example, principal component vectors ranging from dominant face components to exceptional face components can be obtained. The proportion of images in the reference face images that include dominant face shapes can be relatively high. In this example, the dominant face component can correspond to a standard face, and the exceptional face component can correspond to noise. Since the principal component vectors can be orthogonal to each other, face images in all input images can be represented by principal component vectors.

[0049] Principal component vectors can be reinterpreted as facial images, and thus the interpreted facial images can be called eigenfaces. That is, eigenfaces can be determined based on PCA across various (i.e., multiple) facial images, and each principal component vector can correspond to an eigenface. Facial images in the input image can be represented in the augmented appearance space using eigenfaces.

[0050] Reference Figure 4 Facial image A can be generated by λ i A i The sum of 'a's is represented by 'a'. i Let λ represent the principal component vector. i λ represents the coefficients of the principal component vector. i It can also be called an appearance parameter. λ i Can indicate A i The proportion in facial image A. Based on each λ i A i The value of can be represented as follows: Figure 4 The facial image in the image. Index i can have values ​​from 1 to n. A with a smaller i... iIt can be a composition closer to the standard, and A with a larger i. i It could be a component that is closer to noise.

[0051] Figure 5 This illustrates an eye reconstruction process according to an exemplary embodiment. (See reference...) Figure 5 The input image I can range from λ0A0 to λ n A n The sum of the terms indicates that the reconstructed image I' can be derived from λ0A0 to λ m A m The sum is represented as n, where n is a positive integer and m is an integer greater than or equal to 0. n can be greater than m, which can be considered as reducing dimensionality through eye reconstruction. Furthermore, components with relatively large indices (relatively noise-like components) can be removed from the input image I; therefore, noise components (such as high-frequency components or occlusion components) can be removed from the input image I. In the input image I, λ m+1 I m+1 or λ n A n This can be referred to as component removal or noise component removal. Principal component vectors with smaller indices can be represented as having higher priority. In this example, the eye-tracking device can use the principal component vectors (A0 to A10) corresponding to the input image. n At least one principal component vector (A0 to A10) with relatively high priority among them. m (This is used to generate the reconstructed image.)

[0052] Refer to Figure 3 In operation 320, the eye-tracking device can determine the difference between the input image and the reconstructed image. For example, the eye-tracking device can determine the difference by comparing corresponding pixels of the input image and the reconstructed image. Larger coefficients of the various noise components removed during the eye reconstruction process can be interpreted as the presence of significant noise in the input image; in this example, the difference between the input image and the reconstructed image can be determined to be large. Therefore, the likelihood that the reconstructed image or substitute image will be used as the target image increases.

[0053] The difference can be determined based on the eye region. That is, the eye-tracking device can determine the difference by comparing the corresponding pixels of the eye region in the input image with those in the reconstructed image. This is because comparing changes in the eye region is effective when compared to comparing changes across the entire image caused by eye reconstruction, whether using the reconstructed or substitute image.

[0054] For example, eye tracking might be feasible if the user is wearing a mask, but difficult if the user is wearing sunglasses. Therefore, it is essential to detect when the user is wearing sunglasses instead of a mask and to use a reconstructed or alternative image for that situation. In this example, the eye region can be determined based on the eye detection region or the eye tracking region.

[0055] In operation 330, the eye-tracking device compares the difference with a first threshold. If the difference is less than the first threshold, in operation 340, the eye-tracking device identifies the input image as the target image. If the difference is greater than the first threshold, in operation 350, the eye-tracking device compares the difference with a second threshold. If the difference is less than the second threshold, in operation 360, the eye-tracking device identifies the reconstructed image as the target image. If the difference is greater than the second threshold, in operation 370, the eye-tracking device selects a substitute image from the sample images. In operation 380, the eye-tracking device identifies the substitute image as the target image. In operation 390, the eye-tracking device performs eye tracking on the target image.

[0056] Figure 6 This illustrates an operation, according to an exemplary embodiment, of selecting a target image if the difference is less than a second threshold. (Refer to...) Figure 6 A reconstructed image 620 can be generated by eye reconstruction of the input image 610. Through eye reconstruction, high-frequency components, including light reflections from the glasses, can be removed from the input image 610, and the reconstructed image 620 can be represented as relatively smooth when compared with the input image 610.

[0057] After eye reconstruction, the difference between the input image 610 and the reconstructed image 620 can be determined. If the difference is less than a first threshold, the input image 610 can be identified as the target image. If the difference is greater than the first threshold and less than a second threshold, the reconstructed image 620 can be identified as the target image. For example, since high-frequency components included in the input image 610 are removed, the reconstructed image 620 can be identified as the target image.

[0058] Figure 7 This illustrates the operation of selecting a target image if the difference is greater than a second threshold, according to an exemplary embodiment. (Refer to...) Figure 7 A reconstructed image 720 can be generated by eye reconstruction of the input image 710. Through eye reconstruction, high-frequency components including light reflection from sunglasses and occlusion components including sunglasses can be removed from the input image 710, and the reconstructed image 720 can be represented as relatively smooth when compared with the input image 710.

[0059] After eye reconstruction, the difference between the input image 710 and the reconstructed image 720 can be determined. If the difference is less than a first threshold, the input image 710 can be determined as the target image. If the difference is greater than the first threshold and less than a second threshold, the reconstructed image 720 can be determined as the target image. If the difference is greater than the second threshold, a replacement image 740 can be generated. For example, since high-frequency components and occlusion components included in the input image 710 are removed, the difference can be determined to be greater than the second threshold.

[0060] In this example, feature matching can be performed between the input image 710 and each sample image in the sample images 730, and the sample image most similar to the input image 710 can be selected from the sample images 730 as a replacement image 740. The replacement image 740 can then be determined as the target image and used for eye tracking. The sample images 730 can include various facial images that are easy to track the eyes. For example, the sample images 730 can include facial images without glasses and / or facial images with glasses but without light reflection within the glasses. Figure 7 In the example, the sample image containing glasses can be determined to be most similar to the input image 710 containing sunglasses, and therefore the sample image can be selected as the alternative image 740.

[0061] Figure 8 This illustrates an operation of matching features of an input image with features of a sample image according to an exemplary embodiment. (Refer to...) Figure 8 The similarity between the input image 810 and the sample image 820 can be determined by comparing the feature points of the input image 810 with those of the sample image 820. The feature points of both the input image 810 and the sample image 820 can be extracted from regions other than the eyes. For example, feature points can be extracted from the nose, mouth, and facial contours. If sunglasses or other occlusions are present at the eye location, a substitute image is likely to be used. Therefore, extracting feature points from regions other than the eyes is advantageous in obtaining a substitute image similar to the input image 810 and in obtaining an appropriate viewpoint from the substitute image.

[0062] Figure 9 A sample image stored in a database is shown according to an exemplary embodiment. (Refer to...) Figure 9Database 910 may include sample images 921 to 923 (i.e., sample image 921, sample image 922, and sample image 923). Database 910 may reside in the eye-tracking device or in another device different from the eye-tracking device. If database 910 resides in another device, the eye-tracking device may access database 910 via a network to use sample images 921 to 923. Sample images 921 to 923 may each have a state suitable for eye tracking. A state suitable for eye tracking may include a state where high-frequency components or occlusion components are absent.

[0063] Sample images 921 to 923 may correspond to images from which eye tracking has been successfully performed previously. Here, the term "previous" may refer to a time prior to performing eye tracking for the current frame. For example, in the process of manufacturing an eye-tracking device, images suitable as replacement images may be selected from images from which eye tracking has been successfully performed as sample images 921 to 923. In another example, sample images 921 to 923 may be images stored during the process of performing eye tracking for each frame of a previous input image or during the process of performing eye tracking for a previous frame of the current input image. For example, if eye tracking is successful based on the current frame or a reconstructed image corresponding to the current frame, the current frame or a reconstructed image corresponding to the current frame may be stored in database 910 for use as a future replacement image.

[0064] Eye position information 931 to 933 (i.e., eye position information 931, eye position information 932, and eye position information 933) can be mapped to sample images 921 to 923. Since sample images 921 to 923 are sample images for which eye tracking has been successfully performed previously, eye position information 931 to 933 can be ensured relative to sample images 921 to 923 when eye tracking was previously successful. Therefore, when sample images 921 to 923 are used as replacement images, the already ensured eye position information 931 to 933 can be used instead of tracking the eyes in sample images 921 to 923 individually. For example, assuming sample image 921 is selected as a replacement image for the input image, the eye tracking device can immediately output eye position information 931 mapped to sample image 921, instead of generating eye position information by tracking the eyes in sample image 921.

[0065] Figure 10 This is a flowchart illustrating an eye-reconstruction-based eye-tracking method according to an exemplary embodiment. (Refer to...) Figure 10In operation 1010, the eye-tracking device generates a reconstructed image by performing eye reconstruction on the input image. In operation 1020, the eye-tracking device determines the difference between the input image and the reconstructed image. In operation 1030, the eye-tracking device determines a target image by selecting one of the input image, the reconstructed image, and a substitute image based on the determined difference. In operation 1040, the eye-tracking device performs eye tracking based on the target image. Additionally, refer to... Figures 1 to 9 The provided description can be applied to Figure 10 Eye-tracking methods.

[0066] Figure 11 This is a block diagram illustrating an eye-reconstruction-based eye-tracking device according to an exemplary embodiment. (Refer to...) Figure 11 The eye-tracking device 1100 can perform at least one of the operations described and illustrated herein regarding eye tracking, and provide the user with eye position information as a result of eye tracking.

[0067] The eye-tracking device 1100 may include at least one processor 1110 and a memory 1120. The memory 1120 may be connected to the processor 1110 and stores instructions executable by the processor 1110, data to be computed by the processor 1110, or data processed by the processor 1110. The memory 1120 may include a non-transitory computer-readable medium (e.g., high-speed random access memory) and / or a non-volatile computer-readable medium (e.g., at least one disk storage device, flash memory device, or another non-volatile solid-state memory device).

[0068] Processor 1110 can execute instructions to perform reference Figures 1 to 10 At least one of the operations described. If the instructions stored in memory 1120 are executed by processor 1110, processor 1110 can generate a reconstructed image by performing eye reconstruction on the input image, determine the difference between the input image and the reconstructed image, determine a target image by selecting one of the input image, the reconstructed image and the alternative image based on the determined difference, and perform eye tracking based on the target image.

[0069] Figure 12 This is a block diagram illustrating an electronic device including an eye-tracking device according to an exemplary embodiment. (Refer to...) Figure 12 The electronic device 1200 may include reference Figures 1 to 11 The described eye-tracking device, or the one that performs the reference Figures 1 to 12 Describe the functionality of the eye-tracking device.

[0070] Electronic device 1200 may include a processor 1210, a memory 1220, a camera 1230, a storage device 1240, an input device 1250, an output device 1260, and a network interface 1270. The processor 1210, memory 1220, camera 1230, storage device 1240, input device 1250, output device 1260, and network interface 1270 may communicate with each other via a communication bus 1280. For example, electronic device 1200 may include smartphones, tablet PCs, laptops, desktop PCs, wearable devices, smart home appliances, smart speakers, and smart cars. Specifically, electronic device 1200 may be installed in a vehicle to provide functionality for a three-dimensional head-up display (3DHUD).

[0071] Processor 1210 can execute instructions and functions that will be executed in electronic device 1200. For example, processor 1210 can process instructions stored in memory 1220 or storage device 1240. Processor 1210 can execute references Figures 1 to 11 At least one of the operations described.

[0072] Memory 1220 can store references to be processed Figures 1 to 11 Information on at least one of the described operations. Memory 1220 may include a computer-readable storage medium or a computer-readable storage device. Memory 1220 may store instructions to be executed by processor 1210 and related information stored when software or applications run via electronic device 1200.

[0073] Camera 1230 can capture still images, video images, or both. Camera 1230 can capture the face of a user for eye tracking and generate an input image. Camera 1230 can provide a 3D image including depth information related to the object.

[0074] Storage device 1240 may include a computer-readable storage medium or a computer-readable storage device. Storage device 1240 can store a greater amount of information than memory 1220 and store the information for a relatively long time. For example, storage device 1240 may be a magnetic hard disk, optical disk, flash memory, floppy disk, or another type of non-volatile memory known in the art. Storage device 1240 may include... Figure 9 Database 910.

[0075] Input device 1250 can receive input from a user via conventional input methods such as a keyboard or mouse, or newer input methods such as touch input, voice input, and image input. For example, input device 1250 may include a keyboard, mouse, touchscreen, microphone, or any device configured to detect input from the user and send the detected input to electronic device 1200. User data (such as fingerprints, iris scans, voice, sound, and audio) can be input via input device 1250.

[0076] Output device 1260 provides output from electronic device 1200 (e.g., user device) to a user via a visual channel, audio channel, or haptic channel. Output device 1260 may include, for example, a display, touchscreen, speaker, vibration generator, or any device configured to provide output to a user. For example, output device 1260 may include a display panel for implementing a 3D HUD, a 3D optical layer (parallax barrier, biconvex lens, or directional backlight), and an optical system (mirror or lens). Network interface 1270 can communicate with external devices via wired or wireless networks.

[0077] The units described herein may be implemented using hardware components, software components, and / or combinations thereof. The processing device may be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers and arithmetic logic units (ALUs), DSPs, microcomputers, FPGAs, programmable logic units (PLUs), microprocessors, or any other device capable of responding to and executing instructions in a defined manner. The processing device may run an operating system (OS) and one or more software applications running on the OS. In response to the execution of the software, the processing device may also access, store, manipulate, process, and create data. For simplicity, the description of the processing device is used as the singular; however, those skilled in the art will understand that the processing device may include multiple processing elements and various types of processing elements. For example, the processing device may include multiple processors or processors and controllers. Furthermore, different processing configurations (such as parallel processors) are feasible.

[0078] Software may include computer programs, code segments, instructions, or combinations thereof, to independently or jointly instruct or configure a processing device to operate as desired. Software and data may be permanently or temporarily embodied in any type of machine, component, physical or virtual device, computer storage medium, or apparatus, or permanently or temporarily embodied in a propagated signal wave capable of providing instructions or data to or interpreting the processing device. Software may also be distributed across networked computer systems, enabling it to be stored and executed in a distributed manner. Software and data may be stored on one or more non-transitory computer-readable recording media.

[0079] The methods according to the exemplary embodiments described above can be recorded in a non-transitory computer-readable medium including program instructions to implement various operations of the exemplary embodiments described above. The medium may also include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be program instructions specifically designed and constructed for the purposes of the exemplary embodiments, or they may be of types known and available to those skilled in the art of computer software. Examples of non-transitory computer-readable media include: magnetic media (such as hard disks, floppy disks, and magnetic tapes), optical media (such as CD-ROMs, DVDs, and / or Blu-ray discs), magneto-optical media (such as optical discs), and hardware devices specifically configured to store and execute program instructions (such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., USB flash drives, memory cards, memory sticks, etc.)). Examples of program instructions include both machine code generated by a compiler and files containing high-level code that can be executed by a computer using an interpreter. The aforementioned devices may be configured to function as one or more software modules to perform the operations of the exemplary embodiments described above, or vice versa.

[0080] Several exemplary embodiments have been described above. However, it should be understood that various modifications can be made to these exemplary embodiments. For example, suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, other embodiments are within the scope of the claims.

Claims

1. An eye-tracking method, comprising: A reconstructed image is generated by performing eye reconstruction on the input image; Determine the difference between the input image and the reconstructed image; The target image is determined by selecting one of the input image, the reconstructed image, and the replacement image based on a determined difference; as well as Perform eye tracking based on the target image. The step of determining the difference between the input image and the reconstructed image includes: comparing the corresponding pixels of the input image and the reconstructed image to determine the difference between them. The selection steps include: if the determined difference is less than a first threshold, selecting the input image as the target image; if the determined difference is greater than the first threshold and less than a second threshold, selecting the reconstructed image as the target image; and if the determined difference is greater than the second threshold, selecting the replacement image as the target image, wherein the second threshold is greater than the first threshold, and... The eye-tracking method further includes selecting, from multiple sample images stored in a database, the sample image with the highest similarity to the input image as a replacement image.

2. The eye-tracking method according to claim 1, wherein, Eye reconstruction involves reducing noise components in the input image.

3. The eye-tracking method according to claim 1, wherein, The generation steps include: using at least one principal component vector from a plurality of principal component vectors corresponding to the input image, which has a priority higher than a predetermined priority, to generate a reconstructed image, and Each of the multiple principal component vectors corresponds to a predetermined eigenface based on principal component analysis of various facial images.

4. The eye-tracking method according to claim 1, wherein, The alternative image is different from the input image and the reconstructed image.

5. The eye-tracking method according to claim 1, wherein, Similarity is determined based on a comparison between feature points of the input image and feature points of each of the plurality of sample images.

6. The eye-tracking method according to claim 5, wherein, The feature points of the input image and the feature points of each of the plurality of sample images are each extracted from the region other than the eyes.

7. The eye-tracking method according to claim 1, wherein, The multiple sample images correspond to images from which eye tracking has been successfully performed previously.

8. The eye-tracking method according to claim 1, further comprising: If eye tracking is successful based on the input image or the reconstructed image, the input image or the reconstructed image is stored as a sample image in the database.

9. The eye-tracking method according to claim 1, wherein, If the eye detection is successful for the input image, then the generated steps are executed.

10. The eye-tracking method according to claim 1, wherein, If an alternative image is selected as the target image, the steps performed include: performing eye tracking based on eye position information mapped to the alternative image.

11. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the eye-tracking method of any one of claims 1 to 10.

12. An electronic device comprising: processor; The memory is configured to store instructions that can be executed by the processor; as well as The camera is configured to generate an input image by photographing the user. When the instructions are executed by the processor, the processor is configured to: generate a reconstructed image by performing eye reconstruction on the input image; determine the difference between the input image and the reconstructed image; determine a target image by selecting one of the input image, the reconstructed image, and a replacement image based on the determined difference; and perform eye tracking based on the target image. The processor is further configured to compare corresponding pixels of the input image and the reconstructed image to determine the difference between the input image and the reconstructed image. The processor is further configured to: select the input image as the target image if the determined difference is less than a first threshold; select the reconstructed image as the target image if the determined difference is greater than the first threshold and less than a second threshold; and select the substitute image as the target image if the determined difference is greater than the second threshold, wherein the second threshold is greater than the first threshold, and... The processor is also configured to select the sample image with the highest similarity to the input image from among multiple sample images stored in the database as the replacement image.

13. An eye-tracking device, comprising: processor; as well as The memory is configured to store instructions that can be executed by the processor. When the instructions are executed by the processor, the processor is configured to: generate a reconstructed image by performing eye reconstruction on the input image; determine the difference between the input image and the reconstructed image; determine a target image by selecting one of the input image, the reconstructed image, and a replacement image based on the determined difference; and perform eye tracking based on the target image. The processor is further configured to compare corresponding pixels of the input image and the reconstructed image to determine the difference between the input image and the reconstructed image. The processor is further configured to: select the input image as the target image if the determined difference is less than a first threshold; select the reconstructed image as the target image if the determined difference is greater than the first threshold and less than a second threshold; and select the substitute image as the target image if the determined difference is greater than the second threshold, wherein the second threshold is greater than the first threshold, and... The processor is also configured to select the sample image with the highest similarity to the input image from among multiple sample images stored in the database as the replacement image.

14. The eye-tracking device according to claim 13, wherein, The processor is also configured to generate a reconstructed image using at least one principal component vector from a plurality of principal component vectors corresponding to the input image, which has a priority higher than a predetermined priority. Each of the multiple principal component vectors corresponds to a predetermined eigenface based on principal component analysis of various facial images.

15. The eye-tracking device according to claim 13, wherein, Similarity is determined based on a comparison between feature points of the input image and feature points of each of the plurality of sample images, and The feature points of the input image and the feature points of each of the plurality of sample images are each extracted from the region other than the eyes.

16. The eye-tracking device according to claim 13, wherein, If the alternative image is selected as the target image, the processor is also configured to perform eye tracking based on eye position information mapped to the alternative image.

Images