Eye tracking method and apparatus based on eye reconstruction

The eye tracking method employs PCA to reduce noise and select appropriate images based on thresholds, improving accuracy in environments with light issues or occlusions by using a database of sample images for reliable eye position detection.

JP7590057B2Active Publication Date: 2024-11-26SAMSUNG ELECTRONICS CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2020142548
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-12-10
Filing Date
2020-08-26
Publication Date
2024-11-26
Estimated Expiration
2040-08-26

AI Technical Summary

Technical Problem

Existing eye tracking technologies struggle to accurately track eye positions in environments with insufficient light or occlusions, such as sunglasses, due to noise components like light reflections and blockages, which hinder effective eye detection and tracking.

Method used

An eye tracking method that performs eye reconstruction using principal component analysis (PCA) to reduce noise, selects between input, restored, and substitute images based on difference thresholds, and utilizes a database of sample images for successful eye tracking.

Benefits of technology

Enhances eye tracking accuracy by effectively removing noise components and selecting optimal images for tracking, ensuring reliable eye position detection even in challenging lighting conditions or with occlusions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590057000001
    Figure 0007590057000001
  • Figure 0007590057000002
    Figure 0007590057000002
  • Figure 0007590057000003
    Figure 0007590057000003
Patent Text Reader

Abstract

To provide an eye reconstruction-based eye tracking method and device.SOLUTION: An eye tracking method according to an embodiment comprises performing eye reconstruction on an input image to generate a reconstructed image, determining difference between the input image and the reconstructed image, selecting one of the input image, the reconstructed image, and a replacement image on the basis of the determined difference, and performing eye tracking.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The following embodiments relate to an eye tracking method and apparatus based on eye reconstruction. [Background technology]

[0002] A head up display (HUD) device displays a virtual image in front of the driver and provides the driver with various driving information that is useful for driving. Recently, 3D HUD devices have been developed, and for example, 3D HUD devices can use an augmented reality (AR) method. In this case, driving information is displayed so as to be superimposed on real objects, allowing the driver to recognize the driving information more intuitively. Currently, there are various 3D display methods, among which a viewpoint tracking 3D display method with relatively high resolution and flexibility can be applied to 3D HUD devices. Summary of the Invention [Problem to be solved by the invention]

[0003] An objective of the following embodiments is to provide an eye tracking method and apparatus based on eye reconstruction. [Means for solving the problem]

[0004] According to one embodiment, an eye tracking method includes the steps of performing eye restoration on an input image to generate a restored image, determining a difference between the input image and the restored image, determining one of the input image, the restored image, and an alternative image as a target image based on the determined difference, and performing eye tracking based on the target image.

[0005] The eye restoration may include reducing noise components in the input image. The generating of the restored image may include generating the restored image using a portion of principal component vectors having high rankings corresponding to the input image, and the principal component vectors may correspond to predetermined unique faces based on a principal component analysis of various face images.

[0006] The step of selecting the target image includes selecting the input image as the target image if the determined difference is smaller than a first threshold, selecting the restored image as the target image if the determined difference is greater than the first threshold and smaller than a second threshold, and selecting the substitute image as the target image if the determined difference is greater than the second threshold, and the second threshold may be greater than the first threshold.

[0007] The substitute image may be separate from the input image and the restored image. The eye tracking method may further include a step of selecting a sample image having the highest similarity to the input image from among sample images stored in a database as the substitute image. The similarity may be determined based on a comparison between feature points of the input image and feature points of each sample image of the sample images. The feature points of the input image and the feature points of each sample image may be extracted from an area excluding eyes. The sample image may correspond to an image for which eye tracking has been successful in the past. The eye tracking method may further include a step of storing the input image in a database as a sample image when the eye tracking is successful based on the input image or the restored image.

[0008] The generating of the restored image may be performed if eye detection for the input image is successful. If the substitute image is selected as the target image, the performing of the eye tracking may include performing the eye tracking using eye position information mapped to the substitute image.

[0009] According to one embodiment, an eye tracking device includes a processor and a memory containing instructions executable by the processor, and when the instructions are executed by the processor, the processor performs eye restoration on an input image to generate a restored image, determines a difference between the input image and the restored image, determines one of the input image, the restored image, and an alternative image as a target image based on the determined difference, and performs eye tracking based on the target image. Effect of the Invention

[0010] According to the present invention, an eye tracking method and apparatus based on eye reconstruction can be provided. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 illustrates operation of an eye tracking device using an input image, a restored image, and an alternative image according to one embodiment. [Diagram 2] 4 is a flowchart illustrating an eye detection and tracking process according to an embodiment. [Diagram 3] 1 is a flowchart illustrating a process of determining a target image and tracking eyes according to an embodiment. [Figure 4] FIG. 1 illustrates an augmented appearance space according to one embodiment. [Diagram 5] FIG. 1 illustrates an eye restoration process according to an embodiment. [Figure 6] 13 is a diagram illustrating a target image selection operation when a difference value is smaller than a second threshold value according to an embodiment. FIG. [Figure 7] 13 is a diagram illustrating a target image selection operation when a difference value is greater than a second threshold value according to an embodiment. FIG. [Figure 8] FIG. 2 illustrates a feature matching operation between an input video and a sample video according to an embodiment. [Figure 9] FIG. 13 is a diagram showing sample videos stored in a database according to an embodiment. [Figure 10] 1 is a flow chart illustrating an eye tracking method based on eye reconstruction according to an embodiment. [Figure 11] FIG. 1 is a block diagram illustrating an eye tracking device based on eye reconstruction according to an embodiment. [Figure 12] FIG. 1 illustrates an electronic device including an eye tracking device according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Specific structural or functional descriptions disclosed herein are merely exemplary for purposes of describing embodiments, and the embodiments may be embodied in various different forms and are not limited to the embodiments set forth herein.

[0013] Although terms such as first or second may be used to describe multiple components, such terms should be construed only for the purpose of distinguishing one component from the other components. For example, a first component may be named a second component, and similarly, a second component may be named a first component.

[0014] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, the terms "comprise" or "have" and the like indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0015] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as idealized or overly formal unless expressly defined in this specification.

[0016] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and duplicated description thereof will be omitted.

[0017] FIG. 1 is a diagram illustrating the operation of an eye tracking device using an input image, a restored image, and a replacement image according to an embodiment. Referring to FIG. 1, an eye tracking device 100 performs eye tracking on an input image 110 and outputs eye position information as a result. The input image 110 may include a face of a user (e.g., a viewer, a driver). The eye position information is used for various applications such as naked eye 3D display, driver status monitoring, etc. The naked eye 3D display may include various types of displays such as a 3D head up display (HUD), a 3DTV, a 3D mobile device, etc.

[0018] Naked-eye 3D display is a method of showing different images to each eye of a user, and in order to achieve this, information on the positions of the user's eyes is required. For example, to provide a 3D image, a left image for the left eye and a right image for the right eye are provided. In this case, the 3D display device can determine the positions of the user's eyes through the eye position information and provide the left image to the position of the left eye and the right image to the position of the right eye. This method allows the user to view a 3D image.

[0019] When the input image 110 is generated by photographing a user in a state where there is adequate light and the user's eyes are fully visible, the user's eyes can be tracked relatively easily. On the other hand, when the input image 110 is generated by photographing a user in a state where there is insufficient light or the user's eyes are covered by sunglasses or the like and there is light reflection component around the user's eyes, it becomes difficult to track the user's eyes. The eye tracking device 100 can successfully perform eye tracking even in such poor environments by arbitrarily using the input image 110, the restored image 121, and the substitute image 131 according to the situation.

[0020] The input video 110 may be a video containing multiple frames. The input video 111 corresponds to one frame of the input video 110. The eye tracking process described below is performed for each frame of the input video 110 in sequence.

[0021] When the input image 111 is received, the eye tracking device 100 performs eye reconstruction on the input image 111 to generate a reconstructed image 121. The eye reconstruction may include various methods for reducing noise components in the input image 111. For example, the noise components may include high frequency components (e.g., light reflected by glasses, halo, etc.) and occlusion components (e.g., sunglasses, thick glasses, hair, etc.). Dimensionality reduction through principal component analysis (PCA) may be used for eye reconstruction. Specific examples of eye reconstruction will be described in detail later.

[0022] Once the reconstructed image 121 is generated, the eye tracking device 100 compares the input image 111 with the reconstructed image 121 to determine a difference between the input image 111 and the reconstructed image 121. Hereinafter, the determined difference is referred to as a difference value. For example, the eye tracking device 100 compares corresponding pixels of the input image 111 and the reconstructed image 121 with each other to determine a difference between the input image 111 and the reconstructed image 121. Corresponding pixels refer to pixels that exist in the same position in each image.

[0023] The eye tracking device 100 determines a target image by selecting one of the input image 111, the restored image 121, and the substitute image 131 based on the difference value. The target image means an image to be subjected to eye tracking, and corresponds to one of the input image 111, the restored image 121, and the substitute image 131. For example, if the difference value is smaller than a first threshold, the input image 111 may be selected as the target image, if the difference value is larger than the first threshold and smaller than a second threshold, the restored image 121 may be selected as the target image, and if the difference value is larger than the second threshold, the substitute image 131 may be selected as the target image. It is assumed that the second threshold is larger than the first threshold.

[0024] The alternative image 131 may be distinct from the input image 111 and the restored image 121 and selected from the sample images 130. The sample images 130 may include various images in suitable conditions for eye tracking, and the sample images 130 that are most similar to the input image 111 may be selected as the alternative image 131. For example, the similarity between the input image 111 and each sample image of the sample images 130 may be determined based on a comparison between feature points of the input image 111 and feature points of each sample image of the sample images 130.

[0025] The eye tracking device 100 can compare the similarity with a threshold value and select one of the input image 111, the restored image 121, and the alternative image 131 that is most suitable for eye tracking. For example, if the input image 111 corresponds to a state suitable for eye tracking, there is only a small difference (e.g., a difference smaller than a first threshold value) between the input image 111 and the restored image 121, so the input image 111 can be selected as the target image.

[0026] Alternatively, if a noise component such as light reflection is present in the input image 111, the restored image 121 is more suitable for eye tracking than the input image 111. In this case, the corresponding noise component is removed through eye restoration, and a difference (e.g., a difference greater than a first threshold and less than a second threshold) may occur between the input image 111 and the restored image 121 depending on the presence or absence of the noise component. Therefore, the restored image 121 may be selected as the target image.

[0027] Or, if there is a lot of noise in the input image 111, such as when a user is wearing sunglasses, the alternative image 131 is more suitable for eye tracking than the input image 111 and the restored image 121. In this case, a large difference (e.g., a difference larger than a second threshold) may occur between the input image 111 and the restored image 121 during the eye restoration process, and therefore the alternative image 131 may be selected as the target image.

[0028] Among the components constituting the input image 111, components that interfere with eye tracking are removed in the eye restoration process. For example, in the eye restoration process, components corresponding to a typical face are retained, and components that do not correspond to a typical face are removed. In the input image 111, one of the input image 111, the restored image 121, and the alternative image 131 is selected according to the ratio of the components removed in the eye restoration process. For example, in the case of a high frequency component, the ratio of the removed components may be small, so the restored image 121 is selected as the target image, and in the case of a blocking component, the ratio of the removed components may be large, so the alternative image 131 may be selected as the target image.

[0029] When a target image is selected, the eye tracking device 100 performs eye tracking on the target image to generate eye position information. The eye tracking device 100 sequentially receives a plurality of frames and tracks the eyes from each frame. The eye position information may include the positions of the eyes present in each frame. For example, the eye positions may be expressed as the coordinates of the eyes in the image.

[0030] FIG. 2 is a flow chart showing an eye detection process and an eye tracking process according to an embodiment. Referring to FIG. 2, when a first frame of an input image is received, the eye tracking device performs eye detection on the first frame of the input image in step S210. The eye tracking device determines an eye detection area including the user's eyes through the eye detection process. In step S220, the eye tracking device determines whether the eye detection is successful. If the eye detection is unsuccessful, step S210 is performed on the second frame of the input image, and if the eye detection is successful, step S230 is performed. In other words, steps S210 and S220 are repeated for each frame until the eye detection is successful.

[0031] In step S230, the eye tracking device performs eye restoration on the first frame of the input image. After eye restoration, a restored image and an alternative image corresponding to the first frame of the input image are generated, and one of the first frame of the input image, the restored image, and the alternative image is selected as a target image. In step S240, the eye tracking device performs eye tracking on the target image. The eye tracking device determines an eye tracking area for eye tracking based on the eye detection area. The eye tracking device can extract feature points within the eye tracking area and align the extracted feature points to perform eye tracking.

[0032] In step S250, the eye tracking device determines whether eye tracking is successful. If the user's eyes are present within the eye tracking region, it determines that eye tracking is successful, and if the user's eyes are not present within the eye tracking region, it determines that eye tracking is unsuccessful. If eye tracking is successful, the eye tracking device updates the eye tracking region and performs step S230 for the second region of the input image. For example, the eye tracking device can adjust the position of the eye tracking region according to the position of the eyes. As an example, the eye tracking device adjusts the position of the eye tracking region so that the centers of the two eyes are located in the middle of the eye tracking region. The eye tracking device can determine a target image corresponding to the second frame of the input image and continue eye tracking using the target image and the updated eye tracking region.

[0033] Fig. 3 is a flowchart showing a process of determining a target image and tracking eyes according to an embodiment. Steps S310 to S390 shown in Fig. 3 correspond to steps S230 and S240 shown as dotted boxes in Fig. 2, and step S310 is performed when it is determined that eye detection is successful in step S220 shown in Fig. 2.

[0034] Referring to Fig. 3, in step S310, the eye tracking device performs eye reconstruction to generate a reconstructed image. Eye reconstruction includes various methods for reducing noise components in the input image. For example, dimensionality reduction through PCA is applied for eye reconstruction. This will be described in more detail with reference to Figs. 4 and 5.

[0035] FIG. 4 is a diagram illustrating an augmented appearance space according to an embodiment. PCA is performed on various reference face images to obtain principal component vectors. The number of dimensions of the principal component vectors may be the same as the number of dimensions of data (e.g., pixels) of each image. In this case, principal component vectors that reach exceptional face components from dominant face components may be obtained. A ratio of images including dominant face shapes may be high among the reference face images. In this case, the dominant face components correspond to standard faces, and the exceptional face components correspond to noise. Since the principal component vectors are mutually orthogonal, face images in all input images can be represented through the principal component vectors.

[0036] The principal component vectors are again interpreted as face images, and the interpreted face images are called eigenfaces. In other words, eigenfaces can be determined based on PCA for various face images, and each principal component vector corresponds to an eigenface. Face images in the input image are represented as an augmented appearance space using the eigenfaces.

[0037] Referring to FIG. 4, face image A is λ i A i It is expressed as the sum of A i denotes the principal component vector, and λ i means the coefficient of the principal component vector. i λ may be referred to as an appearance parameter. i is face image A and A i Each λ i A i The face image shown in FIG. 4 is expressed by the value of A. The index i has values ​​from 1 to n. i teeth i A smaller value of i means that the component is closer to the standard, and a larger value of i means that the component is closer to noise.

[0038] 5 is a diagram illustrating an eye restoration process according to an embodiment. Referring to FIG. 5, an input image I is a pixel value λ 0A 0 ~λ n A n The restored image I' is expressed as the sum of λ 0 A 0 ~λ m A m n can be assumed to be greater than m, and it can be seen as a reduction in dimension due to eye restoration. In addition, in the input image I, components with large index values ​​(components close to noise) are removed, so noise components such as high frequency components and cutoff components are removed from the input image I. In the input image I, λ m+1 I m+1 ~λ n A n is referred to as a removed component or noise component. The smaller the index value, the higher the priority of the principal component vector. In this case, the eye tracking device calculates the principal component vector A corresponding to the input image. 0 ~A n Among them, the highest-ranked A 0 ~A m The restored image can be generated using

[0039] Referring again to FIG. 3, in step S320, the eye tracking device determines a difference between the input image and the restored image. For example, the eye tracking device determines a difference value by comparing corresponding pixels of the input image and the restored image. The larger the coefficient of each noise component removed in the eye restoration process, the more noise the input image is interpreted to have, and the larger the difference between the input image and the restored image is determined to be. Therefore, the restored image or the alternative image is more likely to be used as the target image.

[0040] According to an embodiment, the difference value may be determined based on the eye region. In other words, the eye tracking device determines the difference value by comparing corresponding pixels of the eye region of the input image and the eye region of the restored image. This is because it is more effective to utilize the restored image or the alternative image by comparing changes that occur in the eye region rather than comparing changes that occur in the entire image due to eye restoration.

[0041] For example, if a user wears a mask, eye tracking is possible, but if the user wears sunglasses, eye tracking becomes difficult. Therefore, it is highly necessary to detect the case where the user wears sunglasses rather than the case where the mask is not worn, and to use restored images or alternative images in this case. In this case, the eye region is determined based on the eye detection region or the eye tracking region.

[0042] In step S330, the eye tracking device compares the difference value with a first threshold. If the difference value is smaller than the first threshold, the eye tracking device determines the input image as the target image in step S340. If the difference value is larger than the first threshold, the eye tracking device compares the difference value with a second threshold in step S350. If the difference value is smaller than the second threshold, the eye tracking device determines the restored image as the target image in step S360. If the difference value is larger than the second threshold, the eye tracking device selects an alternative image from the sample images in step S370 and determines the alternative image as the target image in step S380. In step S390, the eye tracking device performs eye tracking from the target image.

[0043] 6 is a diagram illustrating a target image selection operation when the difference value is smaller than a second threshold value according to an embodiment. Referring to FIG 6, a restored image 620 is generated by eye restoration on an input image 610. High frequency components including light reflection from glasses are removed from the input image 610 by eye restoration, and the restored image 620 appears smoother than the input image 610.

[0044] After the eye restoration, a difference value between the input image 610 and the restored image 620 is determined. If the difference value is smaller than a first threshold, the input image 610 is determined as the target image, and if the difference value is larger than the first threshold and smaller than a second threshold, the restored image 620 is determined as the target image. For example, since high frequency components included in the input image 610 have been removed, the restored image 620 is determined as the target image.

[0045] 7 is a diagram illustrating a target image selection operation when the difference value is greater than a second threshold value according to an embodiment. Referring to FIG 7, a restored image 720 is generated by eye restoration on an input image 710. By eye restoration, high frequency components including light reflections from sunglasses and blocking components including sunglasses are removed from the input image 710, and the restored image 720 appears smoother than the input image 710.

[0046] After eye restoration, a difference value between the input image 710 and the restored image 720 is determined. If the difference value is smaller than a first threshold, the input image 710 is determined as the target image, if the difference value is greater than the first threshold and less than a second threshold, the restored image 720 is determined as the target image, and if the difference value is greater than the second threshold, a substitute image 740 is generated. For example, the difference value is determined to be greater than the second threshold because blocking components and high frequency components contained in the input image 710 have been removed.

[0047] In this case, feature matching is performed between the input video 710 and each sample video of the sample video 730, and the sample video of the sample video 730 that is most similar to the input video 710 is selected as the replacement video 740. The replacement video 740 may then be determined as a target video and used for eye tracking. The sample video 730 may include various face videos that are easy for eye tracking. For example, the sample video 730 may include a face video without glasses, a video with glasses but without light reflection in the glasses, etc. In the embodiment shown in FIG. 7, the sample video with glasses is determined to be most similar to the input video 710 with sunglasses, and therefore the sample video is selected as the replacement video 740.

[0048] FIG. 8 illustrates a feature matching operation between an input video and a sample video according to an embodiment. Referring to FIG. 8, feature points of an input video 810 and feature points of a sample video 820 are compared, and thus a similarity between the input video 810 and the sample video 820 is determined. The feature points of the input video and the feature points of the sample video 820 may be extracted from regions excluding the eyes, respectively. For example, feature points may be extracted by nose, mouth, face contour, etc. The alternative video is likely to be used when there are sunglasses or other occlusions at the eye position. Therefore, extracting feature points from regions excluding the eyes is advantageous to obtain an alternative video similar to the input video and to obtain a suitable viewpoint from the alternative video.

[0049] FIG. 9 is a diagram showing sample videos stored in a database according to an embodiment. Referring to FIG. 9, a database 910 includes sample videos 921-923. The database 910 may exist in the eye tracking device, or may exist in another device separate from the eye tracking device. If the database 910 exists in another device, the eye tracking device can access the database 910 via a network to use the sample videos 921-923. Each of the sample videos 921-923 has a state suitable for eye tracking. The state suitable for eye tracking may include a state in which no high-frequency components or blocking components are present.

[0050] The sample images 921-923 correspond to images in which eye tracking has been successful in the past. Here, "past" means before eye tracking has been performed on the current frame. For example, among images in which eye tracking has been successful during the manufacturing process of the eye tracking device, images suitable as replacement images may be selected as the sample images 921-923. Alternatively, the sample images 921-923 may be stored during the process of performing eye tracking on each frame of a previous input image, or during the process of performing eye tracking on a previous frame of the current input image. As an example, when eye tracking is successful based on the current frame or a restored image corresponding to the current frame, the current frame is stored in the database 910 to be used as a replacement image for the future.

[0051] The eye position information 931-933 is mapped to the sample videos 921-923. Since the sample videos 921-923 have been successfully tracked in the past, the eye position information 931-933 is secured for the sample videos 921-923 when the eye tracking was successful in the past. Therefore, when the sample videos 921-923 are used as substitute videos, the eye position information 931-933 secured in advance can be used instead of separately tracking the eyes in the sample videos 921-923. For example, assuming that the sample video 921 is selected as the substitute video for the input video, the eye tracking device can immediately output the eye position information 931 mapped to the sample video 921 instead of tracking the eyes from the sample video 921 to generate the eye position information.

[0052] Fig. 10 is a flow chart showing an eye tracking method based on eye restoration according to an embodiment. Referring to Fig. 10, the eye tracking device performs eye restoration on an input image to generate a restored image in step S1010, determines a difference between the input image and the restored image in step S1020, selects one of the input image, the restored image, and the alternative image based on the determined difference in step S1030 to determine a target image, and performs eye tracking based on the target image in step S1040. In addition, the matters described with reference to Figs. 1 to 9 may be applied to the eye tracking method shown in Fig. 10.

[0053] 11 is a block diagram illustrating an eye tracking device based on eye reconstruction according to an embodiment. Referring to FIG. 11, an eye tracking device 1100 can perform one or more operations described or illustrated herein in connection with eye tracking and can provide eye position information to a user as an eye tracking result.

[0054] The eye tracking device 1100 includes one or more processors 1110 and a memory 1120. The memory 1120 is coupled to the processor 1110 and stores instructions executable by the processor 1110, data operated on by the processor 1110, or data processed by the processor 1110. The memory 1120 may include a non-transitory computer-readable recording medium, such as a high-speed random access memory and / or a non-volatile computer-readable storage medium (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0055] The processor 1110 executes instructions to perform one or more of the operations described with reference to Figures 1 to 10. According to an embodiment, when the instructions stored in the memory 1120 are executed by the processor 1110, the processor 1110 may perform eye restoration on an input image to generate a restored image, determine a difference between the input image and the restored image, select one of the input image, the restored image, and a substitute image based on the determined difference to determine a target image, and perform eye tracking based on the target image.

[0056] Figure 12 illustrates an electronic device including an eye tracking device according to one embodiment. Referring to Figure 12, an electronic device 1200 may include the eye tracking device described with reference to Figures 1-11 or may perform the functions of the eye tracking device described with reference to Figures 1-12.

[0057] The electronic device 1200 includes a processor 1210, a memory 1220, a camera 1230, a storage device 1240, an input device 1250, an output device 1260, and a network interface 1270. The processor 1210, the memory 1220, the camera 1230, the storage device 1240, the input device 1250, the output device 1260, and the network interface 1270 communicate with each other via a communication bus 1280. For example, the electronic device 1200 may include a smartphone, a tablet PC, a laptop computer, a desktop PC, a wearable device, a smart home appliance, a smart speaker, a smart car, and the like. As an example, the electronic device 1200 may be mounted in a vehicle to provide a function for a three-dimensional HUD.

[0058] The processor 1210 executes functions and instructions to be executed within the electronic device 1200. For example, the processor 1210 processes instructions stored in the memory 1220 or the storage device 1240. The processor 1210 may perform one or more of the operations described with reference to FIGS. 1 to 11.

[0059] The memory 1220 stores information for processing the operation of the neural network. The memory 1220 may include a computer-readable storage medium or a computer-readable storage device. The memory 1220 stores instructions for execution by the processor 1210 and stores relevant information during the execution of software or applications by the electronic device 1200.

[0060] Camera 1230 captures still images, video footage, or both. Camera 1230 captures the user's face for eye tracking and generates input footage. Camera 1230 can also provide 3D footage including depth information about objects.

[0061] Storage device 1240 includes a computer readable storage medium or computer readable storage device. According to one embodiment, storage device 1240 can store a larger amount of information than memory 1220 and can retain information for a longer period of time. For example, storage device 1240 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other forms of non-volatile memory known in the art. Storage device 1240 may include database 910 shown in FIG. 9.

[0062] The input device 1250 receives input from a user through traditional input methods through a keyboard and mouse, and newer input methods such as touch input, voice input, and image input. For example, the input device 1250 may include a keyboard, a mouse, a touch screen, a microphone, or any other device capable of detecting input from a user and communicating the detected input to the electronic device 1200. Data such as a user's fingerprint, iris, speech, voice, and audio may be input via the input device 1250.

[0063] The output device 1260 provides output of the user device 1200 to the user via a visual, audio, or tactile channel. The output device 1260 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device capable of providing output to a user. As an example, the output device 1260 may include a display panel for realizing a 3D HUD, a 3D optical layer (parallax barrier, lenticular lens, directional backlight, etc.), an optical system (mirrors, lenses, etc.), etc. The network interface 1270 communicates with external devices via a wired or wireless network.

[0064] The above-described embodiments may be implemented using hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or other devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, the processing device may be described as being one in which only one processing device is used, but one skilled in the art will appreciate that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0065] Software may include computer programs, codes, instructions, or any combination of one or more thereof, capable of configuring or instructing a processing device to operate as desired, either independently or in combination. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, either permanently or temporarily, to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed across computer systems coupled to a network, and may be stored and executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

[0066] The method according to the present invention may be embodied in the form of program instructions to be executed by various computer means and recorded on a computer-readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and the program instructions may be specially designed and constructed for the purposes of the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code, such as generated by a compiler, but also high-level language code executed by a computer using an interpreter, and the like. The hardware devices may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0067] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted with other components or equivalents to achieve appropriate results. [Explanation of symbols]

[0068] 100, 1100 Eye Tracking Device 111, 610, 710, 810 Input video 121, 620, 720 restored footage 130, 730, 820 Sample video 131, 740 Alternative Video 910 Database 921-923 Sample video 931-933 Eye location information 1110, 1210 processors 1120, 1220 Memory 1200 Electronic equipment 1230 Camera

Claims

1. A processor-implemented method of eye tracking, comprising: performing eye restoration on the input image to generate a restored image; determining a difference between the input image and the reconstructed image; selecting a sample video having the highest similarity to the input video as a replacement video from among sample videos stored in a database, the sample video corresponding to a video that has been successfully subjected to eye tracking in the past; determining one of the input image, the restored image, and the alternative image as a target image based on the determined difference; performing eye tracking based on the target image; Including, generating the restored image includes generating the restored image using a high-ranked portion of principal component vectors corresponding to the input image, the principal component vectors corresponding to predetermined unique faces based on a principal component analysis of various face images; The step of selecting the target image includes selecting the input image as the target image when the determined difference is smaller than a first threshold, selecting the restored image as the target image when the determined difference is larger than the first threshold and smaller than a second threshold, and selecting the substitute image as the target image when the determined difference is larger than the second threshold, the second threshold being larger than the first threshold. Eye tracking method.

2. The eye tracking method of claim 1 , wherein the eye reconstruction reduces noise components in the input video.

3. The eye tracking method of claim 1 , wherein the alternative image is distinct from the input image and the restored image.

4. The eye tracking method of claim 1 , wherein the similarity is determined based on a comparison between feature points of the input image and feature points of each of the sample images of the sample images.

5. The eye tracking method according to claim 4 , wherein the feature points of the input image and the feature points of each of the sample images are extracted from areas excluding eyes.

6. The eye tracking method of claim 1 , further comprising the step of: storing the input video in a database as a sample video if the eye tracking is successful based on the input video or the reconstructed video.

7. The eye tracking method of claim 1 , wherein the step of generating the reconstructed image is performed if eye detection on the input image is successful.

8. The eye tracking method of claim 1 , wherein when the alternative image is selected as the target image, the step of performing eye tracking includes performing the eye tracking using eye position information mapped to the alternative image.

9. A computer readable storage medium having stored thereon one or more programs including instructions for carrying out the method of any one of claims 1 to 8.

10. A processor; a memory containing instructions executable by the processor; A camera that captures an image of a user and generates an input image; Including, When the instructions are executed by the processor, the processor: performing eye restoration on the input image to generate a restored image; determining a difference between the input image and the reconstructed image; Among the sample images stored in the database, a sample image having the highest similarity to the input image is selected as a substitute image, and the sample image corresponds to an image for which eye tracking has been successfully performed in the past; determining one of the input image, the restored image, and the alternative image as a target image based on the determined difference; performing eye tracking based on the target image; generating the restored image includes generating the restored image using a high-ranked portion of principal component vectors corresponding to the input image, the principal component vectors each corresponding to a predetermined unique face based on a principal component analysis of various face images; determining the target image includes selecting the input image as the target image when the determined difference is smaller than a first threshold, selecting the restored image as the target image when the determined difference is larger than the first threshold and smaller than a second threshold, and selecting the substitute image as the target image when the determined difference is larger than the second threshold, the second threshold being larger than the first threshold. electronic equipment.

11. A processor; a memory containing instructions executable by the processor; Including, When the instructions are executed by the processor, the processor: Performing visual restoration on the input image to generate a restored image; determining a difference between the input image and the reconstructed image; Among the sample images stored in the database, a sample image having the highest similarity to the input image is selected as a substitute image, and the sample image corresponds to an image for which eye tracking has been successfully performed in the past; determining one of the input image, the restored image, and the alternative image as a target image based on the determined difference; performing eye tracking based on the target image; generating the restored image includes generating the restored image using a high-ranked portion of principal component vectors corresponding to the input image, the principal component vectors each corresponding to a predetermined unique face based on a principal component analysis of various face images; determining the target image includes selecting the input image as the target image when the determined difference is smaller than a first threshold, selecting the restored image as the target image when the determined difference is larger than the first threshold and smaller than a second threshold, and selecting the substitute image as the target image when the determined difference is larger than the second threshold, the second threshold being larger than the first threshold. Eye tracking device.

12. the similarity is determined based on a comparison between feature points of the input image and feature points of each of the sample images; The eye tracking device of claim 11 , wherein the feature points of the input image and the feature points of each of the sample images are extracted from areas excluding eyes.

13. If the alternative image is selected as the target image, The eye tracking device of claim 11 , wherein the step of performing eye tracking includes performing the eye tracking using eye position information mapped to the alternative image.

Citation Information

Patent Citations

  • Construction assisting device and construction assisting method

    JP2013088895A

  • Method of removing reflection area, eye-tracking method and apparatus thereof

    KR1020190047442A