A line-of-sight fall point stabilization technology

Abstract

The present application provides a line-of-sight detection method, comprising steps SA1 and SA2. The step SA1 performs a dynamic threshold to switch action to correct errors of data generated in a previous face feature detection step. In addition, the step SA2 performs an eye ROI filtering action, which is a pre-processing step, to filter a plurality of eye ROI images and establish a sample set.

Description

Technical Field

[0001] This invention relates to an algorithm, and more particularly to a line-of-sight detection method. Background Technology

[0002] Existing long-distance gaze point detection technologies are mainly used for screen control and public space applications. Non-contact (i.e., non-head-mounted) eye tracking is a method that uses a camera to capture real-time streaming images as input, outputs the user's gaze coordinates through an eye-tracking algorithm, and then uses a distant display to interact with the user interface. Interactions include human-computer interface control and monitoring, information and entertainment interaction, and even latent analysis of the user's gaze's region of interest. The algorithm for this non-contact eye-tracking human-computer interface interaction method needs to consider a significant amount of user information; therefore, the algorithm must include five main steps: face detection, facial feature point detection, head orientation calculation, pupil detection, and gaze point estimation. The second step, facial feature point detection, primarily provides head pose estimation and eye localization.

[0003] However, the stability of long-distance gaze point detection has always been a technical problem that urgently needs to be overcome. The reason for the above-mentioned instability is that long-distance gaze point detection is prone to errors, mainly due to human factors such as different users causing errors in facial feature point detection. The sources of facial feature point detection errors are, for example, (1) the error distribution of different points causes poor head orientation calculation and error sources; (2) failure of eye area positioning causes pupil detection errors, ultimately leading to unstable gaze point.

[0004] Therefore, how to provide a line-of-sight detection method that can solve the above problems is an important issue that the industry needs to consider. Summary of the Invention

[0005] In view of this, the purpose of this invention is to improve the stability of gaze detection point detection in gaze detection methods. One aspect of this disclosure is a gaze detection method comprising steps SA1 and SA2. Step SA1 performs a dynamic threshold to switch operation to correct errors in the data generated by the previous facial feature detection step. Step SA2 performs eye ROI filtering, a preprocessing step, to filter multiple eye ROI images and establish a sample set.

[0006] According to one or more embodiments of this disclosure, step SA1 uses a dynamic threshold as the basis for switching the reference point for calculating a head posture feature.

[0007] According to one or more embodiments of this disclosure, in step SA1, based on a collected contextual image, the Normalized Root Mean Square Error (NRMSE) is calculated, and an NRMSE less than a first threshold is used as the basis for image clustering. Then, the standard deviation of multiple facial features is calculated in each group of images, and the locations of facial features with a standard deviation less than a second threshold are selected as a dynamic threshold setting. It should be noted that the thresholds for NRMSE and standard deviation vary depending on the image collection context; that is, different values ​​can be set for different situations.

[0008] According to one or more embodiments of this disclosure, step SA2 involves preprocessing the image size and pupil feature image to enhance the difference between positive and negative samples and optimize the discrimination effect. Specifically, step SA2 first uses image perspective transformation normalization to unify the image size and orthogonalize the target object, making it suitable for use with a single template. Then, the pupil feature image is preprocessed (e.g., Haar feature analysis, binarization) to emphasize image features and reduce color differences, thereby enhancing the difference between positive and negative samples and optimizing the discrimination effect.

[0009] According to one or more embodiments of this disclosure, step SA2 is set as a criterion for the filter based on a single template matching coefficient being less than a third threshold. It should be noted that the threshold for the single template matching coefficient varies depending on the pupil enhancement method and the template used; that is, it can be set to different values ​​depending on the specific circumstances.

[0010] According to one or more embodiments of this disclosure, step SA1 involves extracting key points corresponding to different angles.

[0011] According to one or more embodiments of this disclosure, step SA1 can reduce the poor head orientation calculation caused by the error distribution of different facial feature points, while step SA2 can filter out the error caused by the failure of eye region positioning of these facial feature points, resulting in pupil detection error, thereby improving the stability of gaze point detection.

[0012] This disclosure also provides a gaze estimation method, comprising: step S1: performing face detection; step S2: performing facial landmark detection; step S3: performing dynamic threshold to switch; step S4: performing head pose estimation; and step S5: performing gaze estimation.

[0013] According to one or more embodiments of this disclosure, step SA1 uses a dynamic threshold as the basis for switching the reference point for calculating a head posture feature.

[0014] According to one or more embodiments of this disclosure, in step SA1, based on a collected contextual image, the Normalized Root Mean Square Error (NRMSE) is calculated, and an NRMSE less than a first threshold is used as the basis for image clustering. Then, the standard deviation of multiple facial features is calculated in each group of images, and the locations of facial features with a standard deviation less than a second threshold are selected as a dynamic threshold setting. It should be noted that the thresholds for NRMSE and standard deviation vary depending on the image collection context; that is, different values ​​can be set for different situations.

[0015] According to one or more embodiments of this disclosure, step SA2 is a preprocessing of the image size and pupil feature image to improve the difference between positive and negative samples and optimize the discrimination effect.

[0016] According to one or more embodiments of this disclosure, step SA2 is set as a criterion for the filter based on a single template matching coefficient being less than a third threshold. It should be noted that the threshold for the single template matching coefficient varies depending on the pupil enhancement method and the template used; that is, it can be set to different values ​​depending on the specific circumstances.

[0017] According to one or more embodiments of this disclosure, step SA1 involves extracting key points corresponding to different angles.

[0018] According to one or more embodiments of this disclosure, step SA1 can reduce the poor head orientation calculation caused by the error distribution of different facial feature points, while step SA2 can filter out the error caused by the failure of eye region positioning of these facial feature points, resulting in pupil detection error, thereby improving the stability of gaze point detection.

[0019] This disclosure also provides a gaze detection method, comprising: step S1: face detection; step S2: facial landmark detection; step SA1: dynamic threshold to switch; step S3: head pose estimation; step SA2: eye ROI filtering; step S4: pupil detection; and step S5: gaze estimation.

[0020] According to one or more embodiments of this disclosure, step SA1 uses a dynamic threshold as the basis for switching the reference point for calculating a head posture feature.

[0021] According to one or more embodiments of this disclosure, in step SA1, based on a collected contextual image, the Normalized Root Mean Square Error (NRMSE) is calculated, and an NRMSE less than a first threshold is used as the basis for image clustering. Then, the standard deviation of multiple facial features is calculated in each group of images, and the locations of facial features with a standard deviation less than a second threshold are selected as a dynamic threshold setting. It should be noted that the thresholds for NRMSE and standard deviation vary depending on the image collection context; that is, different values ​​can be set for different situations.

[0022] According to one or more embodiments of this disclosure, step SA2 is a preprocessing of the image size and pupil feature image to improve the difference between positive and negative samples and optimize the discrimination effect.

[0023] According to one or more embodiments of this disclosure, step SA2 is set as a criterion for the filter based on a single template matching coefficient being less than a third threshold. It should be noted that the threshold for the single template matching coefficient varies depending on the pupil enhancement method and the template used; that is, it can be set to different values ​​depending on the specific circumstances.

[0024] According to one or more embodiments of this disclosure, step SA1 can reduce the poor head orientation calculation caused by the error distribution of different facial feature points, while step SA2 can filter out the error caused by the failure of eye region positioning of these facial feature points, resulting in pupil detection error, thereby improving the stability of gaze point detection. Attached Figure Description

[0025] To make the above and other objects, features, advantages and embodiments of the present invention more readily understood, the accompanying drawings are described below:

[0026] Figure 1 This is a flowchart illustrating a line-of-sight detection method according to an embodiment of the present invention.

[0027] Figure 2 System Description Figure 1 The specific steps of the line-of-sight detection method.

[0028] Figure 3 System Description Figure 1 The specific steps of the line-of-sight detection method.

[0029] Figure 4 System Description Figure 1 A sub-process of a specific step in the line-of-sight detection method.

[0030] Figure 5 This describes the relationship between NRMSE and facial feature points in a gaze detection method according to an embodiment of the present invention.

[0031] Figure 6 This describes the relationship between NRMSE and facial feature points in a gaze detection method according to an embodiment of the present invention.

[0032] Figure 7 This describes the relationship between NRMSE and image ratio in a line-of-sight detection method according to an embodiment of the present invention.

[0033] Figure 8 This describes the relationship between the standard deviation of a face image sample and its respective location in a gaze detection method according to an embodiment of the present invention.

[0034] Figure 9 System Description Figure 1 A sub-process of a specific step in the line-of-sight detection method.

[0035] Figure 10 This describes the image difference between using image perspective transformation normalization and not using it in an embodiment of the present invention.

[0036] Figure 11 This describes the similarity difference of enhanced pupil feature images in a gaze detection method according to an embodiment of the present invention.

[0037] Figures 12A to 12E This describes the effect of image processing using step SA2 in one embodiment of the present invention.

[0038] As is customary practice, the various features and elements in the figures are not drawn to scale, but rather in a manner designed to best represent the specific features and elements related to the present invention. Furthermore, similar elements and components are referred to by the same or similar element symbols across different figures.

[0039] The attached figures are labeled as follows:

[0040] Steps S1, S2, SA1, S3, SA2, S4, S5

[0041] Steps SA11, SA12, SA13, SA14, SA15, SA16, SA17, SA18, SA19

[0042] Steps SA21, SA22, SA23, SA24, SA25, SA26, SA27 Detailed Implementation

[0043] To enable the esteemed review committee to gain a further understanding of the purpose, shape, structural features, and effects of this invention, embodiments are described in detail below with reference to the accompanying drawings.

[0044] The following disclosure provides different embodiments or examples to establish different features of the provided subject matter. The specific examples of the components and arrangements described below are for the purpose of simplifying this disclosure and are not intended to constitute limitation; the size and shape of the elements are not limited by the scope or values ​​disclosed, but may depend on the manufacturing conditions of the elements or the desired characteristics. For example, the technical features of the invention are described using cross-sectional views, which are schematic diagrams of idealized embodiments. Therefore, differences in the shapes illustrated due to manufacturing processes and / or tolerances are foreseeable and should not be limiting.

[0045] Furthermore, spatial relative terms, such as "below," "under," "lower than," "above," and "higher than," are used to easily describe the relationship between the elements or features shown in the accompanying drawings. In addition, spatial relative terms include not only the directions depicted in the drawings but also the different directions in which the elements are used or operated.

[0046] First, it should be specifically noted that "Step SA1" in the specification refers to the step of performing a dynamic threshold to switch operation, used to correct errors in the data generated by the previous facial feature detection step. Secondly, "Step SA2" in the specification refers to the step of performing eye ROI filtering, a preprocessing step used to filter multiple eye ROI images and establish a sample set. In embodiments of the present invention, steps SA1 and / or SA2 help improve the stability of gaze point detection.

[0047] Next, please refer to the following: Figure 1 as well as Figure 2 , Figure 1 This is a flowchart illustrating a line-of-sight detection method according to an embodiment of the present invention; Figure 2 System Description Figure 1 Specific steps of the line-of-sight detection method; Figure 3 System Description Figure 1 The present invention describes specific steps in a gaze detection method. In one embodiment, a gaze detection method includes steps SA1 and SA2. Step SA1 performs a dynamic threshold to switch operation to correct errors in the data generated by the previous facial feature detection step. Step SA2 performs eye ROI filtering, a preprocessing step, to filter multiple eye ROI images and establish a sample set.

[0048] like Figure 1 As shown, one embodiment of the present invention includes at least steps SA1 and SA2, and may further include any one or more of the following steps: performing face detection (S1), performing facial landmark detection (S2), performing head pose estimation (S3), performing pupil center detection (S4), and performing gaze estimation (S5). Later embodiments will be described in more detail.

[0049] like Figure 2As shown, when step SA1 performs the dynamic threshold to switch action, it mainly extracts key points corresponding to different angles. For example, key point 1 corresponds to angle 1; key point 2 corresponds to angle 2; and key point 3 corresponds to angle 3. Furthermore, in this embodiment of the invention, step SA1 uses a dynamic threshold as the basis for switching the reference points for calculating a head pose feature. Moreover, in this embodiment of the invention, step SA1 calculates the normalized root mean square error (NRMSE) based on a collected contextual image, using NRMSE < 0.03 as the basis for image clustering. Then, it calculates the standard deviation of multiple facial features in each group of images and selects the points of facial features with a standard deviation < 0.01 as a dynamic threshold setting.

[0050] like Figure 3 As shown, when step SA2 performs eye ROI filtering, it mainly preprocesses the image size and pupil feature images to filter out positive and negative samples, thereby enhancing the difference between positive and negative samples to optimize the discrimination effect. Specifically, in this embodiment of the invention, when step SA2 performs eye ROI filtering, affine transformation is first performed to orthogonalize the target image and maintain the size and orientation fitting template pattern. Pupil feature enhancement processing is then used to enhance the difference between positive and negative samples to optimize the discrimination effect. Furthermore, in this embodiment of the invention, step SA2 uses a single template matching coefficient <0.6 as a threshold for the filter. Further, in this embodiment of the invention, the collected images come from 400 different people, and since the collected images lack in-camera parameters, general image size normalization is used, and Haar features are specifically used to emphasize pupil image features and reduce the influence of color differences. A single template matching coefficient <0.6 is used as a criterion for the filter. Please refer to [link / reference needed]. Figure 10 , Figure 10 This describes the image difference between using image perspective transformation and normalization in one embodiment of the present invention.

[0051] In addition, in an embodiment of the present invention, step SA1 can reduce the poor head orientation calculation caused by the error distribution of different points of these facial features, while step SA2 can filter out the error caused by the failure of eye region positioning of these facial feature points, resulting in pupil detection error, thereby improving the stability of gaze point detection.

[0052] Next, please refer to the following: Figures 4 to 8 , Figure 4 System Description Figure 1 Sub-processes of specific steps in the line-of-sight detection method; Figure 5 This describes the relationship between NRMSE and facial feature points in a gaze detection method according to an embodiment of the present invention; Figure 6 This describes the relationship between NRMSE and facial feature points in a gaze detection method according to an embodiment of the present invention; Figure 7 This describes the relationship between NRMSE and image ratio in a gaze detection method according to an embodiment of the present invention; Figure 8 This describes the relationship between the standard deviation of a face image sample and its respective location in a gaze detection method according to an embodiment of the present invention.

[0053] like Figure 4 As shown, the sub-process of step SA1 is further explained here, which includes steps SA11 to SA19, as follows:

[0054] In step SA11, the database is collected.

[0055] In step SA12, the NRMSE value of facial landmark from each image is calculated.

[0056] In step SA13, it is determined whether the NRMSE value is less than 0.03 (selecting usable facial feature algorithms with CED≥95%).

[0057] In step SA14, the data is discarded.

[0058] In step SA15, the images are grouped by case according to the object being tested.

[0059] In step SA16, the standard deviation of each facial landmark point within each group is calculated.

[0060] In step SA17, it is determined whether the standard deviation is less than 0.01.

[0061] In step SA18, the point is discarded.

[0062] In step SA19, points are extracted to estimate head pose.

[0063] like Figure 5 As shown, when NRMSE = 0.01, the accuracy of each facial feature point in the face image sample is basically quite high, which can be within the screening criteria (NRMSE < 0.03).

[0064] like Figure 6 As shown, when NRMSE = 0.04, the relative position of each facial feature point in the face image sample is... Figure 5 For the sample, the accuracy was low and did not meet the screening criteria (NRMSE < 0.03).

[0065] Therefore, from Figure 5 and Figure 6 As can be seen, the smaller the NRMSE value, the higher the accuracy of each facial feature point in the face image sample.

[0066] like Figure 7 As shown, it is a cumulative error distribution diagram. It can be seen that the greater the slope of the distribution curve, the greater the proportion of images with small errors, that is, the higher the versatility of this facial feature algorithm.

[0067] In addition, such as Figure 8 As shown, it can be seen that the standard deviation of face image samples with NRMSE < 0.03 for their respective locations can be observed.

[0068] Next, please refer to the following: Figure 9 as well as Figure 11 , Figure 9 System Description Figure 1 Sub-processes of specific steps in the line-of-sight detection method; Figure 11 This describes the similarity difference of enhanced pupil feature images in a gaze detection method according to an embodiment of the present invention.

[0069] like Figure 9 As shown, the sub-process of step SA2 is further explained here, which includes steps SA21 to SA27, as follows:

[0070] In step SA21, eye images are extracted from facial landmarks.

[0071] In step SA22, image perspective transformation normalization is performed.

[0072] In step SA23, the action of reinforcing pupil features is performed.

[0073] In step SA24, the template matching action is performed.

[0074] In step SA25, samples with a similarity of <0.6 are calculated and identified.

[0075] In step SA26, the line-of-sight estimation action is not performed, meaning that some samples are determined to be skipped (Gazeestimation:bypass).

[0076] In step SA27, the gaze estimation action is performed.

[0077] In addition, this disclosure also provides a gaze estimation method, comprising: step S1: performing face detection; step S2: performing facial landmark detection; step S3: performing dynamic threshold to switch; step S4: performing head pose estimation; and step S5: performing gaze estimation.

[0078] In an embodiment of the present invention, step SA1 uses a dynamic threshold as the basis for switching the reference points for calculating a head pose feature. In an embodiment of the present invention, step SA1 calculates the normalized root mean square error (NRMSE) based on a collected context image, and uses NRMSE < 0.03 as the basis for image clustering. Then, it calculates the standard deviation of multiple facial features in each group of images, and selects the points of facial features with a standard deviation < 0.01 as a dynamic threshold setting.

[0079] This disclosure also provides a gaze detection method, comprising: step S1: face detection; step S2: facial landmark detection; step SA1: dynamic threshold to switch; step S3: head pose estimation; step SA2: eye ROI filtering; step S4: pupil detection; and step S5: gaze estimation.

[0080] In an embodiment of the present invention, step SA1 uses a dynamic threshold as the basis for switching the reference points for calculating a head pose feature. In an embodiment of the present invention, step SA1 calculates the normalized root mean square error (NRMSE) based on a collected context image, and uses NRMSE < 0.03 as the basis for image clustering. Then, it calculates the standard deviation of multiple facial features in each group of images, and selects the points of facial features with a standard deviation < 0.01 as a dynamic threshold setting.

[0081] In an embodiment of the present invention, step SA1 involves extracting key points corresponding to different angles. In this embodiment, step SA1 can reduce the poor head orientation calculation caused by the error distribution of different facial feature points, while step SA2 can filter out the error caused by the failure of eye region localization of these facial feature points, resulting in pupil detection errors, thereby improving the stability of gaze point detection.

[0082] In an embodiment of the present invention, step SA2 preprocesses the image size and pupil feature images to enhance the difference between positive and negative samples and optimize the discrimination effect. Specifically, in an embodiment of the present invention, when step SA2 performs eye ROI filtering, it first performs affine transformation to orthogonalize the target image and maintain the size and orientation fitting template pattern, and then enhances the difference between positive and negative samples through pupil feature enhancement processing to optimize the discrimination effect. In an embodiment of the present invention, step SA2 sets a single template matching coefficient < 0.6 as a threshold for the filter. In an embodiment of the present invention, step SA2 preprocesses the image size and pupil feature images to enhance the difference between positive and negative samples and optimize the discrimination effect. In an embodiment of the present invention, step SA2 sets a single template matching coefficient < 0.6 as a threshold for the filter. Furthermore, in an embodiment of the present invention, the collected images are from 400 different people. Since the collected images lack in-camera parameters, they are normalized using standard image size and enhanced with Haar features to emphasize pupil image features and reduce the impact of color differences. A single template matching coefficient <0.6 is set as a criterion for the filter. Figure 11 The aim is to emphasize the difference in similarity between positive and negative samples after image processing.

[0083] like Figure 11 As shown, the first and third curves from top to bottom in the figure represent negative samples (without a complete eye image), while the second and fourth curves from top to bottom represent positive samples (with a complete eye image). Figure 11 The similarity difference between the enhanced pupil feature images and the original images can be observed. The third and fourth curves from the top represent the similarity difference calculated without pupil enhancement processing, while the first and second curves from the top represent the similarity difference after pupil enhancement processing. From the first and second curves from the top, it can be seen that the difference in similarity coefficients between positive and negative samples increases due to the addition of pupil processing.

[0084] Please also refer to the following: Figures 12A to 12E , Figures 12A to 12E This illustrates the effect of image processing using step SA2 in one embodiment of the present invention. As shown in Figure 12, the embodiment of the present invention, using step SA2, can skip blinking and incorrect images, and through transmission conversion, makes single-template detection more suitable for deflection conditions. Furthermore, Figure 12A The image capture is displayed correctly. Figure 12B The blinking image was displayed; Figure 12C This shows the situation where the image is obstructed; Figure 12D This indicates an incorrectly positioned image; Figure 12EThe comparison results show that the image is skewed without perspective conversion calculation, but straightened after perspective conversion calculation. In other words, the results show that the image is better adapted to a single template after perspective conversion calculation.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A line-of-sight detection method, characterized in that, Include: Step SA1: Perform dynamic threshold conversion. Step SA1 is used to correct the errors in the data generated by the previous facial feature detection step. In step SA1, based on the collected context images, the normalized root mean square error is calculated, and the normalized root mean square error is less than a first threshold as the basis for image clustering. The first threshold is 0.

03. Then, the standard deviation of multiple facial features in each group of images is calculated, and the points of the multiple facial features with a standard deviation less than a second threshold are selected as dynamic threshold settings. The second threshold is 0.

01. The dynamic threshold is used as the basis for switching the reference points for calculating head posture features. as well as Step SA2: Perform eye region of interest filtering, including: extracting eye images from facial features; performing image perspective transformation and normalization; enhancing pupil features; performing template matching; calculating and identifying samples that are filtered based on a single template matching coefficient less than a third threshold; the filtered samples do not undergo gaze estimation; the remaining samples undergo gaze estimation. Step SA2 is a preprocessing step, which preprocesses the image size and pupil feature images to improve the difference between positive and negative samples and optimize the discrimination effect, thereby filtering multiple eye region of interest images and establishing a sample set. Step SA2 uses a single template matching coefficient less than the third threshold as the criterion for the filter, and the third threshold is 0.

6.

2. The line-of-sight detection method as described in claim 1, characterized in that, Step SA1 involves extracting key points corresponding to different angles.

3. The gaze detection method as described in claim 1, characterized in that, The step SA1 can reduce the poor head orientation calculation caused by the error distribution of different points of the multiple facial features, while the step SA2 can filter out the error caused by the failure of eye region positioning of the multiple facial feature points, which leads to pupil detection error, thereby improving the stability of gaze point detection.

4. A line-of-sight detection method, characterized in that, Include: Step S1: Face detection; Step S2: Facial feature detection; Step SA1: Dynamic threshold conversion. Based on the collected contextual images, the normalized root mean square error is calculated, and the normalized root mean square error is less than a first threshold as the basis for image clustering. The first threshold is 0.

03. Then, the standard deviation of multiple facial features in each group of images is calculated, and the points of the multiple facial features with a standard deviation less than a second threshold are selected as dynamic threshold settings. The second threshold is 0.

01. The dynamic threshold is used as the basis for switching the reference points for calculating head posture features. Step S3: Head pose prediction; Step SA2: Eye Region of Interest Filtering, including: extracting eye images from facial features; performing image perspective transformation normalization; enhancing pupil features; performing template matching; calculating and identifying samples that are filtered based on a single template matching coefficient less than a third threshold; the filtered samples do not undergo gaze estimation; the remaining samples undergo gaze estimation. Step SA2 is a preprocessing step, which preprocesses the image size and pupil feature images to improve the difference between positive and negative samples and optimize the discrimination effect. Step SA2 uses a single template matching coefficient less than the third threshold as the criterion for the filter, and the third threshold is 0.

6. Step S4: Pupil center detection; and Step S5: Line of sight estimation.

5. The gaze detection method as described in claim 4, characterized in that, The step SA1 can reduce the poor head orientation calculation caused by the error distribution of different points of the multiple facial features, while the step SA2 can filter out the error caused by the failure of eye region positioning of the multiple facial feature points, which leads to pupil detection error, thereby improving the stability of gaze point detection.

Images