Gaze estimation method, system and device, and storage medium

Abstract

A gaze estimation method, system and device, and a storage medium. First human-eye data of a user when gazing each preset marker, and corresponding preset marker position data are collected to obtain a first sampling data set; a data quality test is performed on the first sampling data set to obtain a calibration data set, the data quality of which meets standards; calibration samples in the calibration data set are preprocessed to obtain corresponding first eye movement data and first gaze data as training samples, and a training data set is generated; and the training data set is used to train a constructed first gaze estimation model, so as to obtain a second gaze estimation model.

Description

A sight line estimation method, system, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 18, 2023, with application number 202311201115.2 and invention name “A line of sight estimation method, system, device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present invention relate to the field of computer vision technology, and specifically to a line of sight estimation method, system, device and storage medium. Background Art

[0003] The line represents the direction of eye gaze and the focus of a person's attention. Eye tracking is currently an active research topic, with great potential for application in fields such as human-computer interaction and human cognitive science. Eye tracking is based on estimating the direction of a person's gaze.

[0004] In related technologies, the P-CR method is usually used to determine the eye gaze information. The P-CR method is divided into two types: regression-based P-CR method and 3D-based P-CR method. When these two methods are implemented, they both need to use multiple light sources to form multiple light spots in the user's eyeballs and obtain the user's human eye image. By performing image processing on the human eye image, the light spots corresponding to all light sources are detected on the human eye image, and then the eye gaze information is determined based on the correspondence between the light source and the light spot.

[0005] Existing deep learning-based gaze estimation methods have an error of approximately 5°, making them difficult to meet commercial requirements. Currently available calibration datasets are collected from multiple subjects and do not account for differences in the Kappa angle (the angle between the pupil midline and the visual axis) of each subject, resulting in low-quality calibration data. Summary of the Invention

[0006] To this end, embodiments of the present invention provide a line of sight estimation method, system, device, and storage medium to solve the current technical problem of large line of sight estimation accuracy error.

[0007] In order to achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, an embodiment of the present application provides a line of sight estimation method, the method comprising:

[0009] Constructing a first sight estimation model;

[0010] Collecting first eye data of the user gazing at each preset marker and corresponding preset marker position data as a first sampling example and generating a first sampling data set;

[0011] Performing data quality detection on the first sampling data set;

[0012] Based on the data quality test results, obtain calibration samples with qualified data quality and generate a calibration data set;

[0013] Preprocessing each first eye data and corresponding preset marker position data in the calibration data set to obtain corresponding first eye movement data and first sight line data as training samples, and generating a training data set;

[0014] Using the training data set to train the first sight line estimation model to obtain a second sight line estimation model;

[0015] The step of performing data quality detection on the first sample data set includes:

[0016] Selecting clustered data points from each first human eye data in the first sampling data set;

[0017] Perform cluster analysis on each cluster data point, mark all cluster data points that are not classified into any cluster as noise points, and count the number of clusters and noise points;

[0018] Determining whether the number of clusters is consistent with the number of preset markers and whether the number of noise points is less than a first preset threshold;

[0019] If the number of clusters is consistent with the number of the preset markers, and the number of noise points is less than a first preset threshold, the cluster analysis result of the first sampling data set is qualified; otherwise, the cluster analysis result of the first sampling data set is unqualified, and the first sampling data set is resampled;

[0020] For the first sampling data set with qualified cluster analysis results, first sampling examples corresponding to noise points are removed, and the remaining first sampling examples are used as second sampling examples to update and generate a second sampling data set;

[0021] The second sampling example is used as a calibration example, and the calibration data set is the second sampling data set.

[0022] Furthermore, a line of sight estimation method provided in an embodiment of the present application further includes:

[0023] Collecting second eye data when the user looks at the target marker;

[0024] Preprocessing the second human eye data to obtain corresponding second eye movement data;

[0025] The second eye movement data is input into the second sight line estimation model for processing to obtain second sight line data.

[0026] Furthermore, a line of sight estimation method provided in an embodiment of the present application further includes:

[0027] Saving the training data set to a terminal for storage and management, or uploading the training data set to a cloud for storage and management;

[0028] For the training data set, a two-level directory is used for storage, the first-level directory is the user ID, and the second-level directory is the storage time;

[0029] The training data set obtained by a single user at a time is used as a training data set storage unit;

[0030] Each training data set storage unit includes: a data set description file and a training sample set.

[0031] Furthermore, cluster analysis is performed on each cluster data point, including:

[0032] Randomly select a cluster data point that has not been visited as the starting point, and use the remaining cluster data points as the test points;

[0033] Taking the starting point as the center, generating a corresponding preset neighborhood based on a second preset threshold;

[0034] Comparing the first distance between each test point and the starting point with the second preset threshold value to determine whether there is a test point within a preset neighborhood of the starting point;

[0035] If there is a point to be tested within the preset neighborhood of the starting point, the point to be tested is used as the core point, and it is determined whether there is a cluster data point that has not been visited;

[0036] If there is no point to be tested within the preset neighborhood of the starting point, directly determine whether there is any cluster data point that has not been visited;

[0037] If there is an unvisited cluster data point, the next unvisited cluster data point is randomly selected as the starting point;

[0038] If there are no unvisited cluster data points, each starting point and all core points in the corresponding preset neighborhood are included in the same cluster.

[0039] Furthermore, the first eye data is a relative position reading of the eye obtained by a plurality of spatially sparsely distributed sensor arrays, and preprocessing each first eye data and corresponding preset marker position data in the calibration data set includes:

[0040] Calculate the mean MEAN and variance STD for the relative position readings of the human eyes in each calibration sample;

[0041] The relative eye position readings in each calibration sample are first subtracted from the mean MEAN and then divided by the variance STD to obtain the corresponding first eye movement data;

[0042] Taking the upper left corner or the center marker in the horizontal or vertical direction as the reference point, the distance between the leftmost marker to the rightmost marker in the horizontal direction as the unit length in the horizontal direction, and the distance between the uppermost marker to the lowermost marker in the vertical direction as the unit length in the vertical direction, the coordinates of the preset marker position data in each calibration sample are normalized to obtain the corresponding first line of sight data.

[0043] Furthermore, performing data quality detection on the first sample data set further includes:

[0044] Numbering each cluster in the second sampling data set to obtain a corresponding numbered cluster;

[0045] All clustered data points corresponding to a preset number of numbered clusters are selected to form a first cluster point set, and all clustered data points corresponding to the remaining numbered clusters are selected to form a second cluster point set;

[0046] Performing fitting using the first cluster point set and the second cluster point set to obtain a first fitted ellipse and a second fitted ellipse respectively;

[0047] Calculating the shortest distance between each cluster data point in the first cluster point set and the contour point of the first fitted ellipse as a second distance, and calculating the shortest distance between each cluster data point in the second cluster point set and the contour point of the second fitted ellipse as a third distance;

[0048] Calculate the first square sum S1 of all second distances and the second square sum S2 of all third distances;

[0049] Calculating a first fitting error E1 using the first square sum S1, and calculating a second fitting error E2 using the second square sum S2;

[0050] Determining whether the first fitting error E1 and the second fitting error E2 are both smaller than a first preset error and a second preset error, respectively;

[0051] If the first fitting error E1 is less than a first preset error, and the second fitting error E1 is less than a second preset error, the ellipse fitting error analysis result of the second sampling data set is qualified; otherwise, the ellipse fitting error analysis result of the second sampling data set is unqualified, and the first sampling data set is resampled;

[0052] Determine the second sampling sample in the second sampling data set with a qualified ellipse fitting error analysis result as a third sampling sample, and generate a third sampling data set;

[0053] The third sampling example is used as a calibration example, and the calibration data set is the third sampling data set.

[0054] Furthermore, the first human eye data is human eye image data captured by a camera, and preprocessing each first human eye data and corresponding preset marker position data in the calibration data set includes:

[0055] Detecting pupil center point data or pupil outline point data from the human eye image data in each calibration sample to obtain corresponding first eye movement data;

[0056] Taking the central marker as a reference point and the distance from the outermost marker to the central marker as a unit length, coordinate normalization is performed on the preset marker position data in each calibration sample to obtain corresponding first line of sight data.

[0057] According to a second aspect of an embodiment of the present invention, an embodiment of the present application provides a line of sight estimation system, the system comprising:

[0058] A model building module, used for building a first sight estimation model;

[0059] A collection module, configured to collect first eye data of a user gazing at each preset marker and corresponding preset marker position data as a first sampling example and generate a first sampling data set;

[0060] A detection module, configured to perform a data quality detection on the first sampling data set; based on the data quality detection result, obtain a calibration sample with qualified data quality and generate a calibration data set;

[0061] a preprocessing module, configured to preprocess each first eye data and corresponding preset marker position data in the calibration data set, obtain corresponding first eye movement data and first sight line data as training examples, and generate a training data set;

[0062] a training module, configured to train the first sight line estimation model using the training data set to obtain a second sight line estimation model;

[0063] The step of performing data quality detection on the first sample data set includes:

[0064] Selecting clustered data points from each first human eye data in the first sampling data set;

[0065] Perform cluster analysis on each cluster data point, mark all cluster data points that are not classified into any cluster as noise points, and count the number of clusters and noise points;

[0066] Determining whether the number of clusters is consistent with the number of preset markers and whether the number of noise points is less than a first preset threshold;

[0067] If the number of clusters is consistent with the number of the preset markers, and the number of noise points is less than a first preset threshold, the cluster analysis result of the first sampling data set is qualified; otherwise, the cluster analysis result of the first sampling data set is unqualified, and the first sampling data set is resampled;

[0068] For the first sampling data set with qualified cluster analysis results, first sampling examples corresponding to noise points are removed, and the remaining first sampling examples are used as second sampling examples to update and generate a second sampling data set;

[0069] The second sampling example is used as a calibration example, and the calibration data set is the second sampling data set.

[0070] According to a third aspect of an embodiment of the present invention, a sight line estimation device is provided, the device comprising: a processor and a memory;

[0071] The memory is used to store one or more program instructions;

[0072] The processor is used to run one or more program instructions to execute the steps of a line of sight estimation method as described in any one of the above items.

[0073] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a line of sight estimation method as described in any one of the above items are implemented.

[0074] Compared to existing technologies, the gaze estimation method, system, device, and storage medium provided in the embodiments of the present application collect first eye data and corresponding preset marker position data when a user gazes at each preset marker to obtain a first sampling data set; perform data quality testing on the first sampling data set to obtain a calibration data set with qualified data quality; preprocess the calibration samples in the calibration data set to obtain corresponding first eye movement data and first gaze data as training samples, and generate a training data set; and use the training data set to train the constructed first gaze estimation model to obtain a second gaze estimation model. This achieves high-precision gaze estimation of the user, significantly reducing the accuracy error of gaze estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0076] The structures, proportions, sizes, etc. illustrated in this specification are intended only to complement the contents disclosed herein and to facilitate understanding and reading by persons familiar with the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall still fall within the scope of the technical contents disclosed herein.

[0077] FIG1 is a schematic diagram of the logical structure of a sight line estimation system provided by an embodiment of the present invention;

[0078] FIG2 is a schematic flow chart of a line of sight estimation method provided by an embodiment of the present invention;

[0079] FIG3 is a schematic diagram of human eye image data captured by a camera according to an embodiment of the present invention;

[0080] FIG4 is a schematic diagram showing a visualization effect of the number and position distribution of preset markers corresponding to the human eye image data shown in FIG3 , provided by an embodiment of the present invention;

[0081] FIG5 is a schematic diagram of a plurality of spatially sparsely distributed sensor arrays provided by an embodiment of the present invention;

[0082] FIG6 is a schematic diagram showing the number and position distribution of preset markers corresponding to the sensor array shown in FIG5 according to an embodiment of the present invention;

[0083] FIG7 is a schematic diagram of a process for performing data quality detection on the first sample data set according to an embodiment of the present invention;

[0084] FIG8 is a schematic diagram of a process for performing cluster analysis on each cluster data point according to an embodiment of the present invention;

[0085] FIG9 is a schematic diagram of clustered data points selected from the human eye image data shown in FIG3 according to an embodiment of the present invention;

[0086] FIG10 is a schematic diagram of a process for performing data quality detection on the first sample data set according to another embodiment of the present invention;

[0087] FIG11 is a schematic diagram of a numbering cluster provided in an embodiment of the present invention;

[0088] FIG12 is a schematic diagram of a two-level directory for storing a calibration data set before preprocessing according to an embodiment of the present invention;

[0089] FIG13 is a flow diagram of a two-level directory for storing a training data set obtained after preprocessing according to an embodiment of the present invention. DETAILED DESCRIPTION

[0090] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0091] The embodiments of the present application are applicable to the field of VR / AR eye tracking. For example, they can be applied to VR / AR devices that need to track the user's gaze, providing gaze estimation capabilities to enable visual interaction, gaze point rendering, zoom display, and other functions.

[0092] The purpose of the embodiments of the present application is to generate training data based on calibration data collected by users for eye tracking, and train a high-accuracy gaze estimation model to solve the current technical problem of large accuracy errors in gaze estimation.

[0093] In order to solve the above technical problems, as shown in Figure 1, an embodiment of the present application provides a line of sight estimation system, which specifically includes: a model building module 01, an acquisition module 02, a detection module 03, a preprocessing module 04, a training module 06, and an execution module 07.

[0094] Specifically, in the model training stage, the model construction module 01 is used to construct a first line of sight estimation model 08; the acquisition module 02 is used to collect the first human eye data and the corresponding preset marker position data when the user looks at each preset marker as a first sampling example and generate a first sampling data set; the detection module 03 is used to perform data quality detection on the first sampling data set; based on the data quality detection result, a calibration sample with qualified data quality is obtained and a calibration data set is generated; the preprocessing module 04 is used to preprocess each first human eye data and the corresponding preset marker position data in the calibration data set, and obtain the corresponding first eye movement data and first line of sight data as training examples, and generate a training data set; the training module 06 is used to use the training data set to train the first line of sight estimation model 08 to obtain a second line of sight estimation model 09.

[0095] During the model execution stage, the acquisition module 02 is also used to collect the second human eye data when the user looks at the target marker; the preprocessing module 04 is also used to preprocess the second human eye data to obtain the corresponding second eye movement data; the execution module 07 is used to input the second eye movement data into the second line of sight estimation model 09 for processing to obtain the second line of sight data.

[0096] Furthermore, a line of sight estimation system disclosed in an embodiment of the present invention also includes: a storage module 05, which is used to perform the following steps: saving the training data set to the terminal for storage and management, or uploading the training data set to the cloud for storage and management; for the training data set, a two-level directory method is used for storage, the first-level directory is the user ID, and the second-level directory is the storage time; the training data set obtained by a single user at a time is used as a training data set storage unit; each training data set storage unit includes: a data set description file and a training sample set.

[0097] Compared to existing technologies, the gaze estimation system provided in the embodiments of the present application collects first eye data and corresponding marker position data when the user gazes at various preset markers to obtain a first sampling data set; performs data quality testing on the first sampling data set to obtain a calibration data set with qualified data quality; preprocesses the calibration samples in the calibration data set to obtain corresponding first eye movement data and first gaze data as training samples, and generates a training data set; and uses the training data set to train the constructed first gaze estimation model to obtain a second gaze estimation model. This system achieves high-precision gaze estimation of the user and significantly reduces the accuracy error of gaze estimation.

[0098] Corresponding to the above-disclosed sight line estimation system, an embodiment of the present invention further discloses a sight line estimation method. The following describes in detail the sight line estimation method disclosed in an embodiment of the present invention in conjunction with the above-described sight line estimation system.

[0099] First, taking the first human eye data collected from the user as human eye image data taken by a camera as an example, as shown in Figure 2, the specific steps of a line of sight estimation method provided in an embodiment of the present application are described in detail below.

[0100] The first sight line estimation model 08 is constructed by the model construction module 01 .

[0101] In an embodiment of the present invention, the constructed first sight estimation model 08 is a deep learning model. The backbone network of the first sight estimation model 08 can be of two types. If the input of the first sight estimation model 08 is image data, then the backbone network of the first sight estimation model 08 can be a convolutional neural network. If the input of the first sight estimation model 08 is feature point data, then the backbone network of the first sight estimation model 08 can be a graph convolutional neural network or a graph recurrent neural network. Both backbone networks are connected to a multi-layer perceptron.

[0102] The collection module 02 collects first eye data when the user looks at each preset marker and corresponding preset marker position data as a first sampling example and generates a first sampling data set.

[0103] Specifically, in this embodiment of the present invention, a user sequentially gazes at several preset markers, each for a certain duration. Acquisition module 02 then records at least one frame of first eye data and the corresponding preset marker position data. Referring to FIG3 , the first eye data is the eye image data captured by a camera. Referring to FIG4 , the preset marker position data is the coordinates of each preset dot on a plane. As shown in FIG4 , there are 19 dot markers. The format of the first sampling example is {first eye data, preset marker position data}.

[0104] The detection module 3 performs data quality detection on the first sample data set.

[0105] 7 , the above steps specifically include: selecting cluster data points from each first human eye data in the first sampling data set; performing cluster analysis on each cluster data point, marking all cluster data points that are not classified into any cluster as noise points and counting the number of cluster clusters and the number of noise points; judging whether the number of cluster clusters is consistent with the number of preset markers and whether the number of noise points is less than a first preset threshold; if the number of cluster clusters is consistent with the number of preset markers and the number of noise points is less than the first preset threshold, the cluster analysis result of the first sampling data set is qualified, otherwise the cluster analysis result of the first sampling data set is unqualified, and the first sampling data set is resampled; for the first sampling data set with qualified cluster analysis results, the first sampling samples corresponding to the noise points are eliminated, and the remaining first sampling samples are used as second sampling samples to update and generate the second sampling data set; the second sampling samples are used as calibration samples, and the calibration data set is the second sampling data set.

[0106] Since the first eye data in this case is eye image data captured by a camera, the pupil centers of each eye image in the first sampled dataset are first detected and used as clustering data points to obtain a pupil center point set. The visualization of the pupil center point set is shown in Figure 9. A clustering algorithm is then used to cluster the pupil center point set, obtaining the number of clusters and the number of noise points.

[0107] 8 , the above-mentioned cluster analysis of each cluster data point, that is, the specific steps of the clustering algorithm include: randomly selecting an unvisited cluster data point as the starting point, and taking the remaining cluster data points as test points; taking the starting point as the center, generating a corresponding preset neighborhood based on a second preset threshold; comparing the first distance between each test point and the starting point with the second preset threshold to determine whether the test point exists within the preset neighborhood of the starting point; if the test point exists within the preset neighborhood of the starting point, taking the test point as the core point, and determining whether there is an unvisited cluster data point; if there is no test point within the preset neighborhood of the starting point, directly determining whether there is an unvisited cluster data point; if there is an unvisited cluster data point, randomly selecting the next unvisited cluster data point as the starting point; if there is no unvisited cluster data point, including each starting point and all core points within the corresponding preset neighborhood into the same cluster.

[0108] As described above, the method for determining whether a cluster data point is within the neighborhood of another cluster data point is as follows: determining based on the distance (Euclidean distance or Mahalanobis distance) between the two cluster data points; for each starting point, if other cluster data points exist within the neighborhood of the starting point, then these cluster data points and the starting point are added to the same cluster; and the above steps are repeated until each cluster data point has been visited. Factors that affect the quality of the calibration dataset mainly include: during the acquisition of the first sampling dataset, due to factors such as user inattention, the first sampling dataset contains noise samples, or lacks samples corresponding to certain markers. To address the above issues, the embodiments of the present invention use the above-mentioned cluster analysis of each cluster data point to perform data quality detection, effectively overcoming the impact of the above-mentioned situations on model training.

[0109] Based on the data quality detection result, the detection module 03 obtains a calibration sample with qualified data quality and generates a calibration data set. As mentioned above, at this time, the calibration sample is the second sampling sample and the calibration data set is the second sampling data set.

[0110] Further, with reference to FIG10 , in an embodiment of the present invention, performing data quality detection on the first sampling data set further includes: numbering each cluster in the second sampling data set to obtain corresponding numbered clusters; selecting all clustered data points corresponding to a preset number of numbered clusters to form a first cluster point set, and forming a second cluster point set from all clustered data points corresponding to the remaining numbered clusters; performing fitting using the first cluster point set and the second cluster point set to obtain a first fitted ellipse and a second fitted ellipse, respectively; calculating the shortest distance between each clustered data point in the first cluster point set and the contour point of the first fitted ellipse as a second distance, and calculating the shortest distance between each clustered data point in the second cluster point set and the contour point of the second fitted ellipse as a third distance, wherein the calculation formulas for the second distance and the third distance are respectively:

[0111] Among them, d i 1 is the shortest distance between the i-th cluster data point in the first cluster point set and the contour point of the first fitted ellipse, d j 2 is the shortest distance between the jth cluster data point in the second cluster point set and the contour point of the second fitted ellipse, are the horizontal and vertical coordinates of the i-th cluster data point in the first cluster point set, respectively. are the horizontal and vertical coordinates of the center point of the first fitting ellipse, are the horizontal and vertical coordinates of the j-th cluster data point in the second cluster point set, respectively. are the abscissa and ordinate of the center point of the second fitting ellipse, respectively; i and j are both integers greater than zero; a1 and b1 are the minor axis length and major axis length of the first fitting ellipse, respectively; a2 and b2 are the minor axis length and major axis length of the second fitting ellipse, respectively; θ1 is the rotation angle of the first fitting ellipse, and θ2 is the rotation angle of the second fitting ellipse; calculate the first square sum S1 of all second distances and the second square sum S2 of all third distances, and the calculation formulas for the first square sum S1 and the second square sum S2 are respectively:

[0112] Wherein, m and n are the numbers of clustered data points in the first cluster point set and the second cluster point set, respectively; the first fitting error E1 is calculated using the first square sum S1, and the second fitting error E2 is calculated using the second square sum S2. The calculation formulas for the first fitting error E1 and the second fitting error E2 are respectively:

[0113] Wherein, m and n are the number of clustered data points in the first cluster point set and the second cluster point set, respectively; it is determined whether the first fitting error E1 and the second fitting error E2 are simultaneously less than the first preset error and the second preset error, respectively; if the first fitting error E1 is less than the first preset error, and the second fitting error E1 is less than the second preset error, then the ellipse fitting error analysis result of the second sampling data set is qualified; otherwise, the ellipse fitting error analysis result of the second sampling data set is unqualified, and the first sampling data set is resampled; the second sampling example in the second sampling data set with a qualified ellipse fitting error analysis result is determined as a third sampling example, and a third sampling data set is generated; the third sampling example is used as a calibration example, and the calibration data set is the third sampling data set.

[0114] Factors affecting the quality of the calibration dataset also include: the camera sliding relative to the user's head during the acquisition of the first sample dataset, resulting in dataset distortion. To address this issue, embodiments of the present invention utilize ellipse fitting and error analysis techniques for evaluation. After the first sample dataset is determined to be qualified through cluster analysis, a second sample dataset is obtained. Ellipse fitting is then performed on the selected pupil center point set, and the dataset's qualification is determined based on the ellipse fitting error. Figure 11 shows a schematic diagram of numbered clusters provided by an embodiment of the present invention. Since the first sample dataset is determined to be qualified through cluster analysis, the updated second sample dataset has 19 clusters. According to the above steps, clusters numbered 0 to 18 are selected as the first cluster point set, and clusters numbered 7 to 18 are selected as the second cluster point set. Ellipse fitting is performed on each of these clusters to obtain a first fitted ellipse and a second fitted ellipse. The second and third distances, as well as the first fitting error E1 and the second fitting error E2, are then calculated according to the above steps. If the first fitting error E1 is less than the first preset error and the second fitting error E1 is less than the second preset error, the ellipse fitting error analysis result for the second sample dataset is qualified; otherwise, it is unqualified.

[0115] Similarly, based on the data quality detection result, the detection module 03 obtains a calibration sample with qualified data quality and generates a calibration data set. As mentioned above, at this time, the calibration sample is the third sampling sample and the calibration data set is the third sampling data set.

[0116] Furthermore, a line of sight estimation method provided in an embodiment of the present invention further includes, via storage module 05, saving the pre-processed calibration dataset to the terminal for storage and management, or uploading the pre-processed calibration dataset to the cloud for storage and management. Specifically, the communication basis between the terminal and the cloud is Ethernet, and the communication protocol is TCP / IP.

[0117] Specifically, referring to Figure 12, the calibration dataset before preprocessing is stored in a two-level directory format, where the first-level directory is the user ID and the second-level directory is the storage time; the calibration dataset before preprocessing obtained by a single user at a single time is used as a calibration dataset storage unit; each calibration dataset storage unit includes: a dataset description file and a training sample set.

[0118] At this time, the dataset description file is used to describe the relevant information of the corresponding calibration dataset, including the dataset name, dataset size (number of samples), first eye data type (image data / feature point data), creator, creation time; sample set, that is, a collection of samples. Specifically, the sample naming rule is ID_EYE_POINT_NUM, where ID represents the user ID, EYE represents the left / right eye code, 0 represents the left eye, 1 represents the right eye, POINT represents the marker number, and NUM represents the sample sequence number. For example, 085_0_1_001, user ID is 085, left eye 0, marker number 1, sample sequence number 001, comprehensive understanding: the left eye of the user with ID 085, when looking at marker number 1, obtains sample number 001.

[0119] The preprocessing module 04 preprocesses each first eye data and the corresponding preset marker position data in the calibration data set to obtain corresponding first eye movement data and first sight line data as training samples, and generates a training data set.

[0120] The above steps specifically include: detecting pupil center point data or pupil contour point data from the human eye image data in each calibration sample to obtain the corresponding first eye movement data; using the central marker as a reference point and the distance from the outermost marker to the central marker as the unit length, normalizing the coordinates of the preset marker position data in each calibration sample to obtain the corresponding first line of sight data.

[0121] Similarly, a line of sight estimation method provided in an embodiment of the present invention further includes, via storage module 05, saving the preprocessed training dataset to the terminal for storage and management, or uploading the preprocessed training dataset to the cloud for storage and management. Specifically, the communication basis between the terminal and the cloud is Ethernet, and the communication protocol is TCP / IP.

[0122] Specifically, referring to Figure 13, the training data set obtained after preprocessing is stored in a two-level directory format, where the first-level directory is the user ID and the second-level directory is the storage time; the training data set obtained by a single user at a single time is regarded as a training data set storage unit; each training data set storage unit includes: a data set description file and a training sample set.

[0123] The dataset description file describes the corresponding calibration dataset, including the dataset name, dataset size (number of samples), first eye movement data type (image data / feature point data), first gaze data type (gaze point in 2D plane / gaze starting point and direction in 3D space), creator, and creation time. The sample set is a collection of samples. Specifically, the sample naming convention is ID_EYE_POINT_NUM, where ID represents the user ID, EYE represents the left / right eye designation (0 for the left eye and 1 for the right eye), POINT represents the landmark number, and NUM represents the sample sequence number. For example, 085_0_1_001 indicates user ID 085, left eye 0, landmark number 1, and sample sequence number 001. This means that the left eye of user ID 085, when looking at landmark 1, acquires sample number 001.

[0124] The training module 06 trains the first sight line estimation model 08 using the training data set to obtain the second sight line estimation model 09 .

[0125] Specifically, on the terminal or the cloud, the first line of sight estimation model is trained based on a training data set obtained from a first sampling data set currently collected by the user to obtain a second line of sight estimation model, so as to achieve high-precision line of sight estimation for a specific user.

[0126] On the terminal or cloud, the first line of sight estimation model is trained based on a training data set obtained from the first sampling data set collected multiple times from the same user to obtain a second line of sight estimation model, so as to achieve high-precision line of sight estimation for a specific user.

[0127] On the terminal or in the cloud, the first sight line estimation model is trained based on a training data set obtained from the first sampling data set collected multiple times by multiple users to obtain a second sight line estimation model, so as to achieve high-precision sight line estimation for general users.

[0128] The specific steps of training the first line of sight estimation model 08 using the training data set include: using the first line of sight estimation model to infer the predicted line of sight data based on the first eye movement data in the training data set; determining the parameters of the loss function based on the predicted line of sight data and the first eye movement data; the loss function can be L1Loss or L2Loss, etc.; back propagating based on the parameters of the loss function to adjust the model parameters. If the model is trained based on a calibration data set collected and preprocessed by the same or multiple users, the adjusted parameters are all the parameters of the model. If the training is based on the user's current collected and preprocessed data set, the adjusted parameters tend to select local parameters to shorten the training time. The local parameters can be the parameters of a multi-layer perceptron. Among them, the learning rate related to the adjustment of the model parameters is dynamically adjusted using the Cosine Annealing method. As the number of training rounds epochs increases, the learning rate lr decreases in a cosine form. The formula is:

[0129] Among them, lr_max represents the initial learning rate, lr_min represents the minimum learning rate, epoch represents the current epoch number, and epochs represents the total epoch number.

[0130] Based on the preprocessed training dataset, the above steps are repeated repeatedly, and training ends when preset conditions are met. Preset conditions include but are not limited to: reaching the maximum number of iterations or training rounds, or when the loss is within an acceptable range, or when the accuracy meets the requirement, or when artificial early stopping is performed.

[0131] If the second gaze estimation model is trained in the cloud and based on a pre-processed training dataset collected by the same user, the second gaze estimation model is delivered to the gaze estimation terminal of the user. If the second gaze estimation model is trained in the cloud and based on a pre-processed calibration dataset collected by multiple users, the second gaze estimation model is delivered to the gaze estimation terminals of multiple users.

[0132] The acquisition module 02 acquires the second human eye data when the user looks at the target marker.

[0133] Specifically, in an embodiment of the present invention, when a user gazes at a target marker for a certain period of time, the acquisition module 02 records at least one frame of second eye data and corresponding preset marker position data. The second eye data is the eye image data captured by the camera.

[0134] The preprocessing module 04 preprocesses the second eye data to obtain corresponding second eye movement data.

[0135] 7 , the above steps specifically include: detecting pupil center point data or pupil outline point data from the human eye image data corresponding to the second human eye data to obtain corresponding second eye movement data.

[0136] The execution module 07 inputs the second eye movement data into the second sight line estimation model 09 for processing to obtain the second sight line data.

[0137] In addition, taking the first eye data collected from the user as an example, which is the relative position reading of the human eye obtained by a plurality of spatially sparsely distributed sensor arrays, as shown in Figure 2, the specific steps of a line of sight estimation method provided in an embodiment of the present application are described in detail below.

[0138] The first sight line estimation model 08 is constructed by the model construction module 01 .

[0139] In an embodiment of the present invention, the constructed first sight estimation model 08 is a deep learning model, and the backbone network of the first sight estimation model 08 is a graph convolutional neural network or a graph recurrent neural network, and a multi-layer perceptron is connected after the backbone network.

[0140] The collection module 02 collects first eye data when the user looks at each preset marker and corresponding preset marker position data as a first sampling example and generates a first sampling data set.

[0141] Specifically, in an embodiment of the present invention, a user sequentially gazes at several preset markers, each for a certain duration. Acquisition module 02 then records at least one frame of first eye data and the corresponding preset marker position data. Referring to FIG5 , which is a schematic diagram of a plurality of spatially sparsely distributed sensor arrays provided in an embodiment of the present invention, and FIG6 , the preset marker position data is the coordinates of each preset circular point on a plane. As shown in FIG6 , there are 20 circular markers in total, and the format of the first sampling example is {first eye data, preset marker position data}.

[0142] The detection module 03 performs data quality detection on the first sample data set.

[0143] 7 , the above steps specifically include: selecting cluster data points from each first human eye data in the first sampling data set; performing cluster analysis on each cluster data point, marking all cluster data points that are not classified into any cluster as noise points and counting the number of cluster clusters and the number of noise points; judging whether the number of cluster clusters is consistent with the number of preset markers and whether the number of noise points is less than a first preset threshold; if the number of cluster clusters is consistent with the number of preset markers and the number of noise points is less than the first preset threshold, the cluster analysis result of the first sampling data set is qualified, otherwise the cluster analysis result of the first sampling data set is unqualified, and the first sampling data set is resampled; for the first sampling data set with qualified cluster analysis results, the first sampling samples corresponding to the noise points are eliminated, and the remaining first sampling samples are used as second sampling samples to update and generate the second sampling data set; the second sampling samples are used as calibration samples, and the calibration data set is the second sampling data set.

[0144] Since the first eye data in this case is the relative eye position reading obtained by multiple spatially sparsely distributed sensor arrays, each piece of the first eye data can be considered a point in multidimensional space, called an eye data point. These eye data points are then used as clustering data points to obtain an eye data point set. A clustering algorithm is then used to cluster the eye data point set to determine the number of clusters and the number of noise points.

[0145] 8 , the above-mentioned cluster analysis of each cluster data point, that is, the specific steps of the clustering algorithm include: randomly selecting an unvisited cluster data point as the starting point, and taking the remaining cluster data points as test points; taking the starting point as the center, generating a corresponding preset neighborhood based on a second preset threshold; comparing the first distance between each test point and the starting point with the second preset threshold to determine whether the test point exists within the preset neighborhood of the starting point; if the test point exists within the preset neighborhood of the starting point, taking the test point as the core point, and determining whether there is an unvisited cluster data point; if there is no test point within the preset neighborhood of the starting point, directly determining whether there is an unvisited cluster data point; if there is an unvisited cluster data point, randomly selecting the next unvisited cluster data point as the starting point; if there is no unvisited cluster data point, including each starting point and all core points within the corresponding preset neighborhood into the same cluster.

[0146] As mentioned above, the method to determine whether a cluster data point is in the neighborhood of another cluster data point is: to determine based on the distance between the two cluster data points (Euclidean distance or Mahalanobis distance); for each starting point, if there are other cluster data points in the neighborhood of the starting point, then these cluster data points and the starting point are added to the same cluster; repeat the above steps until each cluster data point has been visited.

[0147] Factors that affect the quality of the calibration dataset include: During the acquisition of the first sample dataset, noise samples may be present in the first sample dataset, or samples corresponding to certain markers may be missing due to factors such as user inattention. To address these issues, the embodiments of the present invention employ the aforementioned cluster analysis of individual clustered data points to perform data quality testing, effectively overcoming the impact of these issues on model training.

[0148] Based on the data quality detection result, the detection module 03 obtains a calibration sample with qualified data quality and generates a calibration data set. As mentioned above, at this time, the calibration sample is the second sampling sample and the calibration data set is the second sampling data set.

[0149] Furthermore, a line of sight estimation method provided in an embodiment of the present invention further includes, via storage module 05, saving the pre-processed calibration dataset to the terminal for storage and management, or uploading the pre-processed calibration dataset to the cloud for storage and management. Specifically, the communication basis between the terminal and the cloud is Ethernet, and the communication protocol is TCP / IP.

[0150] Specifically, referring to Figure 12, the calibration dataset before preprocessing is stored in a two-level directory format, where the first-level directory is the user ID and the second-level directory is the storage time; the calibration dataset before preprocessing obtained by a single user at a single time is used as a calibration dataset storage unit; each calibration dataset storage unit includes: a dataset description file and a training sample set.

[0151] At this point, the dataset description file is used to describe the relevant information of the corresponding calibration dataset, including the dataset name, dataset size (number of samples), first eye data type (relative eye position readings obtained by multiple spatially sparsely distributed sensor arrays), creator, creation time; and sample set, that is, a collection of samples. Specifically, the sample naming rule is ID_EYE_POINT_NUM, where ID represents the user ID, EYE represents the left / right eye code, 0 represents the left eye, 1 represents the right eye, POINT represents the marker number, and NUM represents the sample sequence number. For example, 085_0_1_001, user ID is 085, left eye 0, marker number 1, sample sequence number 001, and the comprehensive understanding is: the left eye of user with ID 085, when looking at marker number 1, obtains sample number 001.

[0152] The preprocessing module 04 preprocesses each first eye data and the corresponding preset marker position data in the calibration data set to obtain corresponding first eye movement data and first sight line data as training samples, and generates a training data set.

[0153] The above steps specifically include: calculating the mean MEAN and variance STD for the relative position readings of the human eyes in each calibration sample; subtracting the mean MEAN from the relative position readings of the human eyes in each calibration sample and then dividing it by the variance STD to obtain the corresponding first eye movement data; using the upper left corner or the center marker in the horizontal or vertical direction as the reference point, the distance from the leftmost marker to the rightmost marker in the horizontal direction as the unit length in the horizontal direction, and the distance from the uppermost marker to the lowermost marker in the vertical direction as the unit length in the vertical direction, normalizing the coordinates of the preset marker position data in each calibration sample to obtain the corresponding first line of sight data.

[0154] Similarly, a line of sight estimation method provided in an embodiment of the present invention further includes, via storage module 05, saving the preprocessed training dataset to the terminal for storage and management, or uploading the preprocessed training dataset to the cloud for storage and management. Specifically, the communication basis between the terminal and the cloud is Ethernet, and the communication protocol is TCP / IP.

[0155] Specifically, referring to Figure 13, the training data set obtained after preprocessing is stored in a two-level directory format, where the first-level directory is the user ID and the second-level directory is the storage time; the training data set obtained by a single user at a single time is regarded as a training data set storage unit; each training data set storage unit includes: a data set description file and a training sample set.

[0156] The dataset description file describes the corresponding calibration dataset, including the dataset name, dataset size (number of samples), first eye movement data type (relative eye position readings obtained by a sparsely distributed array of sensors), first gaze data type (gaze point in 2D space / gaze origin and direction in 3D space), creator, and creation time. The sample set is a collection of samples. Specifically, the sample naming convention is ID_EYE_POINT_NUM, where ID represents the user ID, EYE represents the left / right eye designation (0 for the left eye and 1 for the right eye), POINT represents the landmark number, and NUM represents the sample sequence number. For example, 085_0_1_001 indicates user ID 085, left eye 0, landmark number 1, and sample sequence number 001. This means that the left eye of user ID 085, when looking at landmark 1, generates sample number 001.

[0157] The training module 06 trains the first sight line estimation model 08 using the training data set to obtain the second sight line estimation model 09 .

[0158] Specifically, on the terminal or the cloud, the first line of sight estimation model is trained based on a training data set obtained from a first sampling data set currently collected by the user to obtain a second line of sight estimation model, so as to achieve high-precision line of sight estimation for a specific user.

[0159] On the terminal or cloud, the first line of sight estimation model is trained based on a training data set obtained from the first sampling data set collected multiple times from the same user to obtain a second line of sight estimation model, so as to achieve high-precision line of sight estimation for a specific user.

[0160] On the terminal or in the cloud, the first sight line estimation model is trained based on a training data set obtained from the first sampling data set collected multiple times by multiple users to obtain a second sight line estimation model, so as to achieve high-precision sight line estimation for general users.

[0161] The specific steps of training the first line of sight estimation model 08 using the training data set include: using the first line of sight estimation model to infer the predicted line of sight data based on the first eye movement data in the training data set; determining the parameters of the loss function based on the predicted line of sight data and the first eye movement data; the loss function can be L1Loss or L2Loss, etc.; back propagating based on the parameters of the loss function to adjust the model parameters. If the model is trained based on a calibration data set collected and preprocessed by the same or multiple users, the adjusted parameters are all the parameters of the model. If the training is based on the user's current collected and preprocessed data set, the adjusted parameters tend to select local parameters to shorten the training time. The local parameters can be the parameters of a multi-layer perceptron. Among them, the learning rate related to the adjustment of the model parameters is dynamically adjusted using the Cosine Annealing method. As the number of training rounds epochs increases, the learning rate lr decreases in a cosine form. The formula is:

[0162] Among them, lr_max represents the initial learning rate, lr_min represents the minimum learning rate, epoch represents the current epoch number, and epochs represents the total epoch number.

[0163] Based on the preprocessed training dataset, the above steps are repeated repeatedly, and training ends when preset conditions are met. Preset conditions include but are not limited to: reaching the maximum number of iterations or training rounds, or when the loss is within an acceptable range, or when the accuracy meets the requirement, or when artificial early stopping is performed.

[0164] If the second gaze estimation model is trained in the cloud and based on a pre-processed training dataset collected by the same user, the second gaze estimation model is delivered to the gaze estimation terminal of the user. If the second gaze estimation model is trained in the cloud and based on a pre-processed calibration dataset collected by multiple users, the second gaze estimation model is delivered to the gaze estimation terminals of multiple users.

[0165] The acquisition module 02 acquires the second human eye data when the user looks at the target marker.

[0166] Specifically, in this embodiment of the present invention, when a user gazes at a target marker for a certain period of time, the acquisition module 02 records at least one frame of second eye data and the corresponding preset marker position data. The second eye data is the relative eye position reading obtained by a plurality of spatially sparsely distributed sensor arrays.

[0167] The preprocessing module 04 preprocesses the second eye data to obtain corresponding second eye movement data.

[0168] 7 , the above steps specifically include: calculating the mean MEAN and the variance STD for the relative eye position readings corresponding to the second eye data; subtracting the mean MEAN from the relative eye position readings in each calibration sample and then dividing them by the variance STD to obtain the corresponding second eye movement data.

[0169] The execution module 07 inputs the second eye movement data into the second sight line estimation model 09 for processing to obtain the second sight line data.

[0170] Compared with the existing technology, the gaze estimation method provided in the embodiments of the present application collects first eye data and corresponding preset marker position data when the user looks at each preset marker to obtain a first sampling data set; performs data quality testing on the first sampling data set to obtain a calibration data set with qualified data quality; preprocesses the calibration samples in the calibration data set to obtain corresponding first eye movement data and first gaze data as training samples, and generates a training data set; and uses the training data set to train the constructed first gaze estimation model to obtain a second gaze estimation model. This method achieves high-precision gaze estimation of the user and greatly reduces the accuracy error of gaze estimation.

[0171] In addition, an embodiment of the present invention also provides a line of sight estimation device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a line of sight estimation method as described in any one of the above items.

[0172] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a line of sight estimation method as described in any one of the above items are implemented.

[0173] In the embodiments of the present invention, the processor may be an integrated circuit chip with signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0174] The methods, steps, and logic diagrams disclosed in the embodiments of the present invention can be implemented or executed. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software modules can be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The processor reads the information from the storage medium and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0175] The storage medium may be a memory and may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0176] Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory.

[0177] Volatile memory can be random access memory (RAM), which is used as an external cache memory. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0178] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0179] Those skilled in the art will appreciate that in one or more of the above examples, the functions described herein can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media includes any medium that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0180] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A line of sight estimation method, characterized in that: The method comprises: Constructing a first sight estimation model; Collecting first human eye data when the user gazes at each preset marker and corresponding preset marker position data as a first sampling sample and generating a first sampling data set; Performing data quality detection on the first sampling data set; Based on the data quality test results, calibration samples with qualified data quality are obtained and a calibration data set is generated; Preprocessing each first human eye data and corresponding preset marker position data in the calibration data set to obtain corresponding first eye movement data and first sight line data as training samples, and generating a training data set; Using the training data set to train the first sight line estimation model to obtain a second sight line estimation model; The step of performing data quality detection on the first sample data set includes: Selecting clustered data points from each first human eye data in the first sampling data set; Perform cluster analysis on each cluster data point, mark all cluster data points that are not classified into any cluster as noise points, and count the number of clusters and noise points; Determining whether the number of clusters is consistent with the number of preset markers and whether the number of noise points is less than a first preset threshold; If the number of clusters is consistent with the number of preset markers, and the number of noise points is less than a first preset threshold, the cluster analysis result of the first sampling data set is qualified; otherwise, the cluster analysis result of the first sampling data set is unqualified, and the first sampling data set is resampled; For the first sampling data set with qualified cluster analysis results, first sampling examples corresponding to noise points are removed, and the remaining first sampling examples are used as second sampling examples to update and generate a second sampling data set; The second sampling example is used as a calibration example, and the calibration data set is the second sampling data set.

2. A line of sight estimation method according to claim 1, characterized in that: The method further comprises: Collecting second eye data when the user looks at the target marker; Preprocessing the second human eye data to obtain corresponding second eye movement data; The second eye movement data is input into the second sight line estimation model for processing to obtain second sight line data.

3. A line of sight estimation method as claimed in claim 2, characterized in that: The method further comprises: Saving the training data set to a terminal for storage and management, or uploading the training data set to a cloud for storage and management; The training data set is stored in a two-level directory, where the first-level directory is the user ID and the second-level directory is the storage time; A training data set obtained once by a single user is used as a training data set storage unit; Each training data set storage unit includes: a data set description file and a training sample set.

4. A line of sight estimation method as claimed in claim 3, characterized in that: Perform cluster analysis on each cluster data point, including: Randomly select a cluster data point that has not been visited as the starting point, and use the remaining cluster data points as test points; Taking the starting point as the center, generating a corresponding preset neighborhood based on a second preset threshold; Compare the first distance between each test point and the starting point with the second preset threshold value to determine whether there is a test point within the preset neighborhood of the starting point; If there is a point to be tested within the preset neighborhood of the starting point, the point to be tested is used as a core point, and it is determined whether there is a cluster data point that has not been visited; If there is no point to be tested within the preset neighborhood of the starting point, directly determine whether there is a cluster data point that has not been visited; If there is a cluster data point that has not been visited, randomly select the next cluster data point that has not been visited as the starting point; If there are no unvisited cluster data points, each starting point and all core points in the corresponding preset neighborhood are included in the same cluster.

5. A line of sight estimation method as claimed in claim 4, characterized in that: The first human eye data is a relative position reading of human eyes obtained by a plurality of spatially sparsely distributed sensor arrays, and each first human eye data and corresponding preset marker position data in the calibration data set are preprocessed, including: For the relative position readings of the human eyes in each calibration sample, calculate the mean MEAN and variance STD; The relative eye position readings in each calibration sample are first subtracted from the mean MEAN and then divided by the variance STD to obtain the corresponding first eye movement data; Taking the upper left corner or the center marker in the horizontal or vertical direction as the reference point, the distance between the leftmost marker to the rightmost marker in the horizontal direction as the unit length in the horizontal direction, and the distance between the uppermost marker to the lowermost marker in the vertical direction as the unit length in the vertical direction, the coordinates of the preset marker position data in each calibration sample are normalized to obtain the corresponding first line of sight data.

6. A line of sight estimation method as claimed in claim 4, characterized in that: Performing data quality detection on the first sample data set also includes: Numbering each cluster in the second sampling data set to obtain a corresponding numbered cluster; All clustered data points corresponding to a preset number of numbered clusters are selected to form a first cluster point set, and all clustered data points corresponding to the remaining numbered clusters are selected to form a second cluster point set; Using the first cluster point set and the second cluster point set to perform fitting, respectively obtaining a first fitting ellipse and a second fitting ellipse; Calculating the shortest distance between each cluster data point in the first cluster point set and the contour point of the first fitted ellipse as a second distance, and calculating the shortest distance between each cluster data point in the second cluster point set and the contour point of the second fitted ellipse as a third distance; Calculate the first square sum S1 of all second distances and the second square sum S2 of all third distances; Calculating a first fitting error E1 using the first square sum S1, and calculating a second fitting error E2 using the second square sum S2; Determine whether the first fitting error E1 and the second fitting error E2 are both smaller than a first preset error and a second preset error respectively; If the first fitting error E1 is less than the first preset error, and the second fitting error E1 is less than the second preset error, the ellipse fitting error analysis result of the second sampling data set is qualified; otherwise, the ellipse fitting error analysis result of the second sampling data set is unqualified, and the first sampling data set is resampled; Determine the second sampling sample in the second sampling data set with qualified ellipse fitting error analysis results as a third sampling sample, and generate a third sampling data set; The third sampling example is used as a calibration example, and the calibration data set is the third sampling data set.

7. A line of sight estimation method according to claim 4 or 6, characterized in that: The first human eye data is human eye image data captured by a camera, and each first human eye data and corresponding preset marker position data in the calibration data set are preprocessed, including: Detecting pupil center point data or pupil contour point data from the human eye image data in each calibration sample to obtain corresponding first eye movement data; Taking the central marker as a reference point and the distance from the outermost marker to the central marker as a unit length, coordinate normalization is performed on the preset marker position data in each calibration sample to obtain the corresponding first line of sight data.

8. A line of sight estimation system, characterized in that: The system comprises: A model building module, used for building a first sight estimation model; A collection module, used to collect first human eye data when the user looks at each preset marker and corresponding preset marker position data as a first sampling sample and generate a first sampling data set; A detection module, configured to perform a data quality detection on the first sampling data set; based on the data quality detection result, obtain a calibration sample with qualified data quality and generate a calibration data set; A preprocessing module, used to preprocess each first human eye data and the corresponding preset marker position data in the calibration data set, obtain the corresponding first eye movement data and first sight line data as training samples, and generate a training data set; A training module, used to train the first sight line estimation model using the training data set to obtain a second sight line estimation model; The step of performing data quality detection on the first sample data set includes: Selecting clustered data points from each first human eye data in the first sampling data set; Perform cluster analysis on each cluster data point, mark all cluster data points that are not classified into any cluster as noise points, and count the number of clusters and noise points; Determining whether the number of clusters is consistent with the number of preset markers and whether the number of noise points is less than a first preset threshold; If the number of clusters is consistent with the number of preset markers, and the number of noise points is less than a first preset threshold, the cluster analysis result of the first sampling data set is qualified; otherwise, the cluster analysis result of the first sampling data set is unqualified, and the first sampling data set is resampled; For the first sampling data set with qualified cluster analysis results, first sampling examples corresponding to noise points are removed, and the remaining first sampling examples are used as second sampling examples to update and generate a second sampling data set; The second sampling example is used as a calibration example, and the calibration data set is the second sampling data set.

9. A sight line estimation device, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a line of sight estimation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a line of sight estimation method according to any one of claims 1 to 7 are implemented.

Images