A cross-individual gaze estimation method based on individual differences

Abstract

The present invention discloses a cross-person gaze estimation model based on individual differences, comprising the following steps. First, the facial images in the gaze estimation dataset are preprocessed; secondly, the three-dimensional gaze vector d is obtained by the neural network model CBSP‑Net, and the result is converted into a two-dimensional gaze point (PoG) on the screen according to the camera position when the facial image is taken. Subsequently, the SP module is used to calculate the predicted mean and historical prediction data of all available frame samples of the participant, and correct the prediction deviation; finally, the PST module is used to input the heat map after the participant's data conversion and the heat map converted from the historical data, eliminate the error of the affine transformation, and obtain more accurate heat map data. Finally, the converted two-dimensional prediction result is obtained by SoftMax transformation. The present invention effectively solves the problem of invalid sample data, and at the same time, effectively combines the historical gaze data of the participants, and obtains significant results in the two-dimensional gaze estimation task.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-individual gaze estimation, and in particular to a cross-individual gaze estimation method based on individual differences. Background Art

[0002] Gaze estimation is a very active and challenging research area in computer vision, and has made significant progress in recent years. Gaze estimation requires extracting and fusing facial features from an image to obtain an accurate 2D gaze point. It is highly relevant to many human-computer interaction applications and plays a crucial role in virtual reality interactions, such as estimating user attention distribution and virtual reality eye movement interaction systems. The task of cross-person gaze estimation is defined as evaluating a model on a group of previously unseen participants. Existing gaze estimation techniques do not capture long-range dependencies between eyes, despite significant improvements in gaze estimation accuracy.

[0003] Kappa (the deviation between the optical axis and the visual axis) cannot be detected in images captured using conventional methods. The standard deviation of Kappa for the horizontal and vertical axes in the normal population is approximately 1.8°, resulting in a random error of 2.0° to 2.3° when estimating gaze direction directly from images.

[0004] Existing single-image gaze estimation methods are unable to recover these individual differences, and in theory, their accuracy will reach 2.0°. Many existing methods, such as those based on single images, video sequences, and content information, can predict the gaze position and direction, but the problem of unobservable person-specific differences still exists. In view of these challenges, scholars have proposed many individual adaptation techniques, including using deep learning neural networks to predict differences between people, directly estimating the 6-degree-of-freedom calibration parameters of the human eye, or fine-tuning the model with a small number of labeled test samples, which have achieved good results. However, these methods still fail to solve the Kappa problem well. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention proposes a cross-individual gaze estimation method based on individual differences, which aims to solve the problem of invalid sample data and the long-distance dependency between eyes, and model individual differences in a cross-person setting. This method is specially designed for processing facial images. Three functional modules are designed: DP (data filtering module), SP (auto-calibration module), and PST (prediction transformation module). Among them, the DP module is used to identify invalid samples. The SP and PST modules are used to learn and compensate for individual differences on valid samples, and further improve the accuracy of the prediction results through transformation. This method shows significant performance improvement in the two-dimensional line of sight estimation task.

[0006] The present invention provides a method for estimating inter-individual gaze based on individual differences, comprising the following steps:

[0007] Step (1) selects a gaze estimation dataset and inputs facial images in the dataset into a verification module to remove sample data with closed eyes or partially visible pupils. The DP model is used to denoise the sample data in order to maintain stable detection results under different lighting conditions, facial postures, and image quality.

[0008] Step (2) inputs the denoised samples into the neural network model CBSP-Net to obtain the participant's three-dimensional gaze vector d, calculates the two-dimensional gaze point of the line of sight based on the position transformation of the camera when the sample is collected in the dataset, and obtains the initial prediction point;

[0009] Step (3): Use the SP module to calculate the difference between the predicted mean of all available frame samples of the participant and the historical predicted data, and perform correction learning for the prediction deviation, mainly targeting the Kappa value of the line of sight error offset unique to different individuals, so as to obtain the accurate PoG value;

[0010] Step (4) converts the refined PoG value into a heat map. If historical data exists for the participant, the historical data is also converted into a heat map. The two data are combined and input into the PST module to eliminate the error of the affine transformation and obtain the final predicted heat map data. The final heat map data is then subjected to a SoftMax transformation to obtain the final predicted two-dimensional gaze estimation result.

[0011] Preferably, the DP model denoising method in step 1 is as follows:

[0012] Because samples such as closed eyes or blinking are noisy, and different lighting conditions, facial poses, and image quality can affect model training and prediction, noise samples must be removed. The proposed DP model combines facial feature point detection with iris detection. The model's workflow begins with successful localization of the face region. It then uses 68 facial landmark detection to precisely locate key feature points around the eyes, which serve as a baseline for further ocular structural analysis. Based on these feature points, two independent ocular regions of interest (ROIs) are defined and converted into grayscale images. Next, contrast enhancement and noise reduction techniques are applied to highlight ocular features, particularly the subtle structures of the iris and pupil. Leveraging the specific properties of the pupil and iris, a Hough circle transform is applied to accurately capture iris boundary information. The information gathered from feature point detection is combined with detailed iris features extracted using the Hough transform to calculate the aspect ratio of the eye. Through this series of detailed analyses, combined with the obtained aspect ratio indicators and the specific shape of the iris, the system can efficiently and accurately distinguish whether the eyes are open or closed.

[0013] As a preference, step 2 is specifically implemented as follows:

[0014] After filtering and removing invalid samples in step 1, the filtered images are input into the model, first converted to the feature vector space, and then feature extraction is performed. Because the existing gaze estimation technology does not capture the long-distance dependency between eyes, nor does it combine the spatial attention mechanism well. The proposed neural network model CBSP-Net is based on the L2CS-Net model, adding spatial attention and strip pooling technology to effectively condense information and extract the two components of the three-dimensional gaze vector (yaw angle and pitch angle). The combination of yaw angle and pitch angle can cover all possible gaze directions, thereby providing a comprehensive description of the head or eye posture. The two angles correspond to the natural movement of the head and eyes in daily life. Therefore, combining the two parts can represent the eye and head posture. Combined with the position of the camera during data acquisition, the direction of the camera coordinate system is obtained. The three-dimensional vector is combined with the camera coordinate system and projected onto the two-dimensional screen plane. The intersection of the three-dimensional vector and the two-dimensional plane is the two-dimensional point of gaze (PoG);

[0015] The added spatial attention module is the CBAM module, which is a lightweight convolutional attention module that combines the channel and spatial attention mechanism modules. CBAM contains two sub-modules, CAM and SAM, which perform channel and spatial attention respectively.

[0016] The StripPooling module adds a strip-shaped pooling technique that applies a long strip-shaped pooling kernel in one spatial dimension, which can capture long-range information, especially long-range dependencies. Meanwhile, it maintains a relatively narrow size in the other dimension to avoid capturing irrelevant information.

[0017] As a preference, the specific implementation process of step 3 is as follows:

[0018] Due to the differences in physiological structure between different individuals, there are deviations in the prediction of the gaze point. The SP module mainly learns to compensate for the prediction deviation, models the offset to bridge the gap between the visual axis and the optical axis, and is used to eliminate the overall impact. The distribution of the data set is used to simulate the offset specific to the participant. The prediction mean of all available frame samples of the participant and the calculation of historical prediction data are regarded as errors. The prediction mean of all available frame samples g tr The calculation formula is:

[0019]

[0020] Global average gaze point g tr represents the mean of all sample gaze results, p i Represents the coordinates of the gaze point predicted by each valid sample, and N represents the number of valid samples after screening by the DP module.

[0021] As a preferred method, the prediction formula of the SP module is as follows:

[0022]

[0023] in the formula represents the correction value of the predicted gaze point of the jth participant at the tth time point after calibration by the SP module, p t,j is the predicted value of the initial fixation point, h t,j Represents a historical index set. For online settings (continuous frame input), it contains all historical prediction indexes up to the current time point (t-1); for offline settings, it contains all historical and future frames except the current frame. i,j represents the predicted probability of the i-th sample at position j in the historical data, g t,r Represents the predicted mean of all available frame samples, ω i Represents the weight coefficient of historical forecast data.

[0024] As a preference, where ω i The calculation method is as follows:

[0025]

[0026] In the formula, distance(p i,j,p t,j ) represents the historical prediction p i,j With the current prediction p t,j The distance between them is measured using Euclidean distance. ∈ is a very small integer to prevent the historical prediction from being exactly the same as the current prediction, causing the denominator to become zero and the weight to be infinite.

[0027] Preferably, the PST module in step 4 can model and compensate for the differences in user vision more generally, as follows:

[0028] The PST module predicts the output of the SP module Heatmap of the predictions of the historical refined PoG of the same participant The heat map of the two is used as input, and the heat map of the conversion is output. Finally, it is converted into probability through the SoftMax function. The point with the maximum probability is the point of the improved prediction.

[0029] The present invention has the following characteristics and beneficial effects:

[0030] Using the above technical solution, the present invention first uses the data screening module DP to remove noise samples. It is then input into CBSP-Net to extract features, and the model estimates a three-dimensional gaze vector, which is then converted into a two-dimensional gaze point. Then, by comparing the historical predicted sight line data of the test user and the training Gaze data, the difference in the mean of the two is calculated as the value of the Kappa angle of the user's individual eye difference to correct the user's prediction and eliminate basic differences. Finally, the PST module learns the affine transformation between the user's prediction and the training Gaze, and combines the historical evaluation data of the test user and the previously personalized corrected estimation data to obtain the final improved result. The present invention can model based on the historical data of the predictions of different participants and effectively improve the prediction accuracy. The algorithm proposed in the present invention is independent of the specific sight line estimation model. It is a simple affine transformation that can be applied to any sight line estimation method to effectively improve the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 This is a flow chart of a cross-individual gaze estimation method based on individual differences according to the present invention.

[0033] Figure 2It is a schematic flow diagram of the modules of the method of the present invention.

[0034] Figure 3 Schematic diagram of the CBSP-Net model in the method of the present invention.

[0035] Figure 4 Schematic diagram of the strip pooling module in the CBSP-Net model in the method of the present invention.

[0036] Figure 5 Schematic diagram of the neural network model of the PST module in the method of the present invention.

[0037] Figure 6 3 is a comparison chart of PoG results before and after the method of the present invention improves prediction. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0040] On the contrary, the present invention covers any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention as defined by the claims. Furthermore, to facilitate a better understanding of the present invention, certain specific details are described in detail below in the detailed description of the present invention. Those skilled in the art will be able to fully understand the present invention without these details.

[0041] The present invention provides a method for cross-individual gaze estimation based on individual differences, such as Figure 1 As shown, the model is able to perceive differences between individuals. The processing flow is as follows: First, the input facial image is processed by the DP module to filter out noisy samples such as closed eyes or blinks. Then, the neural network model CBSP-Net is used to extract features. After model processing, a three-dimensional gaze vector is obtained. Combined with the camera parameters, this vector is converted and mapped into a two-dimensional point of gaze (PoG) on the screen. The SP module then calculates the predicted mean PoG for all available frame samples of the participant and compares it with historical prediction data. Corrective learning is performed to obtain a refined PoG value. Finally, the refined PoG value is converted into a heat map and input into the PST module to eliminate errors caused by the affine transformation, resulting in the final predicted heat map data. The resulting heat map data is then subjected to a SoftMax transform to obtain the final predicted two-dimensional gaze estimation result.

[0042] The specific steps are as follows:

[0043] Step (1) Since samples such as closed eyes or blinking are noise samples, and different lighting conditions, different facial postures, and different image quality will affect the training and prediction of the model, the noise samples must be removed. The proposed DP model combines facial feature point detection and iris detection. The model integrates facial feature point detection and iris detection technology. The operation process begins with the successful positioning of the face area, and then uses the 68-point facial landmark detection method to accurately locate the key feature points around the eyes as a baseline reference for further analysis of the eye structure. On this basis, two independent eye regions of interest (ROIs) are defined based on these feature points and converted into grayscale images respectively. Next, by applying contrast enhancement and noise reduction techniques, the eye features, especially the subtle structures of the iris and pupil, are highlighted. Using the specific properties of the pupil and iris, the Hough circle transform is implemented to accurately obtain the iris boundary information. The information summarized from the feature point detection is combined with the detailed iris features extracted by the Hough transform to calculate the aspect ratio of the human eye. Through this series of detailed analyses, combined with the obtained aspect ratio indicators and the specific shape of the iris, the system can efficiently and accurately distinguish whether the eyes are open or closed.

[0044] Step (2), after filtering out invalid samples, the filtered images are input into the model, first converted to vector space, and then basic features are extracted from the input images through the basic convolution extraction module in the neural network model CBSP-Net. Since the existing line of sight estimation technology does not capture the long-distance dependency between eyes, nor does it combine the spatial attention mechanism well. The proposed CBSP-Net is based on the L2CS-Net model, adds spatial attention and strip pooling technology, effectively condenses information, and extracts the two components of the three-dimensional line of sight vector (yaw angle and pitch angle). The combination of the two components can represent the eye and head posture. Combined with the position of the camera during data acquisition, the Zhang Zhengyou calibration method is used to automatically detect the corner points in the picture by taking multiple pictures of a calibration plate with a known geometric pattern. The image coordinates of the corner points are corresponded to the actual world coordinates, and a linear equation system is constructed and solved to calculate the intrinsic parameters of the camera, such as focal length and principal point coordinates. Thus, the direction of the camera coordinate system is obtained. The three-dimensional vector is projected onto the two-dimensional screen plane in combination with the camera coordinate system. The intersection of the three-dimensional vector and the two-dimensional plane is the two-dimensional point of gaze (PoG);

[0045] Specifically, the added spatial attention module is the CBAM module, which is a lightweight convolutional attention module that combines the channel and spatial attention mechanism modules. CBAM contains two sub-modules, CAM and SAM, which perform channel and spatial attention respectively.

[0046] The StripPooling module adds a strip-shaped pooling technique that applies a long strip-shaped pooling kernel in one spatial dimension, which can capture long-range information, especially long-range dependencies. Meanwhile, it maintains a relatively narrow size in the other dimension to avoid capturing irrelevant information.

[0047] Step (3): Use the SP module to calculate the difference between the predicted mean of all available frame samples of the participant and the historical predicted data, and perform correction learning for the prediction deviation, mainly targeting the Kappa value of the line of sight error offset unique to different individuals, so as to obtain the accurate PoG value.

[0048] Specifically, if the participant has historical gaze estimation data, combining the historical estimation data can improve the result of this gaze. The SP module is used to model and compensate for the difference between the individual's specific visual axis and optical axis, and the prediction mean of all available frame samples of the participant and the calculation of historical prediction data are regarded as errors. Then the prediction mean of all available frame samples g tr The calculation formula is:

[0049]

[0050] Global average gaze point g tr represents the mean of all sample gaze results, p i Represents the coordinates of the gaze point predicted by each valid sample, and N represents the number of valid samples after screening by the DP module.

[0051] The prediction formula of the SP module is as follows:

[0052]

[0053] in the formula represents the correction value of the predicted gaze point of the jth participant at the tth time point after calibration by the SP module, p t,j is the predicted value of the initial fixation point, h t,j Represents a historical index set, which contains all historical prediction indexes up to the current time point t-1 for online continuous frame input; for offline settings, it contains all historical and future frames except the current frame. i,j represents the predicted probability of the i-th sample at position j in the historical data, g t,r Represents the predicted mean of all available frame samples, ω i Represents the weight coefficient of historical forecast data.

[0054] Among them, ω i The calculation method is as follows:

[0055]

[0056] In the formula, distance(p i,j ,p t,j ) represents the historical prediction p i,j With the current prediction p t,j The distance between them is measured using Euclidean distance. ∈ is a very small integer to prevent the historical prediction from being exactly the same as the current prediction, causing the denominator to become zero and the weight to be infinite.

[0057] Understandably, differences in gaze point predictions between individuals due to physiological differences can lead to deviations. The SP module primarily learns to compensate for these deviations, modeling the offset to bridge the gap between the visual axis and the optical axis to eliminate the overall effect. It leverages the distribution of the dataset to simulate participant-specific offsets. The error is calculated by combining the mean prediction of all available frame samples for that participant with the historical prediction data.

[0058] Step (4) The prediction output of the SP module is converted into Heatmap of the predictions of the historical refined PoG of the same participant The heat map of the two is taken as input, and the converted heat map is output. Finally, it is converted into probability through the SoftMax function. The point with the maximum probability is the point of the improved prediction.

[0059] The process of generating the heat map first converts the calibrated prediction points Mapped to pixel space, a Dirac delta function is placed at the center of the point, convolved with a deep neural network in the PST module, and an isotropic two-dimensional Gaussian function with fixed variance is applied to form a smooth heat map. The heat map can represent the predicted point and also capture the changes in the surrounding area through the Gaussian distribution.

[0060] The pre-processing of the SP module is crucial. It transforms the initially predicted Point of View (PoG) to a reasonable position, ensuring that the PST module is not disturbed by invalid predictions. The SP module considers and compensates for large translation offsets, making the initial PoG prediction more accurate. Subsequently, the PST module performs more refined processing to compensate for all affine-like transformations. By randomly selecting affine transformations to generate samples for training, the PST module aims to eliminate individual-specific affine-like differences in gaze estimation tasks.

[0061] Among them, the algorithm for generating the historical heat map is as follows:

[0062]

[0063] in, Represents the heat map of the current sample after calibration by the SC module, which is the output of the current step of the model.t,j Represents a heatmap processed with data augmentation (including affine transformation and random noise) to simulate individual-specific differences and possible noise.

[0064] It should be noted that the PST module is a feature processing module with a built-in module called SimAM, which is responsible for extracting attention features. The SimAM module assigns a unique weight to each neuron and calculates the similarity between each pixel and its neighboring pixels in the feature map to better capture global attention features.

[0065] The PST model consists of the following parts: a positioning network for preliminarily determining the location of the gaze point. A SimAM module for extracting attention features. A grid generator that may be used to create a grid structure for transformation. A downsampler for reducing the spatial dimension of the data and increasing computational efficiency. Its network architecture is shown in Table 1. The network architecture of the PST module consists of a basic layer consisting of a basic convolutional layer, a pooling layer, and an activation function. These layers are stacked and finally converted through a Linear layer to obtain the output result. The dimension of the input tensor is Represents the heat map of a single sample, the dimension of the output tensor is Including transformed sample heatmap And the transformed historical heat map data The dimensions of both are R H×W×1 .

[0066] Mathematically, the PST module can be represented as f, and its input-output relationship can be represented as s=f(s t,j ). The loss function of the PST module is designed as a binary cross entropy loss function, and the formula is:

[0067]

[0068] in, Represents the heat map of the current sample after calibration by the SP module, which is the output of the current step of the model. t,j Represents a heatmap processed with data augmentation (including affine transformation and random noise) to simulate individual-specific differences and possible noise.

[0069] To obtain the final Point of Gaze (PoG) for the input image, we need to apply the Softmax function to the output of the SP module. The Softmax function outputs a probability distribution, from which we can find the location corresponding to the maximum probability. This location is then remapped (scaled) to a coordinate system corresponding to the screen size to obtain the final PoG result.

[0070] Table 1. Network architecture parameters of the PST module

[0071]

[0072] In order to verify the performance of the present invention in data post-processing, we tested the method of the present invention on the EVE dataset. The experimental results are shown in Table 2, which includes two key indicators for measuring detection performance: Gaze Dir (°) and PoG (cm / px). Gaze Dir represents the error angle of the three-dimensional gaze direction prediction, while PoG represents the prediction error of the two-dimensional gaze direction, which is defined as the Euclidean distance between the predicted PoG position and the true PoG on the screen plane, measured in centimeters and pixels respectively. It can be seen from the data in Table 2 that the gaze estimation model constructed using the method of the present invention significantly improves the prediction effect.

[0073] To verify the performance of our method for gaze estimation, we tested it on the EVE dataset. Table 2 shows the experimental results, which include two key metrics: Gaze Dir (°) and Point of Gaze (PoG) (cm / px). Gaze Dir represents the error angle of the three-dimensional gaze direction prediction, and PoG represents the two-dimensional gaze direction prediction error, measured in centimeters and pixels. The data in Table 2 shows that the gaze estimation model constructed using our method significantly improves prediction results.

[0074] Table 2. Comparison of the performance of different models on the validation set of the EVE dataset (the best results are marked in bold)

[0075]

[0076] To further test the robustness of our model, we conducted tests on two additional datasets, MPIIGaze and ETH-XGaze. Table 3 shows the test results, with the test annotation being the gaze error (Gaze Dir) of 3D gaze estimation. We also compared our method with other classic gaze estimation methods. The data in Table 3 demonstrates that our method not only effectively models individual differences but also exhibits excellent versatility, achieving excellent performance.

[0077] Table 3. Performance of the model on the MPIIGaze and ETH-XGaze datasets (the best results are marked in bold)

[0078] MPII test set ETH-Gaze ​​test set method GazeDir(°) GazeDir(°) ETH-Gaze 4.8 4.5 Rt-gene 4.8 - Few-shot GazeEstimation 5.2 - EVE-SCPT 4.14 2.88 Method of the present invention 4.01 2.76

[0079] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for cross-individual gaze estimation based on individual differences, characterized in that: The following steps are involved: Step 1: Select the sight estimation dataset and preprocess it; Step 2: Input the denoised samples into the neural network model CBSP-Net to obtain the participant's 3D gaze vector d. Calculate the 2D gaze point of the line of sight based on the position transformation of the camera when the sample was collected in the dataset to obtain the initial prediction point. Step 3: Calculate the difference between the predicted mean of all available frame samples of the participant and the historical predicted data, and use the SP module to correct the prediction deviation to obtain the accurate PoG value; The correction prediction method of the SP module is as follows: (Formula 1) in the formula represents the correction value of the predicted gaze point of the jth participant at the tth time point after calibration by the SP module, is the predicted value of the initial fixation point, Represents a historical index set. For online settings with continuous frame input, it contains all historical prediction indexes up to the current time point t - 1; for offline settings, it contains all historical and future frames except the current frame. Represents the predicted probability of the i-th sample at position j in the historical data, represents the predicted mean of all available frame samples, Represents the weight coefficient of historical forecast data; The weight coefficient is calculated as follows: (Formula 2) in the formula Represents historical forecasts With current forecast The distance between them is measured using Euclidean distance. Is an integer that prevents the historical forecast from being exactly the same as the current forecast, causing the denominator to become zero, resulting in infinite weight; Step 4: Convert the refined PoG value into a heat map and input it into the PST module to eliminate the error of the affine transformation and obtain the final predicted heat map data. Then, the final heat map data is subjected to SoftMax transformation to obtain the final predicted two-dimensional gaze estimation prediction result.

2. The method for cross-individual gaze estimation based on individual differences according to claim 1, characterized in that: The preprocessing method is as follows: inputting facial images in the dataset into the verification module, removing sample data with closed eyes or partially visible pupils, and then using the DP model to denoise the sample data.

3. The method for cross-individual gaze estimation based on individual differences according to claim 1, characterized in that The neural network model CBSP-Net is based on L2CS-Net. In L2CS-Net, the spatial attention mechanism and the strip pooling attention mechanism are added to the main network part of ResNet-50.

4. The method for cross-individual gaze estimation based on individual differences according to claim 3, characterized in that: In step 2, after basic features of the input image are extracted by the basic convolution extraction module in the neural network model CBSP-Net, the channel and spatial attention mechanism is applied to the eye image, and the extracted basic features are compressed in combination with the strip pooling attention mechanism. The output of the model is the yaw angle and pitch angle in the three-dimensional line of sight vector, and the yaw angle and pitch angle respectively represent the state and position of the eye and head posture.

5. A method for estimating inter-individual gaze based on individual differences according to claim 4, characterized in that: The yaw angle and pitch angle are combined with the position of the camera during data collection, and the direction of the camera coordinate system is obtained using the Zhang Zhengyou calibration method. Then the three-dimensional line of sight vector is projected onto a two-dimensional plane in combination with the camera coordinate system. The intersection of the three-dimensional line of sight vector and the two-dimensional plane is the two-dimensional gaze point PoG.

6. The method for estimating inter-individual gaze based on individual differences according to claim 2, characterized in that: In step 3, the predicted mean of all available frame samples is The calculation formula is: (Formula 3) Global average fixation represents the mean of all sample gaze results this time, Represents the coordinates of the gaze point predicted by each valid sample, and N represents the number of valid samples after screening by the DP module.

7. The method for cross-individual gaze estimation based on individual differences according to claim 1, characterized in that: In step 4, if there is historical data for the participant, the historical data is also converted into a heat map, the two data are combined, and then input into the PST module.

8. The method for estimating inter-individual gaze based on individual differences according to claim 1, characterized in that: In step 4, the method for generating the heat map is: first, the calibrated prediction points Mapped to pixel space, a Dirac delta function is then placed at the center of the point, convolved with a deep neural network in the PST module, and an isotropic 2D Gaussian function with fixed variance is applied to form a smooth heat map.

9. The method for cross-individual gaze estimation based on individual differences according to claim 1, characterized in that: The loss function of the PST module is designed as a binary cross entropy loss function, and the formula is (Formula 4) in, Represents the heat map of the current sample after calibration by the SP module, Represents a heatmap after data augmentation to simulate individual-specific differences and possible noise.

Images