Individual difference-based cross-individual fixation estimation method
By introducing data filtering, automatic calibration and prediction transformation modules into the line of sight estimation technology, combined with the spatial attention mechanism, the problem of large gaze point prediction error in the existing technology is solved, and higher accuracy individual difference processing and gaze point estimation are achieved .
Patent Information
- Application Number
- CN202411146510.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-20
AI Technical Summary
Existing line of sight estimation technology cannot effectively capture the long-distance dependence between eyes, resulting in large random errors in gaze point prediction, especially in the case of large individual differences, the Kappa angle cannot be accurately restored, affecting virtual reality and people. Accuracy of computer interaction applications.
A cross-individual gaze estimation method based on individual differences is designed, including a data filtering module (DP), an automatic calibration module (SP) and a prediction transformation module (PST), which combines spatial attention through denoising, feature extraction and deviation correction. Mechanism and affine transformation improve the accuracy of prediction results.
It significantly improves the accuracy of two-dimensional line of sight estimation, reduces prediction errors, can effectively handle line of sight differences between different individuals, and improves the accuracy of virtual reality and human-computer interaction applications.
Smart Images

Figure CN118968606A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cross-individual gaze estimation, and in particular to a cross-individual gaze estimation method based on individual differences. Background Art
[0002] In the field of computer vision, gaze estimation is a very active and challenging research direction, and great progress has been made in recent years. Gaze estimation requires extracting and fusing facial features in images to obtain accurate 2D gaze points. Many human-computer interaction applications are highly related to it, and it plays a very important role in virtual reality interactions, such as user attention distribution estimation, virtual reality eye movement interaction systems, etc. The task of cross-person gaze estimation is defined as the task of evaluating the model on a group of previously unseen participants. Existing gaze estimation techniques do not capture the long-distance dependencies between eyes, although they have greatly improved the accuracy of gaze estimation.
[0003] The angle Kappa (the deviation between the optical axis and the visual axis) cannot be reflected by images taken by traditional methods. The standard deviation of the Kappa angle for the horizontal and vertical axes in the normal population is about 1.8°, resulting in a random error of 2.0° to 2.3° when the gaze method is estimated directly from the image.
[0004] Existing single-image-based gaze estimation methods cannot recover such individual differences. In theory, their accuracy will reach 2.0°. Many existing methods, such as single images, video sequences, and content information, can predict the gaze position direction, but there are still problems with unobservable person-specific differences. In view of these challenges, scholars have proposed many individual adaptation techniques, including using deep learning neural networks to predict differences between people, directly estimating the 6-DOF calibration parameters of the human eye, or fine-tuning the model with a small number of labeled test samples, which have achieved good results. However, these methods still fail to solve the Kappa problem well. Summary of the invention
[0005] In view of the deficiencies of the prior art, the present invention proposes a cross-individual gaze estimation method based on individual differences, which aims to solve the problem of invalid sample data and the long-distance dependency between eyes, and to model individual differences under a cross-person setting. This method is particularly designed for processing facial images. Three functional modules are designed: DP (data filtering module), SP (auto-calibration module), and PST (prediction transformation module). Among them, the DP module is used to identify invalid samples. The SP and PST modules are used to learn and compensate for individual differences on valid samples, and further improve the accuracy of the prediction results through transformation. This method shows significant performance improvement in two-dimensional line of sight estimation tasks.
[0006] The present invention provides a method for estimating inter-individual gaze based on individual differences, comprising the following steps:
[0007] Step (1), selecting a gaze estimation dataset, inputting facial images in the dataset into a verification module, removing sample data with closed eyes or partially visible pupils, and using a DP model to denoise the sample data, with the goal of maintaining stable detection results under different lighting conditions, facial postures, and image quality;
[0008] Step (2), input the denoised samples into the neural network model CBSP-Net, obtain the participant's three-dimensional gaze vector d, calculate the two-dimensional gaze point of the line of sight according to the position transformation of the camera when the sample is collected in the dataset, and obtain the initial prediction point;
[0009] Step (3), using the SP module to calculate the difference between the predicted mean of all available frame samples of the participant and the historical predicted data, and perform correction learning of the prediction deviation, mainly targeting the Kappa value of the line of sight error offset unique to different individuals, so as to obtain the accurate PoG value;
[0010] Step (4), convert the refined PoG value into a heat map. If there is historical data for the participant, convert the historical data into a heat map as well. Combine the two data and input them into the PST module to eliminate the error of the affine transformation and obtain the final predicted heat map data. Then, the final heat map data is subjected to a SoftMax transformation to obtain the final predicted two-dimensional gaze estimation prediction result.
[0011] Preferably, the DP model denoising method in step 1 is as follows:
[0012] Since samples such as closed eyes or blinking are noise samples, and different lighting conditions, different facial postures, and different image quality will affect the training and prediction of the model, it is necessary to remove noise samples. The proposed DP model combines facial feature point detection and iris detection. The model integrates facial feature point detection and iris detection technology. The operation process begins with the successful positioning of the face area, and then uses the 68-point facial landmark point detection method to accurately locate the key feature points around the eyes as a baseline reference for further analysis of the eye structure. On this basis, two independent eye regions of interest (ROIs) are defined based on these feature points and converted into grayscale images respectively. Next, by applying contrast enhancement and noise reduction techniques, the eye features, especially the subtle structures of the iris and pupil, are highlighted. Using the specific properties of the pupil and iris, the Hough circle transform is implemented to accurately obtain the iris boundary information. The information summarized from the feature point detection is combined with the detailed iris features extracted by the Hough transform to calculate the aspect ratio of the human eye. Through this series of detailed analyses, combined with the obtained aspect ratio indicators and the specific shape of the iris, the system can efficiently and accurately distinguish whether the eyes are open or closed.
[0013] Preferably, step 2 is implemented as follows:
[0014] After filtering and removing invalid samples in step 1, the filtered images are input into the model, first converted to the feature vector space, and then feature extraction is performed. Because the existing line of sight estimation technology does not capture the long-distance dependency between eyes, nor does it combine the spatial attention mechanism well. The proposed neural network model CBSP-Net is based on the L2CS-Net model, adds spatial attention and strip pooling technology, effectively condenses information, and extracts two components of the three-dimensional line of sight vector (yaw angle and pitch angle). The combination of yaw angle and pitch angle can cover all possible line of sight directions, thereby providing a comprehensive description of the head or eye posture. The two angles correspond to the natural movement of the head and eyes in daily life. Therefore, combining the two parts can represent the eye and head posture. Combined with the position of the camera during data collection, the direction of the camera coordinate system is obtained. The three-dimensional vector is combined with the camera coordinate system and projected onto the two-dimensional screen plane. The intersection of the three-dimensional vector and the two-dimensional plane is the two-dimensional point of gaze (PoG);
[0015] The added spatial attention module is the CBAM module, which is a lightweight convolutional attention module that combines the channel and spatial attention mechanism modules. CBAM contains two sub-modules, CAM and SAM, which perform channel and spatial attention respectively.
[0016] The added strip pooling technology is the StripPooling module, which applies a long strip pooling kernel in one spatial dimension to capture long-distance information, especially long-distance dependencies. At the same time, it maintains a relatively narrow size in another dimension to avoid capturing some irrelevant information.
[0017] Preferably, the specific implementation process of step 3 is as follows:
[0018] Due to the differences in physiological structure between different individuals, there are deviations in the prediction of the gaze point. The SP module mainly learns to compensate for the prediction deviation, models the offset to bridge the gap between the visual axis and the optical axis, eliminates the overall impact, and uses the distribution of the data set to simulate the offset specific to the participant. The prediction mean of all available frame samples of the participant and the calculation of historical prediction data are used as errors. Then the prediction mean of all available frame samples g tr The calculation formula is:
[0019]
[0020] Global average fixation point g tr represents the mean of all sample gaze results, p i represents the coordinates of the gaze point predicted by each valid sample, and N represents the number of valid samples after screening by the DP module.
[0021] As a preferred method, the prediction formula of the SP module is as follows:
[0022]
[0023] in the formula represents the correction value of the predicted gaze point of the jth participant at the tth time point after calibration by the SP module, p t,j is the predicted value of the initial fixation point, h t,j Represents a historical index set. For the online setting (continuous frame input), it contains all historical prediction indexes up to the current time point (t-1); for the offline setting, it contains all historical and future frames except the current frame. i,j represents the predicted probability of the i-th sample at position j in the historical data, g t,r Represents the predicted mean of all available frame samples, ω i Represents the weight coefficient of historical forecast data.
[0024] As a preference, where ω i The calculation method is as follows:
[0025]
[0026] In the formula, distance(p i,j,p t,j ) represents the historical prediction p i,j With the current prediction p t,j The distance between them is measured using the Euclidean distance. ∈ is a very small integer to prevent the historical prediction from being exactly the same as the current prediction, causing the denominator to become zero, resulting in an infinite weight.
[0027] Preferably, the PST module in step 4 can model and compensate for the differences in the sight lines of more common users, as follows:
[0028] The PST module predicts the output of the SP module Heatmap of the predictions of the historical refined PoG of the same participant The heat map of the two is taken as input, and the heat map of the conversion is output. Finally, it is converted into probability through the SoftMax function. The point with the maximum probability is the point of the improved prediction.
[0029] The present invention has the following characteristics and beneficial effects:
[0030] Using the above technical solution, the present invention first uses the data screening module DP to remove noise samples. Then it is input into CBSP-Net to extract features, and the model budget obtains a three-dimensional gaze vector, which is then converted into a two-dimensional gaze point. Then, by comparing the historical predicted sight line data of the test user and the training Gaze data, the difference in the mean of the two is calculated as the value of the Kappa angle of the user's individual eye difference to correct the user's prediction and eliminate basic differences. Finally, the PST module learns the affine transformation between the user's prediction and the training Gaze, combines the historical evaluation data of the test user and the previous personalized correction estimation data, and obtains the final improved result. The present invention can model according to the historical data of the predictions of different participants, and effectively improve the prediction accuracy. The algorithm proposed in the present invention is independent of the specific sight line estimation model. It is a simple affine transformation that can be applied to any sight line estimation method to effectively improve the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0032] Figure 1 It is a flow chart of a cross-individual gaze estimation method based on individual differences according to the method of the present invention.
[0033] Figure 2It is a schematic flow diagram of the modules of the method of the present invention.
[0034] Figure 3 It is a schematic diagram of the CBSP-Net model in the method of the present invention.
[0035] Figure 4 It is a schematic diagram of the strip pooling module in the CBSP-Net model in the method of the present invention.
[0036] Figure 5 It is a schematic diagram of the neural network model of the PST module in the method of the present invention.
[0037] Figure 6 It is a comparison chart of PoG results before and after the method of the present invention improves prediction. DETAILED DESCRIPTION
[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.
[0040] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.
[0041] The present invention provides a method for cross-individual gaze estimation based on individual differences, such as Figure 1 As shown in the figure, the model can perceive the differences between different individuals. The processing flow is as follows: First, the input facial image is processed by the DP module to filter out noise samples such as closed eyes or blinking. Then, the neural network model CBSP-Net is used to extract features, and the three-dimensional gaze vector is obtained after model processing. Combined with the camera parameters, it is converted and mapped into a two-dimensional gaze point (PoG) on the screen. Then, the SP module is used to calculate the predicted mean PoG of all available frame samples of the participant and the historical prediction data, and the prediction deviation is corrected and learned to obtain the precise PoG value. Finally, the precise PoG value is converted into a heat map and input into the PST module to eliminate the error of the affine transformation and obtain the final predicted heat map data. The final heat map data is then transformed by SoftMax to obtain the final predicted two-dimensional gaze estimation prediction result.
[0042] The specific steps are as follows:
[0043] Step (1), since samples such as closed eyes or blinking are noise samples, and different lighting conditions, different facial postures, and different image quality will affect the training and prediction of the model, the noise samples must be removed. The proposed DP model combines facial feature point detection and iris detection. The model integrates facial feature point detection and iris detection technology. The operation process begins with the successful positioning of the face area, and then uses the 68-point facial landmark point detection method to accurately locate the key feature points around the eyes as a baseline reference for further analysis of the eye structure. On this basis, two independent eye regions of interest (ROIs) are defined based on these feature points and converted into grayscale images respectively. Next, by applying contrast enhancement and noise reduction techniques, the eye features, especially the subtle structures of the iris and pupil, are highlighted. Using the specific properties of the pupil and iris, the Hough circle transform is implemented to accurately obtain the iris boundary information. The information summarized from the feature point detection is combined with the detailed iris features extracted by the Hough transform to calculate the aspect ratio of the human eye. Through this series of detailed analyses, combined with the obtained aspect ratio indicators and the specific shape of the iris, the system can efficiently and accurately distinguish whether the eyes are open or closed.
[0044] Step (2), after filtering and removing invalid samples, the filtered images are input into the model, first converted to vector space, and then basic features are extracted from the input images through the basic convolution extraction module in the neural network model CBSP-Net. The existing sight estimation technology does not capture the long-distance dependency between eyes, nor does it combine the spatial attention mechanism well. The proposed CBSP-Net is based on the L2CS-Net model, adds spatial attention and strip pooling technology, effectively condenses information, and extracts two components of the three-dimensional sight vector (yaw angle and pitch angle). The combination of the two components can represent the eye and head posture. Combined with the position of the camera during data collection, the Zhang Zhengyou calibration method is used to automatically detect the corner points in the picture by taking multiple pictures of the calibration plate with a known geometric pattern, and the image coordinates of the corner points are corresponded to the actual world coordinates. The linear equations are constructed and solved to calculate the intrinsic parameters of the camera, such as focal length and principal point coordinates. Thus, the direction of the camera coordinate system is obtained. The three-dimensional vector is combined with the camera coordinate system and projected onto the two-dimensional screen plane. The intersection of the three-dimensional vector and the two-dimensional plane is the two-dimensional point of gaze (PoG);
[0045] Specifically, the added spatial attention module is the CBAM module, which is a lightweight convolutional attention module that combines the channel and spatial attention mechanism modules. CBAM contains two sub-modules, CAM and SAM, which perform channel and spatial attention respectively.
[0046] The added strip pooling technology is the StripPooling module, which applies a long strip pooling kernel in one spatial dimension to capture long-distance information, especially long-distance dependencies. At the same time, it maintains a relatively narrow size in another dimension to avoid capturing some irrelevant information.
[0047] Step (3): Use the SP module to calculate the difference between the predicted mean of all available frame samples of the participant and the historical predicted data, and perform correction learning of the prediction deviation, mainly targeting the Kappa value of the line of sight error offset unique to different individuals, so as to obtain the accurate PoG value.
[0048] Specifically, if the participant has historical gaze estimation data, combining the historical estimation data can improve the result of this gaze. The SP module is used to model the difference between the individual visual axis and the optical axis, and the prediction mean of all available frame samples of the participant and the calculation of historical prediction data are used as errors. Then the prediction mean of all available frame samples g tr The calculation formula is:
[0049]
[0050] Global average fixation point g tr represents the mean of all sample gaze results, p i represents the coordinates of the gaze point predicted by each valid sample, and N represents the number of valid samples after screening by the DP module.
[0051] The prediction formula of the SP module is as follows:
[0052]
[0053] in the formula represents the correction value of the predicted gaze point of the jth participant at the tth time point after calibration by the SP module, p t,j is the predicted value of the initial fixation point, h t,j Represents a historical index set. For online settings with continuous frame input, it contains all historical prediction indexes up to the current time point t-1; for offline settings, it contains all historical and future frames except the current frame. i,j represents the predicted probability of the i-th sample at position j in the historical data, g t,r Represents the predicted mean of all available frame samples, ω i Represents the weight coefficient of historical forecast data.
[0054] Among them, ω i The calculation method is as follows:
[0055]
[0056] In the formula, distance(p i,j ,p t,j ) represents the historical prediction p i,j With the current prediction p t,j The distance between them is measured using the Euclidean distance. ∈ is a very small integer to prevent the historical prediction from being exactly the same as the current prediction, causing the denominator to become zero, resulting in an infinite weight.
[0057] It is understandable that there are deviations in gaze point prediction due to differences in physiological structure between different individuals. The SP module mainly learns to compensate for the prediction deviation, models the offset to bridge the gap between the visual axis and the optical axis, eliminates the overall impact, and uses the distribution of the data set to simulate the offset specific to the participant. The predicted mean of all available frame samples of the participant and the calculation of historical prediction data are used as errors.
[0058] Step (4), the prediction output of the SP module is converted into Heatmap of the predictions of the historical refined PoG of the same participant The heat map of the two is taken as input, and the converted heat map is output. Finally, it is converted into probability through the SoftMax function. The point with the maximum probability is the point of the improved prediction.
[0059] The process of generating the heat map first converts the calibrated prediction points Mapped to pixel space, a Dirac delta function is then placed at the center of the point, convolved through a deep neural network in the PST module, and an isotropic 2D Gaussian function with fixed variance is applied to form a smooth heat map. The heat map can represent the predicted point and also capture the changes in the surrounding area through the Gaussian distribution.
[0060] The pre-processing of the SP module is crucial, as it transforms the initially predicted PoG to a reasonable position, ensuring that the PST module is not disturbed by invalid predictions when working. The SP module takes into account and compensates for large translation offsets, making the initial PoG prediction more accurate. Subsequently, the PST module performs more sophisticated processing that can compensate for all affine-like transformations. By randomly selecting affine transformations to generate samples for training, the PST module aims to eliminate individual-specific affine-like differences in the gaze estimation task.
[0061] Among them, the algorithm for generating the historical heat map is as follows:
[0062]
[0063] in, Represents the heat map of the current sample after calibration by the SC module, which is the output of the current step of the model.t,j Represents a heatmap processed with data augmentation (including affine transformation and random noise) to simulate individual-specific differences and possible noise.
[0064] It should be noted that the PST module is a feature processing module with a built-in module called SimAM, which is responsible for extracting attention features. The SimAM module assigns a unique weight to each neuron and calculates the similarity between each pixel and its adjacent pixels in the feature map to better capture the global attention features.
[0065] The PST model consists of the following parts: a positioning network, which is used to preliminarily determine the location of the gaze point. A SimAM module, which is used to extract attention features. A grid generator, which may be used to create a grid structure for transformation. A downsampler, which is used to reduce the spatial dimension of the data and increase computational efficiency. Its network architecture is shown in Table 1. The network architecture of the PST module consists of a basic layer composed of basic convolutional layers, pooling layers, and activation functions. These layers are stacked and finally converted through a Linear layer to obtain the output result. The dimension of the input tensor is Represents the heat map of a single sample. The dimension of the output tensor is Includes transformed sample heatmap And the transformed historical heat map data The dimensions of both are R H×W×1 .
[0066] Mathematically, the PST module can be represented as f, and its input-output relationship is represented as s=f(s t,j ). The loss function of the PST module is designed as a binary cross entropy loss function, and the formula is:
[0067]
[0068] in, Represents the heat map of the current sample after calibration by the SP module, which is the output of the current step of the model. t,j Represents a heatmap processed with data augmentation (including affine transformation and random noise) to simulate individual-specific differences and possible noise.
[0069] To obtain the final PoG (Point of Gaze) of the input image, we need to apply the Softmax function to the output of the SP module. The Softmax function will output a probability distribution, from which we can find the position corresponding to the maximum probability. Then, this position is remapped (scaled) to the coordinate system corresponding to the screen size to obtain the final PoG result.
[0070] Table 1. Network architecture parameters of PST module
[0071]
[0072] In order to verify the performance of the present invention in data post-processing, we tested the method of the present invention on the EVE dataset. The experimental results are shown in Table 2, which includes two key indicators for measuring detection performance: Gaze Dir (°) and PoG (cm / px). Gaze Dir represents the error angle of the three-dimensional gaze direction prediction, while PoG represents the prediction error of the two-dimensional gaze direction, which is defined as the Euclidean distance between the predicted PoG position and the true PoG on the screen plane, measured in centimeters and pixels, respectively. It can be seen from the data in Table 2 that the line of sight estimation model constructed by the method of the present invention significantly improves the prediction effect.
[0073] In order to verify the performance of the results of the present invention in line of sight estimation, we tested the method of the present invention on the EVE dataset. The experimental results are shown in Table 2, including two key indicators: Gaze Dir (°) and PoG (cm / px), where Gaze Dir represents the error angle of the three-dimensional gaze direction prediction, and PoG represents the prediction error of the two-dimensional gaze direction, measured in centimeters and pixels. It can be seen from the data in Table 2 that the line of sight estimation model constructed by the method of the present invention significantly improves the prediction results.
[0074] Table 2. Comparison of the performance of different models on the validation set of the EVE dataset (the best results are marked in bold)
[0075]
[0076] To further test the robustness of the model, we tested it on other datasets MPIIGaze and ETH-XGaze. Table 3 shows the test results, and the test annotation is the gaze error Gaze Dir of the three-dimensional gaze estimation. At the same time, we compared the method of the present invention with other classic gaze estimation methods. From the data in Table 3, it can be seen that the method of the present invention can not only effectively simulate the differences between individuals, but also has good versatility and can achieve excellent performance.
[0077] Table 3. Performance of the model on the MPIIGaze and ETH-XGaze datasets (the best results are marked in bold)
[0078] MPII Test Set ETH-Gaze test set method GazeDir(°) GazeDir(°) ETH-Gaze 4.8 4.5 Rt-gene 4.8 - Few-shot Gaze Estimation 5.2 - EVE-SCPT 4.14 2.88 Method of the present invention 4.01 2.76
[0079] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.
Claims
1. A method for estimating inter-individual gaze based on individual differences, characterized in that: The following steps are involved: Step 1: Select the sight estimation dataset and perform preprocessing; Step 2: Input the denoised samples into the neural network model CBSP-Net to obtain the participant's three-dimensional gaze vector d, calculate the two-dimensional gaze point of the line of sight according to the position transformation of the camera when the samples in the dataset were collected, and obtain the initial prediction point; Step 3: Calculate the difference between the predicted mean of all available frame samples of the participant and the historical predicted data, and use the SP module to correct the prediction deviation to obtain the accurate PoG value; Step 4: Convert the refined PoG value into a heat map and input it into the PST module to eliminate the error of the affine transformation and obtain the final predicted heat map data. Then, the final heat map data is subjected to SoftMax transformation to obtain the final predicted two-dimensional gaze estimation prediction result.
2. The method for estimating inter-individual gaze based on individual differences according to claim 1, characterized in that: The preprocessing method is: inputting the facial images in the data set into the verification module, removing the sample data with closed eyes or partially visible pupils, and then using the DP model to denoise the sample data.
3. The method for estimating inter-individual gaze based on individual differences according to claim 1, characterized in that: The neural network model CBSP-Net is based on L2CS-Net. In L2CS-Net, the spatial attention mechanism and the strip pooling attention mechanism are added to the main network part of ResNet-50.
4. The method for estimating inter-individual gaze based on individual differences according to claim 3, characterized in that: In the step 2, after basic features of the input image are extracted by the basic convolution extraction module in the neural network model CBSP-Net, the channel and spatial attention mechanisms are applied to the eye image, and the extracted basic features are compressed in combination with the strip pooling attention mechanism. The output of the model is the yaw angle and pitch angle in the three-dimensional line of sight vector, and the yaw angle and pitch angle respectively represent the state and position of the eye and head posture.
5. A method for estimating inter-individual gaze based on individual differences according to claim 4, characterized in that: The yaw angle and pitch angle are combined with the position of the camera during data collection, and the direction of the camera coordinate system is obtained using the Zhang Zhengyou calibration method. Then the three-dimensional line of sight vector is projected onto a two-dimensional plane in combination with the camera coordinate system. The intersection of the three-dimensional line of sight vector and the two-dimensional plane is the two-dimensional gaze point PoG.
6. The method for estimating inter-individual gaze based on individual differences according to claim 2, characterized in that: In step 3, the predicted mean value g of all available frame samples is tr The calculation formula is: Global average fixation point g tr represents the mean of all sample gaze results, p i represents the coordinates of the gaze point predicted by each valid sample, and N represents the number of valid samples after screening by the DP module.
7. The method for estimating inter-individual gaze based on individual differences according to claim 6, characterized in that: The correction prediction method of the SP module is as follows: in the formula represents the correction value of the predicted gaze point of the jth participant at the tth time point after calibration by the SP module, p t,j is the predicted value of the initial fixation point, g t,j represents the historical index set. For the online setting of continuous frame input, it contains all historical prediction indexes up to the current time point t-1; for the offline setting, it contains all historical and future frames except the current frame, p i,j represents the predicted probability of the i-th sample at position j in the historical data, g t,r Represents the predicted mean of all available frame samples, ω i Represents the weight coefficient of historical forecast data.
8. The method for estimating inter-individual gaze based on individual differences according to claim 7, characterized in that: The weight coefficient ω i The calculation method is as follows: In the formula, distance(p i,j ,p t,j ) represents the historical prediction p i,j With the current prediction p t,j The distance metric between them uses the Euclidean distance, ∈ is a very small integer to prevent the historical prediction from being exactly the same as the current prediction, the denominator becomes zero, and the weight becomes infinite.
9. The method for estimating inter-individual gaze based on individual differences according to claim 1, characterized in that: In step 4, if there is historical data for the participant, the historical data is also converted into a heat map, the two data are combined, and then input into the PST module.
10. The method for estimating inter-individual gaze based on individual differences according to claim 1, characterized in that: In step 4, the method for generating the heat map is: first, the calibrated prediction points Mapped to pixel space, a Dirac delta function is then placed at the center of the point, which is convolved through a deep neural network in the PST module, and an isotropic 2D Gaussian function with fixed variance is applied to form a smooth heat map.
11. The method for estimating inter-individual gaze based on individual differences according to claim 1, characterized in that: The loss function of the PST module is designed as a binary cross entropy loss function, and the formula is: in, Represents the heat map of the current sample after calibration by the SP module, s t,j Heatmap representing data augmentation to simulate individual-specific differences and possible noise.
Citation Information
Patent Citations
Line-of-sight estimation method based on depth appearance gaze network
CN108171218A
Online learning attention tracking method based on sight line estimation and application thereof
CN114120432A
Self-supervision human body key point detection method based on affine transformation
CN117636389A
