Eye movement tracking system and method based on multi-feature fusion

Abstract

According to the method, multiple frames of images of a testee are selected, and firstly, head features, left eye features and right eye features in the multiple frames of images are preliminarily obtained through a backbone network; for head features, assuming that the head is a rigid body, obtaining a parameter model result of a camera through a Zhang Zhengyou calibration method, and further realizing estimation of a free posture of the head through an EPnP algorithm; then, a feature extraction module is used for extracting multi-frame features of the left eye, the right eye and the head, and a predicted value and confidence coefficient corresponding to each frame are further obtained; and finally, global information is captured through a feature fusion module, context information is effectively integrated, the feature fusion module comprises a channel and a space attention mechanism, and the channel and the space attention mechanism can automatically identify and enhance channel features having great contribution to tasks, so that key features of eye movement tracking are highlighted in the feature extraction and fusion process, irrelevant parts are inhibited, and the accuracy of eye movement tracking is improved. The feature expression ability is improved, and accurate eye movement tracking is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer eye tracking, and in particular to an eye tracking system and method based on multi-feature fusion. Background Art

[0002] Eye tracking is a technology that records human eye movements and is used to analyze visual attention and information processing. Early devices relied on head fixation, which limited freedom; modern wearable eye trackers can be flexibly used in a variety of environments. Through cameras and optical sensors, data such as gaze points and scanning patterns are obtained in real time, and are widely used in psychology, human-computer interaction, assisted driving, and offline retail. In psychology, cluster analysis of eye movement data reveals common behaviors of users browsing the web, such as preference for images or price information, improving user experience and business conversion rate; in the field of human-computer interaction, eye tracking helps ALS patients control the interface with their eyes to meet daily needs; in assisted driving, monitoring the driver's blinking frequency and gaze time can assess attention, warn of fatigue or distraction behavior to improve driving safety; in offline retail, eye tracking analyzes consumer shopping behavior, optimizes product layout and marketing strategies, and improves sales results.

[0003] Up to now, eye tracking methods are mainly divided into model-based methods and appearance-based methods. Model-based eye tracking methods rely on the understanding of mathematical models and physiological mechanisms of human eye movement. By establishing an eye movement model, researchers can simulate the movement patterns of the eyes in specific situations. This method usually requires a large amount of data training in advance to capture the eye movement characteristics of different individuals. Model-based systems can provide higher accuracy, but when lighting conditions or head postures change frequently, it is more difficult for this method to obtain the mapping function. In recent years, with the continuous development of deep learning, appearance-based eye tracking methods have been favored by many scholars. It focuses on directly analyzing the visual features of the eyes through image processing technology. This method uses a camera to capture images of the eyes and uses image processing algorithms to identify the position of the eyeball, the size of the pupil, and actions such as blinking. Appearance-based methods usually do not require complex external equipment, and can learn the optimal mapping function on various public test data sets, which is suitable for more practical scenarios and life needs. However, in the research of eye tracking technology, most of the early deep learning-based methods are often carried out under the condition of restricting head posture, which improves the accuracy of line of sight estimation to a certain extent. However, the real-life environment is complex and changeable, and factors such as head posture and lighting constantly affect the acquisition of visual information. Therefore, this limited method is very challenging in practical applications. In order to truly adapt eye tracking technology to complex scenes, it is urgent to develop eye tracking technology that can be integrated with head and eye posture. Therefore, researching and developing eye tracking technology that integrates head and eye posture has important academic value and broad practical significance.

[0004] In summary, the current technology still needs to be improved and enhanced. How to improve the accuracy of eye tracking while reducing the complexity of the model is an urgent problem to be solved. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides an eye tracking system and method based on multi-feature fusion, which achieves accurate eye tracking effect by designing a head free posture estimation module, a feature extraction module and a feature fusion module.

[0006] On one hand, the present invention provides an eye tracking system and method based on multi-feature fusion, comprising:

[0007] The head free pose estimation module first analyzes the characteristics of the head free pose.

[0008] Furthermore, we assume it to be a rigid body and build a model to study its motion state.

[0009] Furthermore, Zhang Zhengyou’s calibration method is used to obtain the camera parameters, and finally the head posture estimation is achieved through the EPnP algorithm.

[0010] Feature Extraction Module,The feature extraction module is responsible for extracting the features of the left eye, right eye, and head,from multiple consecutive frames and generating prediction values ​​and confidence,for each frame.

[0011] Furthermore, the relative position relationship between head posture and eye feature points plays a key role in the extraction process. The changes in these relative positions not only reflect the changes in individual features, but also reveal the interactions between the features;

[0012] The feature fusion module uses channel and spatial attention mechanisms to capture global information and integrate contextual data.

[0013] Furthermore, the channel attention mechanism automatically identifies and strengthens the features that contribute more to the task by adaptively adjusting the importance of each channel, thereby highlighting key features during the extraction process and reducing the interference of irrelevant information.

[0014] Furthermore, the spatial attention mechanism focuses on the spatial position of the feature map in the image, and dynamically adjusts the spatial position in the image, so that the model can focus on important areas at different times and environments.

[0015] Compared with the prior art, the present invention has the following technical effects:

[0016] 1. This application constructs a head free posture estimation module. By combining accurate estimation of head posture and multi-frame feature extraction, the system can achieve more accurate eye tracking. The head free posture estimation module eliminates the interference of head movement on eye tracking accuracy, ensuring that the system can maintain highly accurate eye tracking even in dynamic or complex postures.

[0017] 2. This application designs a multi-feature fusion module, which uses a feature fusion method of channel and spatial attention mechanism, so that the system can adaptively adjust the weights of key features when facing different lighting, angles or posture changes, significantly enhancing the robustness of eye tracking. In particular, the spatial attention mechanism can focus on key areas in the image (such as eyes), ensuring that the system can operate stably in various environments.

[0018] 3. The present invention is not only applicable to standard eye tracking tasks, but can also be extended and applied in complex environments (such as driving scenes, dynamic head postures, remote interactions, etc.). The scalability of the system enables it to adjust feature extraction and fusion strategies according to different needs, thereby achieving wide application in multiple fields (such as psychological research, assisted driving, human-computer interaction, advertising analysis, etc.), with good economic and application prospects.

[0019] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of a free-gesture eye tracking method based on multi-branch feature fusion according to a specific embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of head movement posture according to a specific embodiment of the present invention;

[0022] Figure 3 is a coordinate system transformation diagram of a specific embodiment of the present invention;

[0023] Figure 4 is a transformation diagram of a world coordinate system and a camera coordinate system in a specific embodiment of the present invention;

[0024] Figure 5 is a transformation diagram between a camera coordinate system and an image coordinate system according to a specific embodiment of the present invention;

[0025] Figure 6 is a transformation diagram of an image coordinate system and a pixel coordinate system according to a specific embodiment of the present invention;

[0026] Figure 7 is a schematic diagram of a calibration position of a specific embodiment of the present invention;

[0027] Figure 8 is a network structure diagram of a feature extraction module according to a specific embodiment of the present invention;

[0028] Fig. 9 This is a partial sample graph of the Biwi data set according to a specific embodiment of the present invention;

[0029] Fig.10 This is a graph of EPNP algorithm test results of a specific embodiment of the present invention;

[0030] Fig.11 is a comparative statistical result diagram of a specific embodiment of the present invention;

[0031] Fig.12 This is a graph of the measured data set results of a specific embodiment of the present invention;

[0032] Fig.13 is a comparison diagram of eye tracking results of a specific embodiment of the present invention;

[0033] Fig.14 4 is a comparison result diagram of average angular errors of different methods according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0035] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner, and thus the illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0036] While some exemplary embodiments of the present invention have been described for the purpose of illustration, it should be understood that the present invention may be implemented in other ways not specifically shown in the drawings.

[0037] like Figure 1As shown in the figure, the data set required for the experiment is composed of individuals of different skin colors, ages and genders. An object is randomly selected from them, and several frames of images of the object are used as input. First, the features of the head, left eye and right eye in multiple frames are obtained through the backbone network, and then they are separated to obtain their respective features. Finally, they are fused by modeling in the spatial dimension and channel dimension, and finally the predicted gaze direction is output. Specifically, it mainly involves four steps: 1. Extract the features of the head, left eye and right eye in multiple frames: Use the backbone network to obtain the features of multiple frames, and obtain the feature representation of the head, left eye and right eye in each frame; 2. Extract the features of the head, left eye and right eye in multiple frames: For the features of the head, left eye and right eye extracted from each picture, their features need to be extracted and separated for subsequent feature fusion; 3. Feature fusion in the spatial dimension and channel dimension, model the features of each part in the spatial dimension and channel dimension, perform weighted average on the feature vector, highlight the important features, and fuse the information of different parts; 4. Output the gaze direction result of eye tracking: Input the fused features into the corresponding neural network structure to predict the gaze direction.

[0038] For the input image, where N represents the number of input frames, represents the input frame as an RGB image of size , in order to achieve the feature fusion of the head, left eye and right eye, a multi-branch feature extraction network is designed. Its framework consists of three parts: backbone network, feature extraction module and feature fusion. The backbone network selects FPSA_L2CSNet, which is responsible for extracting the original image features and performing layer-by-layer abstraction; the feature extraction module is used to enhance and strengthen the feature representation of the head, left eye and right eye regions to capture the respective features of the head, left eye and right eye; feature weighted fusion enables the features of different regions to work together, integrate the information of different regions, and improve the model's perception of global and local information. Specifically, this paper uses different symbols to represent different parts. The circle and cube represent the head features after feature fusion and the head features before feature fusion, respectively. The triangle and cuboid represent the left eye features after feature fusion and the left eye features before feature fusion. The pentagon and cuboid represent the right eye features after feature fusion and the right eye features before feature fusion. The feature fusion result will be calculated by weighted sum, and then the final prediction will be made through the feature point regression module. The feature fusion result is calculated by weighted sum, and then the final prediction is made through the feature point regression module.

[0039] like Figure 2The figure shows a schematic diagram of the head movement posture. In the study of human movement, the movement of the head is particularly worthy of attention. The head has the characteristics of a rigid body, which means that its movement can be described by the theory of rigid body motion, thereby reflecting its movement characteristics more deeply. Rigid body motion is mainly divided into two categories: translational motion and rotational motion. Translational motion means that the motion trajectory of each point inside the rigid body is consistent, and the relative positions of all parts remain unchanged. In this state, the translational motion state of the entire rigid body can be inferred by observing only one point of the rigid body. This feature is particularly important when analyzing the translation of the head in space.

[0040] On the other hand, rotational motion is another basic form of rigid body motion. Rotational motion can be further divided into fixed-axis rotation and fixed-point rotation. Fixed-axis rotation refers to the circular motion of all particles in a rigid body around a fixed line, and the position and direction of this line in space remain unchanged. Fixed-point rotation refers to the rotation of all points around a fixed center. Regardless of the type of rotation, the motion of a rigid body can be analyzed by describing its rotation axis and the associated motion trajectory.

[0041] When studying the motion of the human head, the three Euler angles can be used to describe its rotational characteristics. The rotation of the head usually occurs around the neck, so it can be regarded as a rigid body in a resting posture. The basic assumption of this model is that the posture of the person remains stable, which makes the rotational motion of the head describe by a relatively simple physical model. In this context, the three coordinate systems of the center of mass can effectively characterize the translational motion of the rigid body, while the three Euler angles provide an accurate description tool for the rotational motion. Through these methods, the biomechanical characteristics of head motion under motion changes in different postures can be better understood.

[0042] S1. Specifically, this article regards the human head as a rigid body. Therefore, the movement posture of the head can be characterized by three different rotational motions: roll state, pitch state and yaw state. Among them, the roll state refers to the rotation of the head around the front-to-back axis (the axis from the forehead to the back of the head); the pitch state refers to the rotation of the head around the horizontal axis (the axis from the left ear to the right ear), and the head will tilt forward or backward; the yaw state refers to the rotation of the head around the vertical axis (the axis from the top of the head to the bottom), and through the yaw motion, the head will turn left and right.

[0043] S11. Assume that at time t1, the coordinates of the head posture are set to P1 = (X1, Y1, Z1) T After Δt, at time t2, the coordinates of the head posture become P2 = (X2, Y2, Z c ) T , then the expression of its motion process is as follows:

[0044] P2=RP1+T (1)

[0045] Among them, R represents the rotation matrix and T represents the translation matrix.

[0046]

[0047] Therefore, after transformation, we get:

[0048]

[0049] S12. Next, conduct a camera imaging model and parameter acquisition experiment.

[0050] S121, such as Figure 3 As shown in the figure, the camera coordinate system transformation mainly includes four different coordinate systems, which are the pixel coordinate system (O0UV), the camera coordinate system (O c X c Y c Z c ), image coordinate system (O1XY) and world coordinate system (O ω X ω Y ω Z ω ). Among them, P(X ω , Y ω , Z ω ) represents any point in the world coordinate system, and p(x, y) represents its projection in the image coordinate system. The following will introduce the transformation relationship between them in three parts. First, the world coordinate system (O ω X ω Y ω Z ω ) and the camera coordinate system (O c X c Y c Z c ), the specific transformation process is as follows Figure 4 As shown, (X ω , Y ω , Z ω ) and (X c , Y c , Z c ) represent the coordinates of the point in the world coordinate system and the camera coordinate system respectively. The specific expressions are as follows:

[0051]

[0052] S122, then the transformation between the camera coordinate system and the image coordinate system, specifically, the three-dimensional coordinates are converted into two-dimensional coordinates. The specific transformation process is as follows Figure 5 As shown, (X c , Y c , Z c) and (x, y) represent the coordinates of the store in the camera coordinate system and image coordinate system respectively, and the expressions are as follows:

[0053]

[0054] S123, finally, the transformation between the image coordinate system and the pixel coordinate system needs to be obtained through scaling and translation transformation, such as Figure 6 As shown, the expression is as follows:

[0055]

[0056] To sum up, the transformation relationship between the world coordinate system and the pixel coordinate system is

[0057]

[0058] in, is the camera internal parameter, is the camera extrinsic parameter.

[0059] S124. Then, the Zhang Zhengyou calibration method is used to calculate the intrinsic and extrinsic parameters of the camera. The principle is to use the geometric features of the checkerboard: the known corner position and pattern size. By taking multiple checkerboard images, the corner positions under different viewing angles are obtained. By selecting different angles and different distances, multiple pictures are collected for the input image of Zhang Zhengyou calibration. Then, Matlab's generateCheckerboardPoints function is used to generate the world coordinates of the checkerboard, detectCheckerboardPoints is used to extract the corner points in the image, and estimateCameraParameters is used to calculate the internal and external parameters of the camera and the evaluation error. Finally, after the calibration is completed, the calibration position of the camera can be obtained as shown in Figure 7 shown.

[0060] After the calibration is completed, the calibration results of the camera parameters are shown in Table 1.

[0061] Table 1

[0062]

[0063] S2. The free head posture estimation is realized through the EPnP algorithm. The EPnP algorithm was proposed by V. Lepetit et al. in 2007. The EPnP algorithm has good adaptability to different facial feature point layouts and is highly robust to image noise and occlusion. The principle of the algorithm will be studied below.

[0064] S21, EPnP uses virtual points to represent the coordinate points of the three-dimensional world, and then solves the positional relationship of the coordinate system to achieve accurate positioning. Assume that the world coordinate system has m points of three-dimensional coordinates, 4 virtual points, respectively denoted as and Then accurate positioning can be achieved, the expression is as follows:

[0065]

[0066] At the same time, the camera coordinate system can also be expressed by virtual points, the expression is as follows:

[0067]

[0068] Then, combine it with equation 11 to obtain:

[0069]

[0070] From this we can see that

[0071]

[0072] Combining equation 12 with equation 13, we get the following expression:

[0073]

[0074] Next, combine equation 14 with the feature points to obtain the following expression:

[0075]

[0076] Solve the above formula to get the value of X. The expression is as follows:

[0077]

[0078] S22. Then, the value of X can be obtained through formula (4-11) to obtain the values ​​of m feature points in the camera coordinate system, and then the rotation matrix result of the head free posture can be obtained according to the transformation relationship of the coordinate system.

[0079] S23. Finally, the head posture angle is obtained according to Formula 18, as shown in Formulas 19, 20 and 21.

[0080]

[0081] S3, such as Figure 8As shown in the figure, it is the network structure diagram of the feature extraction module. The feature extraction module is used to further extract the multi-frame features of the left eye, right eye and head, and obtain the corresponding prediction value and confidence for each frame. In the feature extraction process, the relative position relationship between the feature points of the head posture, left eye and right eye is an important basis for characterizing the strength relationship of the features. The relative position change between each feature point not only reflects the change of a single feature, but also reveals the mutual influence between different features. For example, the relationship between the head posture (such as pitch, yaw and roll angle) and the eye position can help to more accurately infer the direction of movement or steering angle of the head, while the change in eye position can reflect the focus of the experimenter. Therefore, in order to obtain feature information more accurately, this paper sets filters of different dimensions in the feature extraction module to realize multi-granular feature extraction of input features. For different features F∈{left_eye,right_eye,head}, a multi-layer perceptron is used to indicate the presence or absence of each feature. When the subject's head leaves the camera, the head and eyes cannot be detected, and the confidence is 0 at this time. The expression of the detection method is as follows:

[0082]

[0083] In the above formula, D on the left side of the equation represents the detection result, and the right side of the equation represents the detection result of the feature of each frame in the multi-layer perceptron, where Sigmoid is used to map the result of MLP to the range of (0, 1).

[0084] S31. After the features are obtained, the weighted sum is calculated, which is expressed as follows:

[0085]

[0086] in, refers to the weight from the ith input to the ith hidden layer neuron, refers to the bias term of the neuron, x i is the input feature

[0087] The weighted sum Input into the activation function to get the updated features, expressed as follows:

[0088]

[0089] S32. Calculate the weighted sum of the updated features, expressed as follows:

[0090]

[0091] in, refers to the weight from the jth hidden layer neuron to the kth output layer neuron, Refers to the bias term of this neuron.

[0092] For the three updated features, three different MLP regression prediction results are used, expressed as follows:

[0093]

[0094] In the above formula, O∈R 3 ,F∈{left eye , right eye , head}. Specifically, the reliability of the prediction results obtained from each frame varies in different situations. For example, when the head turns backward, the eyes are invisible, resulting in low reliability of eye tracking using eye features. Therefore, we use three MLPs to predict the confidence levels of the three predicted gazes, expressed as follows:

[0095]

[0096] S33. Then, the results from different features are multiplied by the corresponding confidence levels to obtain the final prediction results of each feature, which are expressed as follows:

[0097] R=FC(O head ×C head , O lefteye ×C lefteye , O righteye ×C righteye ) (28)

[0098] S4, such as Fig. 9 The figure shows the network structure of the feature fusion module. The channel attention mechanism and spatial attention mechanism in the convolutional neural network are key technologies for mining global feature information in images. Therefore, after the convolution layer operation, the original image will generate numerous information channels, representing basic features such as texture, contour and color at different levels in the image. However, there may be redundant or useless information in these channels. The purpose of the channel attention mechanism is to mine the relationship between global information channels and assign different weights so that the entire network model can focus on important feature channels.

[0099] The spatial attention mechanism focuses on different positions in the image or feature map, and strengthens key areas and suppresses unimportant areas based on spatial contextual information. It can identify important local areas in the image (for example, the area where the target is located or the hot spot area of ​​the feature map) by dynamically adjusting the weight of each position, thereby combining local features with global information. Through the collaborative work of channel and spatial attention mechanisms, the model can capture the relationship between features in multiple dimensions: on the one hand, the channel attention mechanism can enhance the dependency between channels, and on the other hand, the spatial attention mechanism can accurately extract the key information of each position.

[0100] Therefore, a channel and spatial attention mechanism is designed to capture global information and effectively integrate contextual information. The channel attention mechanism can automatically identify and enhance channel features that contribute more to the task by adaptively adjusting the importance of each channel, thereby highlighting key features and suppressing irrelevant parts during feature extraction. This helps the model focus more on important semantic information and improves the expressiveness of features.

[0101] S41. For input features Figure X First, the input feature map is processed by 3×3 convolution, so that the local feature information A(x) and B(x) can be obtained; then, in order to strengthen the interaction between features, the local features are fused with the initial features, so that the number of channels is doubled to 2C; then two 1×1 convolution layers are used to learn the feature matrix A1; finally, the feature matrix is ​​multiplied by the input to represent the dependency relationship between pixels in different parts, and then the global feature representation A2 can be obtained, which is expressed as follows:

[0102] A2=A1×Input (22)

[0103] S42, after the local feature information B(x) and the global feature representation A2 are fused, the output is finally obtained. In this way, the context information relationship and feature representation are concentrated into a unified structure, which retains the salient features of eye tracking and suppresses irrelevant information.

[0104] Specifically, in the channel attention mechanism, for a given feature map A∈R C×H×W , three convolutional layers are used to generate corresponding feature maps respectively, so the unified similarity matrix X∈R C×C , the expression is as follows:

[0105] X=f((C×K) T ×(K×C)) (29)

[0106] Where K = H × W. Then apply softmax normalization to f, then the weighted feature map CN of the channel attention mechanism is expressed as follows:

[0107] CN=f(X T ×(C×K)) T +A) (30)

[0108] In the spatial attention mechanism part, similar to the channel attention mechanism part, its unified similarity matrix S∈R K×K , the expression is as follows:

[0109] S=f((K×C) T ×(C×K)) (31)

[0110] The expression of the weighted feature map SN of the spatial attention mechanism is as follows:

[0111] SN=f(S T ×(C×K)) T +A (32)

[0112] S43. Finally, the features SN and CN are combined to generate a global feature map N, which contains channel and spatial context information. The function of N is expressed as follows:

[0113] N=Cat(CN+SN) (33)

[0114] Therefore, the feature fusion module generates a global feature N at each stage. By stacking multiple global features and sharing spatial and channel information, more refined feature extraction can be achieved in complex scenes.

[0115] For example, the model for extracting binocular local features can be regarded as a four-tuple β, which is expressed as follows:

[0116] β=(f A , f B ,ρ,R) (34)

[0117] where f A , f B are two feature extractors, ρ represents the pooling function, is a classifier. The feature extraction process has the following functional relationship, which is expressed as follows:

[0118] f: L×I→R ((hw)×c) (35)

[0119] The feature extractor takes the image as input and combines the location of the feature points The output size is (hw)×c, where h, w and c represent the height, width and number of channels of the image. For the features extracted by the two feature extractors, the feature point locations can be expressed as and respectively, which contain two items: local position information and appearance characteristics.

[0120] The two feature extractors are fused at each position as their output. Specifically, at position Up, f A (l, I) and f B The feature fusion of (l, I) can be expressed as the following functional relationship, which is expressed as follows:

[0121]

[0122] S44, multiply the same position from the two feature extractors, the result of this function can be expressed as position The apparent result at the position, so that by F(l, I l , f A , f B ) strengthens the representation of local features, which is beneficial for the network to obtain the features of all positions from the local area mining results of the final feature discrimination. After ρ weights all the features, the function representation result of the global image is obtained, which is expressed as follows:

[0123]

[0124] When the feature extractor f A and f B When features of sizes K×M and K×N are extracted respectively, the output size is M×N.

[0125] In order to further illustrate the technical solution of the present invention, a comparative description is given below in conjunction with specific experiments.

[0126] First, the experimental results and analysis of free head pose estimation are presented.

[0127] The public beta data set used in this experiment is the BIWI data set, which includes scenes shot from multiple perspectives, including multi-angle videos of people with different backgrounds and lighting conditions. Each frame of the image contains the skeleton information of the person. These labels include the positions of various parts of the body, such as the 3D coordinates of the head, limbs, spine, etc., and contain rich annotation information, providing posture labels, depth information, and scene annotations. In order to obtain sufficient verification, different head postures of subjects with different skin colors, ages, and genders were selected. Some samples of the data set are as follows: Fig. 9These images cover a variety of head postures, including front, side, up and down tilt and other rotation angles, showing the rotation of the head in different directions. In each image, the head posture changes are obvious, such as looking up, looking down, turning left and right, etc., accompanied by different degrees of head rotation. Through these images with different postures, the estimation effect of the EPnP algorithm at various head angles and its robustness in complex environments can be fully tested.

[0128] During the validation process using the Biwi dataset, we encountered a variety of challenging head pose changes, especially when the head pose changes significantly or the angle deflection is large. Although traditional algorithms may accumulate errors due to drastic changes in pose in these cases, thus affecting the estimation accuracy, the EPnP algorithm can effectively avoid these problems and maintain high-precision pose estimation. Fig.10 The results of head posture estimation successfully completed by the EPnP algorithm under these complex postures are shown, showing its robustness and accuracy in dealing with large head deflections. Specifically, when the yaw or pitch angle of the head deviates greatly, the EPnP algorithm can optimize the calculation based on the changes of the two-dimensional feature points in the image and its accurate three-dimensional geometric model. This optimization not only considers the relationship between the points in the two-dimensional image and the points in the three-dimensional world coordinate system, but also can adjust and correct the errors that may occur in the estimation process in real time, so as to accurately reflect the changes in head posture. Compared with some traditional single-point or simple geometric constraint methods, the EPnP algorithm can provide more accurate and stable posture estimation when dealing with large head rotations, avoiding estimation bias caused by error accumulation. It can be seen from the results that when the subject raises his head, the corresponding pitch angle is negative, and when he lowers his head, the corresponding pitch angle is positive; when the subject turns his head to the left, the corresponding yaw angle is positive, and when he turns his head to the right, the corresponding yaw angle is negative, which is consistent with the experimental results, further fully illustrating the effectiveness and accuracy of the algorithm.

[0129] At the same time, in order to further verify the applicability of the algorithm, full verification was carried out on different data sets, and the true value of the data set was compared with the test results, as shown in Table 2. Analysis of Table 2 shows that the mean absolute error value obtained by the EPnP algorithm is within 4°, which can achieve effective head posture estimation.

[0130] Table 2

[0131]

[0132] In order to further verify the accuracy and reliability of the algorithm, this paper compares the performance of the EPnP algorithm with three classic algorithms: the PnP algorithm, the UPnP algorithm, and the POSIT algorithm. Specifically, the experiment conducted comparative analysis on three key posture angles: pitch angle, yaw angle, and roll angle, aiming to evaluate the accuracy and robustness of different algorithms in the head posture estimation task. The experimental results are as follows Fig.11 As shown, the data in the figure clearly reflects the estimation errors of each algorithm at different posture angles, thus providing sufficient basis for selecting the appropriate algorithm.

[0133] As can be seen from the figure, the EPnP algorithm shows the best accuracy among the three, and EPnP can maintain a low error level. Specifically, the EPnP algorithm has significantly lower errors than other algorithms when estimating pitch angles, yaw angles, and roll angles. Compared with the traditional PnP algorithm, EPnP adopts a more efficient solution method, which significantly reduces the projection error by optimizing the mathematical model and utilizing global feature point information. Since the PnP algorithm is usually optimized based on the least squares method, it may face high calculation errors and instability in complex scenes, while EPnP overcomes this problem by introducing simplification and optimization strategies, maintaining high stability and accuracy.

[0134] The comparison results with the UPnP algorithm also show the advantages of EPnP. Although UPnP enhances the robustness of the algorithm by introducing uncertainty analysis, especially in the case of large noise, UPnP shows stronger adaptability, but when processing standard data sets, EPnP still shows higher estimation accuracy. The advantage of UPnP is mainly reflected in the error tolerance under extreme head posture conditions, but in conventional posture estimation tasks, the error of EPnP is significantly smaller than that of UPnP.

[0135] Compared with the POSIT algorithm, EPnP also shows better accuracy and computational efficiency. The POSIT algorithm relies on the assumption that the intrinsic parameters of the camera are known and performs iterative optimization based on the orthogonal projection model. It has better results when the intrinsic parameters are known and the posture changes are small. However, when dealing with complex scenes, dynamic changes, and large angle offsets, POSIT may be affected by initial value sensitivity and convergence problems, resulting in increased calculation errors. EPnP avoids these problems by globally optimizing the position of feature points and can run stably in a wider range of application scenarios.

[0136] Further analysis of the experimental data shows that the mean absolute error of the EPnP algorithm shows the smallest error value, specifically: the pitch angle error is 1.86°, the yaw angle error is 1.93°, and the roll angle error is 2.05°. These results show that the EPnP algorithm can achieve relatively accurate positioning when estimating the three angles of the head posture. Compared with the other three algorithms, the error is smaller and can maintain a higher accuracy.

[0137] In order to further verify the effectiveness of the algorithm, a real data set was used for verification. Fig.12 The experimental results of head pose estimation in a real environment using the EPnP algorithm are shown. The pitch, yaw and roll attitudes are fully tested. The measured data set verifies the effectiveness of the EPnP algorithm in real application scenarios, especially when dealing with large head deflections, it can still maintain a high estimation accuracy. In a real environment, the head pose estimation task faces many complex challenges, such as factors such as head rotation and tilt, which may affect the accuracy of the estimation. However, from the measured results, accurate results can be obtained for different head poses. For example, when the head rotates around the X-axis, that is, when the head is raised and lowered, the corresponding pitch angle is one positive and one negative; when the head rotates around the Y-axis, that is, when the head is turned left and right, the corresponding yaw angle is one positive and one negative; when the head rotates around the Z-axis, the corresponding roll angle is one positive and one negative, which is consistent with the experimental results. Therefore, it can be fully proved that the EPnP algorithm can effectively cope with these challenges and provide accurate head pose estimation.

[0138] Then the experimental results and analysis of the multi-feature fusion eye tracking system

[0139] First, visual images of eye tracking results are given in the MPIIGaze dataset, Gaze360 dataset, ETH-XGaze dataset, and GazeCapture dataset, such as Fig.13 As shown, the red and green arrows in the figure represent the results of the model output and the true value of the dataset, respectively.

[0140] exist Fig.13 The images in the figure cover the diversity of gender, light, gaze angle and skin color. It can be seen from the figure that after the model adds head posture, it can better capture the visual focus of the experimenter, especially in extreme head states (such as the fifth image in the second row and the third image in the third row). The head posture provides contextual information of spatial position and direction, enriches the input features of the model, and improves the overall performance; at the same time, for different lighting conditions (such as the first image in the first row and the fourth image in the fourth row), due to the introduction of head posture, it can be used as an auxiliary feature of eye tracking, reduce the uncertainty of line of sight deviation, adapt to changes in different lighting conditions, and obtain higher prediction accuracy.

[0141] Then, in order to further verify the accuracy of the model, this paper selected six representative eye tracking models and conducted accuracy comparison experiments with them, and obtained the following results: Fig.14 Results shown.

[0142] according to Fig.14 From the experimental results, we can see that compared with other methods and the model proposed in Chapter 3, the model proposed in this chapter has the lowest angular error on four public datasets, and the angular errors on the MPIIGaze dataset, Gaze360 dataset, ETH-XGaze dataset and GazeCapture dataset are 3.26°, 3.14°, 3.32° and 3.53° respectively; this is due to the design idea of ​​multi-feature fusion, that is, extracting the head posture of the experimenter in multiple frames, and realizing spatial and channel fusion with the features of the left and right eyes. In this way, the line of sight drift error caused by head movement alone can be eliminated, and the feature fusion of head and eye posture can be completed.

[0143] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An eye tracking system and method based on multi-feature fusion, characterized in that: include: The head free posture estimation module includes first analyzing the characteristics of the head free posture, assuming it is a rigid body, modeling it, and analyzing its motion state; Then, the camera parameter model result is obtained through Zhang Zhengyou calibration method, and finally the head free posture estimation is realized through EPnP algorithm; The feature extraction module is used to extract multi-frame features of the left eye, right eye and head, and obtain the corresponding prediction value and confidence for each frame. In the feature extraction process, the relative position relationship between the head posture, left eye and right eye feature points is an important basis for characterizing the strength relationship of the features. The relative position change between each feature point not only reflects the change of a single feature, but also reveals the mutual influence between different features. The feature fusion module includes channel and spatial attention mechanisms to capture global information and effectively integrate contextual information. The channel attention mechanism can automatically identify and enhance channel features that contribute more to the task by adaptively adjusting the importance of each channel, thereby highlighting key features and suppressing irrelevant parts during feature extraction.

2. An eye tracking system and method based on multi-feature fusion according to claim 1, characterized in that: Analysis of head free posture features. The system first models the features of the head free posture, assuming that the head is a rigid body, and provides the necessary posture information for subsequent eye tracking by analyzing the movement state of the head. This process can effectively reduce the error caused by changes in head posture.

3. The eye tracking system and method based on multi-feature fusion according to claim 1, characterized in that: Head pose estimation uses Zhang Zhengyou’s calibration method to obtain the camera’s parameter model, and combines it with the EPnP algorithm to accurately estimate the head’s free pose. This technology can accurately track changes in head pose and ensure the accuracy of eye movement data.

4. The eye tracking system and method based on multi-feature fusion according to claim 1, characterized in that: Multi-frame feature extraction,The feature extraction module is responsible for extracting the features of the left eye, right eye and head from multiple consecutive frames.,The feature extraction of each frame not only provides real-time data of eye movement,,but also generates corresponding prediction values ​​and confidence levels to ensure the reliability of the results.

5. The eye tracking system and method based on multi-feature fusion according to claim 1, characterized in that: The multi-feature fusion module effectively integrates feature information from different regions through channel and spatial attention mechanisms. The channel attention mechanism can adaptively adjust the weight of each channel, highlight the features that contribute more to the task, and improve system performance.

6. An eye tracking system and method based on multi-feature fusion according to claim 5, characterized in that: Channel attention mechanism, by adaptively adjusting the importance of each channel, the channel attention mechanism can automatically identify and enhance the feature channels that have a significant impact on the task, while suppressing irrelevant features. This helps to reduce the interference of redundant information and enhance the accuracy and efficiency of the system.

7. An eye tracking system and method based on multi-feature fusion according to claim 5, characterized in that: Spatial attention mechanism,The spatial attention mechanism enhances the spatial information related to gaze estimation by focusing on key areas in the image (such as eyes and head). Through this mechanism, the system can better capture subtle changes in eye movement behavior, thereby improving tracking accuracy.

8. The eye tracking system and method based on multi-feature fusion according to claim 1, characterized in that: Integration of global and contextual information, the system can capture global information and effectively integrate contextual information through channel and spatial attention mechanisms, ensuring that the fusion of various features not only considers single local features, but also integrates the global vision, thereby improving the adaptability and robustness of eye tracking in different environments.

9. An eye tracking system and method based on multi-feature fusion, characterized in that: An eye tracking method based on multi-feature fusion according to any one of claims 1 to 8 is implemented, comprising the following steps: Head free posture estimation module: First, the head free posture is modeled and assumed to be a rigid body, and its motion state is analyzed; then the camera parameters are obtained through the Zhang Zhengyou calibration method, and finally the EPnP algorithm is used to achieve accurate estimation of the head posture; Feature extraction module: used to extract multi-frame features of the left eye, right eye and head, and calculate the prediction value and confidence of each frame. The relative position change between feature points is an important basis for characterizing the strength relationship of features. These changes not only reflect the changes of a single feature, but also reveal the mutual influence between features; Feature fusion module: Combines channel and spatial attention mechanisms to capture global information and effectively integrate context. The channel attention mechanism adaptively adjusts channel weights to highlight key features and suppress irrelevant parts, thereby improving the accuracy of feature extraction.

Images