An eye movement tracking method and system based on multi-modal fusion

Abstract

The application discloses a kind of based on multimodal fusion's eye movement tracking method and system, specifically related to eye movement tracking technical field, including through the RGB camera, infrared camera and IMU of synchronous setting, user face image, eye image and head movement parameter are collected;Parallel processing these signals, respectively extract the line-of-sight direction vector based on face and eye and its reliability measure, and solve head space posture;According to reliability measure dynamic selection high confidence cooperation, master-slave compensation or conflict arbitration strategy, fusion generates initial line-of-sight direction;The direction is mapped to display plane to obtain initial gaze coordinate, and optimization calibration is carried out in combination with the visual saliency analysis of screen content, and the final gaze point coordinate is output.The system includes signal acquisition processing, parallel computing, multi-strategy fusion decision and output optimization module.The application significantly improves the accuracy and robustness of eye movement tracking through multimodal information fusion and dynamic strategy selection.

Description

Technical Field

[0001] This invention relates to the field of eye-tracking technology, and more specifically, to an eye-tracking method and system based on multimodal fusion. Background Technology

[0002] Eye-tracking technology is a key technology in fields such as human-computer interaction, visual analysis, and virtual reality. Its core objective is to accurately and in real time estimate and track the user's gaze point. Existing eye-tracking solutions mostly rely on visual information from a single modality, such as estimating the gaze direction based on global features of facial images, or calculating the gaze point based on the local pupil-corneal reflection spot geometry of near-focus images of the eye.

[0003] However, it still has some drawbacks in practical use. For example, when the user's head moves significantly, the method based on global facial features is prone to increased estimation error due to changes in posture. On the other hand, when the lighting conditions change drastically, the eyes are partially obscured, or the image quality is poor, the method based on local eye features often fails due to the failure to extract the pupil or light spot.

[0004] On the other hand, existing methods generally lack effective measurement and fusion mechanisms for the reliability of line-of-sight estimation results, and cannot adaptively select the optimal estimation source in different scenarios, resulting in insufficient robustness and stability of the system in complex usage environments. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an eye-tracking method and system based on multimodal fusion, which addresses the problems of reliance on a single modality, susceptibility to interference, lack of adaptability, and insufficient robustness mentioned in the background art through the following solutions.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an eye-tracking method based on multimodal fusion, comprising:

[0007] S1: By synchronously setting the image acquisition unit and motion sensing unit, the user's facial image sequence, eye image sequence and head motion parameter sequence are acquired to form the original multimodal signal set, and the original multimodal signal set is preprocessed.

[0008] S2: Based on the preprocessed original multimodal signal set, execute in parallel:

[0009] The facial image sequence is subjected to a first feature analysis to obtain a first gaze direction vector and a first reliability measure based on global facial features;

[0010] A second feature analysis is performed on the eye image sequence to obtain a second gaze direction vector based on local eye features and a second reliability measure;

[0011] The spatial posture transformation matrix is ​​obtained by solving the head motion parameter sequence;

[0012] S3: Based on the numerical relationship between the first reliability metric and the second reliability metric, dynamically select and execute one of the predefined multiple data fusion strategies to combine the first line of sight vector and the second line of sight vector into an initial line of sight direction;

[0013] S4: Map the initial gaze direction and spatial posture transformation matrix to the display plane to obtain the initial gaze coordinates; at the same time, perform visual saliency analysis on the content currently presented on the display plane to generate a probability map; match and verify the initial gaze coordinates with the probability map, and when the matching degree exceeds the set tolerance, optimize and calibrate the initial gaze coordinates according to the probability map to output the final gaze point coordinates.

[0014] Preferably, the present invention further includes an eye-tracking system based on multimodal fusion, characterized in that it comprises:

[0015] Signal acquisition and processing module: used to synchronously acquire the user's facial image sequence, eye image sequence and head motion parameter sequence, and to perform time synchronization and noise filtering on various sequence data;

[0016] Parallel computing module: This parallel computing module includes:

[0017] A global visual analysis unit is used to perform a first feature analysis on the facial image sequence and output a first gaze direction vector and a first reliability measure.

[0018] A local visual analysis unit is used to perform a second feature analysis on the eye image sequence and output a second gaze direction vector and a second reliability metric.

[0019] The motion analysis unit is used to solve the head motion parameter sequence and output the spatial posture transformation matrix;

[0020] Multi-strategy fusion decision module: used to receive the first reliability metric and the second reliability metric; dynamically select a strategy from a variety of pre-stored fusion strategies according to the numerical relationship between the two, and perform fusion processing on the two received line-of-sight direction estimates to output the initial line-of-sight direction;

[0021] Context-aware output optimization module: This module includes:

[0022] A coordinate mapping unit is used to convert the initial gaze direction into initial gaze coordinates on the display plane using the spatial pose transformation matrix;

[0023] The scene analysis unit is used to perform visual saliency analysis on the current content of the display plane and generate a probability map;

[0024] The decision output unit is used to match the initial gaze coordinates with the probability map, and decide whether to optimize the initial gaze coordinates based on the matching result, and output the final gaze point coordinates.

[0025] The technical effects and advantages of this invention are as follows:

[0026] 1. This invention constructs a complementary multimodal signal source by fusing three types of modal information: global facial features, local eye features, and head motion parameters. By dynamically selecting the optimal fusion strategy using reliability metrics, it effectively overcomes the limitations of a single modality in complex scenarios such as large head movements, lighting changes, and partial occlusion, achieving higher accuracy in gaze estimation and significantly improving the stability and adaptability of the system in different usage environments.

[0027] 2. This invention innovatively proposes a multi-strategy fusion mechanism based on reliability metrics, enabling the system to select the most suitable fusion method in real time according to signal quality. This adaptive strategy avoids the rigidity problem of traditional fixed weights or single models, ensuring reliable line-of-sight direction output even under various interference conditions;

[0028] 3. This invention generates an attention probability map by introducing visual saliency analysis of the displayed content and performs matching verification and optimization calibration on the initial gaze coordinates. This mechanism enables the system to combine semantic information of the visual scene to logically correct the physiological gaze estimation results, thereby better conforming to real human gaze behavior and further improving the accuracy and naturalness of the gaze point output. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall method of the present invention;

[0030] Figure 2 This is a schematic diagram of the signal acquisition and preprocessing process of the present invention;

[0031] Figure 3 This is a schematic diagram of the dynamic fusion strategy process of the present invention;

[0032] Figure 4 This is a schematic diagram of the overall modules of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] As attached Figure 1-4 The illustrated eye-tracking method based on multimodal fusion includes:

[0035] S1: By synchronously setting the image acquisition unit and motion sensing unit, the user's facial image sequence, eye image sequence and head motion parameter sequence are acquired to form the original multimodal signal set, and the original multimodal signal set is preprocessed.

[0036] It should be specifically noted that the user's facial image sequence consists of consecutive frame images captured by a front-facing RGB camera;

[0037] The eye image sequence consists of consecutive frame images, including clear pupil and corneal reflective spots, captured by an infrared camera and matching infrared light source at the near-focus of the user's eye.

[0038] The sequence of head motion parameters was collected by an inertial measurement unit fixed to the user's head. The sequence includes triaxial acceleration, triaxial angular velocity, and triaxial Euler angle data calculated by sensor fusion.

[0039] It should be further noted that the RGB camera has a resolution of 1920*1080, a frame rate of 30fps, and a field of view of 78°; the specific data acquired is a continuous RGB frame image, denoted as I. face (t), where t takes values ​​of 1, 2, ..., N, and N represents the number of frames collected.

[0040] The near-focus infrared camera has a resolution of 640*512, a frame rate of 60fps, a focal length of 4mm, and a matching infrared light source with a wavelength of 850nm and a power of 100mW. It acquires continuous infrared frame images containing clear pupil and corneal reflective spots, denoted as I. eye (t).

[0041] It should be further noted that the sampling rate of the inertial measurement unit is 100Hz, the acceleration range is set to ±16g, and the angular velocity range is set to ±2000° / s; the acquired triaxial accelerations are represented by a... x a y a z The angular velocity of the three axes is represented by ω. x ω y ωz The triaxial Euler angles obtained through sensor fusion calculation are represented by α, β, and γ.

[0042] It should be further explained that the data acquisition synchronization control achieves time alignment between the RGB camera, infrared camera, and IMU through hardware trigger signals. For each synchronization pulse generated, the three types of devices simultaneously acquire one set of data, ensuring that the IMU at the same time t is synchronized. face (t), I eye (t), a x (t), a y (t), a z (t), ω x (t), ω y (t), ω z (t), α(t), β(t), γ(t) are in one-to-one correspondence.

[0043] It should be further explained that the preprocessing of the acquired raw multimodal signal set includes noise removal, distortion removal, and spatiotemporal bias removal;

[0044] Image denoising: The RGB image of the face is filtered using a 3*3 Gaussian filter, and the pixel values ​​after denoising are... Where (x, y) are pixel coordinates, and σ is the standard deviation of the Gaussian function, σ = 1.2. These are the original pixel values.

[0045] The infrared image of the eye was filtered using median filtering with a window size of 5x5 to suppress salt-and-pepper noise. The pixel values ​​after denoising were... ,in The value is the original pixel value at coordinates (x, y), and median represents the median value of the pixels within the window.

[0046] It should be further explained that σ=1.2 was determined based on 1000 sets of typical facial image samples, covering control experiments with different illumination intensities of 100-1000 lux, different skin colors, and different head postures; the candidate range of σ was selected as 0.8-2.0, and 3*3 Gaussian filtering was performed on 1000 sets of original facial images with different σ values ​​to obtain multiple sets of filtered images; through statistical analysis of the indicators of all candidate σ values, it was found that when σ=1.2, the noise suppression effect met the standard, effectively filtering sensor noise and illumination fluctuations in RGB images; key facial features were preserved intact; the downstream first gaze vector estimation error was minimized, fully meeting the accuracy requirements of the subsequent "dynamic multi-strategy fusion" stage.

[0047] Image distortion correction is performed on both RGB and infrared cameras. The intrinsic parameter matrix K and distortion coefficients, including radial distortion k1, k2, k3, and tangential distortion p1, p2, are pre-obtained using the Zhang Zhengyou calibration method. The correction process consists of three steps: First, the pixel coordinates are converted to normalized coordinates and radial distortion is eliminated. The calculation formula is as follows: ,in (x, y) are the original distorted pixel coordinates. The coordinates are after radial correction; then, tangential distortion is eliminated, and the calculation formula is: The corrected normalized coordinates are obtained. Finally, the normalized coordinates are converted into corrected pixel coordinates: Where (u,v) are the corrected pixel coordinates, and K is a 3*3 intrinsic parameter matrix in the form of... , where f x and f y The focal lengths are the x-axis and y-axis, respectively. x ,c y () represents the pixel coordinates of the image center.

[0048] In motion parameter processing, the IMU data is first denoised: Kalman filtering is used to suppress zero-drift noise from the accelerometer and gyroscope, and the state vector is set to... The observation vector is the original IMU sample value. The process noise covariance matrix Q and the observation noise covariance matrix R are set to 10 according to experimental standards. -6 I6 and 10 -3 I6, where I6 is a 6th-order identity matrix; then time alignment is performed: since the IMU sampling rate of 100Hz is inconsistent with the image frame rate of 30fps for RGB and 60fps for infrared, a linear interpolation method is used to resample the IMU data. Specifically, for each RGB image frame time t, at an interval of 1 / 30 seconds, two IMU data points before and after it are selected. and And using interpolation formulas The aligned data is calculated, and this process ensures that each frame of the image corresponds to a set of time-synchronized head motion parameters.

[0049] S2: Based on the preprocessed original multimodal signal set, execute in parallel:

[0050] The facial image sequence is subjected to a first feature analysis to obtain a first gaze direction vector and a first reliability measure based on global facial features;

[0051] A second feature analysis is performed on the eye image sequence to obtain a second gaze direction vector based on local eye features and a second reliability measure;

[0052] The spatial posture transformation matrix is ​​obtained by solving the head motion parameter sequence.

[0053] It should be specifically noted that the first feature analysis extracts global facial features through a facial pose robust network and regresses the first gaze direction vector. And the first reliability metric R1; where V1 is a unit vector in the facial coordinate system, representing the direction of the gaze, and R1 takes a value of 0-1, with a higher value indicating stronger reliability.

[0054] It should be further explained that in the network structure of the facial pose robust network, the backbone network uses ResNet-50, and the input is a preprocessed RGB image of the face cropped to 224*224 pixels; a spatial attention module is inserted after the 4th layer of ResNet-50, and this module uses the formula... The attention weights for each pixel in the feature map are calculated, where F is the feature map output by the backbone network, σ is the Sigmoid function, Conv2d is a 1*1 convolution, MaxPool2d is the max pooling function, and AvgPool2d is the average pooling function, thus highlighting key areas such as the eyes and the tip of the nose. The network then splits into two branches. The head rotation estimation branch passes the attention-weighted feature map through three fully connected layers, with output dimensions of 1024, 256, and 3, respectively, to regress the Euler angles of the head in the world coordinate system. , , The gaze vector regression branch combines the estimated head Euler angles with the 3D coordinates of 68 facial key points (left eye corners (36,39), right eye corners (42,45), and nose tip (30)) extracted from the attention module to regress the gaze vector V1 in the facial coordinate system.

[0055] It should be further explained that the 3D coordinates of facial key points are obtained by mapping 2D key points to 3D coordinates through a pre-trained 3D facial model. Where k=1, ..., 68, the origin of the facial coordinate system is set at the tip of the nose, the x-axis is to the right, the y-axis is upward, and the z-axis is forward; the gaze vector regression is based on the center of the eyes. Starting from P, the direction vector is output through the gaze branch of FPR-Net, and then normalized to obtain V1, where P left_center P is the midpoint between key points 36 and 39. right_center The midpoint between key points 42 and 45; the normalization formula is: ,in The original vector output by the network; the first reliability metric R1 combines the accuracy of facial landmark detection with the head rotation estimation error, and is calculated using the following formula: A key To improve the accuracy of facial landmark detection, Erot For the head rotation estimation error, θ max This represents the maximum permissible error.

[0056] It should be further noted that the accuracy rate A for facial landmark detection is... key The calculation is the number of successfully detected keypoints (keypoints with IOU > 0.8) / total number of keypoints, with a value ranging from 0 to 1; head rotation estimation error E rot Calculation for predicting Euler angles , , Compared with the true value α gt ,β gt γ gt Euclidean distance, Maximum permissible error θ max Set to 10°, experimental verification was conducted when E rot When the angle is less than 10°, the line-of-sight estimation error is ≤2°. It takes values ​​from 0 to 1.

[0057] It should be further explained that the accuracy weight for facial keypoint detection is 0.6 and the error term weight is 0.4. This was achieved by selecting 1000 sets of samples covering complex scenes, with illumination ranging from 100-1000 lux, head angular velocity from 0-60° / s, and 50 users aged 20-50 years. Each sample set includes: a pre-processed facial RGB image and the actual gaze error E obtained through an external high-precision motion capture system. gt E gt Given the angle between V1 and the true view vector, calculate R1 and E under different weight combinations. gt negative correlation coefficient R 2 R 2 The closer R is to 1, the more accurately R1 reflects the reliability of V1; experiments show that when the weights are 0.6:0.4, R... 2 Reaching a maximum of 0.85 and having the smallest average line-of-sight error in high-confidence scenarios, this weight combination demonstrates that it can most accurately quantify the confidence of V1 and is the optimal solution verified by experiments.

[0058] It should be specifically noted that the second feature analysis uses a two-stage eye analysis network to segment the pupil and corneal reflective spot in the eye region, and combines this with geometric modeling to obtain the second gaze direction vector. And the second reliability metric R2, with a value of 0-1; the two-stage process of the two-stage eye analysis network includes the following stages:

[0059] Stage 1: Pupil and Light Spot Segmentation

[0060] The input is a preprocessed infrared image of the eye, cropped to 128*128 pixels, focusing on the monocular region. U-Net is used as the segmentation network, and the output is a 2-channel feature map: Channel 1: pupil probability map, Channel 2: spot probability map. Pupil and spot masks are obtained through threshold segmentation with a threshold set to 0.5. , M pupil For the pupil mask, M glint For the light spot mask, P pupil (x,y), P glint (x,y) represents the probability value output by the segmentation network;

[0061] Phase 2: Geometric Modeling and Line-of-Sight Vector Calculation

[0062] First, let's look at the mask M. pupil M glint Calculate its centroid as the central coordinate to obtain the pupil center. The origin of the eye coordinate system is the corneal vertex, the x-axis is to the right, the y-axis is upward, and the z-axis is outward; , , Given the distance from the pupil to the corneal apex Corneal reflective spot center The light spot is the reflection point of the infrared light source on the corneal surface. Given that the corneal radius of curvature r = 7.8 mm and the distance from the light source to the corneal apex L = 100 mm, , , Derived from the geometric reflection law, the z-axis coordinate is a fixed value of 8.47mm; the second line of sight direction vector V2 is calculated, and the line of sight direction coincides with the backward extension of the line from the pupil center to the light spot center. Therefore, V2 is the normalized vector of (PC), as shown in the formula: ;

[0063] The second reliability metric, R², combines the segmentation quality with the geometric fit residuals. IOU pupil The intersection-union ratio (IUU) for pupil segmentation is calculated as the area of ​​intersection between the segmented pupil region and the manually labeled region, divided by the area of ​​union, with a value between 0 and 1; E fit The geometric fitting residual is calculated as the distance from the actual pupil edge point to a pupil radius r centered at point P. pupil =4mm average distance of a circle edge is the set of actual pupil edge points, δ max The maximum permissible residual is set to 1 mm, therefore It takes values ​​from 0 to 1.

[0064] It should be specifically noted that the spatial attitude transformation matrix adopts a vision-inertial tight coupling algorithm, which fuses the preprocessed IMU data and visual attitude data to calculate the spatial attitude transformation matrix T as a 4*4 matrix, realizing the transformation from the facial coordinate system to the world coordinate system and the display plane coordinate system.

[0065] It should be further explained that in the vision-inertial tightly coupled algorithm, the state vector is defined as... Where α, β, γ are the Euler angles of the head, and t x , t y , t z This is the head translation vector. To achieve zero bias in the IMU accelerometer, To ensure zero bias in the gyroscope, during the prediction step, the prior estimate of the state vector is updated based on the angular velocity ω and acceleration a through IMU integration. The specific formula is as follows: , , , Where Δt = 0.01s is the IMU sampling interval. The x-axis velocity is obtained by integrating the velocity and acceleration from the previous moment; in the update step, the head Euler angles output by FPR-Net are used. As observed values, the observation residuals are calculated. The prior estimate is corrected using the update equation of the extended Kalman filter, and the posterior estimate is finally obtained. This refers to the head pose and position results output by the algorithm.

[0066] The spatial attitude transformation matrix T consists of a rotation matrix R (3*3) and a translation vector t (3*1), and its homogeneous coordinate form is as follows: The rotation matrix R is the Euler angle output by the VITC algorithm. , , The calculations show a rotation sequence following roll α-pitch β-yaw γ, with the basic rotation matrices for each axis defined as follows: , , The overall rotation matrix is The translation matrix t is derived from the head translation vector output by VITC. , , Composition, that is The unit is millimeters, and the origin of the world coordinate system is set to the center of the display plane.

[0067] S3: Based on the numerical relationship between the first reliability metric and the second reliability metric, dynamically select and execute one of the predefined multiple data fusion strategies to combine the first line-of-sight vector and the second line-of-sight vector into an initial line-of-sight direction.

[0068] It should be specifically noted that the various data fusion strategies include:

[0069] High-confidence collaboration strategy: When both the first and second reliability metrics are higher than the target, the first and second line-of-sight direction vectors are fused to generate an initial line-of-sight direction by maximizing directional consistency.

[0070] Master-slave compensation strategy: When the difference between two reliability metrics is greater than a preset difference threshold, the high-confidence vector is used as the master, and the low-confidence vector is fused after nonlinear compensation using the spatial attitude transformation matrix.

[0071] Conflict arbitration strategy: When both reliability metrics are below the low threshold, select the vector with higher relevance as the initial gaze direction based on the head movement trend.

[0072] It should be further explained that the fusion threshold was set through statistical analysis of 1000 experimental samples, covering different lighting conditions and head movement states, and the following threshold was set:

[0073] High threshold When R ≥ 0.8, the estimation error of the line-of-sight vector is ≤ 1°;

[0074] low threshold When R ≤ 0.4, the estimation error of the line-of-sight vector is ≥ 3°;

[0075] Difference threshold :when At that time, the credibility of the two vectors differed significantly.

[0076] It should be further explained that the high-confidence cooperation strategy is when R1≥T high And R2≥T high In this case, the strategy aims to maximize the directional consistency between V1 and V2, employing a weighted average fusion method based on reliability metrics; the fusion formula is as follows: ,in Transform the vector V1 in the facial coordinate system to the world coordinate system. Transform the vector V2 in the eye coordinate system to the world coordinate system, where This is the eye-face offset matrix obtained through device calibration.

[0077] The master-slave compensation strategy is when In this strategy, the high-confidence vector is prioritized, and the low-confidence vector is compensated for deviation through the spatial pose transformation matrix. First, the master-slave vectors are determined: if... ,set up , Otherwise, switch master and slave roles; then perform nonlinear compensation on the slave vector. The compensation coefficient k was experimentally calibrated to be 0.7. This represents the projection of vector a onto the direction of vector b; finally, the output is fused. The weight of the principal vector is set to 0.8, and the weight of the compensated slave vector is 0.2.

[0078] It should be further explained that the principal vector weight was set to 0.8, and the compensated slave vector weight was 0.2. This was achieved by constructing 1000+ sets of experimental samples, covering typical scenarios such as "slow head movement (angular velocity ≤30° / s)," "fast head movement (angular velocity 30-60° / s)," "strong light interference (1000-2000 lux)," and "weak light interference (100-300 lux)." For each weight combination, the average gaze error and standard deviation of the error were calculated for each combination in all scenarios. By comparing the indicators of each combination, it was found that when the principal vector weight was set to 0.8 and the compensated slave vector weight was 0.2, the average gaze error was the smallest, the standard deviation of the error was the smallest, and the robustness was the best.

[0079] The conflict arbitration strategy is when R1≤T low And R2≤T low At this time, this strategy selects the vector with higher correlation to the direction of head movement as the output based on the head movement trend. First, the head movement direction vector is calculated: based on the IMU angular velocity vector Normalization yields Next, the correlation is calculated, specifically V1, V2, and V... motion The cosine similarity, i.e. , Finally, select the output; if S1 > S2, then... ,otherwise .

[0080] S4: Map the initial gaze direction and spatial posture transformation matrix to the display plane to obtain the initial gaze coordinates; at the same time, perform visual saliency analysis on the content currently presented on the display plane to generate a probability map; match and verify the initial gaze coordinates with the probability map, and when the matching degree exceeds the set tolerance, optimize and calibrate the initial gaze coordinates according to the probability map to output the final gaze point coordinates.

[0081] It should be further explained that, to calculate the initial gaze coordinates, the initial gaze direction vector V is first... initMapping the spatial attitude transformation matrix T onto the display plane yields the initial gaze coordinates (u). pre v pre The display plane is rectangular, and its parameters include resolution W*H and physical size W. phy *H phy Its coordinate system origin is located at the top left corner, the x-axis is to the right, the y-axis is downward, and the z-axis is perpendicular to the screen and points outward, consistent with the z-axis of the world coordinate system. The mapping process consists of the following steps: First, construct the line-of-sight ray equation in the world coordinate system. ,in Let T be the translation matrix from T, where s≥0 is the ray parameter; next, find the intersection point of the ray and the display plane, assuming the z-axis coordinate of the display plane in the world coordinate system is z. disp , let P ray The z-coordinate of (s) is equal to z disp Solving for Because the gaze is directed towards the screen, v initz >0, s is positive; then, the world coordinates of the intersection point are... Convert to display plane physical coordinates , Finally, the physical coordinates are mapped to pixel coordinates. , Ensure that the pixel coordinates corresponding to the center of the screen are (W / 2, H / 2).

[0082] It should be further explained that visual saliency analysis uses a pre-trained visual saliency model to analyze the current content on the display plane and generate a heatmap representing the probability of each region attracting attention. ,in A higher value indicates that the region is more likely to be viewed; the model input is the current frame image of the display plane, with a resolution of W*H, consistent with the screen, and the output is a grayscale image of the same resolution. The grayscale value of each pixel is obtained through... Converted to probability values; the model was trained using the MIT Saliency Benchmark dataset, which contains 10,000 natural images and manually annotated salient regions. The backbone network was ResNet-18, and the output layer used 1×1 convolution and the Sigmoid activation function to ensure the output probability range. The loss function was binary cross-entropy, the optimizer was Adam, and the learning rate was 1e-4.

[0083] It should be further explained that during the matching and verification phase, the preliminary gaze coordinates (u) are calculated. pre v pre The probability value corresponding to ) ( (This indicates rounding down), and is compared with the tolerance threshold T. match =0.5 Comparison: If S pre <Tmatch If the initial coordinates are not reliable, then the reliability is low, and the coordinates should be output directly (u). pre v pre ) as the final coordinates; if S pre ≥T match Then, coordinate optimization is performed. During the optimization phase, the probability map is used as the weight to perform a weighted average of the initial coordinates' 3×3 pixel neighborhood. , The weight of each pixel in the neighborhood is its saliency probability, making the optimized coordinates closer to the high attention region; finally, ufinal and vfinal are cropped to ensure they are within the range [0, W) and [0, H), and the output (u final v final () serves as the final gaze point coordinate for single-frame eye tracking.

[0084] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0085] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An eye-tracking method based on multimodal fusion, characterized in that, include: S1: By synchronously setting the image acquisition unit and motion sensing unit, the user's facial image sequence, eye image sequence and head motion parameter sequence are acquired to form the original multimodal signal set, and the original multimodal signal set is preprocessed. S2: Based on the preprocessed original multimodal signal set, execute in parallel: The facial image sequence is subjected to a first feature analysis to obtain a first gaze direction vector and a first reliability measure based on global facial features; The first feature analysis is implemented through a facial pose robust network. This network extracts and spatially corrects facial key points through a spatial attention mechanism, and then integrates head rotation estimation to jointly regress the first gaze direction vector and its first reliability measure. A second feature analysis is performed on the eye image sequence to obtain a second gaze direction vector based on local eye features and a second reliability measure; The second feature analysis is implemented through a two-stage network. The network first segments the pupil and light spot in the eye region, then converts the segmentation result into a second gaze direction vector through geometric modeling, and calculates the second reliability metric by combining the segmentation quality and the fitting residual. The spatial posture transformation matrix is ​​obtained by solving the head motion parameter sequence; S3: Based on the numerical relationship between the first reliability metric and the second reliability metric, dynamically select and execute one of the predefined multiple data fusion strategies to combine the first line of sight vector and the second line of sight vector into an initial line of sight direction; The various data fusion strategies include: High-confidence collaboration strategy: When both the first and second reliability metrics are higher than the high threshold, the first and second line-of-sight direction vectors are fused to generate the initial line-of-sight direction by maximizing directional consistency. Master-slave compensation strategy: When the difference between two reliability metrics is greater than a preset difference threshold, the high-confidence vector is used as the master, and the low-confidence vector is fused after nonlinear compensation using the spatial attitude transformation matrix. Conflict arbitration strategy: When both reliability metrics are below the low threshold, select the vector with higher relevance as the initial gaze direction based on the head movement trend; S4: Map the initial gaze direction and spatial posture transformation matrix to the display plane to obtain the initial gaze coordinates; at the same time, perform visual saliency analysis on the content currently presented on the display plane to generate a probability map; match and verify the initial gaze coordinates with the probability map, and when the matching degree exceeds the set tolerance, optimize and calibrate the initial gaze coordinates according to the probability map to output the final gaze point coordinates.

2. The eye-tracking method based on multimodal fusion according to claim 1, characterized in that: The user's facial image sequence consists of consecutive frame images captured by a front-facing RGB camera; The eye image sequence consists of consecutive frame images, including clear pupil and corneal reflective spots, captured by an infrared camera and matching infrared light source at the near-focus of the user's eye. The sequence of head motion parameters was collected by an inertial measurement unit fixed to the user's head. The sequence includes triaxial acceleration, triaxial angular velocity, and triaxial Euler angle data calculated by sensor fusion.

3. The eye-tracking method based on multimodal fusion according to claim 1, characterized in that: The solution of the head motion parameter sequence adopts a vision-inertial tightly coupled algorithm, which calculates the spatial attitude transformation matrix by fusing inertial measurement unit data and visual attitude data.

4. The eye-tracking method based on multimodal fusion according to claim 1, characterized in that: The visual saliency analysis uses a pre-trained deep learning model to extract features from the content currently presented on the display plane, generating a visual saliency heatmap that represents the probability of each region in the image attracting attention.

5. An eye-tracking system based on multimodal fusion, characterized in that, include: Signal acquisition and processing module: used to synchronously acquire the user's facial image sequence, eye image sequence and head motion parameter sequence, and to perform time synchronization and noise filtering on various sequence data; Parallel computing module: This parallel computing module includes: A global visual analysis unit is used to perform a first feature analysis on the facial image sequence and output a first gaze direction vector and a first reliability measure. The first feature analysis is implemented through a facial pose robust network. This network extracts and spatially corrects facial key points through a spatial attention mechanism, and then integrates head rotation estimation to jointly regress the first gaze direction vector and its first reliability measure. A local visual analysis unit is used to perform a second feature analysis on the eye image sequence and output a second gaze direction vector and a second reliability metric. The second feature analysis is implemented through a two-stage network. The network first segments the pupil and light spot in the eye region, then converts the segmentation result into a second gaze direction vector through geometric modeling, and calculates the second reliability metric by combining the segmentation quality and the fitting residual. The motion analysis unit is used to solve the head motion parameter sequence and output the spatial posture transformation matrix; Multi-strategy fusion decision module: used to receive the first reliability metric and the second reliability metric; dynamically select a strategy from a variety of pre-stored fusion strategies according to the numerical relationship between the two, and perform fusion processing on the two received line-of-sight direction estimates to output the initial line-of-sight direction; The various data fusion strategies include: High-confidence collaboration strategy: When both the first and second reliability metrics are higher than the high threshold, the first and second line-of-sight direction vectors are fused to generate the initial line-of-sight direction by maximizing directional consistency. Master-slave compensation strategy: When the difference between two reliability metrics is greater than a preset difference threshold, the high-confidence vector is used as the master, and the low-confidence vector is fused after nonlinear compensation using the spatial attitude transformation matrix. Conflict arbitration strategy: When both reliability metrics are below the low threshold, select the vector with higher relevance as the initial gaze direction based on the head movement trend; Context-aware output optimization module: This module includes: A coordinate mapping unit is used to convert the initial gaze direction into initial gaze coordinates on the display plane using the spatial pose transformation matrix; The scene analysis unit is used to perform visual saliency analysis on the current content of the display plane and generate a probability map; The decision output unit is used to match the initial gaze coordinates with the probability map, and decide whether to optimize the initial gaze coordinates based on the matching result, and output the final gaze point coordinates.

Images