A method and system for calculating line-of-sight coordinates based on multimodal fusion

Abstract

This invention discloses a method and system for calculating gaze point coordinates based on multimodal fusion, specifically relating to the field of human-computer interaction technology. The invention obtains a first multimodal gaze dataset through an image acquisition subunit, a sensor acquisition subunit, and a physiological signal acquisition subunit. This dataset is then preprocessed to obtain a second multimodal gaze dataset. Based on the second multimodal gaze dataset, local gaze data fusion is performed to obtain fused eye feature data, followed by global gaze data fusion to obtain global gaze confidence coordinates. Finally, gaze point coordinates are calculated and optimized using a pre-established individual model library and an adaptive algorithm model, resulting in optimized gaze point coordinates that are output to the interactive system for human-computer interaction. This invention improves the stability and adaptability of computation by employing a hierarchical multimodal data fusion approach. Local fusion improves the accuracy of eye feature localization, while further fusion at the global level.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a method and system for calculating gaze point coordinates based on multimodal fusion. Background Technology

[0002] Eye-tracking technology, a key technology in the field of human-computer interaction, captures and analyzes eye movement information, such as eye / pupil position, pupil changes, iris angle, gaze trajectory, and corneal reflection point. Then, through algorithms, it measures the gaze direction, fixation point position, and eye movement pattern in real time, realizing intelligent control of "gaze as interaction".

[0003] Existing gaze tracking solutions mainly rely on a single modality, such as: image-based solutions, which capture eye images through a camera and extract features such as the pupil and corneal reflection points to calculate the gaze direction; and inertial sensor-based solutions, which measure the angular velocity and acceleration of eye movements using gyroscopes and accelerometers to achieve gaze tracking. Existing technologies have attempted to simply combine two modalities, but these are usually just simple data-level complementarity and fail to address the differences in characteristics and spatiotemporal scales of data from different modalities, as well as the potential impact of changes in the user's physiological state on model performance.

[0004] While existing gaze tracking solutions can address the issue of tracking gaze direction, several problems remain. Firstly, single-modal data acquisition is susceptible to environmental interference and individual differences, leading to insufficient accuracy and stability in gaze point coordinate calculation. Furthermore, differences in individual eye physiological characteristics, such as pupil size and corneal curvature, also interfere with gaze point coordinate calculation. Secondly, traditional gaze point coordinate calculation methods often lack effective fusion and utilization of multimodal data, failing to fully leverage the complementarity between different modalities and limiting the performance improvement of eye-tracking technology. Therefore, a gaze tracking solution capable of deep collaborative fusion and adaptive to changes in environment and user state is needed. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a method and system for calculating line-of-sight coordinates based on multimodal fusion, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for calculating line-of-sight coordinates based on multimodal fusion, comprising:

[0007] S1: Through data acquisition technology, multimodal gaze data is acquired in real time, including image acquisition subunit, sensor acquisition subunit and physiological signal acquisition subunit, to obtain the first multimodal gaze dataset;

[0008] S2: Preprocess the acquired multimodal gaze dataset, including image data denoising, sensor data filtering, and physiological signal feature extraction, to obtain the second multimodal gaze dataset;

[0009] S3: Based on the second multimodal gaze dataset, the denoised image data and the filtered sensor data are locally fused to obtain the fused eye feature data;

[0010] S4: Globally fuse the fused eye feature data with the physiological signal features in the second multimodal gaze dataset to construct a multimodal attention state mapping model and obtain the global gaze confidence coordinates of the eye feature data;

[0011] S5: Based on the global gaze confidence coordinates of eye feature data, the gaze point coordinates are calculated and optimized by combining the pre-established individual model library and adaptive algorithm model to obtain the optimized gaze point coordinates;

[0012] S6: Output the optimized gaze point coordinates to the interactive system in real time for human-computer interaction.

[0013] Preferably, a line-of-sight coordinate calculation system based on multimodal fusion includes:

[0014] Multimodal gaze data acquisition module: Through data acquisition technology, it acquires multimodal gaze data in real time, including image acquisition subunit, sensor acquisition subunit and physiological signal acquisition subunit, to obtain the first multimodal gaze dataset, and transmits it to the multimodal gaze data preprocessing module;

[0015] Multimodal gaze data preprocessing module: preprocesses the acquired multimodal gaze dataset, including image data denoising, sensor data filtering and physiological signal feature extraction, to obtain a second multimodal gaze dataset, which is then transmitted to the local gaze data fusion module;

[0016] Local gaze data fusion module: Based on the second multimodal gaze dataset, the denoised image data and the filtered sensor data are locally fused to obtain fused eye feature data, which is then transmitted to the global gaze data fusion module;

[0017] Global gaze data fusion module: Globally fuses the fused eye feature data with the physiological signal features in the second multimodal gaze dataset, constructs a multimodal attention state mapping model, obtains the global gaze confidence coordinates of the eye feature data, and transmits them to the gaze point coordinate calculation module;

[0018] The gaze point coordinate calculation module calculates and optimizes the global gaze confidence coordinates based on eye feature data, combined with a pre-established individual model library and adaptive algorithm model. The optimized gaze point coordinates are then transmitted to the gaze point coordinate interaction module.

[0019] The gaze point coordinate interaction module outputs the optimized gaze point coordinates to the interaction system in real time for human-computer interaction.

[0020] The technical effects and advantages of this invention are as follows:

[0021] 1. This invention collects eye image data, eye movement data, and physiological signal data through an image acquisition subunit, a sensor acquisition subunit, and a physiological signal acquisition subunit, respectively, to obtain a multimodal gaze dataset and perform preprocessing, avoiding the error of single data prediction, accurately capturing the user's true gaze intention, and providing data support for subsequent clustering modeling and adaptive weighting.

[0022] 2. This invention employs a layered fusion approach, fusing image-based eye feature data and sensor-based eye movement data. Sensor data is used to correct and supplement image data, improving the accuracy of eye feature localization. At the global level, the fused eye feature data is further fused with physiological signal-based data, thereby improving computational stability and adaptability.

[0023] 3. This invention uses an adaptive algorithm model to calculate the coordinates of the line of sight. This model can automatically adjust parameters and calculation strategies according to different environments and individual differences. The model will adaptively match and adjust according to a pre-established individual model library to ensure the accuracy of the line of sight coordinate calculation, thereby realizing real-time human-computer interaction. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0025] Figure 2 This is a schematic diagram of the method flow of the present invention.

[0026] Figure 3 This is a schematic diagram of the preprocessing flow for the multimodal gaze dataset of the present invention.

[0027] Figure 4 This is a schematic diagram of the line-of-sight point coordinate calculation process of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see Figure 1 As shown, the present invention provides a line-of-sight coordinate calculation system based on multimodal fusion, including a multimodal line-of-sight data acquisition module, a multimodal line-of-sight data preprocessing module, a local line-of-sight data fusion module, a global line-of-sight data fusion module, a line-of-sight coordinate calculation module, and a line-of-sight coordinate interaction module.

[0030] The multimodal line-of-sight data acquisition module is connected to the multimodal line-of-sight data preprocessing module, the local line-of-sight data fusion module is connected to both the multimodal line-of-sight data preprocessing module and the global line-of-sight data fusion module, and the line-of-sight point coordinate calculation module is connected to both the global line-of-sight data fusion module and the line-of-sight point coordinate interaction module.

[0031] Multimodal gaze data acquisition module: Through data acquisition technology, it acquires multimodal gaze data in real time, including image acquisition subunit, sensor acquisition subunit and physiological signal acquisition subunit, to obtain the first multimodal gaze dataset, and transmits it to the multimodal gaze data preprocessing module;

[0032] Multimodal gaze data preprocessing module: preprocesses the acquired multimodal gaze dataset, including image data denoising, sensor data filtering and physiological signal feature extraction, to obtain a second multimodal gaze dataset, which is then transmitted to the local gaze data fusion module;

[0033] Local gaze data fusion module: Based on the second multimodal gaze dataset, the denoised image data and the filtered sensor data are locally fused to obtain fused eye feature data, which is then transmitted to the global gaze data fusion module;

[0034] Global gaze data fusion module: Globally fuses the fused eye feature data with the physiological signal features in the second multimodal gaze dataset, constructs a multimodal attention state mapping model, obtains the global gaze confidence coordinates of the eye feature data, and transmits them to the gaze point coordinate calculation module;

[0035] The gaze point coordinate calculation module calculates and optimizes the global gaze confidence coordinates based on eye feature data, combined with a pre-established individual model library and adaptive algorithm model. The optimized gaze point coordinates are then transmitted to the gaze point coordinate interaction module.

[0036] The gaze point coordinate interaction module outputs the optimized gaze point coordinates to the interaction system in real time for human-computer interaction.

[0037] Please see Figure 2 As shown, the method for calculating gaze point coordinates based on multimodal fusion includes: S1: Real-time acquisition of multimodal gaze data, including image acquisition subunit, sensor acquisition subunit, and physiological signal acquisition subunit, to obtain a first multimodal gaze dataset; S2: Preprocessing the acquired multimodal gaze dataset, including image data denoising, sensor data filtering, and physiological signal feature extraction, to obtain a second multimodal gaze dataset; S3: Locally fusing the denoised image data and filtered sensor data based on the second multimodal gaze dataset to obtain fused eye feature data; S4: Globally fusing the fused eye feature data with the physiological signal features in the second multimodal gaze dataset to construct a multimodal attention state mapping model, obtaining global gaze confidence coordinates of the eye feature data; S5: Calculating and optimizing gaze point coordinates based on the global gaze confidence coordinates of the eye feature data, combined with a pre-established individual model library and adaptive algorithm model, to obtain optimized gaze point coordinates; S6: Real-time output of the optimized gaze point coordinates to the interactive system for human-computer interaction.

[0038] S1: Multimodal gaze data is acquired in real time using data acquisition technology, including an image acquisition subunit, a sensor acquisition subunit, and a physiological signal acquisition subunit, to obtain the first multimodal gaze dataset. The image acquisition subunit uses an eye-tracking camera to acquire a real-time eye image sequence I(x,y,t), where (x,y) represents pixel coordinates and t is a timestamp. The sensor acquisition subunit acquires the angular velocity w(t) (in radians / second) and acceleration a(t) (in radians / second) of the eyeball through a miniature inertial measurement unit. 2 ), angular velocity w(t) includes the components of angular velocity data on the three coordinate axes (x, y, z). x (t), w y (t) and w z (t), acceleration a(t) includes the acceleration data components a on the three coordinate axes (x, y, z). x (t), a y (t) and a z (t); The physiological signal acquisition subunit acquires the initial physiological signal E through electroencephalogram (EEG) electrodes and electrooculogram (EOG) electrodes. EEG (t) and E EOG (t); Finally, the multimodal gaze dataset D(t) is obtained, D(t)=[I(x,y,t),w(t),a(t),E EEG (t),E EOG (t)];

[0039] This embodiment specifically addresses the requirements of the eye-tracking camera, including high frame rate and high resolution. Electroencephalogram (EEG) signals are recorded by electrodes placed on the scalp, recording the electrical activity of neurons in the brain. This method is widely used in consumer and research-grade devices (such as NeuroSky, Emotiv, and medical EEG equipment). Electrooculogram (EOG) signals are recorded by electrodes placed around the eyeball, recording the potential difference between the cornea and retina during eye movements. This is a very stable and reliable signal, widely used in sleep monitoring and eye-tracking research. Energy in specific frequency bands (e.g., δ, θ, α, β, etc.) is extracted from these signals. For example, an increase in the energy of theta waves (4–8 Hz) is positively correlated with drowsiness and fatigue.

[0040] Please see Figure 3 As shown, S2: Preprocessing the acquired multimodal gaze dataset, including image data denoising, sensor data filtering, and physiological signal feature extraction, to obtain the second multimodal gaze dataset, includes the following steps:

[0041] S2.1: Image data denoising: The MedianFilter algorithm is used to denoise the eye image sequence I(x,y,t) in the multimodal gaze dataset, resulting in the denoised eye image sequence I. de (x,y,t), I de (x,y,t)=MedianFilter[I(x,y,t),kernel_size], where kernel_size is the size of the median filter window (e.g., 3×3, 5×5, etc., usually an odd number to ensure central symmetry), kernel_size=2×round(σI / 2)+1, where σI represents the standard deviation of Gaussian noise in the eye image, and round represents the rounding function;

[0042] S2.2: Sensor data filtering: The Kalman filter algorithm is used to filter the eye angular velocity w(t) and acceleration a(t) in the multimodal gaze dataset. The filtering is achieved by minimizing the mean square error argmin(e) between the filtered state estimate and the true state. k The filtered optimal eye sensor fusion data x is obtained. h k|k This includes the following steps:

[0043] S2.2.1: The filtered eye sensor fusion data x h k|k Predicting the state equation x using Kalman filtering h k|k-1 and observation update equation x h k|k The two-step iteration yields x hk|k-1 =A k ×x h k-1|k-1 +B k ×u k +W k A k Let x be the state transition matrix at time k. h k-1|k-1 B represents the filtered result of the eye sensor fusion data at time k-1. k Let u be the control input matrix at time k. k The sensor input data at time k includes angular velocity w(k) and acceleration a(k), W k The process noise follows a Gaussian distribution N(0,Q) k ), Q k This is a process covariance diagonal matrix, composed of diagonal elements representing the variances of each element in the state vector, with dimensions 6×6, x h k|k =x h k|k-1 +K k ×(z k -H k ×x h k|k-1 ), K k The Kalman gain matrix, with dimensions 6×6, is calculated from the covariance matrix and reflects the confidence weights between prediction and observation. z k Let z be the sensor observation at time k. k =H k ×x h k|k +V k H k The observation matrix is ​​a 6×6 identity matrix, because the observed values ​​directly correspond to the state vector, V. k To observe the noise, we assume it follows a Gaussian distribution N(0,R). k ), R k To observe the covariance diagonal matrix, the diagonal elements are determined based on the measured noise of the sensor (e.g., the observed noise variance of the accelerometer is 0.01m). 2 / s 4 The observation noise variance of the gyroscope is 0.001 rad. 2 / s 2 );

[0044] This embodiment requires specific explanation of the state transition matrix A when the system is in a stable state (e.g., eye movement speed ≤ 50° / s). k Given a diagonal matrix with diagonal elements all equal to 1, control the input matrix B. kIt is a 6×6 identity matrix; when the system is in a dynamically changing state (e.g., eye movement rate > 50° / s or subjected to external disturbances), A k and B k The element values ​​are dynamically adjusted based on the real-time angular velocity w(t) and acceleration a(t), and the adjustment rule is: A k (i,i)=1+α×|w i (k)|,A k (i,i) represents the element in the i-th row and i-th column, α is the scaling factor (e.g., 0.001), i=1,2,3 correspond to the dimensions of the triaxial acceleration, and w i (k) represents the angular velocity in the i-th dimension at time k, where i = 4, 5, 6 correspond to the dimensions of the three-axis angular velocities. k (j,j)=exp(-δ×σ u (j)), B k (j,j) is the element in the j-th row and j-th column, σ u (j) represents the standard deviation of the j-th dimension sensor input, j=1,2,...,6, and δ is the attenuation coefficient (e.g., δ=50 determined experimentally). In dynamic scenarios such as eye tracking, changes in the angular velocity of the eyeball will affect the evolution of acceleration through physical coupling (such as centrifugal force).

[0045] S2.2.2: By minimizing the mean square error argmin(e k The filtered optimal eye sensor fusion data x is obtained. h k|k x h k|k =[w h (k),a h (k)]=[w h x (k),w h y (k),w h z (k),a h x (k),a h y (k),a h z (k)],w h (k) and a h (k) represents the filtered angular velocity and acceleration fusion values ​​at the three coordinates at time k; e k =E[||x k -x h k|k || 2 E[·] represents the mathematical expectation, ||·|| 2 Let x represent the square norm of a vector.k Let P be the true state vector measured by the sensor; when the error covariance matrix P k|k The trace (sum of diagonal elements) is less than the pre-filtered predicted state covariance matrix P. k|k-1 And less than the observation noise covariance R k If the trace is found, it indicates that the optimality has been verified; otherwise, continue updating and iterating until the optimality is satisfied. k|k =(IK k ×H k )×P k|k-1 I is the identity matrix, and P is the predicted state covariance matrix. k|k-1 The dimension is 6×6, P k|k-1 =A k ×P k-1|k-1 ×A k T +Q k P k-1|k-1 Let be the covariance matrix of the filtered state at time k-1;

[0046] S2.3: Physiological signal feature extraction: Wavelet transform is used to extract physiological signal features, including the EEG feature vector F. EEG and electrooculogram feature vector F EOG F EEG =[mean(β),var(θ)], where β and θ represent the 13-30Hz frequency band signal (typically associated with a conscious, active cognitive state) and the 4-8Hz frequency band signal (a transitional region between full wakefulness and full sleep), respectively. mean(β) is the mean of the signal in this frequency band, and var(θ) is the variance of the signal in this frequency band. F EOG =[peak(E EOG (t)),dur(E EOG (t))],peak(E EOG (t) represents the peak value of the electrooculogram (EOG) signal, reflecting the amplitude of eye movement (the higher the peak value, the greater the possible amplitude of eye movement), dur(E EOG (t) represents the duration of the electrooculogram signal, reflecting the duration of eye movement (the longer the duration, the longer the eye movement process).

[0047] S2.4: Based on the denoised eye image sequence I de (x,y,t), the filtered optimal eye sensor fusion data x h k|k EEG feature vector F EEG and electrooculogram feature vector F EEG This yields the second multimodal gaze dataset;

[0048] S3: Based on the second multimodal gaze dataset, the denoised image data and the filtered sensor data are locally fused to obtain the fused eye feature data, including the following steps:

[0049] S3.1: Based on the second multimodal gaze dataset, from the denoised eye image sequence I de Extracting the pupil position from (x,y,t) using P_img(t)=(x_p,y_p) T The location of the corneal reflex point is G_img(t)=(x_g,y_g). T Then calculate the eye feature vector V_img(t), V_img(t) = P_img(t) - G_img(t), where V_img(t) has a dimension of 2×1. x ,v y ) T , (v x ,v y () represents the coordinates of the local eye's line of sight.

[0050] This embodiment requires a specific explanation of the fact that extracting the pupil center and corneal reflection point coordinates from an eye image using image processing algorithms is a well-known technique in the field. For example, a feasible approach includes: segmenting the image to identify the pupil region, and obtaining the pupil center coordinates P_img(t) using an ellipse fitting algorithm; simultaneously, obtaining the corneal reflection point coordinates G_img(t) through threshold segmentation and centroid calculation.

[0051] S3.2: Based on the second multimodal gaze dataset, at the image frame interval Δt, the filtered sensor data w... h (k) and a h (k) Predict eye movements and construct an eye movement prediction model Δθ pre , Δθ pre =w h (k)×Δt+1 / 2×a h (k)×(Δt) 2 , Δθ pre The three-dimensional rotational angular displacement of the eyeball within a time interval is Δθ, with dimensions of 3×1. pre =(Δθ x ,Δθ y ,Δθ z ) T ;

[0052] S3.3: Based on the three-dimensional rotational angular displacement Δθ of the eyeball pre The eye feature vector V_img(t) is corrected using the Jacobian matrix J and the gain matrix K1 to obtain the fused eye feature data V. fus (t), Vfus (t)=V_img(t)+K1×J×Δθ pre K1 is a diagonal matrix with diagonal elements k1 x and K1 y K1, with a dimension of 2×2, is used to control the weights of the correction and depends on the image quality (e.g., contrast, sharpness). The lower the image quality, the larger the K1 value, and the more dependent it is on sensor data. When the image is clear (high confidence), the K1 value will decrease, making the system more confident in the image measurement results. J, with a dimension of 2×3, is a linear mapping matrix responsible for transforming small rotations (3×1 vectors) in three-dimensional physical space into vector changes (2×1 vectors) on the two-dimensional image plane.

[0053] In this embodiment, the Jacobian matrix J and the gain matrix K1 need to be specifically explained. Each element in matrix J is a partial derivative, representing the amount of pixel displacement in the image coordinate direction caused by rotation in each direction; a feasible implementation of the gain matrix K1 is... k1 x and K1 y The fusion weights used to control the x and y directions are, for example, the average gradient value G of the pupil region is used as the image quality evaluation index Q. G is normalized to the [0,1] interval to obtain Q, K1=K1 max ×(1-Q), K1 max For a preset maximum gain value, including k1 x and K1 y .

[0054] S4: Globally fuse the fused eye feature data with the physiological signal features in the second multimodal gaze dataset to construct a multimodal attention state mapping model and obtain the global gaze confidence coordinates of the eye feature data, including the following steps:

[0055] S4.1: Based on the physiological signal features in the second multimodal gaze dataset, the EEG feature vector F EEG and electrooculogram feature vector F EEG They are respectively quantified as EEG attention weights Att EEG (t) and the electrooculogram attention weight Att EOG (t), Att EEG(t)=a×σ(Norm(mean(β)))+(1-a)×(1-σ(Norm(var(θ))), where a is the EEG dual-feature fusion coefficient (e.g., 0.5), taking values ​​∈ [0,1], σ is the Sigmoid function, and Norm() is the data obtained through the Min-Max normalization method. The minimum and maximum values ​​of the data in the Min-Max normalization method are determined by offline collection of EEG and EEG data in different scenarios to determine the maximum and minimum values ​​of the mean of the β wave and the maximum and minimum values ​​of the variance of the θ wave, while also determining the normalization interval of the EEG features; Att EOG (t)=b×σ(1-Norm(peak(E EOG (t))))+(1-b)×(1-σ(Norm(dur(E EOG (t)))), b is the electrooculogram dual-feature fusion coefficient (e.g., 0.5), with values ​​∈ [0,1]; combined with EEG attention weight Att EEG (t) and the electrooculogram attention weight Att EOG (t), construct a multimodal attention state mapping model Att(t), Att(t) = c × Att EEG (t)+(1-c)×Att EOG (t), where c is the EEG-EOG cross-modal fusion coefficient (e.g., 0.6), with values ​​∈ [0,1];

[0056] S4.2: Combining the multimodal attention state mapping model Att(t) and the fused eye feature data V fus (t), to obtain the global gaze confidence coordinates W_glo of the eye feature data, W_glo=V fus (t)×Att(t)+λ×Norm(peak(E EOG (t)),dur(E EOG (t)))+ε, where W_glo has a dimension of 2×1, and ε is a two-dimensional basic compensation factor (e.g., ε=(0.01,0.01)). T To ensure the reasonableness of the weight values, λ is the compensation coefficient for the scalar electrooculogram characteristics (e.g., λ=0.3).

[0057] Please see Figure 4 As shown, S5: Based on the global gaze confidence coordinates of eye feature data, the gaze point coordinates are calculated and optimized using a pre-established individual model library and adaptive algorithm model to obtain the optimized gaze point coordinates, including the following steps:

[0058] S5.1: Pre-establish an individual model library:

[0059] S5.1.1: Data Construction Phase: Collect physiological signal characteristics (including EEG and EEG characteristics), global gaze confidence coordinates W_glo, and corresponding real gaze point coordinates G from at least m1 users (m1≥50) in different scenarios. tr =(x tr ,y tr Construct a training dataset D, D={(W_glo L G tr,L )} L=1 M M is the total number of samples (M≥1000), and L is the Lth sample.

[0060] In this embodiment, it should be specifically explained that different scenarios refer to diverse scenarios covering different attention states, environmental conditions, and task types; different attention states include focused scenarios, distracted scenarios, neutral scenarios, etc.; different environmental conditions include lighting scenarios (strong light, weak light, normal environment, etc.), noise scenarios, and scenarios with varying distances, etc.; different task types include cognitive tasks (such as mathematical calculations and logical reasoning), visual tasks (such as image browsing and video viewing), and motor tasks (such as limb manipulation and active eye movements), etc.

[0061] S5.1.2: Individual Model Classification and Training: Using a clustering algorithm (e.g., K-means), users are divided into n classes based on their physiological signal characteristics by calculating the similarity of physiological feature vectors using Euclidean distance. The subset D of the L1 class users is then used for... L1 ⊆D, respectively train the viewpoint mapping function f L1 (•), training methods include linear regression, multinomial regression, support vector regression, etc.; then, the mean squared error loss function f is minimized. L1 (W_glo) represents the error between the output gaze point coordinates and the actual gaze point coordinate components of that user class. When the error is less than or equal to the corresponding threshold, the trained gaze point mapping function f for each user class is obtained. L1 (•) and the corresponding model parameters;

[0062] This embodiment requires specific explanation of the physiological signal feature classification types: for example, users of type 1 may share common characteristics such as high mean β wave, low variance θ wave, small peak EOG value, and short duration (corresponding to a highly focused group); users of type 2 may share common characteristics such as low mean β wave, high variance θ wave, large peak EOG value, and long duration (corresponding to a easily distracted group), etc.; for example, training a linear function f on the x-axis component of user type i. L1,x (W_glo)=a L1,x ×W_glo L1,x +b L1,x ×W_glo L1,y +c L1,x a L1,xb L1,x and c L1,x These are the parameters obtained from training; similarly, the function f for training the y-axis components is used. L1,y (•).

[0063] S5.1.3: Establish an individual model library: Map the gaze point function f for each user type L1 (•) model parameters and feature center U of this category L1 Stored together in the individual model library, feature center U L1 It is obtained by the mean value of the global gaze confidence coordinates W_glo of all samples in this category;

[0064] This embodiment requires specific explanation of f. L1 (W_glo) is the L1st gaze point mapping function pre-stored in the individual model library. This function library is trained by pre-collecting calibration data of users with different physiological signal characteristics (multimodal data W_global corresponding to when the user gazes at a known screen coordinate point); each mapping function f L1 (W_glo) is responsible for establishing a non-linear mapping relationship from the fused global feature vector W_global to the screen gaze point coordinates G, representing the gaze behavior pattern of a certain type of user with specific physiological characteristics.

[0065] S5.2: Based on the global gaze confidence coordinates W_glo(t) of the current eye feature data, the adaptive algorithm model G(t) is used to calculate the gaze point coordinates. f L1 (W_glo) represents the coordinates of the line-of-sight point output when W_glo(t) is input, ω L1 (t) is the L1-th type of viewpoint mapping function f in the individual model library. L1 The weight of (•), , sim(W_glo(t),U L1 () represents the cosine similarity. γ is the similarity coefficient; then, γ is adjusted according to the light intensity L to optimize the line-of-sight coordinates G(t), γ(L)=γ0×(1+ζ×log2(L / L0)), ζ is the light intensity sensitivity coefficient (e.g. ζ=0.2); finally, the optimized line-of-sight coordinates G(t) are obtained, including the x-axis component and the y-axis component;

[0066] S6: The optimized gaze point coordinates are output to the interactive system in real time for human-computer interaction, realizing gaze-based human-computer interaction functions. For example, in VR games, users can control the viewpoint and operation of the game character by moving their gaze; in smart office scenarios, users can select menu options and trigger operation commands by looking at their eyes.

[0067] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0068] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for calculating line-of-sight coordinates based on multimodal fusion, characterized in that: include: S1: Through data acquisition technology, multimodal gaze data is acquired in real time, including image acquisition subunit, sensor acquisition subunit and physiological signal acquisition subunit, to obtain the first multimodal gaze dataset; S2: Preprocess the acquired multimodal gaze dataset, including image data denoising, sensor data filtering, and physiological signal feature extraction, to obtain the second multimodal gaze dataset; S3: Based on the second multimodal gaze dataset, the denoised image data and the filtered sensor data are locally fused to obtain the fused eye feature data; The fused eye feature data in S3 includes: S3.1: Based on the second multimodal gaze dataset, from the denoised eye image sequence I de Extracting the pupil position from (x,y,t) using P_img(t)=(x_p,y_p) T The location of the corneal reflex point is G_img(t)=(x_g,y_g). T Then calculate the eye feature vector V_img(t), V_img(t) = P_img(t) - G_img(t), where V_img(t) has a dimension of 2×1. x ,v y ) T , (v x ,v y () represents the coordinates of the local eye's line of sight. S3.2: Based on the second multimodal gaze dataset, at the image frame interval Δt, the filtered sensor data w... h (k) and a h (k) Predict eye movements and construct an eye movement prediction model Δθ pre , Δθ pre =w h (k)×Δt+1 / 2×a h (k)×(Δt) 2 , Δθ pre The three-dimensional rotational angular displacement of the eyeball within a time interval is Δθ, with dimensions of 3×1. pre =(Δθ x ,Δθ y ,Δθ z ) T ; The fused eye feature data in S3 also includes: S3.3: based on the three-dimensional rotational angular displacement Δθ of the eyeball pre The eye feature vector V_img(t) is corrected using the Jacobian matrix J and the gain matrix K1 to obtain the fused eye feature data V. fus (t), V fus (t)=V_img(t)+K1×J×Δθ pre K1 is a diagonal matrix with diagonal elements k1 x and K1 y The dimension of matrix J is 2×2, and the dimension of matrix J is 2×3. Each element in matrix J is a partial derivative, which represents the amount of pixel displacement caused by rotation in each direction in the image coordinate direction. S4: Globally fuse the fused eye feature data with the physiological signal features in the second multimodal gaze dataset to construct a multimodal attention state mapping model and obtain the global gaze confidence coordinates of the eye feature data; S5: Based on the global gaze confidence coordinates of eye feature data, the gaze point coordinates are calculated and optimized by combining the pre-established individual model library and adaptive algorithm model to obtain the optimized gaze point coordinates; S6: Output the optimized gaze point coordinates to the interactive system in real time for human-computer interaction.

2. The method for calculating line-of-sight coordinates based on multimodal fusion according to claim 1, characterized in that: The image acquisition subunit described in S1 employs an eye-tracking camera to acquire a real-time eye image sequence I(x,y,t), where (x,y) represents pixel coordinates and t is a timestamp. The sensor acquisition subunit acquires the angular velocity w(t) and acceleration a(t) of the eyeball via a miniature inertial measurement unit. The angular velocity w(t) includes components of the angular velocity data along the three coordinate axes (x,y,z). x (t), w y (t) and w z (t), acceleration a(t) includes the acceleration data components a on the three coordinate axes (x, y, z). x (t), a y (t) and a z (t); The physiological signal acquisition subunit acquires the initial physiological signal E through electroencephalogram (EEG) electrodes and electrooculogram (EOG) electrodes. EEG (t) and E EOG (t); Finally, the multimodal gaze dataset D(t) is obtained, D(t)=[I(x,y,t),w(t),a(t),E EEG (t),E EOG (t)].

3. The method for calculating line-of-sight coordinates based on multimodal fusion according to claim 1, characterized in that: The implementation of S2 includes: S2.1: Image data denoising: The MedianFilter algorithm is used to denoise the eye image sequence I(x,y,t) in the multimodal gaze dataset to obtain the denoised eye image sequence I. de (x,y,t), I de (x,y,t)=MedianFilter[I(x,y,t),kernel_size], where kernel_size is the size of the median filter window, kernel_size=2×round(σI / 2)+1, where σI represents the standard deviation of Gaussian noise in the eye image, and round represents the rounding function; S2.2: Sensor data filtering: The Kalman filter algorithm is used to filter the eye angular velocity w(t) and acceleration a(t) in the multimodal gaze dataset. The filtering is achieved by minimizing the mean square error argmin(e) between the filtered state estimate and the true state. k The filtered optimal eye sensor fusion data x is obtained. h k|k ; S2.3: Physiological signal feature extraction: Wavelet transform is used to extract physiological signal features, including the EEG feature vector F. EEG and electrooculogram feature vector F EOG F EEG =[mean(β),var(θ)], where β and θ represent the 13-30Hz and 4-8Hz frequency band signals, respectively. mean(β) is the mean of the signal in that frequency band, and var(θ) is the variance of the signal in that frequency band. F EOG =[peak(E EOG (t)),dur(E EOG (t))],peak(E EOG (t) represents the peak value of the electrooculogram signal, dur(E) EOG (t) represents the duration of the electrooculogram signal.

4. The method for calculating line-of-sight coordinates based on multimodal fusion according to claim 1, characterized in that: The global gaze confidence coordinates of the eye feature data in S4 include: S4.1: Based on the physiological signal features in the second multimodal gaze dataset, the EEG feature vector F EEG and electrooculogram feature vector F EEG They are respectively quantified as EEG attention weights Att EEG (t) and the electrooculogram attention weight Att EOG (t), Att EEG (t)=a×σ(Norm(mean(β)))+(1-a)×(1-σ(Norm(var(θ))), where a is the EEG dual-feature fusion coefficient, taking values ​​∈[0,1], σ is the Sigmoid function, and Norm() is the data obtained by the Min-Max normalization method; Att EOG (t)=b×σ(1-Norm(peak(E EOG (t))))+(1-b)×(1-σ(Norm(dur(E EOG (t)))), b is the fusion coefficient of the two features of electrooculography, with a value ∈ [0,1]; combined with the attention weight Att of EEG. EEG (t) and the electrooculogram attention weight Att EOG (t), construct a multimodal attention state mapping model Att(t), Att(t) = c × Att EEG (t)+(1-c)×Att EOG (t), where c is the EEG-EOG cross-modal fusion coefficient, with values ​​∈ [0,1]; S4.2: Combining the multimodal attention state mapping model Att(t) and the fused eye feature data V fus (t), to obtain the global gaze confidence coordinates W_glo of the eye feature data.

5. The method for calculating line-of-sight coordinates based on multimodal fusion according to claim 1, characterized in that: The pre-established individual model library in S5 includes: S5.1.1: Data construction phase: collecting physiological signal characteristics, global gaze confidence coordinates W_glo, and corresponding real gaze point coordinates G of at least m1 users in different scenarios. tr =(x tr ,y tr Construct a training dataset D, D={(W_glo L G tr,L )} L=1 M M is the total number of samples, and L is the Lth sample; S5.1.2: Individual Model Classification and Training: A clustering algorithm is used to divide users into n classes based on their physiological signal characteristics by calculating the similarity of physiological feature vectors using Euclidean distance. The subset D of users in the L1 class is then used. L1 ⊆D, respectively train the viewpoint mapping function f L1 (•); then minimize f using the mean squared error loss function. L1 (W_glo) represents the error between the output gaze point coordinates and the actual gaze point coordinate components of that user class. When the error is less than or equal to the corresponding threshold, the trained gaze point mapping function f for each user class is obtained. L1 (•) and the corresponding model parameters; S5.1.3: Establish an individual model library: Map the gaze point function f for each user type L1 (•) model parameters and feature centers of the categories U L1 Stored together in the individual model library, feature center U L1 It is obtained by the mean value of the global gaze confidence coordinates W_glo of all samples in this category.

6. The method for calculating line-of-sight coordinates based on multimodal fusion according to claim 5, characterized in that: The optimized gaze point coordinates in S5 are calculated using an adaptive algorithm model G(t) based on the global gaze confidence coordinates W_glo(t) of the current eye feature data. f L1 (W_glo) represents the coordinates of the line-of-sight point output when W_glo(t) is input, ω L1 (t) is the L1-th type of viewpoint mapping function f in the individual model library. L1 The weight of (•), , sim(W_glo(t),U L1 Let ) be the cosine similarity and γ be the similarity coefficient; then, adjust γ according to the light intensity L to optimize the line-of-sight coordinates G(t), γ(L)=γ0×(1+ζ×log2(L / L0)), where ζ is the light intensity sensitivity coefficient; finally, the optimized line-of-sight coordinates G(t) are obtained, including the x-axis component and the y-axis component.

7. A line-of-sight coordinate calculation system based on multimodal fusion, used to calculate line-of-sight coordinates using the method for calculating line-of-sight coordinates based on multimodal fusion as described in any one of claims 1-6, comprising: Multimodal gaze data acquisition module: Through data acquisition technology, it acquires multimodal gaze data in real time, including image acquisition subunit, sensor acquisition subunit and physiological signal acquisition subunit, to obtain the first multimodal gaze dataset, and transmits it to the multimodal gaze data preprocessing module; Multimodal gaze data preprocessing module: preprocesses the acquired multimodal gaze dataset, including image data denoising, sensor data filtering and physiological signal feature extraction, to obtain a second multimodal gaze dataset, which is then transmitted to the local gaze data fusion module; Local gaze data fusion module: Based on the second multimodal gaze dataset, the denoised image data and the filtered sensor data are locally fused to obtain fused eye feature data, which is then transmitted to the global gaze data fusion module; Global gaze data fusion module: Globally fuses the fused eye feature data with the physiological signal features in the second multimodal gaze dataset, constructs a multimodal attention state mapping model, obtains the global gaze confidence coordinates of the eye feature data, and transmits them to the gaze point coordinate calculation module; The gaze point coordinate calculation module calculates and optimizes the global gaze confidence coordinates based on eye feature data, combined with a pre-established individual model library and adaptive algorithm model. The optimized gaze point coordinates are then transmitted to the gaze point coordinate interaction module. The gaze point coordinate interaction module outputs the optimized gaze point coordinates to the interaction system in real time for human-computer interaction.

Images