Hybrid wireless eye movement tracking method and system based on multi-sensor fusion, electronic equipment and storage medium

Abstract

The invention provides a hybrid wireless eye movement tracking method and system based on multi-sensor fusion, electronic equipment and a storage medium, and relates to the technical field of eye movement tracking. The method comprises the following steps: firstly, acquiring various data by using various different types of sensors such as a Q-Var sensor, an inertial measurement unit and a high-speed panoramic camera; secondly, preprocessing the various sensor data by means of graph Fourier transform, a Kalman filter, a spectral clustering technology and the like; fusing the preprocessed sensor data based on a graph theory method; and finally, performing fixation track prediction based on a convolutional neural network and a gating circulation unit. According to the technical scheme, the performance of eye movement tracking under different environment conditions is improved, the eye movement tracking precision is higher, and the stability is better. In addition, the technology of the application also uses a 5G millimeter wave technology or an IEEE 802.11 ac protocol to transmit gaze tracking data, thereby achieving the purpose of bandwidth enhancement and low-delay data transmission.

Description

Technical Field

[0001] The present invention relates to the field of eye movement tracking technology, and particularly to a hybrid wireless eye movement tracking method, system, electronic device and storage medium based on multi-sensor fusion. Background Art

[0002] As a key technical means for exploring visual and human behaviors in multiple fields, eye tracking technology is becoming increasingly important. By virtue of its ability to construct a more natural and intuitive gaze interface and provide accurate analysis, eye tracking technology has strongly promoted significant progress in many fields such as human-computer interaction, virtual reality and augmented reality, medical diagnosis, and psychological research. With the continuous development and technological evolution of eye tracking technology, the demand for new systems adapted to different usage scenarios in related fields is also becoming increasingly urgent. These fields and scenarios require the system to have accurate and robust tracking performance, high-speed data transmission capabilities, energy-saving features, and the ability to achieve more natural and intuitive human-computer interaction, etc.

[0003] However, traditional eye tracking systems generally rely on single-modal sensors, resulting in inherent limitations in tracking accuracy and stability. Especially in dynamic environments or when dealing with head movements, such limitations are extremely significant. Moreover, traditional eye tracking systems also have obvious shortcomings in key aspects such as real-time wireless data transmission efficiency and power efficiency.

[0004] In view of this, the present application proposes a hybrid wireless eye movement tracking technical solution based on multi-sensor fusion, aiming to at least solve one or more of the above-mentioned restrictive factors to optimize and break through the existing eye tracking technology. Summary of the Invention

[0005] (1) Technical Problems to be Solved

[0006] Aiming at the deficiencies of the prior art, the present invention provides a hybrid wireless eye movement tracking method, system, electronic device and storage medium based on multi-sensor fusion, which solves at least the technical problems of poor tracking accuracy and poor stability of the existing eye tracking technology.

[0007] (2) Technical Solutions

[0008] To achieve the above object, the present invention is realized through the following technical solutions:

[0009] In a first aspect, the present application first proposes a hybrid wireless eye movement tracking method based on multi-sensor fusion, and the method includes:

[0010] Collect comprehensive data of the user's eye movement and head position based on a multi-sensor including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera;

[0011] Perform data fusion on the comprehensive data;

[0012] Estimate the user's gaze direction based on the fused sensor data.

[0013] In one embodiment, before performing data fusion on the comprehensive data, preprocess the comprehensive data; the preprocessing includes:

[0014] Perform graph Fourier transform on the data collected by the Q-Var sensor; and / or

[0015] Perform Kalman filtering on the data collected by the inertial measurement unit; and / or

[0016] Perform motion blur compensation on the data collected by the high-speed panoramic camera.

[0017] Preferably, the preprocessing further includes:

[0018] Perform adaptive thresholding on the Q-Var sensor data after graph Fourier transform based on spectral clustering technology.

[0019] In one embodiment, performing data fusion on the comprehensive data includes:

[0020] Fuse the preprocessed data of the Q-Var sensor, inertial measurement unit, and high-speed panoramic camera based on a graph theory method.

[0021] In one embodiment, the method further includes: extracting key eye movement features from the fused data.

[0022] In one embodiment, estimating the user's gaze direction based on the fused sensor data includes:

[0023] Predict the user's gaze trajectory by analyzing the fused sensor data based on a convolutional neural network and a gated recurrent unit.

[0024] In one embodiment, the method further includes:

[0025] Wirelessly transmit the gaze tracking data using at least one of 5G millimeter wave technology or IEEE 802.11ac protocol.

[0026] Preferably, the wireless transmission method is configured to be low power.

[0027] In one embodiment, the method further includes:

[0028] Set up an application layer with integrated edge computing capabilities to enable various real-time eye movement tracking applications.

[0029] In one embodiment, the method further includes:

[0030] Implement dynamic user interface adaptation based on the estimated gaze direction of the user.

[0031] Preferably, implement a personalized eye comfort mode with adaptation based on game theory.

[0032] In a second aspect, the present application further proposes a hybrid wireless eye movement tracking system based on multi-sensor fusion, the system comprising:

[0033] A data acquisition module configured to acquire comprehensive data of the user's eye movement and head position based on a multi-sensor including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera;

[0034] A data fusion module configured to perform data fusion on the comprehensive data;

[0035] A gaze estimation module configured to estimate the user's gaze direction based on the fused sensor data.

[0036] When the data acquisition module, the data fusion module, and the gaze estimation module and their associated modules execute a program, they implement the steps of the hybrid wireless eye movement tracking method based on multi-sensor fusion as described in any one of the above.

[0037] In a third aspect, the present application also proposes an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the hybrid wireless eye movement tracking method based on multi-sensor fusion as described in any one of the above.

[0038] In a fourth aspect, the present application finally proposes a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the hybrid wireless eye movement tracking method based on multi-sensor fusion as described in any one of the above.

[0039] (III) Advantageous Effects

[0040] The present invention provides a hybrid wireless eye movement tracking method, system, electronic device, and storage medium based on multi-sensor fusion. Compared with the prior art, it has the following advantageous effects:

[0041] A hybrid wireless eye movement tracking method based on multi-sensor fusion proposed in this application first collects comprehensive data of the user's eye movement and head position based on multi-sensors including Q-Var sensors, inertial measurement units, and high-speed panoramic cameras; then performs data fusion on the comprehensive data; and finally estimates the user's gaze direction based on the fused sensor data. The method of hybrid wireless eye movement tracking based on multi-sensor fusion in this application improves the performance of eye movement tracking technology under different environmental conditions, with higher eye movement tracking accuracy and better stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0043] Figure 1 It is a flowchart of a hybrid wireless eye movement tracking method based on multi-sensor fusion proposed in an embodiment of this application;

[0044] Figure 2 It is a flowchart of a hybrid wireless eye movement tracking method based on multi-sensor fusion proposed in another embodiment of this application;

[0045] Figure 3 It is a flowchart of the Q-Var sensor data acquisition, processing, and transmission process in an embodiment of this application;

[0046] Figure 4 It is a flowchart of the Q-Var sensor data preprocessing in an embodiment of this application

[0047] Figure 5 It is a flowchart of a hybrid wireless eye movement tracking method based on multi-sensor fusion proposed in yet another embodiment of this application;

[0048] Figure 6 It is a logic diagram of a hybrid wireless eye movement tracking method based on multi-sensor fusion proposed in another embodiment of this application;

[0049] Figure 7 It is a principle block diagram of a hybrid wireless eye movement tracking system based on multi-sensor fusion proposed in an embodiment of this application;

[0050] Figure 8 It is a schematic diagram of the principle of a hybrid wireless eye movement tracking system based on multi-sensor fusion proposed in an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] Glossary:

[0053] Eye tracking: Eye tracking technology is a technique for recording what a subject is looking at, mainly a "non-invasive" technique based on eye video analysis. This technique uses a near-infrared light source to generate a reflected image on the cornea and pupil of the user's eye, and then uses two image sensors to collect the eye and the reflected image. Through image processing algorithms and a three-dimensional eyeball model, the position of the eye in space and the line-of-sight position are accurately calculated. Eye tracking technology has become one of the technical means for visual behavior and human behavior in multiple fields, such as psychology, neuromarketing, neurocognition, user experience, basic research, and market research.

[0054] Inertial measurement unit: An inertial measurement unit (IMU) is a device used to measure the motion state of an object. It usually consists of sensors such as accelerometers, gyroscopes, and magnetometers, and can measure information such as the acceleration, angular velocity, and magnetic field strength of an object to determine the attitude, position, and motion speed of the object.

[0055] High-speed panoramic camera: A high-speed panoramic camera is a camera device that can quickly capture high-resolution panoramic images, with characteristics such as high resolution, fast shooting, panoramic coverage, real-time transmission and processing, and is widely used in fields such as security monitoring, sports event broadcasting, autonomous driving, virtual reality (VR) and augmented reality (AR), and industrial inspection.

[0056] Motion blur compensation: Motion blur is an image blur phenomenon caused by relative motion between the camera or the shooting object during exposure. Motion blur compensation is to reduce or eliminate this blur through various technologies and methods to improve the clarity and quality of the image.

[0057] By providing a hybrid wireless eye tracking method, system, electronic device, and storage medium based on multi-sensor fusion, the embodiments of the present application at least solve the technical problems of poor tracking accuracy and poor stability in the existing eye tracking technology, and achieve the objectives of high-precision, high-stability, high-efficiency, energy-saving, and intelligent user gaze estimation.

[0058] The overall idea of the technical solutions in the embodiments of the present application to solve the above technical problems is as follows:

[0059] To address the challenges often faced by traditional eye-tracking technologies in terms of accuracy, real-time performance, and power efficiency, especially the problems of poor accuracy and stability in dynamic environments or when dealing with head movements, the technology of this application combines multiple sensor modalities and advanced processing techniques to overcome these limitations and push the boundaries of eye-tracking technology. The technology of this application mainly includes: combining multiple types of sensors such as Q-Var sensors, inertial measurement units (IMUs), and high-speed panoramic cameras to improve gaze tracking accuracy, head movement compensation, and real-time data transmission. To enhance gaze estimation accuracy and stability, data preprocessing techniques including graph Fourier transform (GFT), Kalman filtering, motion blur compensation, etc., and techniques such as sensor fusion based on graph theory are used to process and fuse multi-sensor data, and gaze trajectory estimation is achieved based on convolutional neural networks (CNNs) and gated recurrent units (GRUs). To enable data transmission, this application also combines wireless transmissions using 5G mmWave and IEEE 802.11ac to enhance bandwidth and low-latency data transmission, providing energy-efficient and real-time gaze estimation under different environmental conditions. At the same time, this application also has a game theory model for gaze adaptation and improving user interaction in dynamic environments, with functions such as an adaptive personalized eye comfort mode.

[0060] To better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0061] Example 1:

[0062] In the first aspect, the present invention first proposes a hybrid wireless eye-tracking method based on multi-sensor fusion, see Figure 1 , which includes:

[0063] Collect comprehensive data on the user's eye movement and head position based on multiple sensors including Q-Var sensors, inertial measurement units, and high-speed panoramic cameras;

[0064] Perform data fusion on the comprehensive data;

[0065] Estimate the user's gaze direction based on the fused sensor data.

[0066] A hybrid wireless eye-tracking method based on multi-sensor fusion proposed in this embodiment first collects comprehensive data on the user's eye movement and head position based on multiple sensors including Q-Var sensors, inertial measurement units, and high-speed panoramic cameras; then performs data fusion on the comprehensive data; and finally estimates the user's gaze direction based on the fused sensor data. This method improves the performance of eye-tracking under different environmental conditions, with higher eye-tracking accuracy and better stability.

[0067] The implementation process of an embodiment of the present invention will be described in detail below in combination with the explanations of the above specific steps.

[0068] A hybrid wireless eye movement tracking method based on multi-sensor fusion proposed in this embodiment is shown in Figure 1-4 and Figure 6 The implementation steps mainly include:

[0069] S1. Collect comprehensive data of the user's eye movement and head position based on multi-sensors including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera.

[0070] Traditional eye tracking techniques usually rely on unimodal data collected by unimodal sensors, which may have limitations in the accuracy of user annotation estimation, especially in dynamic environments or when dealing with head movements. Based on this, the hybrid wireless eye movement tracking method based on multi-sensor fusion proposed in this embodiment uses multi-sensors including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera to collect comprehensive data of the user's eye movement and head position. Specifically:

[0071] First, use a Q-Var sensor to capture the user's electrooculogram (EOG) signal. The Q-Var sensor can detect the tiny potential difference generated by eye movement. The EOG signal captured by the Q-Var sensor can provide information about eye position and movement with high temporal resolution.

[0072] Then, use an inertial measurement unit (IMU) including at least a gyroscope and an accelerometer to collect the user's head movement data. Among them, the gyroscope is used to measure the changes in the angular velocity and direction of the user's head, and the accelerometer is used to detect linear acceleration and inclination.

[0073] In a preferred implementation, the above inertial measurement unit (IMU) further includes a magnetometer, and the magnetometer is used to calibrate the IMU data. The magnetometer calibration process involves compensating for the hard iron effect and soft iron effect that may distort the magnetometer readings. Appropriate magnetometer calibration can improve the accuracy of the heading information provided by the magnetometer component of the IMU, and further ensure the accuracy of the head position and direction estimation from the IMU data.

[0074] When collecting the user's head movement data, these IMU components such as the gyroscope, accelerometer, and magnetometer cooperate with each other to more accurately track the movement and position of the user's head.

[0075] Finally, use a high-speed panoramic camera to capture an image of the environment where the user is located in front of the user. The high-speed panoramic camera can provide context information for associating eye movement with objects or regions of interest in the user's field of view.

[0076] Of course, in actual applications, other types of sensors can be added according to actual needs. Therefore, the above multi-sensors include but are not limited to Q-Var sensors, inertial measurement units, and high-speed panoramic cameras. Correspondingly, the collected comprehensive data includes but is not limited to user eye movement and head position data, and may also be data of other parts or organs of the user's head. The movements of this part or organ are associated with the user's gaze movement.

[0077] It can be seen that compared with the existing single-sensor solution, the integration of multi-sensors of Q-Var sensors, IMUs, and high-speed panoramic cameras in this embodiment provides more robust and accurate raw data for gaze tracking estimation, helps improve the accuracy of subsequent gaze estimation, and this method of collecting comprehensive data by multi-sensors allows better compensation for head movement and improves performance under different environmental conditions, making the gaze estimation more stable.

[0078] S2. Perform data fusion on the comprehensive data.

[0079] After collecting the comprehensive data of user eye movement and head position using multi-sensors such as Q-Var sensors, inertial measurement units, and high-speed panoramic cameras, the data from the Q-Var sensors, IMUs, and high-speed panoramic cameras are fused. The data fusion process can utilize the advantages of each sensor mode to generate a more robust and accurate representation of eye movement.

[0080] In one embodiment, a graph theory-based method is used to combine the data from Q-Var sensors, IMUs, and high-speed panoramic cameras. This sensor data fusion process includes constructing a graph representation of multi-modal data, where nodes represent different sensor inputs and edges represent the relationships between these inputs. The graph theory-based sensor data fusion method can more effectively integrate heterogeneous sensor data and potentially improve the overall accuracy of gaze estimation.

[0081] In a preferred embodiment, refer to Figure 2-4 , before performing data fusion on the comprehensive data, the above hybrid wireless eye movement tracking method based on multi-sensor fusion further includes:

[0082] S20. Preprocess the above comprehensive data; the preprocessing includes:

[0083] Perform graph Fourier transform on the data collected by the Q-Var sensor; and / or

[0084] Perform Kalman filtering on the data collected by the inertial measurement unit; and / or

[0085] Perform motion blur compensation on the data collected by the high-speed panoramic camera.

[0086] In specific implementation, comprehensive consideration is made according to the actual needs of accuracy and data processing efficiency, and the above-mentioned one or more types of sensor data can be preprocessed. For example, for Q-Var sensor data, graph Fourier transform and spectral clustering techniques are used for preprocessing; and / or Kalman filter filtering and magnetometer calibration are used to preprocess IMU data; and / or motion blur compensation processing is performed on the high-speed panoramic camera information stream. Specifically:

[0087] Perform graph Fourier transform (GFT) preprocessing on Q-Var sensor data to convert the original sensor data into a form suitable for accurate gaze estimation. As Figure 3 - Figure 4 shown. The GFT component can be applied to Q-Var sensor data to convert the time-domain signal of Q-Var sensor data into the graph frequency domain. This conversion allows the separation of low-frequency fixation points from high-frequency noise and microsaccades. The GFT processing of Q-Var sensor data includes: constructing a graph representation of Q-Var data, calculating the Laplacian of the graph, and applying a transform to convert the signal into the spectral domain.

[0088] For IMU data, a Kalman filter is used to implement Kalman filtering of IMU data. The Kalman filter can be used to estimate the true state of the IMU sensor by combining a prediction based on the previous state and new measurement values to estimate the true state of the IMU sensor. Kalman filtering helps reduce noise and improve the accuracy of head position and orientation estimation from IMU data.

[0089] For the data collected by the high-speed panoramic camera, motion blur compensation technology is used for processing. Motion blur compensation analyzes the high-speed panoramic camera information stream to detect and correct the blur caused by rapid head or eye movements. Motion blur compensation helps maintain clear images from the high-speed panoramic camera and is important for correlating the user's gaze direction with environmental features.

[0090] In a more preferred embodiment, for the preprocessing of Q-Var sensor data, after GFT, it further includes further processing using spectral clustering technology applied to the Q-Var data after graph Fourier transform. Spectral clustering technology can classify different eye movement events, such as fixation, saccade, and blink, through adaptive thresholding. Spectral clustering can use the frequency representation of the graph to identify different clusters corresponding to different eye movement states.

[0091] The data preprocessing steps in the above embodiments of the present application can preprocess one or more types of sensor data, preparing better-quality data for subsequent sensor data fusion and gaze estimation processes. By processing sensor-specific noise and artifacts in the data preprocessing stage, it helps improve the overall accuracy and robustness of the hybrid wireless eye tracking system proposed in the present application.

[0092] In a further preferred embodiment, referring to Figure 5-6 , the hybrid wireless eye movement tracking method based on multi-sensor fusion proposed in the above embodiment further includes feature extraction of the fusion data, analyzing the pre-processed sensor data for fusion to extract key eye movement features.

[0093] Feature extraction can receive inputs from the pre-processing stages of the Q-Var sensor and the IMU. During feature extraction, the pre-processed Q-Var sensor data can be analyzed to determine the pupil size. The electrical signals captured by the Q-Var sensor may be related to changes in the pupil diameter. By examining the patterns and amplitudes in the processed Q-Var data, the feature extraction module can estimate the relative pupil size over time. Feature extraction can also extract information about the fixation duration from the sensor data. In some cases, the relatively stable periods of the pre-processed Q-Var and IMU signals are analyzed. This feature extraction step can identify the continuous periods during which the eye position remains relatively constant, indicating visual fixation on a specific point or object. Additionally, feature extraction can process the sensor data to detect and quantify microsaccades. Microsaccades are small and rapid eye movements that occur during visual fixation. In some cases, this feature extraction method can apply frequency analysis techniques to the pre-processed Q-Var sensor data to identify these high-frequency, low-amplitude eye movements. The feature extraction can track the frequency and characteristics of the detected microsaccades over time.

[0094] Furthermore, the graph Fourier transform applied during the pre-processing of the Q-Var sensor can also facilitate the feature extraction process. In some cases, the transformed signal can allow for more effective separation of different eye movement components. The graph Fourier transform can achieve adaptive thresholding of the Q-Var sensor data, thereby improving the accuracy of feature extraction. Spectral clustering techniques can be applied to the transformed Q-Var data for adaptive thresholding. In some cases, this method may help distinguish different eye movement states and events. Spectral clustering can enable the feature extraction module to more accurately classify fixation periods, saccade periods, and other eye movement periods.

[0095] By extracting these key features - pupil size, fixation time, and microsaccade frequency - from the pre-processed sensor data, rich eye movement features can be provided. The extracted information can be used as input data for subsequent fixation estimation and analysis processes in the hybrid wireless eye movement tracking system.

[0096] S3. Estimate the user's gaze direction based on the fused sensor data.

[0097] Based on the fused data or features extracted from the fused data, combined with the data collected by the high-speed panoramic camera after motion blur compensation, they are jointly used as inputs to determine the gaze direction.

[0098] In a preferred embodiment, when estimating the user's gaze direction, a Convolutional Neural Network (CNN) and a Gated Recurrent Unit (GRU) are utilized for gaze trajectory prediction. The CNN component can analyze the spatial features extracted from the sensor data, while the GRU can process time series to predict the gaze trajectory that changes over time.

[0099] During the gaze estimation process, data from the fusion or features extracted from the fused data can be combined, including information about pupil size, fixation time, and microsaccade frequency. In addition, the fused data or features extracted from the fused data can also contain motion compensation data from the high-speed panoramic camera, providing an environmental background for gaze estimation.

[0100] In a more preferred embodiment, during gaze estimation, the CNN applies multiple convolutional layers to extract relevant features from the fused data or features extracted from the fused data, as well as the input data of the motion blur compensation unit. These convolutional layers can detect patterns and features indicating the gaze direction in the spatial domain.

[0101] In a further embodiment, the GRU component of gaze estimation can process the feature sequence over time. In a specific implementation, the GRU maintains an internal state, allowing it to capture the temporal dependencies in the eye movement patterns. This recurrent structure may enable the module to predict future gaze trajectories based on past and current inputs.

[0102] Gaze estimation can fuse the outputs from the CNN and GRU components to produce a final estimate of the gaze direction. In a specific implementation, the fusion process involves a weighted combination of spatial and temporal features to produce a comprehensive gaze prediction.

[0103] Gaze estimation can also incorporate an adaptive mechanism to improve the accuracy of the gaze estimation it produces over time. In one embodiment, a hybrid wireless eye tracking system based on multi-sensor fusion can use feedback from a high-speed panoramic camera or other sensors to calibrate and improve the gaze estimation model during use.

[0104] By leveraging the combined capabilities of the spatial analysis of the CNN and the temporal processing of the GRU, gaze estimation can provide accurate and robust gaze direction predictions. This gaze estimation method can enable the hybrid wireless eye tracking system to handle complex eye movement patterns and adapt to various user behaviors and environmental conditions.

[0105] In a further embodiment, refer toFigure 6 , Figure 8 , the above-mentioned hybrid wireless eye movement tracking method based on multi-sensor fusion further includes:

[0106] Using at least one of 5G millimeter wave technology or IEEE 802.11ac protocol to wirelessly transmit gaze tracking data.

[0107] The wireless transmission corresponding module 802.11ac can be configured to use multiple wireless protocols to transmit data. Among them, the wireless transmission method includes but is not limited to using at least one of 5G millimeter wave technology or IEEE 802.11ac protocol to wirelessly transmit gaze tracking data.

[0108] In one embodiment, the wireless transmission method can utilize 5G millimeter wave (mmWave) technology for data transmission. 5G mmWave technology can operate at high frequencies (between 30 GHz and 300 GHz). Compared with low-frequency wireless technologies, this high-frequency operation allows for increased bandwidth and reduced latency.

[0109] In another preferred embodiment, the wireless transmission method further includes the IEEE 802.11ac protocol for data transmission. In some implementation scenarios, the 802.11ac protocol can operate in the 5GHz frequency band and provide high-throughput wireless communication.

[0110] By combining 5G mmWave and IEEE 802.11ac technologies, the wireless transmission method can be flexible in terms of data transmission. Specifically in implementation, dynamic switching is performed between these wireless transmission methods according to factors such as signal strength, bandwidth requirements, and power consumption. The dual transmission ability can provide advantages in various usage scenarios. For example, 5G mmWave technology may be suitable in cases where extremely low latency and high bandwidth are required, such as real-time virtual reality applications. On the contrary, IEEE 802.11ac may be more suitable for indoor environments where mmWave signals may have difficulty penetrating walls.

[0111] In a more preferred embodiment, the wireless transmission method can be configured for low-power operation. At this time, the wireless transmission method is based on technologies such as adaptive power control, and its transmission power is adjusted according to signal quality and the distance to the receiver. The wireless transmission method can also implement energy-saving protocols to reduce energy consumption during inactive periods.

[0112] The wireless transmission method can support real-time data transmission of gaze tracking information. In some cases, time-sensitive data packets are prioritized, and a quality of service (QoS) mechanism is implemented to ensure consistent and low-latency transmission of key eye movement tracking data.

[0113] By leveraging the advantages of 5G mmWave and IEEE 802.11ac technologies, the wireless transmission method can provide robust, flexible, and efficient data transmission capabilities for a hybrid wireless eye-tracking system. This dual-mode approach enables the system to adapt to various environmental conditions and application requirements while maintaining high performance and energy efficiency.

[0114] In one embodiment, the above-mentioned hybrid wireless eye-tracking method based on multi-sensor fusion further includes:

[0115] Set up an application layer with integrated edge computing capabilities to enable various real-time eye-tracking applications.

[0116] As Figure 6 and Figure 8 shown, the application layer can receive eye-tracking data transmitted via the wireless transmission method. In some cases, the application layer can implement dynamic user interface adaptation based on real-time gaze behavior. The hybrid wireless eye-tracking method based on multi-sensor fusion can analyze the gaze-tracking data to determine the location where the user is looking on the display or in the environment. Based on this analysis, the application layer can dynamically adjust the user interface elements to optimize the user experience. For example, the application layer can zoom in on or highlight the interface elements that the user is currently viewing, or move important information to the areas where the user's gaze is often directed. In some implementations, the system can use prediction algorithms to predict the next possible locations that the user may view and preload content in these areas for faster response times.

[0117] In one embodiment, the hybrid wireless eye-tracking method based on multi-sensor fusion further includes: implementing dynamic user interface adaptation based on the estimated gaze direction of the user.

[0118] The application layer can receive the transmitted user gaze-tracking data and utilize the edge computing resources of various applications to implement an adaptive personalized eye comfort mode.

[0119] In a preferred embodiment, an adaptive personalized eye comfort mode is implemented based on game theory.

[0120] In some cases, the application layer can use evolutionary game theory to implement an adaptive personalized eye comfort mode. This eye comfort mode can dynamically adjust the display parameters or content presentation according to the user's gaze pattern and environmental conditions. The evolutionary game theory approach in the eye comfort mode may involve treating different comfort-related parameters as strategies in the game. These strategies can be iteratively updated based on user feedback and gaze behavior, evolving towards the optimal comfort configuration over time. This adaptive mechanism may allow the system to provide a personalized eye-tracking experience for individual users while taking into account changing environmental factors.

[0121] The hybrid wireless eye tracking method based on multi-sensor fusion proposed in this embodiment can be implemented in various fields that require precise eye tracking and fixation estimation, such as augmented reality (AR) and virtual reality (VR) applications. By providing more accurate and sensitive eye tracking, it can enhance the immersion and user experience in the AR / VR environment, enable more natural interaction with virtual objects, and improve the overall performance of these technologies.

[0122] In the medical field, the hybrid wireless eye tracking method based on multi-sensor fusion proposed in this embodiment can significantly improve the diagnostic ability for neurological and ophthalmological diseases. The characteristics of high-precision fixation tracking and real-time data transmission make it particularly suitable for the early detection of eye movement abnormalities related to diseases such as Parkinson's disease, multiple sclerosis, or certain types of brain tumors.

[0123] In potential applications in the automotive industry, especially in advanced driver assistance systems (ADAS) and autonomous vehicles, the hybrid wireless eye tracking method based on multi-sensor fusion proposed in this embodiment can enhance safety functions, improve driver monitoring systems, and contribute to the development of more complex vehicle-human interfaces by accurately tracking the driver's gaze and attention.

[0124] In the field of human-computer interaction, the hybrid wireless eye tracking method based on multi-sensor fusion proposed in this embodiment can revolutionize the way users interact with various devices and interfaces. Combining precise gaze estimation with a game theory model-based adaptive user interface can bring more intuitive and effective interaction methods to computers, smartphones, and other electronic devices, potentially replacing or supplementing traditional input methods such as mice and keyboards.

[0125] Embodiment 2:

[0126] In a second aspect, the present invention also provides a hybrid wireless eye tracking system based on multi-sensor fusion. Refer to Figure 7 , the system mainly includes:

[0127] A data acquisition module configured to collect comprehensive data on the user's eye movement and head position based on a multi-sensor including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera;

[0128] A data fusion module configured to perform data fusion on the comprehensive data;

[0129] A fixation estimation module configured to estimate the user's gaze direction based on the fused sensor data.

[0130] The following combines the attached Figure 6-8, and the functions of the specific modules of the system proposed in this embodiment, as well as the explanations of the specific connection relationships between the modules, to elaborate in detail on the implementation process of an embodiment of the present invention.

[0131] As Figure 7 shown, Figure 7 The principle block diagram of the architecture of a hybrid wireless eye movement tracking system based on multi-sensor fusion is shown. This hybrid wireless eye movement tracking system based on multi-sensor fusion combines multiple sensor technologies and advanced data processing technologies to provide accurate gaze tracking in a wireless configuration. Specifically:

[0132] The system includes a data acquisition module. Among them, the data acquisition module at least includes a Q-Var sensor for capturing electrooculogram signals, an inertial measurement unit (IMU), and a high-speed panoramic camera. These multiple types of sensors work together to jointly collect comprehensive data on eye movement and head position from different angles, providing a rich data set for accurate gaze tracking.

[0133] The system also includes a data fusion module, which is configured to fuse the data from the Q-Var sensor, IMU, and high-speed panoramic camera. When performing data fusion, the data fusion module utilizes the advantages of each sensor mode to generate a more robust and accurate representation of eye movement.

[0134] The system further includes a gaze estimation module, which is configured to analyze the fused sensor data to determine the user's gaze direction. The gaze estimation process uses machine learning techniques to interpret the complex relationships between various sensor inputs.

[0135] In one embodiment, the data acquisition module includes a Q-Var sensor for capturing electrooculogram (EOG) signals. The Q-Var sensor can detect the small potential differences generated by eye movement. The EOG signals captured by the Q-Var sensor can provide information about eye position and movement with high temporal resolution.

[0136] In one embodiment, the inertial measurement unit (IMU) includes a gyroscope and an accelerometer. The gyroscope is used to measure the changes in the angular velocity and direction of the user's head, and the accelerometer is used for linear acceleration and inclination.

[0137] In a preferred embodiment, the inertial measurement unit (IMU) further includes a magnetometer for magnetometer calibration of IMU data. The magnetometer calibration process involves compensating for the hard iron effect and soft iron effect that may distort the magnetometer readings. Appropriate magnetometer calibration can improve the accuracy of the heading information provided by the magnetometer component of the IMU, and further ensure the accuracy of the head position and direction estimation from the IMU data.

[0138] These IMU components such as the gyroscope, accelerometer, and magnetometer cooperate with each other to more precisely track the movement and position of the user's head.

[0139] The high-speed panoramic camera in the data acquisition module can capture images of the user's environment in front of the user. The high-speed panoramic camera can provide context information for associating eye movements with objects or regions of interest in the user's field of view.

[0140] The Q-Var sensor, IMU, and high-speed panoramic camera work together to collect different data as an input set. The Q-Var sensor can focus on specific eye movements, while the IMU can track broader head movements, and the high-speed panoramic camera can add environmental context to the above sensor data.

[0141] By combining these different sensor modalities, the data acquisition module can capture a comprehensive image of eye and head dynamics. This method of collecting different data from multiple types of sensors from different angles and levels allows the system to collect complementary data streams for fusion and analysis to produce accurate gaze tracking results.

[0142] In one embodiment, the above-mentioned hybrid wireless eye tracking system based on multi-sensor fusion further includes a data fusion module, which can fuse the data from the Q-Var sensor, IMU, and high-speed panoramic camera. When performing data fusion, the data fusion module utilizes the advantages of each sensor modality to produce a more robust and accurate representation of eye movements.

[0143] In a preferred embodiment, the data fusion module integrates algorithms and programs including graph theory-based methods to fuse the data from the Q-Var sensor, IMU, and high-speed panoramic camera.

[0144] In one embodiment, the above-mentioned hybrid wireless eye tracking system based on multi-sensor fusion further includes a data preprocessing module, which is configured to preprocess the raw data obtained from the above different sensors in advance before the multi-sensor data is fused. Specifically:

[0145] In one embodiment, the data preprocessing module includes separate components for processing data of one or more sensor types, that is, according to actual needs, preprocessing can be performed on data of one or more types of sensors. For example, for Q-Var sensor data, preprocessing is performed using graph Fourier transform and spectral clustering techniques; and / or Kalman filter filtering and magnetometer calibration are used to preprocess magnetometer data; and / or the information flow of the high-speed panoramic camera is processed through motion blur compensation. Specifically,

[0146] For Q-Var sensor data, the data preprocessing module includes a graph Fourier transform (GFT) component for preprocessing. The GFT component can be applied to Q-Var sensor data to convert the time-domain signal of the Q-Var sensor data into the graph frequency domain. This conversion allows the separation of low-frequency fixations from high-frequency noise and microsaccades. GFT processing of Q-Var sensor data includes constructing a graph representation of the Q-Var data, calculating the Laplacian of the graph, and applying the transform to convert the signal to the spectral domain.

[0147] For IMU data, the data preprocessing module includes a Kalman filter to perform Kalman filtering on the IMU data. The Kalman filter can be used to estimate the true state of the IMU sensor by combining a prediction based on the previous state and new measurements to estimate the true state of the IMU sensor. Kalman filtering helps reduce noise and improve the accuracy of head position and orientation estimation from IMU data.

[0148] For data collected by a high-speed panoramic camera, the data preprocessing module further includes a motion blur compensation unit configured to process the data collected by the high-speed panoramic camera using motion blur compensation techniques. Motion blur compensation involves analyzing the information flow of the high-speed panoramic camera to detect and correct blurs caused by rapid head or eye movements. Motion blur compensation helps maintain clear images from the high-speed panoramic camera, which is important for correlating the user's gaze direction with environmental features.

[0149] In a preferred embodiment, for the preprocessing of Q-Var sensor data, after GFT, spectral clustering techniques are applied to the further processing of the Q-Var data after graph Fourier transform. Spectral clustering techniques can classify different eye movement events, such as fixations, saccades, and blinks, through adaptive thresholding. Spectral clustering can utilize the frequency representation of the graph to identify different clusters corresponding to different eye movement states.

[0150] The data preprocessing module proposed in the above embodiments of the present application, for the preprocessing steps of one or more types of sensor data, can prepare data for subsequent sensor fusion and gaze estimation processes. By processing sensor-specific noise and artifacts at this stage, the data preprocessing module contributes to the overall accuracy and robustness of the hybrid wireless eye tracking system proposed in the present application.

[0151] In a further preferred embodiment, the hybrid wireless eye tracking system proposed in the above embodiments further includes a feature extraction module configured to process the sensor data fused after preprocessing to extract key eye movement features. Such as Figure 6 and Figure 8As shown, the feature extraction module can receive inputs from the preprocessing stage of the Q-Var sensor and the IMU.

[0152] The feature extraction module can analyze the preprocessed Q-Var sensor data to determine pupil size. The electrical signals captured by the Q-Var sensor may be related to changes in pupil diameter. By examining the patterns and amplitudes in the processed Q-Var data, the feature extraction module can estimate the relative pupil size over time.

[0153] The feature extraction module can also extract information about fixation duration from the sensor data. In some cases, it analyzes the relatively stable periods of the preprocessed Q-Var and IMU signals. The feature extraction module can identify the durations during which the eye position remains relatively constant, indicating visual fixation on a particular point or object.

[0154] In addition, the feature extraction module can process the sensor data to detect and quantify microsaccades. Microsaccades are small and rapid eye movements that occur during visual fixation. In some cases, the feature extraction module can apply frequency analysis techniques to the preprocessed Q-Var sensor data to identify these high-frequency, low-amplitude eye movements. The feature extraction module can track the frequency and characteristics of the detected microsaccades over time.

[0155] In addition, the graph Fourier transform applied during the preprocessing of the Q-Var sensor can also facilitate the feature extraction process. In some cases, the transformed signal can allow for more effective separation of different eye movement components. The graph Fourier transform can achieve adaptive thresholding of the Q-Var sensor data, thereby improving the accuracy of feature extraction. Spectral clustering techniques can be applied to the transformed Q-Var data for adaptive thresholding. In some cases, this method may help distinguish different eye movement states and events. Spectral clustering can enable the feature extraction module to more accurately classify fixation periods, saccade periods, and other eye movement periods.

[0156] By extracting these key features - pupil size, fixation time, and microsaccade frequency - from the preprocessed sensor data using the feature extraction module, the feature extraction module can provide rich eye movement features. The extracted information can serve as input data for subsequent fixation estimation and analysis processes in the hybrid wireless eye movement tracking system.

[0157] In one embodiment, the hybrid wireless eye movement tracking system based on multi-sensor fusion further includes a fixation estimation module configured to determine the fixation direction based on data directly from the data fusion module or the feature extraction module, in cooperation with the input of the motion blur compensation unit. The fixation estimation module can receive processed data from multiple sources to perform accurate fixation tracking.

[0158] In a preferred embodiment, the gaze estimation module includes a Convolutional Neural Network (CNN) and a Gated Recurrent Unit (GRU) for gaze trajectory prediction. The CNN component can analyze the spatial features extracted from sensor data, while the GRU can process time series to predict the gaze trajectory that changes over time.

[0159] The gaze estimation module can combine the inputs from the feature extraction module, which may include information about pupil size, fixation time, and microsaccade frequency. In addition, the module can also incorporate motion compensation data from a high-speed panoramic camera to provide an environmental context for gaze estimation.

[0160] In a more preferred embodiment, the CNN within the gaze estimation module applies multiple convolutional layers to extract relevant features from the data of the data preprocessing module or the feature extraction module, as well as the input data of the motion blur compensation unit. These convolutional layers can detect patterns and features indicating the gaze direction in the spatial domain.

[0161] In a further embodiment, the GRU component of the gaze estimation module can process the feature sequence over time. In a specific implementation, the GRU maintains an internal state that allows it to capture the temporal dependencies in the eye movement patterns. This recurrent structure may enable the module to predict future gaze trajectories based on past and current inputs.

[0162] The gaze estimation module can fuse the outputs from the CNN and GRU components to produce a final estimate of the gaze direction. In a specific implementation, the fusion process involves a weighted combination of spatial and temporal features to produce a comprehensive gaze prediction.

[0163] The gaze estimation module can also incorporate an adaptive mechanism to improve the accuracy of the gaze estimation it produces over time. In one embodiment, a hybrid wireless eye tracking system based on multi-sensor fusion can use feedback from a high-speed panoramic camera or other sensors to calibrate and improve the gaze estimation model during use.

[0164] By leveraging the combined capabilities of the spatial analysis of the CNN and the temporal processing of the GRU, the gaze estimation module can provide accurate and robust gaze direction predictions. This gaze estimation method can enable the hybrid wireless eye tracking system to handle complex eye movement patterns and adapt to various user behaviors and environmental conditions.

[0165] In a further embodiment, the above-mentioned hybrid wireless eye tracking system based on multi-sensor fusion further includes a wireless transmission module for transmitting gaze tracking data. As Figure 7 shown, the wireless transmission module 802.11ac can be configured to use multiple wireless protocols to transmit data.

[0166] In one embodiment, the wireless transmission module may utilize 5G millimeter wave (mmWave) technology for data transmission. 5G mmWave technology can operate at high frequencies (between 30 GHz and 300 GHz). Compared with low-frequency wireless technologies, this high-frequency operation allows for increased bandwidth and reduced latency.

[0167] In another preferred embodiment, the wireless transmission module further includes the IEEE 802.11ac protocol for data transmission. In some implementations, the 802.11ac protocol can operate in the 5 GHz band and provide high-throughput wireless communication.

[0168] By combining 5G mmWave and IEEE 802.11ac technologies, the wireless transmission module can be flexible in data transmission. Specifically, in implementation, the hybrid wireless eye tracking system based on multi-sensor fusion can dynamically switch between these transmission methods according to factors such as signal strength, bandwidth requirements, and power consumption. The dual transmission capability can provide advantages in various usage scenarios. For example, 5G mmWave technology may be suitable for situations that require extremely low latency and high bandwidth, such as real-time virtual reality applications. In contrast, IEEE 802.11ac may be more suitable for indoor environments where mmWave signals may have difficulty penetrating walls.

[0169] In a more preferred embodiment, the wireless transmission module can be configured for low-power operation. At this time, the wireless transmission module is based on technologies such as adaptive power control, and its transmission power is adjusted according to signal quality and the distance to the receiver. The wireless transmission module can also implement energy-saving protocols to reduce energy consumption during inactive periods.

[0170] The wireless transmission module can support real-time data transmission of gaze tracking information. In some cases, time-sensitive data packets are prioritized, and a quality of service (QoS) mechanism is implemented to ensure consistent and low-latency transmission of critical eye tracking data.

[0171] By leveraging the advantages of 5G mmWave and IEEE 802.11ac technologies, the wireless transmission module can provide robust, flexible, and efficient data transmission capabilities for the hybrid wireless eye tracking system. This dual-mode approach enables the system to adapt to various environmental conditions and application requirements while maintaining high performance and energy efficiency.

[0172] In one embodiment, the above-mentioned hybrid wireless eye tracking system based on multi-sensor fusion further includes an application layer with integrated edge computing capabilities to enable various real-time eye tracking applications. Such as Figure 6 and Figure 8As shown, the application layer can receive the eye-tracking data transmitted through the wireless transmission module.

[0173] In some cases, the application layer can implement dynamic user interface adaptation based on real-time gaze behavior. The system can analyze the incoming gaze-tracking data to determine the location where the user is looking on the display or in the environment. Based on this analysis, the application layer can dynamically adjust the user interface elements to optimize the user experience. For example, the application layer can zoom in on or highlight the interface element that the user is currently viewing, or move important information to the area where the user's gaze is often directed. In some implementations, the system can use predictive algorithms to predict the next possible location that the user will view and preload content in these areas for faster response times.

[0174] The wireless transmission capabilities of the hybrid wireless eye-tracking system based on multi-sensor fusion proposed in the above embodiments can facilitate various applications in artificial intelligence (AI), ophthalmology, and human-computer interaction (HCI). In some cases, the low-latency transmission of gaze-tracking data may enable real-time AI analysis of eye movements to be used in medical diagnosis or user behavior research.

[0175] For ophthalmology applications, the system can provide continuous monitoring of eye movements and pupil responses, potentially allowing for the early detection of certain eye conditions or the evaluation of treatment effectiveness. In an HCI environment, real-time gaze data may enable new forms of hands-free computer control or augmented reality interactions.

[0176] The application layer can utilize edge computing resources to locally process the gaze-tracking data, reducing latency and enhancing privacy. In some implementations, an edge AI model can run on a local device to further process eye movements before transmitting an initial analysis of the aggregated results to a cloud service. This edge computing approach may allow for more responsive applications, as certain gaze-based interactions can be processed and acted upon locally without the need for round-trip communication with a remote server. Additionally, processing sensitive eye movement data at the edge can help address privacy concerns by reducing the amount of raw data transmitted over the network.

[0177] Integrating the wireless transmission module with an AI edge cloud application can enable a hybrid processing model. In some cases, the system may perform an initial gaze analysis on an edge device and then utilize cloud-based AI for more complex or resource-intensive computations. This hybrid approach can allow for scalable and flexible gaze-driven applications that can adapt to different computational requirements and network conditions.

[0178] By combining real-time gaze tracking capabilities with edge computing and flexible wireless transmission, the hybrid wireless eye tracking system can support a wide range of applications across multiple domains. The system is capable of providing low-latency, context-aware gaze data, which may enable human-computer interaction and gaze-based analysis in areas such as healthcare, user experience design, and augmented reality.

[0179] Next, the operating principle of this system will be elaborated in detail by briefly describing the specific execution steps and processes of the above-mentioned hybrid wireless eye tracking system based on multi-sensor fusion proposed in this embodiment:

[0180] After the data acquisition module collects data, a graph representation of the Q-Var sensor signal is constructed in the data preprocessing module. This graph construction step may include treating each time sample as a node and establishing connections between nodes based on signal similarity or temporal adjacency. After graph construction, the graph Fourier transform can be applied to the Q-Var sensor data to convert the time-domain signal into the graph frequency domain. In some cases, this conversion may allow separating low-frequency gazes from high-frequency noise and microsaccades. Additionally, the GFT process involves calculating the Laplacian of the graph, which helps define the signal smoothness on the graph. In some implementations, the system can simultaneously calculate the standard form and the normalized form of the Laplacian determinant of the graph. After GFT, for IMU data, a Kalman filter is used to perform Kalman filtering on the IMU data to eliminate high-frequency noise components. This filtering process may involve applying the Kalman filter in the graph frequency domain to maintain smooth gazes while eliminating unnecessary high-frequency variations. Additionally, adaptive thresholding spectral clustering can also be performed. This clustering method can dynamically classify different eye movement events, such as gazes, saccades, and blinks, based on the frequency components of the graph.

[0181] By implementing this graph Fourier transform-based method and other data preprocessing methods, the hybrid wireless eye tracking system based on multi-sensor fusion can effectively process Q-Var sensor data, IMU measurement data, and high-speed panoramic camera data, and extract meaningful eye movement information. These processed data may contribute to accurate gaze estimation and enable various applications in areas such as human-computer interaction and medical diagnosis.

[0182] After the data preprocessing actions performed by the data preprocessing module, the preprocessed data from the Q-Var sensor, inertial measurement unit (IMU), and high-speed panoramic camera are combined for comprehensive gaze estimation.

[0183] The hybrid wireless eye-tracking system based on multi-sensor fusion integrates multiple components to provide accurate and real-time gaze tracking functionality. The system combines data from Q-Var sensors, inertial measurement units (IMUs), and high-speed panoramic cameras to generate comprehensive eye-tracking information.

[0184] In some cases, the system can implement a graph-theory-based method to combine preprocessed data from Q-Var sensors, IMUs, and high-speed panoramic cameras. This sensor data fusion process may include constructing a graph representation of the multimodal data, where nodes represent different sensor inputs and edges represent the relationships between these inputs. The graph-based fusion method can more effectively integrate heterogeneous sensor data, potentially improving the overall accuracy of gaze estimation.

[0185] The data fusion module can apply graph-based algorithms to analyze the interconnected sensor data. In some implementations, this may involve using graph spectral analysis techniques to identify important features and patterns between different sensor modalities. The graph-theory-based method enables the system to capture the complex dependencies between eye movements, head position, and environmental context.

[0186] After sensor data fusion, the gaze estimation module can process the combined data to determine the user's gaze direction. Then, the estimated gaze information can be wirelessly transmitted using the system's dual-mode transmission capabilities, as Figure 6 and Figure 8 shown.

[0187] The hybrid wireless eye-tracking system based on multi-sensor fusion can incorporate a wireless transmission module with its dual-mode transmission capabilities to implement a power-saving scheme. As Figure 6 and Figure 8 shown, the system can simultaneously utilize 5G millimeter-wave technology and IEEE802.11ac protocol for wireless data transmission.

[0188] In some cases, the power-saving scheme may involve dynamic switching between the two transmission modes based on various factors. The system can evaluate the current power level, data transmission requirements, and environmental conditions to determine the most energy-efficient transmission method at any given time.

[0189] 5G millimeter wave technology may offer high bandwidth and low latency, which may be beneficial for quickly transmitting large amounts of eye tracking data. However, compared to other wireless protocols, this technology may consume more power. In some implementations, the system may reserve 5G millimeter wave transmissions that require high data throughput or extremely low latency. Instead, the IEEE 802.11ac protocol may provide a more energy-efficient option for data transmission in certain cases. In some cases, when lower bandwidth is sufficient or when saving battery life becomes a priority, the system may switch to 802.11ac.

[0190] The power-saving scheme may also include an adaptive power control mechanism. In some implementations, the system can adjust the transmission power based on the distance between the eye tracking device and the receiving unit. When the receiver is nearby, by reducing the transmission power, the system can save energy without compromising data integrity. In some cases, the system may implement a sleep mode or a low-power state during inactivity. The eye tracking sensor and the wireless transmission module can enter these power-saving states when not actively collecting or transmitting data, thus further extending the battery life.

[0191] Dual-mode transmission capabilities may improve energy efficiency by allowing the system to optimize its power consumption according to the specific requirements of different eye tracking applications. For example, applications that require continuous high-frequency data transmission may utilize 5G millimeter wave technology, while less demanding tasks may rely on the more energy-efficient 802.11ac protocol.

[0192] In some implementations, the system can adopt intelligent scheduling algorithms to balance power consumption and performance. These algorithms can analyze usage patterns and application requirements to predict the optimal time to switch between transmission modes or enter a low-power state.

[0193] The power-saving scheme can also utilize edge computing capabilities to reduce the amount of data that needs to be wirelessly transmitted. In some cases, the preliminary processing of eye tracking data can be performed locally, and only the relevant results or compressed data are transmitted, thus reducing the total power consumption associated with wireless communication.

[0194] By combining these energy-saving technologies with the flexibility of dual-mode transmission, a hybrid wireless eye tracking system can achieve a balance between energy efficiency and high performance. This approach can extend the operating time of battery-powered devices while maintaining the ability to provide high-quality eye tracking data when needed.

[0195] The application layer of the system can receive the transmitted gaze tracking data and utilize the edge computing resources of various applications. In some cases, the application layer can implement an adaptive personalized eye comfort mode using evolutionary game theory. This eye comfort mode can dynamically adjust the display parameters or content presentation according to the user's gaze pattern and environmental conditions.

[0196] An evolutionary game theory approach in eye comfort mode may involve treating different comfort-related parameters as strategies in a game. The system can iteratively update these strategies based on user feedback and gaze behavior, evolving towards an optimal comfort configuration over time. This adaptive mechanism may allow the system to provide a personalized eye-tracking experience for individual users while taking into account changing environmental factors.

[0197] In summary, the overall data flow of the hybrid wireless eye-tracking system based on multi-sensor fusion can go from pre-processed sensor acquisition, sensor fusion, gaze estimation, wireless transmission, and finally to application-level processing. Each stage in this pipeline contributes to the system providing accurate, real-time eye-tracking data.

[0198] In some implementations, the system can use feedback loops between different components to continuously improve and enhance performance. For example, the application layer can provide feedback to the sensor fusion module to adjust fusion parameters based on application-specific requirements or detected gaze patterns.

[0199] A graph theory-based sensor fusion method, combined with an evolutionary game theory-driven eye comfort mode, can enable the hybrid wireless eye-tracking system to adapt to different usage scenarios and user needs. These advanced processing techniques, along with the system's wireless capabilities and edge computing integration, may allow for flexible and efficient eye-tracking applications across different domains.

[0200] It is understandable that the hybrid wireless eye-tracking system based on multi-sensor fusion proposed in the embodiments of the present invention corresponds to the above-mentioned hybrid wireless eye-tracking method based on multi-sensor fusion. For the explanations, examples, beneficial effects, etc. of the relevant content, reference can be made to the corresponding content in the hybrid wireless eye-tracking method based on multi-sensor fusion, which will not be elaborated here.

[0201] Embodiment 3:

[0202] In a third aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the hybrid wireless eye-tracking method based on multi-sensor fusion as described in any one of the above embodiments and their preferred embodiments. The method mainly includes:

[0203] S1. Collect comprehensive data on the user's eye movement and head position based on a multi-sensor including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera;

[0204] S2. Perform data fusion on the comprehensive data;

[0205] S3. Estimate the user's gaze direction based on the fused sensor data.

[0206] It can be understood that the hybrid wireless eye movement tracking electronic device based on multi-sensor fusion provided by the embodiments of the present invention corresponds to the above-mentioned hybrid wireless eye movement tracking method and system based on multi-sensor fusion. For the explanations, examples, beneficial effects, etc. of the relevant content, reference can be made to the corresponding content in the hybrid wireless eye movement tracking method and system based on multi-sensor fusion, which will not be elaborated here.

[0207] Embodiment 4:

[0208] Fourthly, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the hybrid wireless eye movement tracking method based on multi-sensor fusion as described in any one of the above embodiments and their preferred embodiments. The method includes:

[0209] S1. Collect comprehensive data of the user's eye movement and head position based on a multi-sensor including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera;

[0210] S2. Perform data fusion on the comprehensive data;

[0211] S3. Estimate the user's gaze direction based on the fused sensor data.

[0212] It can be understood that the storage medium of the hybrid wireless eye movement tracking based on multi-sensor fusion provided by the embodiments of the present invention corresponds to the above-mentioned hybrid wireless eye movement tracking method and system based on multi-sensor fusion. For the explanations, examples, beneficial effects, etc. of the relevant content, reference can be made to the corresponding content in the hybrid wireless eye movement tracking method and system based on multi-sensor fusion, which will not be elaborated here.

[0213] In summary, compared with the prior art, the following beneficial effects are achieved:

[0214] 1. A hybrid wireless eye movement tracking technology based on multi-sensor fusion proposed in this application first collects comprehensive data of the user's eye movement and head position based on a multi-sensor including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera; then performs data fusion on the comprehensive data; and finally estimates the user's gaze direction based on the fused sensor data. The method of this application improves the performance of eye movement tracking under different environmental conditions, with higher eye movement tracking accuracy and better stability.

[0215] 2. A hybrid wireless eye tracking technology based on multi-sensor fusion proposed in this application performs graph Fourier transform on the data collected by the Q-Var sensor; and / or performs Kalman filtering on the data collected by the inertial measurement unit; and / or performs motion blur compensation on the data collected by the high-speed panoramic camera, and further processes the Q-Var data after graph Fourier transform using spectral clustering technology. By processing sensor-specific noise and artifacts in the data preprocessing stage, it helps improve the overall accuracy and robustness of the hybrid wireless eye tracking system proposed in this application.

[0216] 3. A hybrid wireless eye tracking technology based on multi-sensor fusion proposed in this application uses a graph theory-based method to combine data from the Q-Var sensor, IMU, and high-speed panoramic camera, which can more effectively integrate heterogeneous sensor data and potentially improve the overall accuracy of gaze estimation.

[0217] 4. A hybrid wireless eye tracking technology based on multi-sensor fusion proposed in this application can provide rich eye movement features by extracting these key features - pupil size, fixation time, and microsaccade frequency - from the preprocessed sensor data.

[0218] 5. A hybrid wireless eye tracking technology based on multi-sensor fusion proposed in this application can provide accurate and robust gaze direction prediction through the combined capabilities of spatial analysis using CNN and temporal processing using GRU for gaze estimation. This gaze estimation method enables the hybrid wireless eye tracking system to handle complex eye movement patterns and adapt to various user behaviors and environmental conditions.

[0219] 6. A hybrid wireless eye tracking technology based on multi-sensor fusion proposed in this application can provide a robust, flexible, and efficient data transmission capability for the hybrid wireless eye tracking system through the advantages of 5GmmWave and IEEE 802.11ac technologies. This dual-mode method enables the system to adapt to various environmental conditions and application requirements while maintaining high performance and energy efficiency.

[0220] 7. A hybrid wireless eye tracking technology based on multi-sensor fusion proposed in this application realizes dynamic user interface adaptation based on the estimated gaze direction of the user, and realizes an adaptive personalized eye comfort mode based on game theory. These strategies can be iteratively updated according to user feedback and gaze behavior, evolving towards the optimal comfort configuration over time. This adaptive mechanism may allow the system to provide a personalized eye tracking experience for individual users while taking into account changing environmental factors.

[0221] It should be noted that, in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0222] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid wireless eye movement tracking method based on multi-sensor fusion, characterized in that, The method includes: Collecting comprehensive data of the user's eye movement and head position based on multi-sensors including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera; Performing data fusion on the comprehensive data; Estimating the user's gaze direction based on the fused sensor data.

2. The hybrid wireless eye movement tracking method based on multi-sensor fusion according to claim 1, wherein, Before performing data fusion on the comprehensive data, preprocessing the comprehensive data; the preprocessing includes: Performing graph Fourier transform on the data collected by the Q-Var sensor; and / or Performing Kalman filtering on the data collected by the inertial measurement unit; and / or Performing motion blur compensation on the data collected by the high-speed panoramic camera.

3. The hybrid wireless eye movement tracking method based on multi-sensor fusion according to claim 2, wherein The preprocessing further includes: Performing adaptive thresholding on the Q-Var sensor data after graph Fourier transform based on spectral clustering technology.

4. The hybrid wireless eye movement tracking method based on multi-sensor fusion according to any one of claims 2-3, characterized in that, Performing data fusion on the comprehensive data includes: Fusing the preprocessed data of the Q-Var sensor, the inertial measurement unit, and the high-speed panoramic camera based on a graph theory method.

5. The hybrid wireless eye movement tracking method based on multi-sensor fusion according to any one of claims 1-4, characterized in that, The estimating the user's gaze direction based on the fused sensor data includes: Predicting the user's gaze trajectory by analyzing the fused sensor data based on a convolutional neural network and a gated recurrent unit.

6. The hybrid wireless eye movement tracking method based on multi-sensor fusion according to any one of claims 1-5, characterized in that, The method further includes: Wirelessly transmitting the gaze tracking data using at least one of 5G millimeter wave technology or IEEE 802.11ac protocol.

7. The hybrid wireless eye movement tracking method based on multi-sensor fusion according to any one of claims 1-6, characterized in that The method further includes: Implementing dynamic user interface adaptation based on the estimated gaze direction of the user; and Implementing a personalized eye comfort mode with adaptability based on game theory.

8. A hybrid wireless eye movement tracking system based on multi-sensor fusion, characterized in that, The system includes: A data acquisition module configured to collect comprehensive data of the user's eye movement and head position based on multi-sensors including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera; A data fusion module configured to perform data fusion on the comprehensive data; A gaze estimation module configured to estimate the user's gaze direction based on the fused sensor data.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the steps of the hybrid wireless eye movement tracking method based on multi-sensor fusion as described in any one of claims 1-7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the hybrid wireless eye movement tracking method based on multi-sensor fusion as described in any one of claims 1-7 are implemented.

Images