A desktop eye movement signal capture and interaction system

Abstract

The application provides a desktop eye movement signal capturing and interaction system, comprising: a display integrated acquisition terminal, containing a display interaction device integrated with an eye movement tracking sensor, used for capturing an eye reflection image of a user and transmitting the eye reflection image to a multi-modal data processing unit; an environment and physiological signal acquisition unit, used for synchronously acquiring relevant environment and physiological data and transmitting the environment and physiological data to the multi-modal data processing unit; and the multi-modal data processing unit, used for converting the received eye reflection image into a real-time gaze point positioning coordinate, identifying a control intention based on the environment and physiological data, and generating visual feedback information and a cursor control signal, which are fed back to the display interaction device. According to the technical scheme of the application, the stability, the intelligent level and the application range of the desktop eye movement signal capturing and interaction system can be significantly improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and biosignal processing technology, specifically to a desktop eye-tracking signal capture and interaction system. Background Technology

[0002] With the rapid development of artificial intelligence and human-computer interaction technologies, eye-tracking technology, with its non-contact, intuitive, and efficient characteristics, has shown great application potential in fields such as industrial control, medical assistance, virtual reality, and market research. Desktop eye trackers, currently the most widely used form, typically capture the user's gaze through sensors integrated into a display, thereby enabling quantitative analysis and interactive control of visual attention.

[0003] However, despite the relatively mature understanding of the basic principles of eye-tracking technology, it still faces many challenges in practical applications, severely restricting its widespread adoption and reliability in complex environments and high-level interaction scenarios. First, existing systems have poor environmental adaptability. Traditional eye-tracking devices mostly use fixed-mode infrared light sources, lacking the ability to dynamically perceive and adjust to changes in ambient light. When the ambient light is too strong or too weak, it easily leads to blurred pupil edges and loss of feature points, resulting in decreased data acquisition accuracy or even tracking failure. Second, the interaction dimension is limited, and intent recognition is ambiguous. Current mainstream systems mainly rely on simple gaze coordinate data for interaction, making it difficult for the system to accurately distinguish between "unintentional gaze" and "intentional control" by the user, resulting in a high rate of misoperation in medical assistance or complex command operations. Moreover, the existing technology is clearly lacking in functional integration and scenario adaptability. Most existing eye-tracking interaction systems are presented as single acquisition terminals, lacking dedicated decoding modules and third-party auxiliary interaction mechanisms for specific scenarios (such as medical assistance). This results in significant obstacles for existing devices when exchanging data with external environmental sensors or analysis platforms, limiting their in-depth application in the field of multimodal data analysis and real-time feedback.

[0004] Therefore, a technical solution is needed to provide a desktop eye-tracking system that is highly integrated, environmentally adaptive, and supports multimodal interaction. Summary of the Invention

[0005] This application aims to provide a desktop eye-tracking signal capture and interaction system that can significantly improve the stability, intelligence level and applicability of the desktop eye-tracking signal capture and interaction system by introducing an ambient light adaptive adjustment mechanism, constructing a multimodal data fusion processing framework, and integrating interactive functions for specific scenarios (such as medical assistance).

[0006] According to one aspect of this application, a desktop eye-tracking signal capture and interaction system is provided, comprising: an integrated acquisition and display terminal, an environmental and physiological signal acquisition unit, and a multimodal data processing unit.

[0007] The integrated acquisition and display terminal includes a display interaction device with an integrated eye-tracking sensor, used to capture the user's eye reflection image and transmit the eye reflection image to the multimodal data processing unit;

[0008] An environmental and physiological signal acquisition unit is used to synchronously acquire relevant environmental and physiological data and transmit them to the multimodal data processing unit;

[0009] The multimodal data processing unit is equipped with an eye-tracking decoding algorithm. The eye-tracking decoding algorithm converts the received eye reflection image into real-time gaze point positioning coordinates, and identifies the control intention based on the environmental and physiological data, thereby generating visual feedback information and cursor control signals, which are then fed back to the display interaction device.

[0010] According to some embodiments, it also includes:

[0011] The auxiliary interaction unit includes an auxiliary interaction device independent of the display interaction device, the auxiliary interaction device being used to output relevant visual feedback information to a third party.

[0012] According to some embodiments, the multimodal data processing unit includes: an external device synchronization interface, which connects to the external environment and physiological signal acquisition devices via the PsyHUBTPSN protocol, wherein...

[0013] Physiological signal acquisition devices include one or more of electroencephalogram (EEG) acquisition devices, magnetic resonance imaging (GSR) acquisition devices, and facial expression capture cameras; environmental signal acquisition devices include one or more of temperature and humidity sensors, light sensors, and noise sensors.

[0014] According to some embodiments, the multimodal data processing unit includes: a FACS (Facial Action Coding System) expression recognition engine, which identifies the user's facial action units based on the input of the facial expression capture camera, thereby identifying the user's expression state, performing correlation analysis with eye movement interest area data, and generating an emotion-gaze correlation report.

[0015] According to some embodiments, the eye-tracking sensor supports a dual-light source switching mode with bright and dark pupils, and the switching is controlled by the light balance algorithm module built into the multimodal data processing unit.

[0016] The light balance algorithm module is configured as follows:

[0017] Real-time monitoring of light sensor data from the environmental and physiological data;

[0018] When the ambient light level exceeds the preset light threshold, it automatically switches to dark-eye tracking mode, and / or

[0019] When the ambient light intensity is lower than the preset light threshold, it automatically switches to the Bright Eye Tracking mode and dynamically adjusts the light source gain.

[0020] According to some embodiments, the integrated acquisition and display terminal includes a magnetic filter assembly, wherein magnetic adsorption sites are arranged on the front side of the lens of the eye-tracking sensor, thereby allowing the magnetic filter assembly to be detachably installed.

[0021] According to some embodiments, the multimodal data processing unit includes: a medical assistance decoding module that supports medical assistance scene modes. The medical assistance decoding module is configured with a preset medical instruction set. When in medical assistance scene mode, it supports users to select symptoms and locations through eye-tracking gaze, and then packages the selection results into alarm information and pushes it to the auxiliary interaction device.

[0022] According to some embodiments, the multimodal data processing unit further includes a data development interface (API) to output raw waveform data of raw eye movement coordinates, pupil diameter, head posture angle and other external physiological signals to external analysis devices in real time.

[0023] According to some embodiments, the multimodal data processing unit supports the automatic generation function of regions of interest (AOIs). Based on the ilab visual attention model tool, it performs saliency calculation on the display content of the display interaction device, automatically generates eye-tracking regions of interest (AOIs), and counts the gaze time and reversion rate for each word.

[0024] According to some embodiments, the auxiliary interaction device includes a touch input module, which allows a third party to confirm or respond to the user's eye-tracking selection through the touch input module, and to synchronize the response information back to the integrated acquisition and display terminal through the multimodal data processing unit.

[0025] According to another aspect of this application, a computing device is provided, comprising:

[0026] Processor; and

[0027] A memory storing a computer program that, when executed by the processor, causes the processor to perform the method described in any of the preceding methods.

[0028] According to another aspect of this application, a non-transitory computer-readable storage medium is provided, having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any of the preceding claims.

[0029] According to embodiments of this application, the synchronous acquisition and fusion analysis of eye-tracking, physiological, and environmental data is achieved through the collaborative work of an integrated terminal for acquisition and display, an environmental and physiological signal acquisition unit, and a multimodal data processing unit. By utilizing an eye-tracking decoding algorithm to convert eye reflection images into high-precision gaze coordinates, and combining environmental and physiological data to accurately identify user control intentions, this not only effectively overcomes the vulnerability of single-modality data to interference and improves interaction stability in complex environments, but also simplifies the deployment process through an integrated design, achieving low-latency real-time visual feedback and natural, smooth cursor control, significantly reducing user cognitive load and system application costs.

[0030] According to some embodiments, by fusing multimodal data from eye-tracking sensors, environmental and physiological signal acquisition units, and the FACS facial expression recognition engine, accurate quantitative analysis of users' emotional states and attention distribution is achieved, enabling a more comprehensive capture of users' psychological activity characteristics in complex scenarios.

[0031] According to some embodiments, by employing cross-modal correlation analysis technology, a dynamic mapping relationship between eye movement, physiological, and facial expression data is established, solving the misjudgment problem caused by data isolation in traditional methods. Furthermore, the data development interface provided by this invention supports seamless integration with third-party systems, enabling the emotion-attention analysis results to be quickly applied to fields such as intelligent driving and mental health monitoring, providing reliable technical support for cross-industry intelligent applications.

[0032] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0034] Figure 1 The diagram illustrates a desktop eye-tracking signal capture and interaction system according to an example embodiment.

[0035] Figure 2 This diagram illustrates the workflow of a desktop eye-tracking signal capture and interaction system according to an example embodiment.

[0036] Figure 3 A block diagram of a computing device according to an exemplary embodiment is shown. Detailed Implementation

[0037] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0038] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0041] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0042] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0043] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0044] Eye-tracking technology, as an important branch of human-computer interaction, has been widely used in market research, medical assistance, and virtual reality. Current desktop eye trackers primarily use sensors integrated into a display to capture gaze, enabling non-contact operation.

[0045] However, existing systems still have significant shortcomings in practical applications: First, they have poor environmental adaptability. Traditional devices mostly use fixed-mode light sources, which make it difficult to maintain stable tracking performance when ambient light levels change drastically, easily leading to data loss. Second, the interaction methods are limited, relying solely on eye-tracking data and lacking synchronous acquisition and fusion analysis of physiological signals (such as EEG and EEG) and environmental data, resulting in ambiguous control intent recognition and a high rate of misoperation. Third, they lack scalability, lacking standardized external device synchronization and development interfaces, making it difficult to meet the needs of multimodal data analysis and in-depth applications in specific scenarios (such as medical assistance).

[0046] To address this, this application proposes a desktop eye-tracking signal capture and interaction system. By introducing an ambient light adaptive adjustment mechanism, constructing a multimodal data fusion processing framework, and integrating interactive functions for specific scenarios (such as medical assistance), the system significantly improves its stability, intelligence level, and applicability. According to embodiments, this invention achieves accurate analysis of a user's emotional state and attention distribution through the fusion of multimodal data from eye movements, physiological signals, and facial expressions. The system employs cross-modal correlation analysis technology, solving the problem of misjudgment caused by single data sources. Furthermore, the system provides standardized data interfaces for easy integration with third-party platforms. This system enables objective and real-time monitoring of human psychological states, providing reliable technical support for intelligent applications across multiple industries.

[0047] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application.

[0048] Figure 1 The diagram illustrates a desktop eye-tracking signal capture and interaction system according to an example embodiment.

[0049] See Figure 1The figure shows a desktop eye-tracking signal capture and interaction system, including an integrated acquisition and display terminal 01, an environmental and physiological signal acquisition unit 02, and a multimodal data processing unit 03. Through the collaborative work of the integrated acquisition and display terminal 01, the environmental and physiological signal acquisition unit 02, and the multimodal data processing unit 03, the system uses a light balance algorithm to automatically switch between light and dark pupil modes according to the ambient light, ensuring the stability of eye tracking. At the same time, it integrates eye-tracking, physiological, and environmental multimodal data to accurately identify the user's control intentions to reduce the error rate. Combined with a medical assistance module and standardized interfaces, it achieves highly adaptable and high-precision multi-scenario eye-tracking interaction.

[0050] According to some embodiments, the integrated acquisition and display terminal 01 includes a display interaction device with an integrated eye-tracking sensor for capturing the user's eye reflection image and transmitting the eye reflection image to the multimodal data processing unit 03. Specifically, the integrated acquisition and display terminal 01 serves as the core human-computer interaction interface of the system, and its display interaction device is preferably a liquid crystal display screen or a touch screen. Optionally, the eye-tracking sensor can be embedded in the display area or frame of the display interaction device, specifically including a near-infrared light source and an image acquisition module. The near-infrared light source emits infrared light of a specific wavelength towards the user's eyes, forming a reflected light spot at the cornea and pupil; the image acquisition module (such as a high-speed CMOS camera) is used to acquire the eye reflection image including the pupil center and corneal reflection point.

[0051] According to some embodiments, the integrated acquisition and display terminal 01 also integrates a signal preprocessing circuit, which is used to perform analog-to-digital conversion and preliminary noise reduction on the acquired eyeball reflection image, and transmit the processed digital image signal to the multimodal data processing unit 03 in real time through a high-speed data transmission interface (such as USB 3.0 or Type-C) for subsequent line-of-sight coordinate calculation and interactive logic analysis.

[0052] According to some embodiments, the integrated acquisition and display terminal 01 includes a magnetically attached filter assembly. Magnetic adsorption points are arranged on the front side of the lens of the eye-tracking sensor, allowing for detachable installation of the magnetically attached filter assembly. Further, the magnetically attached filter assembly is detachably disposed on the front side of the lens of the eye-tracking sensor. Magnetic adsorption points are arranged around the periphery of the lens of the eye-tracking sensor or on the corresponding inner side of the terminal housing. Magnetic components or magnetic conductive components adapted to the magnetic adsorption points are correspondingly disposed on the edge of the magnetically attached filter assembly. Through the magnetic attraction between the magnetic adsorption points and the magnetically attached filter assembly, quick installation and removal of the filter assembly on the front side of the lens are achieved. The magnetic filter component is preferably a near-infrared bandpass filter, used to filter visible light and stray infrared light in ambient light, allowing only the specific wavelength of light required for eye tracking to pass through, thereby effectively improving the signal-to-noise ratio of image acquisition and ensuring the accuracy and stability of eye tracking in strong light or complex lighting environments. The magnetic design makes it easy for users to flexibly replace or remove the filter component according to the actual usage scenario, improving the equipment's maintenance convenience and environmental adaptability.

[0053] Correspondingly, the multimodal data processing unit 03 is also equipped with an optical state monitoring module and a parameter adaptive loading module. The optical state monitoring module is electrically connected to the magnetic adsorption site and is used to monitor the connection status of the magnetic filter component in real time. When the removal or installation of the magnetic filter component is detected, the optical state monitoring module generates a corresponding state switching signal. In response to the state switching signal, the parameter adaptive loading module automatically retrieves the optical correction parameters matching the state from the pre-stored configuration database. Specifically, when the installation of the magnetic filter component is detected, a first set of correction parameters including infrared refractive index compensation and optical path offset correction is loaded; when the removal of the magnetic filter component is detected, the system switches to a second set of correction parameters suitable for full-spectrum or visible light environments. By automatically loading the corresponding optical correction parameters, the multimodal data processing unit 03 can correct the changes in optical path focal length and imaging distortion caused by the introduction or removal of the filter in real time, ensuring that the eye-tracking algorithm can acquire high-precision feature data under different optical configurations, achieving seamless switching and calibration-free operation.

[0054] According to some embodiments, the environmental and physiological signal acquisition unit 02 is used to synchronously acquire relevant environmental and physiological data and transmit them to the multimodal data processing unit 03. Specifically, in some embodiments, the environmental and physiological signal acquisition unit 02 may include an environmental sensor module and a physiological sensor module. The environmental sensor module is used to acquire physical parameters of the environment in which the integrated acquisition and display terminal 01 is located in real time, specifically including a light intensity sensor, a temperature and humidity sensor, and an environmental noise sensor to obtain environmental data such as ambient light intensity, temperature, and background noise levels. The physiological sensor module is used to acquire user-related physiological indicators, specifically including an electroencephalogram (EEG) acquisition device, a skin conductance response acquisition device, and a facial expression capture camera to obtain physiological data such as the user's EEG waveform, skin conductance level, and facial micro-expression images.

[0055] Optionally, according to some embodiments, the environmental and physiological signal acquisition unit 02 can transmit the acquired environmental and physiological data to the multimodal data processing unit 03 in real time via a wired interface (such as RS-485 or USB) or a wireless communication module (such as Wi-Fi or Bluetooth). The multimodal data processing unit 03 adds a timestamp synchronized with the eye-tracking data to each sensor data, achieving millisecond-level time alignment of environmental, physiological, and eye-tracking signals, providing a data foundation for subsequent multimodal data fusion analysis.

[0056] According to some embodiments, the multimodal data processing unit 03 is configured to align the timestamps of eye-tracking data with the timestamps of the external physiological signals to generate a fused data stream. The multimodal data processing unit 03 is equipped with a high-precision time synchronization engine for time-axis alignment of data streams from different sources. Specifically, the multimodal data processing unit 03 receives an eye-tracking data stream and an external physiological signal data stream accessed through the external device synchronization interface. The time synchronization engine parses the timestamps of the eye-tracking data and the external physiological signals. Based on the parsed timestamps, the time synchronization engine employs interpolation compensation and a sliding window alignment algorithm to precisely match the eye-tracking data and the external physiological signals in the time dimension, eliminating clock drift and acquisition delay differences caused by device heterogeneity. After alignment, the multimodal data processing unit 03 generates a fused data stream containing eye-tracking features, physiological signal features, and a unified time label, providing a time-consistent data foundation for subsequent multimodal correlation analysis, cognitive state recognition, and human-computer interaction decision-making.

[0057] According to some embodiments, the multimodal data processing unit 03 is configured with an eye-tracking decoding algorithm. This algorithm converts the received eyeball reflection image into real-time gaze point positioning coordinates and, based on the environmental and physiological data, identifies the control intent, thereby generating visual feedback information and cursor control signals, which are then fed back to the display interaction device. Specifically, in a specific embodiment, the multimodal data processing unit 03 first preprocesses the received eyeball reflection image using a built-in image processing module, including grayscale conversion, filtering and noise reduction, and contrast enhancement. Then, it uses a feature detection submodule in the eye-tracking decoding algorithm (such as the findPupil and findPerkinje algorithms) to accurately locate the pupil center coordinates and the corneal reflection point position. Based on the pupil-corneal reflection vector method (PCCR), a geometric mapping model is used to convert the relative displacement of the aforementioned eye feature points into the user's real-time gaze point positioning coordinates in the screen coordinate system.

[0058] Furthermore, the multimodal data processing unit 03 introduces a decision-level fusion strategy, performing timestamp alignment and correlation analysis between the real-time gaze point positioning coordinates and the physiological data (such as EEG signals and ductal nerve signals) and environmental parameters transmitted by the environmental and physiological signal acquisition unit 02. A trained intent recognition model (such as a Hidden Markov Model or Convolutional Neural Network) is used to classify the multimodal feature vectors, identifying the user's true control intent (such as "confirm selection," "scroll page," or "emergency help"), effectively distinguishing between unintentional gaze and operational intent, and reducing the error rate. This generates visual feedback information including gaze heatmaps and area of ​​interest (AOI) statistics, and outputs cursor movement commands or function trigger signals to the display interaction device, realizing closed-loop human-computer interaction control based on eye movement and physiological feedback.

[0059] According to some embodiments, the multimodal data processing unit 03 supports automatic region of interest (AOI) generation. Based on the ilab visual attention model tool, it performs saliency calculation on the displayed content of the display interaction device, automatically generating AOIs and calculating the gaze time and return rate per character. The display content of the display interaction device is acquired in real-time or pre-loaded, and the ilab visual attention model tool is used to perform saliency calculation on the displayed content. The saliency calculation is based on visual cortical mechanism simulation, generating a saliency map by analyzing the color, brightness, direction, and texture features of the image, thereby automatically identifying and delineating high-attention areas in the displayed content. Based on the saliency map, AOIs are automatically generated, including but not limited to polygonal, rectangular, or circular regions, and assigned corresponding region labels. During eye-tracking data acquisition, the user's gaze coordinates are mapped to the automatically generated AOIs for refined statistical analysis. Specifically, the statistical analysis includes calculating the user's gaze time (i.e., gaze time per character) for each character or text unit within the AOIs, and calculating the return rate when the user's gaze leaves and then returns to the region. Through the automatic generation function and refined statistics, the system can objectively quantify the reading depth and cognitive processing difficulty of users for different text content, eliminating the need for manual drawing of interest areas and significantly improving the efficiency and standardization of data analysis.

[0060] According to some embodiments, the multimodal data processing unit 03 includes: an external device synchronization interface, which connects to external environment and physiological signal acquisition devices via the PsyHUB TPSN protocol. The physiological signal acquisition devices include one or more of an electroencephalogram (EEG) acquisition device, a magnetic resonance imaging (GSR) device, and a facial expression capture camera. The environmental signal acquisition devices include one or more of a temperature and humidity sensor, a light sensor, and a noise sensor. The multimodal data processing unit 03 is configured with a dedicated external device synchronization interface, which is built based on the PsyHUB TPSN protocol to achieve high-precision time synchronization and data communication with the external environment and physiological signal acquisition devices. Through this interface, the system can synchronously access data from multiple types of sensors to construct a comprehensive multimodal perception environment. Specifically, the physiological signal acquisition devices include, but are not limited to, one or more combinations of an EEG acquisition device, a GSR device, and a facial expression capture camera, used to capture the user's physiological and emotional states. The environmental signal acquisition devices include, but are not limited to, one or more combinations of a temperature and humidity sensor, a light sensor, and a noise sensor, used to monitor the physical environment parameters of the user's surroundings. The multimodal data processing unit 03 uses the PsyHUB TPSN protocol to uniformly schedule and collect data from the aforementioned devices, ensuring that data from different sources are precisely aligned on the timeline, thus providing an accurate and synchronized data foundation for subsequent multimodal data analysis and fusion.

[0061] According to some embodiments, the multimodal data processing unit 03 includes: a FACS (Facial Action Coding System) expression recognition engine. The FACS expression recognition engine identifies the user's facial action units based on the input from the facial expression capture camera, thereby identifying the user's expression state, and performs correlation analysis with eye-tracking interest region data to generate an emotion-gaze correlation report. The FACS expression recognition engine is communicatively connected to the facial expression capture camera and configured to receive facial image sequences captured by the facial expression capture camera. The FACS expression recognition engine extracts features from the facial image sequences based on deep learning algorithms, identifies the user's facial action units (AUs) according to the Facial Action Coding System standard. These facial action units include, but are not limited to, micro-expression features such as raised eyebrows, closed eyelids, and raised corners of the mouth. Furthermore, through the combined analysis of action units, the engine identifies the user's real-time expression state (such as happiness, sadness, anger, surprise, disgust, fear, or neutrality).

[0062] Furthermore, the multimodal data processing unit 03 is also equipped with a cross-modal correlation analysis module. This module spatiotemporally correlates the identified facial expression states with the eye movement interest area (AOI) data in the eye-tracking data. Specifically, the cross-modal correlation analysis module analyzes the trend and intensity of facial expression changes when the user gazes at a specific AOI, and calculates the correlation between facial expression states and the duration of gaze and gaze sequence. Based on the results of the correlation analysis, the multimodal data processing unit 03 generates an emotion-gaze correlation report. This report visually displays the emotional fluctuation spectrum of the user when focusing on different target areas, thereby providing multi-dimensional quantitative evidence for assessing the user's cognitive load, emotional preferences, and attention allocation.

[0063] According to some embodiments, the eye-tracking sensor supports a dual-light source switching mode for both bright and dark pupils. This switching is controlled by a light balance algorithm module built into the multimodal data processing unit 03. The light balance algorithm module is configured to: monitor light sensor data from the environmental and physiological data in real time; automatically switch to dark pupil tracking mode when the ambient illuminance exceeds a preset light threshold, and / or automatically switch to bright pupil tracking mode when the ambient illuminance is below a preset light threshold, and dynamically adjust the light source gain. Specifically, the eye-tracking sensor supports a dual-light source switching mode for both bright and dark pupils, and this switching control is executed by the light balance algorithm module built into the multimodal data processing unit 03. The light balance algorithm module is configured to monitor light sensor data from the environmental and physiological data in real time to obtain the current ambient light intensity information. The light balance algorithm module compares the light intensity information with a preset light threshold. When the ambient light intensity exceeds the preset light threshold (e.g., in strong light or outdoor environments), the light balancing algorithm module controls the eye-tracking sensor to automatically switch to dark pupil tracking mode, using dark pupil technology to suppress ambient light interference and ensure clear extraction of pupil features. When the ambient light intensity is below the preset light threshold (e.g., in a dark room or low-light environment), the light balancing algorithm module controls the eye-tracking sensor to automatically switch to bright pupil tracking mode to enhance the contrast of eye features. Furthermore, while switching modes, the light balancing algorithm module is also configured to dynamically adjust the light source gain to optimize the infrared illumination intensity, ensuring high signal-to-noise ratio eye-tracking image acquisition under different lighting conditions, thereby guaranteeing the stability and accuracy of eye tracking.

[0064] According to some embodiments, the multimodal data processing unit 03 includes: a medical assistance decoding module, supporting a medical assistance scenario mode. The medical assistance decoding module is configured with a preset medical instruction set. When in the medical assistance scenario mode, it allows users to select symptoms and affected areas through eye-tracking gaze, and then packages the selection results into alarm information and pushes it to the auxiliary interaction device. Specifically, the medical assistance decoding module is configured with a preset medical instruction set, which includes digital mappings of various common symptom descriptions (such as pain, difficulty breathing, palpitations, etc.) and human anatomical locations (such as head, chest, abdomen, limbs, etc.). Optionally, when the system is in the medical assistance scenario mode, the medical assistance decoding module is configured to display a symptom selection interface or a human model atlas through the display interaction device. Users select corresponding symptom descriptions or affected areas by eye-tracking gaze at a specific area (i.e., the gaze duration exceeds a preset time threshold). The medical assistance decoding module analyzes the user's gaze coordinates in real time, identifies the user's selection intent, and logically combines the results of multiple selections. Subsequently, the module packages the selection results to generate standardized alarm information, which includes user identification, symptom code, and body location code. Finally, the medical assistance decoding module pushes the alarm information to the auxiliary interaction device (such as a nurse station terminal, emergency center server, or family member's mobile terminal), thereby enabling contactless emergency medical assistance and rapid reporting of the patient's condition.

[0065] According to some embodiments, the multimodal data processing unit 03 further includes a data development interface (API) to output raw waveform data of raw eye movement coordinates, pupil diameter, head posture angle, and other external physiological signals to an external analysis device in real time. Specifically, the data development interface supports real-time streaming transmission of multimodal raw data to the external analysis device via RESTful, WebSocket, or gRPC communication protocols. The transmitted raw data includes at least: raw eye movement coordinate data (such as X and Y coordinates of the fixation point) collected by the eye-tracking sensor, pupil diameter change data, head posture angle data (including pitch angle, yaw angle, and roll angle), and raw waveform data of other external physiological signals (such as time-series waveforms of electroencephalograms, electrocardiograms, electromyograms, or skin conductance responses) obtained through the external device synchronization interface. Through the data development interface, the external analysis device can acquire uncompressed or downsampled raw signal streams for higher-precision offline analysis, custom algorithm verification, or deep machine learning model training, thereby expanding the application scope and secondary development capabilities of the system.

[0066] According to some embodiments, the system further includes: the auxiliary interaction unit, comprising an auxiliary interaction device independent of the display interaction device, the auxiliary interaction device being used to output relevant visual feedback information to a third party (such as a guardian, medical staff, or operating assistant), thereby achieving transparency and collaboration in the interaction process. Specifically, when a user selects a preset interaction command (such as "confirm," "return," "help," etc.) on the display interaction device via eye tracking, the multimodal data processing unit 03 not only executes the corresponding operation logic but also parses and transforms the interaction command into intuitive visual feedback information (such as command icon highlighting, operation path animation, status prompt bar, etc.). The visual feedback information is synchronously displayed on the auxiliary interaction device in real time via network or wired transmission, enabling the third party to instantly grasp the user's operational intent and current interaction status. Preferably, the auxiliary interaction device can be a tablet computer, a monitoring terminal, or a remote display device, whose display interface is logically synchronized with the display interaction device but physically isolated, avoiding visual interference to the user, thereby achieving information collaboration between the user and the third party, improving the system's usability and security, and is particularly suitable for application scenarios such as medical monitoring, special education, and remote assistance.

[0067] According to some embodiments, the auxiliary interaction device includes a touch input module, which allows a third party to confirm or respond to the user's eye-tracking selections via the touch input module. The response information is then back-synchronized to the integrated acquisition and display terminal 01 via the multimodal data processing unit 03. Specifically, during collaborative interaction, the multimodal data processing unit 03 transmits the target information or operation request selected by the user through eye-tracking to the auxiliary interaction device for display in real time. A third party (such as medical personnel, guardians, or assistant operators) observes the graphical user interface and uses the touch input module to perform secondary confirmation, rejection, or response operations on the eye-tracking selections. The response information (including confirmation signals, text responses, or instruction codes) generated by the touch input module is fed back to the multimodal data processing unit 03. The multimodal data processing unit 03 is configured with a bidirectional communication link for protocol encapsulation and parsing of the response information and back-synchronization to the integrated acquisition and display terminal 01. The integrated acquisition and display terminal 01 responds to the reply information and presents corresponding visual feedback (such as "Operation confirmed", "Message sent" or specific reply content) on the display interface, thereby constructing a closed-loop interaction process of "eye-tracking input - touch confirmation - visual feedback", which effectively prevents instruction errors caused by eye movement tremors or misoperation, and significantly improves the reliability and security of human-computer interaction.

[0068] According to some embodiments, the desktop eye-tracking signal capture and interaction system includes an adaptive calibration subsystem that supports multiple calibration modes: a standard mode, which displays geometric calibration points; a toddler mode, which uses dynamic cartoon images and background music from a multimedia library to guide children to fixate on calibration points; and a special subject mode, which initiates a gain scanning program to expand the pupil search range and extend the single-point fixation time threshold. The adaptive calibration subsystem automatically selects or switches between multiple calibration modes based on the characteristics of the subject to ensure the accuracy and applicability of eye tracking. Specifically, in the standard mode, the adaptive calibration subsystem controls the display interaction device to present a series of geometric calibration points (such as circles or crosses) to guide adults or subjects with normal cognitive abilities to fixate sequentially, thereby completing gaze mapping. In the toddler mode, the adaptive calibration subsystem uses a pre-stored multimedia library containing dynamic cartoon images and background music; the system attracts children's attention by playing the dynamic cartoon images and guides them to fixate on specific positions on the screen with sound effects, thus completing the calibration process in a fun and interactive way, reducing the difficulty of cooperation for child subjects. In the special subject mode, the adaptive calibration subsystem initiates a gain scanning procedure, which is configured to expand the pupil search range to accommodate atypical eye features (such as ptosis or strabismus) and extend the single-point fixation time threshold to tolerate subject fixation instability, thereby ensuring the success rate of data acquisition.

[0069] Furthermore, the preschool mode also includes a physical assistive component interface. This interface is mounted on the housing of the integrated acquisition and display terminal 01 or the display interaction device and is configured to detachably mount a head-fixing bracket. The head-fixing bracket includes adjustable lateral support pads and a chin rest. The position and spacing of the lateral support pads are adjustable to accommodate the head sizes of children of different ages. By installing the head-fixing bracket, significant head movement in preschool subjects during experiments can be effectively limited, assisting the adaptive calibration subsystem in acquiring high-precision eye-tracking reference data under more stable physical conditions.

[0070] Figure 2 This diagram illustrates the workflow of a desktop eye-tracking signal capture and interaction system according to an example embodiment.

[0071] According to some embodiments, see Figure 2During the data acquisition phase, eye-tracking sensors capture the user's gaze trajectory, fixation point coordinates, and fixation duration, while simultaneously recording dynamic eye parameters such as blink frequency. The environmental and physiological signal acquisition unit integrates a skin conductance sensor, a heart rate monitoring module, and an ambient temperature / light sensor to collect raw data such as skin conductance levels, heart rate variability, ambient temperature, and light intensity in real time. The two types of sensors use timestamp alignment technology to ensure the synchronization of multi-source data, providing a foundation for subsequent analysis.

[0072] Next, expression recognition and feature extraction are performed. The FACS expression recognition engine is based on computer vision algorithms and performs real-time analysis of facial video streams captured by cameras. It uses facial action unit detection technology to identify micro-expression changes in key areas such as eyebrows, eyes, and mouth, and extracts feature parameters such as expression intensity, duration, and combination patterns to form standardized expression feature vectors.

[0073] The system then generates a dynamic attention region. Based on the density and duration of gaze points in the eye-tracking data, combined with scene information provided by environmental sensors (such as screen content area and physical spatial layout), the system dynamically divides the core and peripheral areas of user attention, generating a real-time updated spatial heatmap to identify the specific location where the user's attention is focused.

[0074] Furthermore, cross-modal correlation analysis is performed. Eye-tracking data (gaze trajectory, pupil diameter), physiological data (skin conductance, heart rate), and facial expression feature vectors are input into a multimodal fusion algorithm. Through time series analysis and pattern recognition techniques, the correlation between different modalities is established. For example, when the system detects a combination of pupil dilation (eye movement) + increased skin conductance (physiological) + slightly raised corners of the mouth (facial expression), it identifies it as "high arousal positive emotion"; if there is a combination of gaze dispersion (eye movement) + decreased heart rate (physiological) + furrowed brows (facial expression), it is identified as "low arousal confused state".

[0075] Finally, the execution results are output and interacted with. The fusion analysis results are output in the form of a structured data stream, generating an associated report containing emotional state labels (such as "focused," "anxious," and "pleasant"), gaze heatmaps, and physiological indicator trends. Control intentions are identified, thereby generating visual feedback information and cursor control signals, which are then fed back to the display interaction device or a third-party interactive device. When an abnormal state is detected (such as persistent inattention or emotional fluctuations exceeding a threshold), the system automatically triggers an alarm message, which is pushed to external devices (such as teaching management systems or intelligent driving assistance systems) through a visual interface or API interface.

[0076] In addition, the system provides standardized APIs through data development interfaces, supporting the real-time transmission of fused data streams to third-party platforms, enabling seamless integration with education assessment systems, human-computer interaction devices, or health monitoring platforms, and providing data support for application scenarios such as personalized teaching, driving safety warnings, or mental health interventions.

[0077] According to some embodiments, this invention achieves precise quantitative analysis of a user's emotional state and attention distribution through multimodal data fusion using an eye-tracking sensor, an environmental and physiological signal acquisition unit, and a FACS facial expression recognition engine. Compared to single-modal detection technologies, this system, by simultaneously acquiring gaze trajectories, physiological indicators, and facial micro-expressions, and combining this with a dynamic attention region generation algorithm, can more comprehensively capture the psychological activity characteristics of users in complex scenarios. For example, in driving scenarios, the system can identify driver fatigue or anxiety in advance through combined analysis of pupil diameter changes, skin conductance levels, and micro-expressions at the corners of the mouth, thereby triggering an early warning mechanism and effectively improving driving safety.

[0078] According to some embodiments, the system of this invention employs cross-modal correlation analysis technology to establish a dynamic mapping relationship between eye movement, physiological, and facial expression data, solving the misjudgment problem caused by data isolation in traditional methods. Through timestamp alignment and feature vector fusion, the system can output an emotion-attention correlation report in real time, not only labeling the user's current emotion type (such as focus, confusion, and pleasure) but also locating the specific area of ​​their attention concentration, providing refined data support for educational assessment, human-computer interaction, and other scenarios. Furthermore, the data development interface provided by this invention supports seamless integration with third-party systems, enabling the emotion-attention analysis results to be quickly applied to fields such as intelligent driving and mental health monitoring.

[0079] Figure 3 A block diagram of a computing device according to an example embodiment of this application is shown.

[0080] like Figure 3 As shown, the computing device 30 includes a processor 12 and a memory 14. The computing device 30 may also include a bus 22, a network interface 16, and an I / O interface 18. The processor 12, memory 14, network interface 16, and I / O interface 18 can communicate with each other via the bus 22.

[0081] Processor 12 may include one or more general-purpose CPUs (Central Processing Units), microprocessors, or application-specific integrated circuits, for executing relevant program instructions. According to some embodiments, computing device 30 may also include a high-performance display adapter (GPU) 20 for accelerating processor 12.

[0082] Memory 14 may include a machine-readable medium in the form of volatile memory, such as random access memory (RAM), read-only memory (ROM), and / or cache memory. Memory 14 is used to store one or more programs containing instructions, as well as data. Processor 12 may read the instructions stored in memory 14 to perform the methods described above according to embodiments of this application.

[0083] The computing device 30 can also communicate with one or more networks via the network interface 16. The network interface 16 can be a wireless network interface.

[0084] Bus 22 can include address bus, data bus, control bus, etc. Bus 22 provides a path for exchanging information between components.

[0085] It should be noted that, in specific implementations, the computing device 30 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the device described above may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0086] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), network storage devices, cloud storage devices, or any type of medium or device suitable for storing instructions and / or data.

[0087] This application also provides a computer program product including a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0088] Those skilled in the art will clearly understand that the technical solutions of this application can be implemented using software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware capable of independently performing or cooperating with other components to perform a specific function, where the hardware may be, for example, a field-programmable gate array (FPGA), integrated circuit, etc.

[0089] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0090] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0092] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0094] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.

[0095] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0096] The exemplary embodiments of this application have been specifically shown and described above. It should be understood that this application is not limited to the detailed structures, arrangements, or implementation methods described herein; rather, this application is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended provisions.

Claims

1. A desktop eye-tracking signal capture and interaction system, characterized in that, include: The system includes an integrated data acquisition and display terminal, an environmental and physiological signal acquisition unit, and a multimodal data processing unit. The integrated acquisition and display terminal includes a display interaction device with an integrated eye-tracking sensor, used to capture the user's eye reflection image and transmit the eye reflection image to the multimodal data processing unit; An environmental and physiological signal acquisition unit is used to synchronously acquire relevant environmental and physiological data and transmit them to the multimodal data processing unit; The multimodal data processing unit is equipped with an eye-tracking decoding algorithm. The eye-tracking decoding algorithm converts the received eye reflection image into real-time gaze point positioning coordinates, and identifies the control intention based on the environmental and physiological data, thereby generating visual feedback information and cursor control signals, which are then fed back to the display interaction device.

2. The desktop eye-tracking signal capture and interaction system according to claim 1, characterized in that, Also includes: The auxiliary interaction unit includes an auxiliary interaction device independent of the display interaction device, the auxiliary interaction device being used to output relevant visual feedback information to a third party.

3. The desktop eye-tracking signal capture and interaction system according to claim 1, characterized in that, The multimodal data processing unit includes: an external device synchronization interface, which connects to the external environment and physiological signal acquisition equipment via the PsyHUB TPSN protocol, wherein... Physiological signal acquisition devices include one or more of electroencephalogram (EEG) acquisition devices, electrodermal conductance (EDA) acquisition devices, and facial expression capture cameras; environmental signal acquisition devices include one or more of temperature and humidity sensors, light sensors, and noise sensors.

4. The desktop eye-tracking signal capture and interaction system according to claim 3, characterized in that, The multimodal data processing unit includes an FACS facial expression recognition engine, which identifies the user's facial movements based on the input from the facial expression capture camera, thereby identifying the user's facial expression state, performing correlation analysis with eye movement interest area data, and generating an emotion-gaze correlation report.

5. The desktop eye-tracking signal capture and interaction system according to claim 3, characterized in that, The eye-tracking sensor supports a dual-light source switching mode with both bright and dark pupils, and the switching is controlled by the light balance algorithm module built into the multimodal data processing unit. The light balance algorithm module is configured as follows: Real-time monitoring of light sensor data from the environmental and physiological data; When the ambient light level exceeds the preset light threshold, it automatically switches to dark-eye tracking mode, and / or When the ambient light intensity is lower than the preset light threshold, it automatically switches to the Bright Eye Tracking mode and dynamically adjusts the light source gain.

6. The desktop eye-tracking signal capture and interaction system according to claim 1, characterized in that, The integrated acquisition and display terminal includes a magnetic filter assembly, with magnetic adsorption points arranged on the front side of the lens of the eye-tracking sensor, so that the magnetic filter assembly can be detachably installed.

7. The desktop eye-tracking signal capture and interaction system according to claim 2, characterized in that, The multimodal data processing unit includes a medical assistance decoding module that supports medical assistance scenario modes. The medical assistance decoding module is configured with a preset medical instruction set. When in medical assistance scenario mode, it supports users to select symptoms and locations through eye-tracking gaze, and then packages the selection results into alarm information and pushes it to the auxiliary interaction device.

8. The desktop eye-tracking signal capture and interaction system according to claim 3, characterized in that, The multimodal data processing unit also includes a data development interface, which outputs raw waveform data of raw eye movement coordinates, pupil diameter, head posture angle and other external physiological signals to external analysis devices in real time.

9. The desktop eye-tracking signal capture and interaction system according to claim 1, characterized in that, The multimodal data processing unit supports the automatic generation of regions of interest. Based on the ilab visual attention model tool, it performs saliency calculation on the displayed content of the display interaction device, automatically generates eye-tracking regions of interest, and counts the gaze time and reversion rate for each word.

10. The desktop eye-tracking signal capture and interaction system according to claim 1, characterized in that, The auxiliary interaction device includes a touch input module, which allows third parties to confirm or respond to the user's eye-tracking selections through the touch input module, and to synchronize the response information back to the integrated acquisition and display terminal through the multimodal data processing unit.

Images