ADHD neural feedback training system and method based on bimodal fusion
By combining EEG signals and eye movement behavior, a multidimensional attention state classification model was constructed, which solved the problem of recognition distortion in complex scenarios in existing ADHD neurofeedback training systems, achieving more accurate attention state recognition and personalized feedback, and improving training effectiveness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-03-13
AI Technical Summary
Existing ADHD neurofeedback training systems have a single dimension when recognizing attentional states and lack the ability to comprehensively analyze individual gaze objects and visual guidance factors, resulting in distorted recognition results and failure of feedback regulation in complex scenarios.
By combining EEG signals and eye-tracking behavior, a multidimensional attention state classification model is constructed. Data is collected using a BrainLink Lite monochannel EEG instrument and an ASeeGlasses eye tracker. Feature fusion and supervised learning algorithms are employed to construct a 2×2 attention state classification model, enabling more comprehensive and accurate identification of attention states. Real-time feedback is provided through an intelligent vehicle.
It enables finer-grained recognition of attentional states, improves recognition accuracy and adaptability, provides personalized attention intervention, and enhances training effectiveness and engagement.
Smart Images
Figure CN121647673A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of electroencephalography (EEG) and eye-tracking technology, specifically to an ADHD neurofeedback training system and method based on bimodal fusion. Background Technology
[0002] EEG is a non-invasive technique for recording brain electrical activity, widely used in neuroscience research and clinical monitoring. It reflects the characteristics of an individual's neural activity under different cognitive states by collecting electrical potential changes caused by the firing of neurons in the cerebral cortex. In attention studies, different frequency bands of brain waves (such as Delta, Theta, Alpha, and Beta waves) are closely related to attention levels. In particular, the Theta / Beta ratio has been considered a potential biomarker for assessing ADHD.
[0003] Eye-tracking (ET) technology, as a non-invasive measurement technique, reveals the distribution characteristics of an individual's visual attention during task execution by accurately recording indicators such as fixation point, gaze duration, saccade path, and pupillary changes. Compared to EEG, eye-tracking data has good spatial resolution, clearly showing whether the subject is focusing on the target object, reflecting the individual's spatial attention behavior, and can help determine the visual attention focus area, thus having auxiliary value in the assessment of attention disorders such as ADHD.
[0004] Neurofeedback training is a non-invasive psychological intervention method that targets brain activity and achieves self-regulation through real-time feedback. Its basic principle is to convert key parameters of brain electrical signals (such as frequency band energy) into visual or auditory feedback, prompting subjects to actively adjust their brain electrical activity under feedback guidance, thereby optimizing and reshaping neural function. This manifests directly as improved cognitive or behavioral symptoms caused by abnormal brain waves; in the long term, unconscious behavioral changes gradually transform into the ability to actively regulate neural rhythms, thus achieving effective control over attention, emotions, and behavior. This process relies on the brain's neuroplasticity, meaning that with continuous training, brain electrical activity patterns can be gradually optimized to form a more stable and efficient functional state. In ADHD intervention, neurofeedback training has been widely validated as an effective treatment method.
[0005] The inventors discovered that some existing attention training technologies based on EEG neurofeedback attempt to assist children with ADHD in rehabilitation training through virtual tasks, mini-game control, or smart hardware interaction. These systems can improve children's participation and enjoyment in attention training to some extent. However, these systems generally suffer from a single dimension of attention state recognition; that is, existing "attention states" are mostly defined and focused on the high level of focus shown on EEG. Current neurofeedback training systems mainly rely on changes in EEG energy bands to determine attention states, lacking the ability to comprehensively analyze individual objects of focus, eye gaze distribution, and visual guidance factors. Therefore, when faced with complex scenarios such as task interference, visual wandering, or environmental changes, the system often fails to accurately distinguish distraction states, leading to distorted recognition results and ineffective feedback modulation. Existing technologies struggle to simultaneously capture information on both "whether the subject is focused" and "what they are focused on," limiting the accuracy and individual adaptability of neurofeedback training systems in attention intervention. Therefore, there is an urgent need to develop a multimodal recognition method that integrates EEG neural signals and eye movement behavior data to achieve more comprehensive and accurate recognition and intervention feedback of attention states. Summary of the Invention
[0006] The purpose of this invention is to assist individuals with Attention Deficit and Hyperactivity Disorder (ADHD) in attention enhancement training. This invention combines electroencephalogram (EEG) signals with eye movement behavior to construct a multidimensional attention state classification model, and proposes a neurofeedback training method that integrates eye movements to achieve more accurate and targeted attention training.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: an ADHD neurofeedback training system based on bimodal fusion, characterized in that it includes: a computer server, an intelligent vehicle, and further includes:
[0008] Data acquisition module: The data acquisition module is used to acquire electroencephalogram (EEG) signal data and eye movement signal data;
[0009] Feature fusion module: The feature fusion module is used to preprocess the two sets of data to construct bimodal features;
[0010] Classification and Recognition Module: The classification and recognition module outputs classification results based on the fused features;
[0011] Real-time feedback module: The real-time feedback module coordinates with the intelligent vehicle based on the classification results;
[0012] The computer server is connected to the data acquisition module, feature fusion module, classification and recognition module, real-time feedback module, and intelligent vehicle communication module.
[0013] Furthermore, the data acquisition module includes an electroencephalogram (EEG) signal acquisition device and an eye movement signal acquisition device. The EEG signal acquisition device is a BrainLink Lite single-channel EEG meter, and the eye movement signal acquisition device is an ASeeGlasses eye tracker.
[0014] A further method for an ADHD neurofeedback training system based on bimodal fusion is as follows:
[0015] S1. The neural activity and gaze behavior of the subject during the task execution process are collected by the EEG signal acquisition device and the eye movement signal acquisition device, and multi-dimensional data including EEG signals and eye movement signals are acquired in real time.
[0016] S2. The EEG signal data and eye movement signal data are preprocessed through the feature fusion module to obtain EEG features and eye movement features. The two types of features are synchronized based on the timestamp. By setting different sliding window sizes, a bimodal feature pair is constructed.
[0017] S3. Based on the fused feature vectors, the classification and recognition module constructs a 2×2 attention state classification model using a supervised learning algorithm, outputting four states. Then, a decision-level fusion structure and an intermediate fusion structure are used to output the classification results.
[0018] S4. The real-time feedback module provides real-time visual feedback on physical objects based on the classification results, dynamically realizing real-time interaction and linkage with the intelligent vehicle.
[0019] Furthermore, the preprocessing includes denoising, segmenting, and feature extraction of EEG signal data to obtain EEG features reflecting neurocognitive states, and cleaning, aggregating, and extracting features of eye movement signal data to obtain eye movement features reflecting visual fixation behavior.
[0020] Furthermore, the four states include:
[0021] Focus: This includes both eye contact and highly concentrated brain activity;
[0022] Pseudo-focus: The eyes are fixed on the target, but the brain signals show a relaxed state;
[0023] Distraction: When the eyes are not focused on the target, cognitive activity in the brain is reduced;
[0024] False distraction: Eye movements are not focused on the target, but brain signals remain highly concentrated;
[0025] It can not only determine whether someone is focused, but also identify whether their attention is directed at a target specified by the system, achieving a more granular recognition than traditional single EEG classification schemes.
[0026] Furthermore, the real-time interactive linkage specifically means that when the user is identified as being focused, the intelligent vehicle will accelerate as a reward incentive; when the user is identified as being in a state of pseudo-focus, pseudo-distraction, or distraction, the vehicle speed will be reduced or stopped to prompt the user to adjust their attention. The feedback content is based on the joint decision-making of the current gaze behavior and neural state, and has a dual regulation function of "state-oriented + goal-oriented", realizing a highly personalized attention intervention mechanism.
[0027] The beneficial effects of this invention are:
[0028] 1. Breaking through the limitations of the traditional single recognition dimension in ADHD neurofeedback regulation, it achieves more granular judgment by integrating EEG and eye movement behavior information, upgrading the judgment index to a multi-dimensional decoding that integrates visual gaze and neural activity, comprehensively decoding attention state, and significantly improving the accuracy and adaptability of attention state recognition.
[0029] 2. A dual-modal time synchronization acquisition and alignment mechanism is proposed to solve the delay and error problems existing in the modal fusion stage of the current system;
[0030] 3. A complete "identification-feedback-training" closed loop has been constructed, creating a wearable, scalable, and low-burden intervention platform for people with ADHD and other conditions. It can be widely applied to non-pharmacological treatment of children and adolescents with ADHD and has good practical application potential and promotion value. Attached Figure Description
[0031] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the specific embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 This is a system flowchart of the present invention. It illustrates the basic process structure of data acquisition, identification, and feedback.
[0033] Figure 2 This is a picture of an actual EEG signal acquisition device;
[0034] Figure 3 This is a schematic diagram of an eye-tracking device;
[0035] Figure 4 Label the region of interest (AOI) map;
[0036] Figure 5 A visual illustration of the gaze trajectory;
[0037] Figure 6 This is a schematic diagram of the system's composition and structure;
[0038] Figure 7 This is a schematic diagram illustrating the principle of EEG signal acquisition.
[0039] Figure 8 This is a physical image of a smart car model;
[0040] Figure 9 A schematic diagram illustrating the construction of a 2×2 attention state classification model;
[0041] Figure 10 This is a diagram showing the system usage state in the experimental scenario;
[0042] Figure 11 This is a schematic diagram of a gaze heatmap;
[0043] Figure 12 This is a flowchart of the EEG signal acquisition and processing process;
[0044] Figure 13 This is a diagram of the classification model structure.
[0045] Figure 14 The confusion matrix diagram for a multi-class classification model;
[0046] Figure 15 This is a classification performance evaluation table. Detailed Implementation
[0047] The present invention will be described in detail below with reference to specific embodiments and examples, thereby making the advantages and various effects of the present invention more clearly apparent. Those skilled in the art should understand that these specific embodiments and examples are for illustrative purposes only and are not intended to limit the present invention.
[0048] Throughout this specification, unless otherwise specified, the terminology used herein should be understood to have the meaning commonly used in the art. Therefore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. In the event of any conflict, this specification shall prevail.
[0049] Unless otherwise specified, all raw materials, reagents, instruments and equipment used in this invention can be purchased from the market or prepared by existing methods.
[0050] like Figure 1-15 As shown, the present invention has the following specific embodiments.
[0051] Example 1
[0052] A bimodal fusion-based ADHD neurofeedback training system consists of four core modules: a data acquisition module, a feature fusion module, a classification and recognition module, and a real-time feedback module. The data acquisition module includes a BrainLink Lite mono-channel EEG device. Figure 2 , Figure 7 ) and ASeeGlasses eye tracker ( Figure 3 The system collects user neural electrical signals and gaze behavior separately, and the computer server is responsible for receiving and synchronously processing the bimodal data. Figure 6 ), and generate attention classification judgments, controlling multiple modules to drive the intelligent vehicle (feedback terminal, Figure 8 It provides behavioral feedback by adjusting its driving status.
[0053] Example 2
[0054] Data acquisition and preprocessing process ( Figure 12 The EEG signal was acquired at a frequency of 512Hz. After preprocessing by front-end hardware and filtering algorithms, the system used the Python platform to parse the raw data stream. Figure 12 The collected EEG signals were then saved as CSV files. The signals were further processed by filtering and ICA independent component analysis to remove artifacts and eliminate interference factors such as electromyography and blinking. Subsequently, the signals were divided into different time windows (Epochs) according to the experimental trigger events, and feature values reflecting cognitive load, such as event-related spectral perturbation (ERSP), power spectral density (PSD), and time-frequency energy, were extracted.
[0055] Simultaneously, the eye-tracking system can acquire behavioral parameters such as the subject's fixation point position, pupil diameter, and fixation duration in the experimental scene. The system then sets the region of interest (AOI) based on the task screen. Figure 4 ), and combined with gaze trajectory ( Figure 5 ) and gaze heatmap ( Figure 11 The subjects' visual attention behavior was spatially encoded to determine whether they were focused on the target object or whether there was visual wandering. All EEG and eye movement data were aligned based on a unified timestamp to ensure that the dual-channel signals were semantically comparable within the same time window.
[0056] Example 3
[0057] Attention state classification model construction: An attention state classification system based on a two-dimensional division of "whether gazing + whether focusing" (e.g., ...) Figure 9 As shown, the current cognitive state of an individual is divided into four categories: focus (both eye contact and highly concentrated brain activity), pseudo-focus (eyes are on the target but brain signals show a relaxed state), pseudo-distraction (eye movement is not on the target but brain signals remain highly concentrated), and distraction (eyes are not focused on the target while brain cognitive activity is low). This classification framework breaks through the traditional binary judgment of "focus" and "distraction" in neurofeedback systems, and can more accurately distinguish the deviation between visual behavior and cognitive state, providing a more detailed basis for intervention feedback.
[0058] The classification model employs a decision-level fusion structure and a mid-term fusion structure. Figure 13 This involves training independent classifiers (such as random forests or support vector machines, SVMs) for EEG and eye-tracking features respectively, with each classifier outputting intermediate judgment results. These results are then combined using a weighted voting mechanism or decision function to form the final classification result. This structure enhances the system's ability to independently model and jointly infer different modal features, improving the model's robustness and generalization under multimodal input conditions, and providing a model foundation for subsequent comparison work.
[0059] Example 4
[0060] This section compares and evaluates four common classification algorithms: Logistic Regression, Random Forest, Support Vector Machine (SVM), and XGBoost. Figure 14 As shown, the confusion matrix results for each model across the four attentional states demonstrate that Random Forest and XGBoost exhibit better discriminative ability. Figure 15 The corresponding classification performance comparison table shows that Random Forest outperforms other models in both precision and recall, making it the preferred algorithm for subsequent training and deployment. This evaluation experiment was based on subjects in various experimental scenarios (such as...). Figure 10 The data collected (as shown) were used for modeling and validation, covering various cognitive states such as gaze, conversation, distraction, and task execution, which verified the practicality and stability of the system in natural environments.
[0061] After the classification results are output, the system inputs the judgment results to the feedback control module to achieve real-time interactive linkage with the intelligent vehicle model. Figure 8 In a positive feedback scenario, when the intelligent vehicle detects that the user is focused, it will accelerate as a reward incentive. When it detects that the user is in a state of pseudo-focus, pseudo-distraction, or distraction, it will slow down or stop, prompting the user to adjust their attention. This interactive closed-loop mechanism not only enhances the sense of participation and behavioral response during the training process, but also effectively promotes the user's ability to regulate their own neural rhythms, thereby achieving the goal of psychological intervention.
[0062] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.
[0063] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps specifying the functions in one or more boxes. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An ADHD neurofeedback training system based on bimodal fusion, characterized in that, include: Computer servers, smart cars, and also: Data acquisition module: The data acquisition module is used to acquire electroencephalogram (EEG) signal data and eye movement signal data; Feature fusion module: The feature fusion module is used to preprocess the two sets of data to construct bimodal features; Classification and Recognition Module: The classification and recognition module outputs classification results based on the fused features; Real-time feedback module: The real-time feedback module coordinates with the intelligent vehicle based on the classification results; The computer server is connected to the data acquisition module, feature fusion module, classification and recognition module, real-time feedback module, and intelligent vehicle communication module.
2. The system according to claim 1, characterized in that: The data acquisition module includes an electroencephalogram (EEG) signal acquisition device and an eye movement signal acquisition device. The EEG signal acquisition device is a BrainLink Lite single-channel EEG meter, and the eye movement signal acquisition device is an ASeeGlasses eye tracker.
3. The method for an ADHD neurofeedback training system based on bimodal fusion according to claims 1-2, characterized in that: Specifically as follows: S1. The neural activity and gaze behavior of the subject during the task execution process are collected by the EEG signal acquisition device and the eye movement signal acquisition device, and multi-dimensional data including EEG signals and eye movement signals are acquired in real time. S2. The EEG signal data and eye movement signal data are preprocessed through the feature fusion module to obtain EEG features and eye movement features. The two types of features are synchronized based on the timestamp. By setting different sliding window sizes, a bimodal feature pair is constructed. S3. Based on the fused feature vectors, the classification and recognition module constructs a 2×2 attention state classification model using a supervised learning algorithm, outputting four states. Then, a decision-level fusion structure and an intermediate fusion structure are used to output the classification results. S4. The real-time feedback module provides real-time visual feedback on physical objects based on the classification results, dynamically realizing real-time interaction and linkage with the intelligent vehicle.
4. The method according to claim 3, characterized in that: The preprocessing includes denoising, segmenting, and feature extraction of EEG signal data to obtain EEG features reflecting neurocognitive states, and cleaning, aggregating, and extracting features of eye movement signal data to obtain eye movement features reflecting visual fixation behavior.
5. The method according to claim 3, characterized in that: The four types of states include: Focus: This includes both eye contact and highly concentrated brain activity; Pseudo-focus: The eyes are fixed on the target, but the brain signals show a relaxed state; Distraction: When the eyes are not focused on the target, cognitive activity in the brain is reduced; False distraction: Eye movements are not focused on the target, but brain signals remain highly concentrated.
6. The method according to claim 3, characterized in that: The real-time interactive linkage is as follows: when the intelligent vehicle detects that the user is in a focused state, it will accelerate as a reward incentive; when it detects that the user is in a pseudo-focused, pseudo-distracted, or distracted state, it will slow down or stop to prompt the user to adjust their attention.