Assistant communication system for disturbance of consciousness integrating eye movement tracking and electroencephalogram

Abstract

The invention discloses a disturbance of consciousness auxiliary communication system fusing eye movement tracking and electroencephalogram, comprising a data acquisition device connected with a multi-mode BCI communication system and used for collecting scalp electroencephalogram signals and eye movement signals; the multi-modal BCI communication system can identify the answer (yes / no) of the subject by detecting and analyzing the multi-modal data collected from the communication normal form, so as to realize online binary communication; the multi-modal BCI communication system comprises a multi-modal target identification network, and the multi-modal target identification network specifically comprises a data preprocessing module used for preprocessing and synchronizing different modal data; the double-flow feature extraction module is composed of two independent multi-scale convolutional neural networks which are respectively used for EEG and eye movement data so as to carry out feature extraction on data of different modals; according to the invention, a novel BCI alternating normal form integrating P300 potential and eye movement tracking is designed; the P300 detection and the eye movement tracking detection have irreplaceability and complementarity.

Description

Technical Field

[0001] The present invention belongs to the fields of brain-computer interface, eye tracking and artificial intelligence technology, and specifically relates to a consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram. Background Art

[0002] Disorders of consciousness (DOC) is a serious clinical disease characterized by varying degrees of wakefulness and abnormalities in consciousness and cognition. According to the level of retained consciousness, disorders of consciousness can be divided into: coma, vegetative state (VS) / unresponsive wakefulness syndrome (UWS), and minimally conscious state (MCS). VS patients are unconscious and may wake up from a coma, with only reflex movements retained in behavior. MCS patients retain a certain degree of consciousness and may show reproducible non-reflex movements. Locked-in syndrome (LIS) patients retain self-awareness and near-normal cognitive abilities, but have severe sensory and motor deficits. Misdiagnosis of a patient's state of consciousness can have serious medical and ethical consequences.

[0003] The key sign of diagnosing consciousness disorder is whether the patient can follow instructions, so communication ability is an important indicator for assessing the level of consciousness. At present, the main means of consciousness detection for patients with consciousness disorder include: behavioral scale assessment represented by the Coma Recovery Scale-Revised (CRS-R) and neuroimaging assessment centered on functional magnetic resonance imaging (fMRI). However, these consciousness diagnosis methods have some defects. In the behavioral scale assessment, doctors mainly evaluate the patient's various functions based on the patient's different behavioral responses to different stimuli (for example, CRS-R includes six major functional assessments: vision, hearing, movement, speech, communication, and awakening). This diagnostic method is highly subjective, lacks quantifiable indicators, and is heavily dependent on the patient's lack of motor behavior ability. These contradictions may lead to the misdiagnosis of patients with residual consciousness. Studies have shown that the misdiagnosis rate of behavioral assessment-based methods is as high as 40%. Functional magnetic resonance imaging can show the patient's brain activity pattern in specific activity tasks, so it can provide an objective result for the diagnosis of patients with consciousness disorder. However, the equipment is expensive, inconvenient, and complicated to operate. It requires professional operation and analysis, which limits its application in clinical settings and places high technical requirements on operators and analysts. In order to avoid misdiagnosis caused by movement disorders and other reasons and to improve the accuracy of assessment, it is necessary to introduce simpler, objective and effective methods to supplement the assessment of consciousness status, so that doctors can more easily obtain accurate information on the patient's consciousness status in a clinical setting. In addition, communication is a basic ability that conscious patients generally lack but urgently need. Providing a new communication channel for patients with hidden consciousness or potential communication ability is of great significance to assist medical personnel in assessing the level of consciousness disorders, determining prognosis, protecting patients' basic human rights, and promoting personalized treatment.

[0004] Brain-computer interfaces (BCIs) can detect brain activity through electroencephalograms (EEGs), and therefore can provide an objective and reliable method for assessing patients with impaired consciousness. In addition, compared with functional magnetic resonance imaging, brain-computer interface devices are lighter and less expensive to operate. In recent years, brain-computer interfaces have shown good performance in the field of impaired consciousness. More and more BCIs are being used to supplement single-item assessments in the augmented behavior scale (such as communication assessments and auditory assessments), but their shortcomings still exist. Most BCIs for patients with impaired consciousness rely on a single modality, which may limit the information used for decision-making. In addition, due to the particularity of the application objects, BCIs for patients with impaired consciousness also have special requirements for experimental paradigm design. Most BCIs fail to achieve the following at the same time: a simple user graphical interface (GUI), mild paradigm stimulation, and simple and easy-to-operate tasks. Most importantly, most of these BCIs require patients to complete continuous cognitive tasks, which is a challenge for DOC patients with limited attention. Currently, BCI research related to communication is also very limited, with only 24% of BCIs for DOC patients being used for communication applications. This type of brain-computer interface is still in its infancy, and designing a stable BCI with a low threshold for use is expected to improve this situation.

[0005] In terms of recognition algorithms, previous studies on brain-computer interfaces for consciousness disorder detection and communication have mainly focused on traditional machine learning, such as support vector machines (SVMs), linear discriminant analysis (LDA), decision trees, and random forests. These methods may not be able to effectively mine deep information from the data, which may be the reason why the performance of this type of BCI is not ideal. In recent years, with the development of deep learning technology, neural networks (such as convolutional neural networks (CNNs)) have gradually shown satisfactory performance in the field of BCI. Although these methods have produced good results in healthy subjects, their accuracy depends to a certain extent on the quantity and quality of training data, which may not be suitable for DOC patients. Because DOC patients are prone to fatigue and lack of concentration, it is difficult to collect enough high-quality data from them. Although most EEG signal classification models use batch normalization (BN) and dropout strategies to improve generalization, overfitting may still be unavoidable on patient datasets with particularly small data volumes. Different components may coexist in EEG data for a period of time, reflecting different stages of neural activity. This may indicate that feature information at different scales represents different components and contextual features, but few studies have paid attention to this. In addition, most multimodal BCIs for DOC adopt decision-level fusion strategies, which may have difficulty integrating rich semantic information from each modality and depicting the associations between them. It remains a challenge to propose reliable BCI technology for clinical application in patients with DOC.

[0006] In addition to communication ability, the patient's visual ability is also an important evaluation indicator for the diagnosis of consciousness disorder. However, the visual assessment of CRS-R (such as visual tracking, visual positioning, etc.) mainly relies on doctors to make manual judgments, which lacks reliability, and the patient's weak eye movements may not be discerned by doctors. Eye tracking technology that can detect various eye movements (such as gaze positioning, pupil changes, and eye saccades) may be a useful auxiliary means. In addition, due to the rapid development of computer vision and wearable devices in recent years, communication applications can be realized using simple shooting devices (such as cameras) or glasses-type eye trackers. Unfortunately, systems based on eye tracking technology alone cannot interact accurately with patients (patient data cannot be accurately classified based on eye tracking signals alone). Some recent studies have shown that it is possible to improve system performance by fusing eye tracking data and EEG data. However, these studies or inventions of integrated eye tracking technology mainly focus on patients with movement disorders, ignoring DOC patients with more special conditions. The user interfaces of these systems or devices are more complex, more difficult to operate, and require a higher level of cognition, which is not suitable for patients with consciousness disorders who have low levels of consciousness, poor comprehension, limited attention, and easy fatigue. Moreover, most of these inventions or studies only involve healthy subjects and lack testing on patient groups. Although they enhance confidence in interaction by integrating eye movements and EEG signals, the application of such multimodal devices in disorders of consciousness remains to be verified. In addition, almost all interactive devices based on eye tracking use only single eye movement features (e.g., only pupil reactions or gaze positioning). Combining multiple eye movement features for decision-making may be another improvement to improve the accuracy of interaction with patients. Summary of the invention

[0007] The purpose of the present invention is to provide a consciousness disorder auxiliary communication system that integrates eye tracking and electroencephalography, so as to solve the problems in the field of consciousness disorders raised in the above background technology, such as high misdiagnosis rate of traditional diagnostic methods, unsatisfactory performance of related BCI, high threshold and difficulty of use.

[0008] To achieve the above object, the present invention provides the following technical solution: a consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram, comprising:

[0009] Data acquisition equipment, connected to the multimodal BCI communication system, used to collect scalp EEG signals and eye movement signals;

[0010] The multimodal BCI communication system can detect and analyze the multimodal data collected from the communication paradigm and identify the subject's response (yes / no), thereby achieving online binary communication;

[0011] The multimodal BCI communication system includes a multimodal target recognition network, and the multimodal target recognition network specifically includes:

[0012] Data preprocessing module, used for preprocessing and synchronization of data of different modalities;

[0013] The dual-stream feature extraction module consists of two independent multi-scale convolutional neural networks, one for EEG and one for eye movement data, to extract features from different modalities of data;

[0014] The multimodal attention module is used to integrate the multimodal signal features output by the dual-stream feature extraction module to achieve feature complementarity;

[0015] The prototype classifier, which is composed of a prototype network based on cosine distance, is used to improve the classification performance of the system for small sample data.

[0016] Preferably, the dual-stream feature extraction module includes:

[0017] EEG feature extraction module, the multi-scale convolutional neural network of this EEG feature extraction module mainly includes 6 layers:

[0018] Eye movement feature extraction module, the multi-scale convolutional neural network of this eye movement feature extraction module consists of a 7-layer architecture.

[0019] Preferably, the 6-layer structure of the multi-scale convolutional neural network is as follows:

[0020] Spatial convolution layer: It uses weighted stacking averaging and common spatial filtering to improve the signal-to-noise ratio of the EEG signal and eliminate redundant spatial information. It consists of a convolution kernel of size (30, 1). The size of the convolution kernel is equal to the number of electrodes, and the step size of the convolution is (1, 1). The calculation process of this layer is as follows:

[0021]

[0022] Among them, f tanh is the hyperbolic tangent activation function, I c,i represents the i-th moment element value of the c-th channel, represents the convolution kernel matrix, represents additive bias;

[0023] Temporal convolution layer: Two parallel convolution operations and batch normalization operations are introduced to capture more differentiated features; the two convolutions have the same convolution step size (1,1) and convolution kernels of different sizes [(1,2), (1,4)]; this strategy can help extract different temporal features and improve the effectiveness of these features; batch normalization is used to reduce internal covariate bias, avoid saturation problems, and ensure faster training speed; the formula of this layer is as follows:

[0024]

[0025] and represents the feature map obtained from the receptive fields of different scales of different convolution kernels, which contains contextual features of different scales in the EEG data; f BN represents batch normalization, and Especially two different convolution kernel matrices and biases;

[0026] Integration layer: Cascade the temporal features extracted in the previous layer to integrate the local details of the EEG data with the global information; the fused features will contain more global information while retaining the local information of the EEG;

[0027] Max pooling layer: Use a kernel of size (1,2) to pool the integrated features to expand the receptive field and reduce redundancy, thereby improving the performance of the network;

[0028] Fully connected layer: Flatten the multi-dimensional features to prepare for the subsequent fully connected (FC) operation; to enhance the generalization ability of the network, the Dropout strategy is adopted in this layer; the formula of this layer is summarized as follows:

[0029] v=Bernoulli(P)

[0030] a [5] =f tanh ((v×a [4] )×w [5] +b [5] )

[0031] The Bernoulli function randomly generates a 01 vector with a probability of P. By multiplying the vector with the input feature map value by value, the purpose of suppressing P% of neurons can be achieved; tanh is the tanh activation function, a [4] represents the feature output of the L4 maximum pooling layer, w [5] and b [5] They are the convolution kernel matrix and bias respectively;

[0032] Fully connected layer: Fully connected operations are performed in this layer to generate representative EEG features for all input samples; the formula is summarized as follows:

[0033] a [6] =f tanh (a [5] × [6] +b [6] )

[0034] where f tanh is the tanh activation function, w [6] and b [6]are the convolution kernel matrix and bias respectively.

[0035] Preferably, the multimodal attention module operates as follows:

[0036] First, element-by-element summation is performed to preliminarily fuse the extracted representative features of each modality;

[0037] F fuse =F eeg +F eye

[0038] Among them, F eeg ∈R 1×X and F eye ∈R 1×X are representative features of EEG data and eye movement data;

[0039] In order to achieve accurate adaptive feature selection, the compact feature z∈R is introduced 1×d ; z is implemented through the FC layer to reduce dimensionality and improve efficiency; z is calculated as follows:

[0040] z=FC(F fuse ) = f relu (f BN (F fuse × [2] ))

[0041] Among them, f relu is the linear rectification activation function, f BN represents batch normalization, w [2] is the convolution kernel;

[0042] Subsequently, convolution operations and Softmax functions are used to achieve cross-channel soft attention to information of different spatial scales. Specifically, two convolutions are performed under the guidance of the compact feature z to obtain the corresponding weights of each feature. The Softmax function is then applied at the channel level to dynamically focus on information of different spatial scales. The calculation process is summarized as follows:

[0043] W [3,1] =f relu (z×w [3,1] +b [3,1] )

[0044] W [3,2] =f relu (z×w [3,2] +b [3,2] )

[0045]

[0046] Where W [3,1] and W [3,2]is the feature map calculated by different convolution kernels, w [3,1] and w [3,2] represents the convolution kernel, b [3 ,1] and b [3,2] Indicates bias; and W [3,1] and v [4,1] The xth element of [4,1] ∈R 1×X is the feature map (soft attention vector) obtained after the Softmax operation; similarly, and v [4,2] It also follows a similar pattern; finally, each original feature is weighted with the soft attention vector to obtain the final fused feature;

[0047] F=v [4,1] ·F eeg +v [4,2] ·F eye ,v [4,1] +v [4,2] =1;

[0048] The fused feature F is the final representative feature of the input sample. That is, the embedding of all samples (non-linear mapping of input samples to embedding space) is calculated at this layer.

[0049] Preferably, the prototype classifier works by clustering the embeddings of samples around their class prototypes, and classifying the embeddings F of the query sample by identifying the most similar class prototype;

[0050] The task of the prototype classifier can be described as a binary classification task, that is, distinguishing between positive classes (targets, labeled 1) and negative classes (non-targets, labeled -1); the prototype is obtained by averaging the features of all samples from the same class in the training set; the positive prototype F positive The calculation is as follows:

[0051]

[0052] For identifying the "most similar class prototype", the model performs identification and classification by calculating the distance between the class prototype and the embedding of the sample in the metric space; the cosine distance is selected as the similarity calculation standard; the calculation formula is as follows:

[0053]

[0054] Among them, D positive ∈[-1,1] represents the embedding F and the positive prototype F positive The cosine distance between negativeThe definition of is the same; note that when the prototype classifier predicts the sample label, only the embedding of the input sample and the positive class prototype F are calculated positive The cosine similarity between positive The closer the value of is to 1, the more similar the shape of the input sample is to the shape of the positive sample, and the input sample is more likely to be a positive sample; on the contrary, if D positive If it is close to -1, it means that the sample is more likely to be a negative sample; therefore, in this study, the prediction result of a single sample is defined as:

[0055]

[0056] Among them, Pred represents the label prediction function, and S is the sample to be predicted.

[0057] Preferably, the data acquisition device is based on a 32-electrode EEG cap and SynAmps of the international 10-20 system. 2 The amplifier recorded EEG signals at a sampling rate of 250 Hz, and the reference electrode was placed on the right mastoid. To remove noise, a 50 Hz notch filter was performed, and eye-tracking glasses were used to record eye movement data. The sampling rate of the scene view and eye view was 60 Hz, with resolutions of 1280 × 960 and 320 × 240, respectively.

[0058] Preferably, the multimodal object recognition network is implemented by the PyTorch framework, and trained using mean square error (MSE) loss and AdamW optimizer, and the loss function is defined as:

[0059] loss = f MSE (D positive ,y)+f MSE (D negative ,-y)

[0060] Among them, y represents the true label of the sample, the learning rate decay uses the cosine annealing method, the batch size used for model training is 30, and the number of training times is 100.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] (1) In the present invention, a new BCI communication paradigm is designed that integrates P300 potential and eye tracking. P300 detection and eye tracking detection are irreplaceable and complementary. Multimodal fusion reduces the threshold and difficulty of use, increases the information used for decision-making, and helps promote clinical communication.

[0063] (2) In the present invention, a new multimodal target recognition network is proposed. The dual-stream feature extraction module can maximize the extraction of rich multi-scale features of EEG and eye movement signals. The multimodal attention module integrates the dual-stream output and can adaptively emphasize important features and eliminate redundancy. In order to solve the overfitting problem and enhance the classification of small sample data, the present invention adopts a prototype network based on cosine distance as a classifier.

[0064] (3) In the present invention, a multimodal hybrid BCI online communication system is developed, which provides medical personnel with a more accurate and convenient means of assessing the state of consciousness (communication level), and also provides a channel for ensuring the basic human rights of patients and personalized treatment and nursing decisions for patients. The effective communication application of multimodal online communication BCI in patients with consciousness disorders provides a new concept and method for introducing BCI integrated eye tracking into the clinical practice of consciousness disorder detection and communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the external arrangement of the present invention;

[0066] Figure 2 The overall composition block diagram of the system of the present invention is shown in FIG.

[0067] Figure 3 It is a schematic diagram of the overall architecture of the multi-modal communication BCI system of the present invention;

[0068] Figure 4 It is a GUI schematic diagram of the multi-modal hybrid BCI online communication system of the present invention;

[0069] Figure 5 This is a schematic diagram of the online communication BCI paradigm of the present invention;

[0070] Figure 6 It is a structural diagram of the EEG feature extraction module in the dual-stream feature extraction module of the present invention;

[0071] Figure 7 It is a structural diagram of the eye movement feature extraction module in the dual-stream feature extraction module of the present invention;

[0072] Figure 8 This is a structural diagram of the multimodal attention module of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] See also Figures 1 to 8 The present invention provides a technical solution: a consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram, comprising

[0075] Data acquisition equipment, connected to the multimodal BCI communication system, used to collect scalp EEG signals and eye movement signals;

[0076] The multimodal BCI communication system can detect and analyze the multimodal data collected from the communication paradigm and identify the subject's response (yes / no), thereby achieving online binary communication;

[0077] The multimodal BCI communication paradigm is as follows

[0078] In this study, the present invention designed a communication paradigm that can simultaneously induce P300 event-related potential (ERP) and eye movement response (including pupil response, gaze); by detecting and analyzing the multimodal data collected from the communication paradigm, the subject's answer (yes / no) can be identified, thereby realizing online binary communication; the specific paradigm design is as follows;

[0079] GUI: Figure 4 The graphical user interface of the multimodal hybrid BCI online communication system designed by the present invention is shown; a question and instruction are displayed at the top of the screen; all questions are designed based on the CRS-R communication subscale to assist doctors in communication assessment; for example: "Is your name Haley?", "Is this a cup / comb?", "Is the doctor clapping now?"; the instruction instructs the patient to focus on the correct answer (target); below the instruction, two text blocks of "yes" and "no" are displayed on the left and right sides respectively; each text block is a rectangle with a width of 12.05° and a height of 11.42°; the distance between the two text blocks in the visual field is about 6.01°; during visual stimulation, this size and distance design can not only induce discernible eye movements and EEG responses of the subject, but also is not easy to cause fatigue and discomfort to the subject; in addition, in order to help the subject focus, the present invention sets a point at the center of each text block and between the text blocks;

[0080] Stimulation mode: Two text blocks were used as stimuli to induce P300 signals and eye tracking signals in the form of random flashes; when the stimulus was presented, the text color of the corresponding text block changed from blue to green, the background color changed from black to white, and the brightness changed from 1.27 cd / m 2 Increased to 90.76cd / m 2; At the same time, the audio corresponding to the text is played at a volume of 65dB; The change in the brightness of the text block will change the subject's pupil size (Pupillary Light Reflex (PLR): Pupil size fluctuates with the brightness of the visual stimulus. The higher the brightness, the pupil contracts, and the lower the brightness, the pupil dilates), and the degree of this change will increase with the concentration of attention; Random audio-visual stimulation can induce P300 ERP; By detecting and analyzing the P300 potential and eye movement response, the text block selected by the subject can be identified;

[0081] BCI Paradigm: Online Communication Paradigm Figure 5 As shown in the figure, the experiment begins with a question and instruction phase. In this phase, the computer displays and issues predetermined questions and instructions. For example, "Is this a cup? Pay attention to the answer button and count the number of times it flashes." After 8 seconds of questions and instructions, the two text blocks flash in a random order, accompanied by a corresponding audio broadcast. Each flash lasts for 300 milliseconds, with an interval of 700 milliseconds. Therefore, the duration of a round of stimulation is 2000 milliseconds in total, that is, the period in which the left and right text blocks flash once each. The duration of the stimulation phase is set to 20 seconds. After real-time calculation by the MTRN, the results of this trial will be echoed on the screen at the end of the stimulation. If the MTRN successfully identifies the target (i.e., the correct answer), the system will issue applause as encouraging feedback to provide positive reinforcement. After a short break of 4 seconds, the next trial begins.

[0082] Each experiment consists of 10 trials, one trial for each question. To prevent position deviation, the text blocks containing the correct answer appear the same number of times on the left and right sides.

[0083] The multimodal BCI communication system includes a multimodal target recognition network, and the multimodal target recognition network specifically includes:

[0084] The data preprocessing module is used for preprocessing and synchronizing data of different modalities. For the preprocessing of EEG data, the present invention first uses a 0.1-20 Hz bandpass filter to filter the signal to remove noise; then, within 800 ms after the start of the stimulus, the epoch corresponding to each stimulus of each channel is extracted; the epoch is baseline calibrated using the data 100 ms before the start of the stimulus to reduce signal offset; finally, channel screening is performed to remove the reference electrode and downsampled at a rate of 4; to avoid the influence of dimensions between data of different modalities, the present invention normalizes the epoch at the channel level; in addition, the present invention captures the electrooculograms of two pairs of electrodes "HEOR" and "HEOL" and "VEO" and "VEOL" to filter out eye movement artifacts in the EEG data; the structure of the preprocessed EEG data is [N c ×Nt ]; where N c =30 is the number of electrode channels, N t =50 is the time dimension;

[0085] For the preprocessing of eye movement data, the present invention firstly adopts the cubic spline interpolation method to compensate for the data loss caused by blinking or gaze position beyond the detection range, so as to ensure the integrity of the eye movement data; then, for each flickering stimulation of the text block, the epoch of eye movement data from 0 to 800ms is extracted; considering the individual differences in pupil size and the scale of the data, the present invention standardizes each epoch; in addition, to be consistent with the EEG data, the present invention downsamples the eye movement epoch; the present invention selects the size of the left and right pupils and the distance between the gaze point and the current flickering text block as eye movement features; the distance feature is calculated by the gaze coordinates and the center coordinates of the flickering text block; after preprocessing, the shape of the data is [N c ×N t ]; where N c =3 is the number of eye movement features, N t =50 is the time scale;

[0086] The dual-stream feature extraction module consists of two independent multi-scale convolutional neural networks, one for EEG and one for eye movement data, to extract features from different modalities of data;

[0087] The multimodal attention module is used to integrate the multimodal signal features output by the dual-stream feature extraction module to achieve feature complementarity;

[0088] The prototype classifier, which is composed of a prototype network based on cosine distance, is used to improve the classification performance of the system for small sample data.

[0089] In this embodiment, the dual-stream feature extraction module includes:

[0090] EEG feature extraction module, the multi-scale convolutional neural network of the EEG feature extraction module mainly includes a 6-layer structure, such as Figure 6 As shown:

[0091] Eye movement feature extraction module, the multi-scale convolutional neural network of the eye movement feature extraction module consists of a 7-layer architecture. The structure of the network is similar to that of the EEG feature extraction network, except that the parameter settings are different and a new convolution layer is added for fine-grained feature extraction. This newly added layer contains a convolution kernel of size (1, 3) and a batch normalization operation. The calculation formula is as follows:

[0092]

[0093] Among them, f tanhis the tanh activation function, f BN represents batch normalization, a [4] It is the feature map output by the L4 maximum pooling layer. [5] and b [5] are the convolution kernel matrix and bias respectively.

[0094] Figure 7 The network architecture for eye tracking feature extraction is shown. The network consists of the following 7 layers: L1 spatial convolution layer, L2 temporal convolution layer, L3 integration layer, L4 maximum pooling layer, L5 fine-grained feature extraction layer, L6 and L7 two fully connected layers. The L7 layer outputs representative eye tracking features of all input samples.

[0095] In this embodiment, the 6-layer structure of the multi-scale convolutional neural network is as follows:

[0096] Spatial convolution layer: It uses weighted stacking averaging and common spatial filtering to improve the signal-to-noise ratio of the EEG signal and eliminate redundant spatial information. It consists of a convolution kernel of size (30, 1). The size of the convolution kernel is equal to the number of electrodes, and the step size of the convolution is (1, 1). The calculation process of this layer is as follows:

[0097]

[0098] Among them, f tanh is the hyperbolic tangent activation function, I c,i represents the i-th moment element value of the c-th channel, represents the convolution kernel matrix, represents additive bias;

[0099] Temporal convolution layer: Two parallel convolution operations and batch normalization operations are introduced to capture more differentiated features; the two convolutions have the same convolution step size (1,1) and convolution kernels of different sizes [(1,2), (1,4)]; this strategy can help extract different temporal features and improve the effectiveness of these features; batch normalization is used to reduce internal covariate bias, avoid saturation problems, and ensure faster training speed; the formula of this layer is as follows:

[0100]

[0101] and represents the feature map obtained from the receptive fields of different scales of different convolution kernels, which contains contextual features of different scales in the EEG data; f BN represents batch normalization, and Especially two different convolution kernel matrices and biases;

[0102] Integration layer: Cascade the temporal features extracted in the previous layer to integrate the local details of the EEG data with the global information; the fused features will contain more global information while retaining the local information of the EEG;

[0103] Max pooling layer: Use a kernel of size (1,2) to pool the integrated features to expand the receptive field and reduce redundancy, thereby improving the performance of the network;

[0104] Fully connected layer: Flatten the multi-dimensional features to prepare for the subsequent fully connected (FC) operation; to enhance the generalization ability of the network, the Dropout strategy is adopted in this layer; the formula of this layer is summarized as follows:

[0105] v=Bernoulli(P)

[0106] a [5] =f tanh ((v×a [4] )×w [5] +b [5] )

[0107] The Bernoulli function randomly generates a 01 vector with a probability of P. By multiplying the vector with the input feature map value by value, the purpose of suppressing P% of neurons can be achieved; tanh is the tanh activation function, a [4] represents the feature output of the L4 maximum pooling layer, w [5] and b [5] They are the convolution kernel matrix and bias respectively;

[0108] Fully connected layer: Fully connected operations are performed in this layer to generate representative EEG features for all input samples; the formula is summarized as follows:

[0109] a [6] =f tanh (a [5] × [6] +b [6] )

[0110] where f tanh is the tanh activation function, w [6] and b [6] are the convolution kernel matrix and bias respectively.

[0111] In this embodiment, the operation of the multimodal attention module is as follows:

[0112] Its structure is as Figure 8 As shown;

[0113] First, element-by-element summation is performed to preliminarily fuse the extracted representative features of each modality;

[0114] F fuse =F eeg +F eye

[0115] Among them, F eeg ∈R 1×X and F eye ∈R 1×X It is the representative features of EEG data and eye movement data;

[0116] In order to achieve accurate adaptive feature selection, the compact feature z∈R is introduced 1×d ; z is implemented through the FC layer to reduce dimensionality and improve efficiency; z is calculated as follows:

[0117] z=FC(F fuse ) = f relu (f BN (F fuse × [2] ))

[0118] Among them, f relu is the linear rectification activation function, f BN represents batch normalization, w [2] is the convolution kernel; in order to study the influence of d on the model efficiency, the present invention uses a reduction ratio r to control its value;

[0119] The present invention uses convolution operation and Softmax function to achieve cross-channel soft attention to information of different spatial scales; specifically, two convolutions are performed under the guidance of compact feature z to obtain the corresponding weight of each feature; then, Softmax function is applied at the channel level to dynamically pay attention to information of different spatial scales; the calculation process is summarized as follows:

[0120] W [3,1] =f relu (z×w [3,1] +b [3,1] )

[0121] W [3,2] =f relu (z×w [3,2] +b [3,2] )

[0122]

[0123] Where W [3,1] and W [3,2] is the feature map calculated by different convolution kernels, w [3,1] and w [3,2] represents the convolution kernel, b [3 ,1] and b [3,2] Indicates bias; and W [3,1] and v [4,1] The xth element of [4,1] ∈R 1×X is the feature map (soft attention vector) obtained after the Softmax operation; similarly, and v [4,2] It also follows a similar pattern;

[0124] Finally, each original feature is weighted with the soft attention vector to obtain the final fused feature;

[0125] F=v [4,1] ·F eeg +v [4,2] ·F eye ,v [4,1] +v [4,2] =1

[0126] The fused features F are the final representative features of the input samples; in other words, this layer computes the embeddings (non-linear mapping of the input to the embedding space) of all samples.

[0127] In this embodiment, the working principle of the prototype classifier is that the embeddings of the samples are clustered around their class prototypes, and the embeddings F of the query samples are classified by identifying the most similar class prototypes;

[0128] The task of the prototype classifier can be described as a binary classification task, that is, distinguishing between positive classes (targets, labeled 1) and negative classes (non-targets, labeled -1); the prototype is obtained by averaging the features of all samples from the same class in the training set; the positive prototype F positive The calculation is as follows:

[0129]

[0130] For identifying the "most similar class prototype", the model performs recognition and classification by calculating the distance between the class prototype and the embedding of the sample in the metric space; cosine distance is selected as the similarity calculation standard because it is more sensitive to small-sized data; its calculation formula is as follows:

[0131]

[0132] Among them, D positive ∈[-1,1] represents the embedding F and the positive prototype F positive The cosine distance between negative The definition of is the same; note that when the prototype classifier predicts the sample label, only the embedding of the input sample and the positive class prototype F are calculated positive The cosine similarity between positiveThe closer the value of is to 1, the more similar the shape of the input sample is to the shape of the positive sample, and the input sample is more likely to be a positive sample (i.e., the target); on the contrary, if D positive If it is close to -1, it means that the sample is more likely to be a negative sample; therefore, in this study, the prediction result of a single sample is defined as:

[0133]

[0134] Among them, Pred represents the label prediction function, and S is the sample to be predicted.

[0135] In this embodiment, the data acquisition equipment is based on the 32-electrode EEG cap and SynAmps of the international 10-20 system. 2 The amplifier recorded EEG signals at a sampling rate of 250 Hz, and the reference electrode was placed on the right mastoid. To remove noise, a 50 Hz notch filter was performed, and eye-tracking glasses were used to record eye movement data. The sampling rate of the scene view and eye view was 60 Hz, with resolutions of 1280 × 960 and 320 × 240, respectively;

[0136] This self-collected data set includes 10 healthy subjects (mean age 24.0; 5 males) and 10 patient subjects (mean age 38.8 years; 5 males), including 9 DOC patients and 1 LIS patient;

[0137] Table 1. Patient information

[0138]

[0139] The CRS-R scoring subscales include: vision, hearing, motor, speech, communication, and arousal functions;

[0140] The main screen of the experiment is a 24-inch monitor with a refresh rate of 60Hz, which is about 1 meter away from the subject; the 20-inch secondary screen is used for monitoring and calibration; in order to obtain more accurate gaze data, all subjects need to undergo eye tracking calibration; however, considering that there is no guarantee that the patient will follow the instructions to gaze at the calibration point, the present invention does not manually calibrate the patient to maintain a consistent gaze deviation; the present invention sends event triggers to two servers through the parallel port of the computer to synchronize EEG data and eye tracking data; because the patient is easily fatigued and cannot stay awake for a long time, the experiment is arranged in two days, with two days of rest in between;

[0141] The experiment is divided into two stages: offline training and online testing. Figure 5As shown in the figure; Offline training: Before the online communication experiment, each subject completed a training experiment consisting of 10 trials to collect data to train MTRN; In offline communication, the model does not provide result feedback; Online test: Each participant conducted 5 online communication tests, each test consisting of 10 trials; However, due to physical discomfort, patients P1, P4 and P8 only completed 3 and 4 online communication tests, respectively; The setting and process of online testing are similar to offline training, except that in online experiments, MTRN will process the collected data in real time and feedback the test results (i.e., the patient's answer choice) at the end of stimulation;

[0142] To verify the rationality and superiority of the design of the present invention, the researchers tested the accuracy of MTRN in a single-modal mode (canceling the multimodal attention module and using only the feature extraction network and prototype classifier corresponding to the modality in the dual-stream feature extraction module) offline, as well as the performance of three representative models (SVM, EEGNet, and EyeNet) in each modality. SVM is a classification algorithm commonly used in BCI for impaired consciousness. EEGNet is a classic network for EEG recognition. Since there is no network specifically used for eye movement data processing and classification, the researchers designed an eye movement feature extraction network EyeNet with reference to the classic convolutional neural network AlexNet. EyeNet consists of three convolution blocks and one fully connected block. Each convolution block contains convolution, normalization, activation, and pooling operations. The fully connected block includes a Dropout operation and three fully connected operations. In addition, in order to verify the effectiveness of the multimodal attention module and the prototype classifier, the researchers tested MTRN-A and MTRN-B under multimodal conditions. MTRN-A and MTRN-B are variants of the MTRN network in the present invention. MTRN-A adopts a data layer fusion strategy (i.e., the collected multimodal data are directly merged after preprocessing), while MTRN-B abandons the prototype classifier and uses the Softmax classifier instead.

[0143] Table 2 lists the experimental accuracy of 10 healthy subjects in detail; the results show that the online communication accuracy of all healthy subjects reached 100%, which is far above the significance threshold (P≤0.05); for all healthy subjects, the accuracy of the system based on single mode exceeded the significance level of 64%; in addition, whether SVM, MTRN-A, MTRN-B or MTRN is used, the multimodal system has higher accuracy than the system based on P300 or eye tracking only; this finding shows that the multimodal system is better than the single modal system, confirming the effectiveness of MTRN and multimodal communication design; although the accuracy of SVM, MTRN-A, MTRN-B and MTRN based on multimodality all reached 100%, it cannot be compared; but under the single modality condition, MTRN is better than SVM and EEGNet / EyeNet, which shows that MTRN has superior feature extraction and classification capabilities;

[0144] Table 2. Accuracy of online and offline experiments for 10 healthy subjects (50 trials)

[0145]

[0146]

[0147] Table 3 lists the accuracy of online and offline experiments of 10 patients; among them, 5 patients (4 MCS and 1 LIS) achieved significant results in the online communication experiment (P≤0.05), with an average accuracy of 76.1±7.9%; the significant results of patient P4 proved the robustness of this study; because P4's eyeballs could not move left and right, but only slightly up and down, P4 may communicate only through pupil response (for most vision-based BCI binary, P4 may not meet its use conditions); according to whether the online accuracy is achieved When the significance threshold was reached, the patients were divided into a responding group (P1, P4, P5, P8 and P10) and a non-responding group (the other 5 patients); in the responding group, the multimodal MTRN achieved the highest accuracy; in addition, under both unimodal and multimodal conditions, the overall accuracy of MTRN was significantly higher than that of SVM, EEGNet / EyeNet, MTRN-A and MTRN-B, which further verified the effectiveness of the multimodal attention module; the average accuracy of the non-responding group was 60.0±2%, slightly higher than the probability level of 50%;

[0148] In addition, it is worth noting that patients P5 and P10 achieved 84% and 64% accuracy in the online communication experiment, respectively, far exceeding the significance threshold; this indicates that P5 and P10 were considered to be communicative in the BCI assessment and were able to conduct binary yes-no communication; however, they scored 0 on the communication subscale of the CRS-R, which means that they may lack communication ability; this contradictory result suggests that the doctor's judgment of the patient's communication ability may have been wrong, and it also shows that the hybrid BCI of the present invention has the potential to improve brain function testing in this challenging patient group;

[0149] After conducting the online communication experiment, the present invention feeds back the situation of the patient with impaired consciousness to the doctor. These results can serve as an auxiliary basis for the doctor to judge the consciousness state (communication level) of the patient with impaired consciousness, and assist the doctor in diagnosis and treatment and prognosis judgment; the doctor can focus on the patients with communication ability, and can appropriately adjust the treatment strategy according to the patient's wishes, so that the patient can recover better;

[0150] Table 3. Accuracy of online and offline tests in 10 patients

[0151]

[0152]

[0153] The P values ​​in the table correspond to the online accuracy obtained by MTRN (multimodality) for each patient; the accuracy exceeding the significance threshold (P≤0.05) is marked in bold;

[0154] The present invention calculates some common indicators to further evaluate the classification performance of MTRN and the contribution of its different components; the true positive rate (also called sensitivity) is usually used to estimate the sensitivity of detection; the higher the sensitivity, the lower the probability of missed detection; the model of the present invention mainly identifies targets by detecting P300 and eye movement responses, so the present invention should pay more attention to the sensitivity of the method; in addition to sensitivity, accuracy, specificity, F1 score and precision are also important metrics; Table 4 shows the average classification performance of each model on 10 healthy subjects; the sensitivity of MTRN in each mode exceeds that of other models, which illustrates its classification ability for P300 and eye movement responses, and also proves the multi-scale and prototype-based The effectiveness of the classification technology; the multimodal-based MTRN outperformed the MTRN-A that abandoned the multimodal attention module and the MTRN-B that replaced the prototype classifier in all indicators, which showed that the adaptive attention strategy and the prototype classifier could promote the fusion and classification of small sample data; in addition, the accuracy of MTRN under unimodal and multimodal conditions was statistically compared in the present invention; in view of the small number of subjects, the present invention adopted the Wilcoxon signed rank test for analysis; the results showed that there was a significant difference in accuracy between the multimodal-based MTRN and the unimodal-based MTRN in both healthy subjects and patients (P≤0.05), which showed that communication through the integration of eye tracking and EEG was effective;

[0155] Table 4. Comparison of mean classification performance of ten healthy subjects;

[0156]

[0157] In the table, TP, TN, FP, and FN are the number of true positives, true negatives, false positives, and false negatives, respectively. In this embodiment, the multimodal target recognition network is implemented by the PyTorch framework, and is trained using the mean square error (MSE) loss and the AdamW optimizer, and the loss function is defined as:

[0158] loss = f MSE (D positive ,y)+f MSE (D negative ,-y)

[0159] Among them, y represents the true label of the sample, the learning rate decay uses the cosine annealing method, the batch size used for model training is 30, and the number of training times is 100.

[0160] The system adopts a new BCI binary communication paradigm that integrates P300 signals and eye movement signals. The binary yes / no communication paradigm is simple and convenient, and is more suitable for communication with patients with impaired consciousness. P300 event-related potential (ERP) is one of the current mainstream BCI paradigms. Compared with other mainstream paradigms, such as steady-state visual evoked potential (SSVEP) and motor imagery (MI), P300 stimulation is milder and more stable, there is no risk of inducing potential diseases in patients, and its induction conditions and difficulty are lower, which is more suitable for patients with impaired consciousness. In order to solve the problem of continuous active cognition required for inducing P300, and further reduce the threshold and difficulty of using this type of BCI, the present invention integrates eye movement signals for comprehensive judgment. That is, communication is carried out through eye movement reactions (pupil reactions, gaze). Multimodal fusion can achieve data information complementarity, which is conducive to improving BCI communication performance. In addition, the induction of eye movement reactions only requires the patient to be able to open his eyes and have visual ability, which is consistent with the prerequisites for stimulating visual P300 ERP, so it will not increase the threshold for using BCI. And it does not require the subject to perform additional active tasks other than fixation (counting tasks need to be completed to induce P300), which reduces the difficulty of execution. Even if the patient cannot maintain continuous attention or fixation behavior during the stimulation phase (intermittent attention; unable to complete the active task of inducing P300), it will not affect the final recognition result. Because the algorithm model of the present invention can identify the patient's choice only by detecting the eye movement response generated during the stimulation process. In addition, the two eye movement features selected by the present invention are complementary, which can improve the robustness of the system. Because for patients whose eyeballs cannot move, the present invention can still be identified by his pupil response.

[0161] In general, the patient of the present invention mainly conducts binary yes / no communication through attention and gaze: the system presents a situation-oriented question to the DOC patient, and simultaneously displays text blocks "yes" and "no" on the left and right sides below the question. Randomly flashing text blocks and corresponding voice announcements serve as visual and auditory stimuli. The experiment requires the patient to selectively focus on the stimulus with the correct answer. The text block that the patient chooses to focus on can be identified by detecting and analyzing the P300 signal and eye movement signal generated in the stimulus.

[0162] In order to enhance feature extraction, maximize the use of features from each modality, and improve the recognition accuracy of small sample data, the present invention proposes a multimodal target recognition network (MTRN) consisting of three modules. The dual-stream feature extraction module uses multiple convolutions with different receptive fields to learn features of different scales from EEG and eye tracking data. Compared with ordinary convolutional networks, it can capture more contextual feature information contained in EEG. The present invention uses a cross-channel soft attention mechanism to fuse the multimodal features extracted by the dual-stream feature extraction module. This mechanism can adaptively capture the most valuable information in multimodal data. Finally, considering the characteristics of patient data, the present invention introduces a prototype network to classify small sample data. This strategy can avoid the problem of low model accuracy when the amount of data is insufficient.

[0163] The comparison results show that the method not only reaches the level of mainstream models in terms of accuracy, but also demonstrates the potential for wide application in medical diagnosis, opening up new avenues for clinical practice research. The results of practical applications also show that the system increases the likelihood that patients will provide evidence of residual brain function to examiners and can serve as an auxiliary tool for clinical communication with DOC patients.

[0164] Although embodiments of the present invention have been shown and described (see the above detailed description for details), it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram, characterized by: include Data acquisition equipment, connected to the multimodal BCI communication system, used to collect scalp EEG signals and eye movement signals; The multimodal BCI communication system can detect and analyze the multimodal data collected from the communication paradigm and identify the subject's response (yes / no), thereby achieving online binary communication; The multimodal BCI communication system includes a multimodal target recognition network, and the multimodal target recognition network specifically includes: Data preprocessing module, used for preprocessing and synchronization of data of different modalities; The dual-stream feature extraction module consists of two independent multi-scale convolutional neural networks, one for EEG and one for eye movement data, to extract features from different modalities of data; The multimodal attention module is used to integrate the multimodal signal features output by the dual-stream feature extraction module to achieve feature complementarity; The prototype classifier, which is composed of a prototype network based on cosine distance, is used to improve the classification performance of the system for small sample data.

2. The consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram according to claim 1, characterized in that: The dual-stream feature extraction module includes: EEG feature extraction module, the multi-scale convolutional neural network of this EEG feature extraction module mainly includes 6 layers: Eye movement feature extraction module, the multi-scale convolutional neural network of this eye movement feature extraction module consists of a 7-layer architecture.

3. The consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram according to claim 2, characterized in that: The 6-layer structure of the multi-scale convolutional neural network is as follows: Spatial convolution layer: It uses weighted stacking averaging and common spatial filtering to improve the signal-to-noise ratio of the EEG signal and eliminate redundant spatial information. It consists of a convolution kernel of size (30, 1). The size of the convolution kernel is equal to the number of electrodes, and the step size of the convolution is (1, 1). The calculation process of this layer is as follows: Among them, f tanh is the hyperbolic tangent activation function, I c,i represents the i-th moment element value of the c-th channel, represents the convolution kernel matrix, represents additive bias; Temporal convolution layer: Two parallel convolution operations and batch normalization operations are introduced to capture more differentiated features; the two convolutions have the same convolution step size (1,1) and different sizes of convolution kernels [(1,2), (1,4)]; this strategy can help extract different temporal features and improve the effectiveness of these features; batch normalization is used to reduce internal covariate bias, avoid saturation problems, and ensure faster training speed; the formula of this layer is as follows: and represents the feature map obtained from the receptive fields of different scales of different convolution kernels, which contains contextual features of different scales in the EEG data; f BN represents batch normalization, and They are two different convolution kernel matrices and biases respectively; Integration layer: Cascade the temporal features extracted in the previous layer to integrate the local details of the EEG data with the global information; the fused features will contain more global information while retaining the local information of the EEG; Max pooling layer: Use a kernel of size (1,2) to pool the integrated features to expand the receptive field and reduce redundancy, thereby improving the performance of the network; Fully connected layer: Flatten the multi-dimensional features to prepare for the subsequent fully connected (FC) operation; to enhance the generalization ability of the network, the Dropout strategy is adopted in this layer; the formula of this layer is summarized as follows: v=Bernoulli(P) a [5] =f tanh ((v×a [4] )×w [5] +b [5] ) The Bernoulli function randomly generates a 01 vector with a probability of P. By multiplying the vector with the input feature map value by value, the purpose of suppressing P% of neurons can be achieved; tanh is the tanh activation function, a [4] represents the feature output of the L4 maximum pooling layer, w [5] and b [5] They are the convolution kernel matrix and bias respectively; Fully connected layer: Fully connected operations are performed in this layer to generate representative EEG features for all input samples; the formula is summarized as follows: a [6] =f tanh (a [5] ×w [6] +b [6] ) where f tanh is the tanh activation function, w [6] and b [6] are the convolution kernel matrix and bias respectively.

4. The consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram according to claim 1, characterized in that: The multimodal attention module operates as follows: First, element-by-element summation is performed to preliminarily fuse the extracted representative features of each modality; F fuse =F eeg +F eye Among them, F eeg ∈R 1×X and F eye ∈R 1×X It is the representative features of EEG data and eye movement data; In order to achieve accurate adaptive feature selection, the compact feature z∈R is introduced 1×d ; z is implemented through the FC layer to reduce dimensionality and improve efficiency; z is calculated as follows: z=FC(F fuse )=f relu (f BN (F fuse ×w [2] )) Among them, f relu is the linear rectification activation function, f BN represents batch normalization, w [2] is the convolution kernel; Subsequently, convolution operations and Softmax functions are used to achieve cross-channel soft attention to information of different spatial scales. Specifically, two convolutions are performed under the guidance of the compact feature z to obtain the corresponding weights of each feature. The Softmax function is then applied at the channel level to dynamically focus on information of different spatial scales. The calculation process is summarized as follows: W [3,1] =f relu (z×w [3,1] +b [3,1] ) W [3,2] =f relu (z×w [3,2] +b [3,2] ) Where W [3,1] and W [3,2] is the feature map calculated by different convolution kernels, w [3,1] and w [3,2] represents the convolution kernel, b [3,1] and b [3,2] Indicates bias; and W [3,1] and v [4,1] The xth element of [4,1] ∈R 1×X is the feature map (soft attention vector) obtained after the Softmax operation; similarly, and v [4,2] It also follows a similar pattern; Finally, each original feature is weighted with the soft attention vector to obtain the final fused feature; F=v [4,1] ·F eeg +v [4,2] ·F eye ,v [4,1] +v [4,2] =1; The fused feature F is the final representative feature of the input sample; that is, the embedding of all samples (non-linear mapping of input samples to embedding space) is calculated at this layer.

5. The consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram according to claim 1, characterized in that: The prototype classifier works by clustering the embeddings of samples around their class prototypes, and classifying the embedding F of the query sample by identifying the most similar class prototype; The task of the prototype classifier can be described as a binary classification task, that is, distinguishing between positive classes (targets, labeled 1) and negative classes (non-targets, labeled -1); the prototype is obtained by averaging the features of all samples from the same class in the training set; the positive prototype F positive The calculation is as follows: For identifying the "most similar class prototype", the model performs identification and classification by calculating the distance between the class prototype and the sample embedding in the metric space; the cosine distance is selected as the similarity calculation standard; the calculation formula is as follows: Among them, D positive ∈[-1,1] represents the embedding F and the positive prototype F positive The cosine distance between negative The definition of is the same; it should be noted that when the prototype classifier predicts the sample label, only the embedding of the input sample and the positive class prototype F are calculated positive The cosine similarity between positive The closer the value of is to 1, the more similar the shape of the input sample is to the shape of the positive sample, and the input sample is more likely to be a positive sample; on the contrary, if D positive If it is close to -1, it means that the sample is more likely to be a negative sample; therefore, in this study, the prediction result of a single sample is defined as: Among them, Pred represents the label prediction function, and S is the sample to be predicted.

6. The consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram according to claim 1, characterized in that: The data acquisition equipment is based on the 32-electrode EEG cap and SynAmps of the international 10-20 system. 2 The amplifier recorded EEG signals at a sampling rate of 250 Hz, and the reference electrode was placed on the right mastoid. To remove noise, a 50 Hz notch filter was performed, and eye-tracking glasses were used to record eye movement data. The sampling rate of the scene view and eye view was 60 Hz, with resolutions of 1280 × 960 and 320 × 240, respectively.

7. The consciousness disorder auxiliary communication system integrating eye tracking and electroencephalogram according to claim 1, characterized in that: The multimodal object recognition network is implemented by the PyTorch framework and trained using mean square error (MSE) loss and AdamW optimizer, and the loss function is defined as: loss=f MSE (D positive ,y)+f MSE (D negative ,-y) Among them, y represents the true label of the sample, the learning rate decay uses the cosine annealing method, the batch size used for model training is 30, and the number of training times is 100.

Images