Brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning

Abstract

The present invention discloses a brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning, including: an auditory-tactile perception stimulation paradigm design module, which combines the multi-sensory stimulation paradigm and the question-answering process based on the brain-computer interface to form assisted communication; an EEG signal induction and acquisition module, which is used to induce the P300 EEG signal of patients with impaired consciousness and pre-process the data; a data enhancement module, which is used to expand the data scale and further improve the classification performance of EEG signals; an assisted communication training module, which uses synthetic EEG data vectors to train the neural network to generate a neural network model with the best performance that can assist communication; an assisted communication prediction module, which uses the neural network model to process the EEG signals of the patient with impaired consciousness to be detected in real time to achieve the function of assisted communication. The present invention combines the multi-sensory stimulation paradigm with deep learning technology, and can carry out brain-computer interface detection for patients with impaired consciousness with visual function damage.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and brain-computer interface, and in particular to a brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning. Background Art

[0002] With the advancement of critical care technology, the lives of many patients with severe brain injuries have been saved. However, many patients fail to regain consciousness after the initial coma phase (2–4 weeks), entering a state known as chronic disorder of consciousness (DOCs). Depending on the severity of DOCs, patients with DOCs are categorized as experiencing a vegetative state, unresponsive wakefulness syndrome, or minimally conscious state. The internationally accepted gold standard for assessing the level of consciousness in patients with DOCs is behavioral rating scales, particularly the Coma Recovery Scale-Revised (CRS-R). This behavioral assessment method assigns scores to each functional module based on whether the patient demonstrates specific responses. However, due to the impairment of motor and cognitive function in most DOCs patients and their inability to maintain a stable state during the assessment phase, this behavioral assessment method results in a significant misdiagnosis rate. With technological advancements and interdisciplinary collaboration, electroencephalography (EEG) and brain-computer interface (BCI) technologies have been used in many recent studies to assess patient consciousness due to their clinical applicability, portability, and high reproducibility.

[0003] By delivering sensory stimuli, patients with DOCs who lack overt motor activity can also generate specific EEG signals. These signals can then be analyzed using EEG to assess the patient's latent ability to follow commands and their level of consciousness. Several studies have proposed brain-computer interface (BCI) systems for monitoring consciousness in patients with impaired consciousness, using temporal and semantic consistency of visual and auditory perceptual stimuli. Experimental results have demonstrated that BCI technology has provided more accurate assessments of some of these patients.

[0004] However, brain-computer interfaces based on the visual perception stimulation paradigm require the subjects to keep their eyes focused on the visual interface. Some DOCs patients have impaired visual function. This type of brain-computer interface based on visual perception stimulation cannot achieve good results, and patients with impaired visual function cannot use it normally.

[0005] Many studies have applied deep learning methods to the field of brain-computer interfaces by adjusting network structures and EEG data input dimensions. This has enabled the analysis of EEG data for classification of ERPs, SMRs, and SSVEPs, among other applications. Some studies have combined different neural network structures to improve the performance of machine learning methods, furthering the practical application of brain-computer interfaces.

[0006] However, when applied to the field of patients with impaired consciousness, different experimental conditions and the patients' own conditions make data collection more difficult, resulting in large differences in the amount of data used to train the network. Directly using the complex network model trained in the large data set may produce poor results.

[0007] The use of brain-computer interface technology combined with electroencephalography (EEG) and its application to the field of patients with impaired consciousness can obtain more accurate results than behavioral assessments through auxiliary diagnosis and evaluation based on computer processing. Taking into account the motor function impairment and inability of patients with impaired consciousness to maintain a stable state for a long time, the experimental paradigm designed for such subjects needs to make multiple considerations to cover different types of patients with impaired consciousness, such as the use of tactile and auditory stimulation that does not rely on vision. In addition, in order to improve the classification accuracy of EEG data in experiments and obtain more accurate results, many studies in recent years have begun to use EEG data processing methods based on neural networks. Compared with traditional machine learning methods, the research results have been improved to a certain extent. However, when applied to the field of patients with impaired consciousness, it is also necessary to consider problems such as the small sample size of the patient data set and the poor data quality, which may affect the performance of the neural network. Summary of the Invention

[0008] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology, and proposes a brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning. Taking into account the visual function damage of some patients with impaired consciousness, an innovative paradigm design is made from the starting point of not relying on vision. In order to solve the problem of data scarcity in brain-computer interface experiments for patients with impaired consciousness, a generative adversarial method is used for data enhancement to generate high-quality EEG data, thereby further improving the performance of the neural network classifier.

[0009] To achieve the above objectives, the present invention provides a technical solution: a brain-computer interface-assisted communication system based on auditory-tactile perception stimulation and deep learning, comprising:

[0010] The auditory-tactile perceptual stimulation paradigm design module designs a multi-sensory stimulation paradigm and combines it with a brain-computer interface-based question-answering process to form assisted communication. The multi-sensory stimulation includes spatial orientation hearing and muscle pulse electrotactile sensation. The brain-computer interface-based question-answering process includes question instructions, stimulation, feedback, and rest.

[0011] The EEG signal induction and acquisition module uses a designed multi-sensory stimulation paradigm and a brain-computer interface-based question-answering process to induce P300 EEG signals in patients with impaired consciousness, and then uses a brain-computer interface device to collect the signals. The collected data is then pre-processed to form a P300 EEG data vector;

[0012] A data augmentation module uses an improved generative adversarial network to process P300 EEG data vectors to generate high-quality artificial EEG data vectors, which are then concatenated with the P300 EEG data vectors to obtain synthetic EEG data vectors. The improved generative adversarial network generates artificial EEG data vectors using resting-state EEG data vectors from patients with impaired consciousness, improves the activation functions of the generator and discriminator output layers to linear functions, and uses Wasserstein distance as a loss function to balance the training speeds of the discriminator and generator.

[0013] The communication-assisted training module designs a neural network composed of multi-scale convolution, channel attention mechanism, and deep convolution. This neural network is trained using synthetic EEG data vectors to generate a neural network model with optimal performance for assisting communication. The multi-scale convolution is used to process the synthetic EEG data vectors at different time scales, the channel attention mechanism is used to learn and amplify the weights of important channels in the synthetic EEG data vectors, and the deep convolution is used to further extract the spatial features of the synthetic EEG data vectors.

[0014] The auxiliary communication prediction module uses the neural network model with the best performance obtained from the auxiliary communication training module to process the EEG signals of the patient with impaired consciousness in real time, obtain answers to questions in the question-and-answer process based on the brain-computer interface, so as to realize the function of auxiliary communication and provide an assessment of the consciousness level of the patient with impaired consciousness.

[0015] Furthermore, the auditory-tactile perception stimulation paradigm design module includes a spatial orientation auditory module, a muscle pulse electrotactile module, a spatial-temporal consistency multi-sensory stimulation module, and a brain-computer interface-based question-answering process module, wherein:

[0016] The spatial orientation auditory module uses the head-related transfer function and a public dataset to perform convolution processing at different angles on the audio files of the Chinese characters "yes", "no", and "empty" to generate a spatial orientation effect. When patients with impaired consciousness wear headphones in both ear canals, they can perceive the sound "yes" coming from the right front, the sound "no" coming from the left front, and the sound "empty" coming from the front.

[0017] The muscle pulse electrotactile module is connected to a pulse stimulation instrument. After the positive and negative silicone electrodes that transmit pulse electrotactile sensation are attached to the skin of the patient with impaired consciousness, the intensity of the electrotactile stimulation is controlled by a computer. The electrodes are installed on the right and left wrists and abdomen of the patient with impaired consciousness, respectively, corresponding to the right, left, and front of the spatial orientation hearing.

[0018] The spatial-temporal coherent multi-sensory stimulation module integrates auditory and tactile sensory stimulation and, through program control, delivers auditory and tactile stimulation in the same direction to the consciousness disorder patient at a certain moment during the stimulation phase;

[0019] The brain-computer interface-based question-answering process module includes a complete online brain-computer interface experimental process: first, the question-asking instruction link, then the multi-sensory stimulation link, followed by the brain-computer interface feedback link, and finally the rest link.

[0020] Furthermore, the EEG signal induction and acquisition module includes an EEG signal induction module, a data loading module and a data preprocessing module, wherein:

[0021] The EEG signal induction module uses auditory and tactile perception stimulation and a designed brain-computer interface-based question-answering process to induce P300 EEG signals, and uses a brain-computer interface device to collect them;

[0022] The data loading module can receive EEG data collected in real time during online experiments, and can also read EEG data files saved in different formats;

[0023] The data preprocessing module reduces the data volume and computational complexity by selecting a time window, performing baseline correction, filtering, and downsampling operations, and selects the P300 EEG data vector to be processed.

[0024] Furthermore, the data enhancement module performs the following operations:

[0025] 1) Obtain artificial EEG data vector based on the improved generator:

[0026] Compared with the original generative adversarial network, the generator receives random noise. The improved method is to receive the EEG data vector of patients with consciousness disorders in the resting state to generate artificial EEG data vectors. The activation function of the original generator output layer sigmoid is modified to ReLU to avoid the gradient disappearance and gradient explosion problems. The artificial EEG data vector Generator (D rest ):

[0027] Generator(D rest )=ReLU(BN(W3·ReLU(BN(W2·ReLU(BN(W1·D rest )))))

[0028] Where D rest represents the EEG data vector in the resting state, W1, W2, and W3 correspond to the weight matrices of each fully connected layer in the generator, BN represents the batch normalization operation, and ReLU represents the rectified linear unit activation function;

[0029] 2) The discriminator based on the double-layer LSTM calculates the discrimination result:

[0030] The artificial EEG data vector generated by the generator is input into the discriminator based on the double-layer LSTM together with the P300 EEG data vector, and is processed with Dropout and ReLU to obtain the discriminator result Discriminator(x):

[0031] Discriminator(x)=W dense ReLU(W lstm2 Dropout(ReLU(W lstm1 x+b lstm1 ))+b lstm2 )+b dense Where x is the P300 EEG data vector D P300 Or artificial EEG data vector Generator (D rest ), W lstm1 、W lstm2 Represents the weight matrices of the two LSTM layers in the discriminator hidden layer, W dense represents the weight matrix of the output layer, b lstm1 、b lstm2 、b dense Represent the bias vectors of the two LSTM layers and the output layer respectively;

[0032] 3) Generative Adversarial Networks Based on Wasserstein Distance:

[0033] Wasserstein distance is used as a loss function to calculate the loss of the discriminator and generator during training, which is used to update the parameters of the discriminator and generator, and continuously optimize the discriminant ability of the discriminator and the generation quality of the generator. The Wasserstein distance is expressed as:

[0034]

[0035] Where, W(P P300 ,P g ) represents the distribution distance between the P300 EEG data vector and the artificial EEG data vector, P P300 Represents the distribution of P300 EEG data vector, P g Represents the distribution of artificial EEG data vector, Π(PP300 ,P g ) is all from P P300 to P g The set of joint distributions of , γ represents P P300 to P g A joint distribution of , (x, y) represents a pair of data points sampled from the joint distribution, ||xy|| represents the distance between the data point pairs;

[0036] 4) By repeatedly iterating the above steps 1), 2) and 3), the generation ability of the generator is gradually improved, and finally a high-quality artificial EEG data vector D with a distribution similar to that of the P300 EEG data vector is obtained. g , the two EEG data vectors are spliced ​​together to obtain the synthetic EEG data vector D s , achieving the expansion of the EEG dataset:

[0037] D s =concatenate([D p300 ,D g ])

[0038] In the formula, concatenate represents the concatenation operation.

[0039] Furthermore, the auxiliary communication training module performs the following operations:

[0040] 1) The shape of the synthesized EEG data vector is changed through the reshape operation to form the input of the convolutional layer:

[0041] Input=D s .reshape(C,T,1)

[0042] Where C and T represent the number of channels and data points of the synthetic EEG data vector respectively;

[0043] 2) Use three different convolution kernels to process EEG data of different time periods respectively. After each convolution, use BN, ELU and Dropout to prevent overfitting, and obtain the feature F after the first multi-scale convolution layer. (i) :

[0044] F (i) =Dropout(ELU(BN(Conv2d(Input,kernel_size=(1,s i )))))

[0045] Where i = {1, 2, 3}, corresponding to three different convolution kernels, F (i)Represents the features obtained by the i-th convolution kernel after the first multi-scale convolution layer. ELU is a linear unit activation function. Conv2d represents the convolution operation. (1, s i ) represents the size of the i-th convolution kernel, s i Represents the size of the i-th convolution kernel in the time dimension;

[0046] 3) The different features obtained by multi-scale convolution are all learned through the channel attention mechanism to learn the information of important channels, and the feature F (i) Perform global average pooling and global maximum pooling respectively to avoid the influence of spatial information on attention, and then obtain the channel attention mask M of global average pooling through ReLU activation function and FC full connection layer. GAP and the global maximum pooled channel attention mask M GMP :

[0047] M GAP =FC(ReLU(f GAP (F (i) ))

[0048] M GMP =FC(ReLU(f GMP (F (i) ))

[0049] Where, f GAP represents the global average pooling operation, f GMP represents the global maximum pooling operation, and FC represents the fully connected operation;

[0050] The two channel attention masks obtained by different pooling operations are transformed by a variable weight matrix, and the channel weight map is output using the sigmoid activation function. The output is multiplied with the original feature to adjust the channel importance and obtain the feature after the channel attention mechanism.

[0051]

[0052] In the formula, σ represents the sigmoid activation function, α and β represent the variable weight matrices respectively;

[0053] 4) Feature F obtained by merging the channel attention mechanism 4 :

[0054]

[0055] Where, and Respectively represent the features obtained by three different convolution kernels after the channel attention mechanism;

[0056] The spatial features are further extracted through a deep convolution with the same size as the number of channels of the synthetic EEG data vector, and BN, ELU and Dropout are used to prevent overfitting to obtain the feature F 5 :

[0057] F 5 =Dropout(ELU(BN(DepthwiseConv2d(F 4 ,kernel_size=(C,1)))))

[0058] Where DepthwiseConv2d represents depth convolution, and the convolution kernel size is (C, 1);

[0059] 5) The features are average-pooled to gradually extract the most important features. The remaining features are linearly transformed through flattening and fully connected layers. Finally, the probability Prob of the predicted category distribution is obtained through the softmax activation function:

[0060] Prob=softmax(FC(flatten(f ap (F 5 )))

[0061] In the formula, flatten represents the flattening operation, and softmax represents an activation function that can obtain classification results;

[0062] Calculate the cross entropy loss Loss between the categories and labels obtained by the neural network, which is used to further train and update the parameters of the neural network and save the optimal neural network model:

[0063]

[0064] Where N represents the sample size of synthetic EEG data vector, Label j represents the original label of the jth synthetic EEG data vector, log represents the logarithmic calculation, Prob j Represents the predicted category distribution probability of the j-th synthetic EEG data vector.

[0065] Furthermore, the auxiliary AC prediction module performs the following operations:

[0066] 1) Using the multi-sensory stimulation of the auditory-tactile perception paradigm design module and the question-answering process based on the brain-computer interface, combined with the EEG signal induction and acquisition module to carry out experiments. After receiving EEG signals online or loading saved EEG data files, the P300 EEG data vector is obtained through preprocessing;

[0067] 2) The P300 EEG data vector obtained in step 1) is used to generate a high-quality artificial EEG data vector through a data enhancement module, and then the synthetic EEG data vector is obtained by splicing.

[0068] 3) The synthesized EEG data vector obtained in step 2) is input into the trained neural network model, and the EEG signals collected in the online communication experiment are processed in real time to obtain answers to the questions, thereby realizing the function of assisting communication. The classification accuracy of the EEG signals is subsequently calculated, and the level of consciousness of the patient with impaired consciousness is evaluated based on the recorded classification accuracy.

[0069] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0070] 1. A vision-independent auditory-tactile stimulation paradigm, combined with a brain-computer interface, can be used to conduct brain-computer interface experiments for patients with impaired consciousness and visual impairment. This can analyze the specific EEG signals induced by this auditory-tactile stimulation and further assess their level of consciousness. This can be further combined with visual stimulation to enhance the effectiveness of multisensory stimulation and be applied to a wider population, leading to more accurate patient assessments and reduced costs for patients' families.

[0071] 2. Using a neural network composed of multi-scale convolution, channel attention mechanism, and deep convolution, we extract more discriminative EEG features while maximizing computational speed. Furthermore, through a new data augmentation method, we generate EEG data with more similar features and higher quality, further improving EEG signal classification performance.

[0072] 3. Design a system that integrates stimulation paradigms and EEG signal processing algorithms to achieve real-time communication assessment, providing patients with impaired consciousness with a simple channel for communication with the outside world, and assisting the Coma Recovery Scale in assessing the patient's level of consciousness. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 Schematic diagram of the relationship between the various modules of the system of the present invention.

[0074] Figure 2 Flowchart of training and prediction of the system of the present invention.

[0075] Figure 3 Schematic diagram of the auditory-tactile perception stimulation paradigm designed for the system of the present invention.

[0076] Figure 4 Schematic diagrams of three different multisensory stimulations designed for the system of the present invention.

[0077] Figure 5This is a complete flow chart of the communication experiment used by the system of the present invention; in the figure, Asking and Instruction is the questioning and instruction stage, Stimulation is the stimulation stage, which presents auditory and tactile multi-sensory stimulation to patients with consciousness disorders in the form of oddballs, Feedback is the feedback stage, and Rest is the rest stage.

[0078] Figure 6 Schematic diagram of the structure of the improved generative adversarial network used in the system of the present invention.

[0079] Figure 7 Schematic diagram of the structure of the neural network used in the system of the present invention; in the figure, Input Sample represents the input of the neural network, Conv represents ordinary convolution, Channel Attention Module represents the channel attention mechanism, MaxPool represents the global maximum pooling operation, AvgPool represents the global average pooling operation, Shared MLP represents the shared multi-layer perceptron, Concatenate represents the merging operation, DepthwiseConv represents the depth convolution, AveragePooling represents the average pooling operation, Flatten represents the flat fully connected operation, and Softmax represents the activation function used for classification. DETAILED DESCRIPTION

[0080] The present invention will be further described below with reference to specific embodiments.

[0081] This embodiment discloses a brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning. It is an assisted communication system developed using Python language and Microsoft visual studio platform and can be run on Windows devices. The relationship between the various modules of the system is as follows: Figure 1 As shown, the system training and prediction process is as follows Figure 2 shown.

[0082] It includes:

[0083] The auditory-tactile perceptual stimulation paradigm design module designs a multi-sensory stimulation paradigm and combines it with a brain-computer interface-based question-answering process to form assisted communication. The multi-sensory stimulation includes spatial orientation hearing and muscle pulse electrotactile sensation. The brain-computer interface-based question-answering process includes question instructions, stimulation, feedback, and rest.

[0084] The EEG signal induction and acquisition module uses a designed multi-sensory stimulation paradigm and a brain-computer interface-based question-answering process to induce P300 EEG signals in patients with impaired consciousness, and then uses a brain-computer interface device to collect the signals. The collected data is then pre-processed to form a P300 EEG data vector;

[0085] A data augmentation module uses an improved generative adversarial network to process P300 EEG data vectors to generate high-quality artificial EEG data vectors, which are then concatenated with the P300 EEG data vectors to obtain synthetic EEG data vectors. The improved generative adversarial network generates artificial EEG data vectors using resting-state EEG data vectors from patients with impaired consciousness, improves the activation functions of the generator and discriminator output layers to linear functions, and uses Wasserstein distance as a loss function to balance the training speeds of the discriminator and generator.

[0086] The communication-assisted training module designs a neural network composed of multi-scale convolution, a channel attention mechanism, and deep convolution. This neural network is trained using synthetic EEG data vectors to generate a neural network model with optimal performance for assisting communication. The multi-scale convolution is used to process synthetic EEG data vectors at different time scales, the channel attention mechanism is used to learn and amplify the weights of channels in important synthetic EEG data vectors, and the deep convolution is used to further extract the spatial features of the synthetic EEG data vectors.

[0087] The auxiliary communication prediction module uses the neural network model with the best performance obtained from the auxiliary communication training module to process the EEG signals of the patient with impaired consciousness in real time, obtain answers to questions in the question-and-answer process based on the brain-computer interface, so as to realize the function of auxiliary communication and provide an assessment of the consciousness level of the patient with impaired consciousness.

[0088] Specifically, the auditory-tactile perception stimulation paradigm design module includes a spatial orientation auditory module, a muscle pulse electrotactile module, a spatial-temporal consistency multi-sensory stimulation module, and a brain-computer interface-based question-answering process module. Figure 3 As shown, where:

[0089] The spatial orientation auditory module uses the head-related transfer function and a public dataset to perform convolution processing at different angles on the audio files of the Chinese characters "yes", "no", and "empty" to generate a spatial orientation effect. Patients with impaired consciousness wearing binaural headphones can perceive the sound "yes" coming from the right front, the sound "no" coming from the left front, and the sound "empty" coming from the front. "Yes" and "no" can be used as answers to simple questions, while "empty" can be used as a "skip" meaning or interference stimulus.

[0090] The muscle pulse electrotactile module is connected to a pulse stimulation instrument. After the positive and negative silicone electrodes that transmit pulse electrotactile sensation are attached to the skin of the patient with impaired consciousness, the intensity of the electrotactile stimulation is controlled by a computer. The electrodes are installed on the left and right wrists and abdomen of the patient with impaired consciousness, respectively, corresponding to the right, left, and front of the spatial hearing sense.

[0091] The spatial-temporal consistent multi-sensory stimulation module integrates auditory and tactile sensory stimulation and, through program control, delivers auditory and tactile stimulation of the same orientation to the consciousness disorder patient at a certain moment during the stimulation phase. The consciousness disorder patient will feel three different multi-sensory stimulations during the stimulation phase, such as Figure 4 As shown in a, when the "yes" stimulus is given, the earphones sense a "yes" audio in the right front, and at the same time, the right wrist receives an electrical tactile stimulus; Figure 4 As shown in b, when the "no" stimulus is given, the earphones sense a "no" audio signal from the left front, and at the same time, the left wrist receives an electrical tactile stimulus; Figure 4 As shown in the middle c, during the "empty" stimulation, the earphones sense an "empty" audio directly in front of them, and at the same time, an electrotactile stimulation is transmitted to the abdomen;

[0092] The simple question-answering process module contains a complete online brain-computer interface experimental process, such as Figure 5 As shown, in the questioning and instruction stage, patients with impaired consciousness are asked a simple question, such as "Have you ever worked in a bank?", and are asked to focus on the correct answer. In the stimulation stage, patients with impaired consciousness feel auditory and tactile stimulation conveyed in the form of oddballs. In the feedback stage, the collected EEG signals are processed in real time to obtain the answers to the questions from patients with impaired consciousness, and then they enter the rest stage.

[0093] Specifically, the EEG signal induction and acquisition module includes an EEG signal induction module, a data loading module and a data preprocessing module, wherein:

[0094] The EEG signal induction module uses auditory and tactile perception stimulation and a designed brain-computer interface-based question-answering process to induce P300 EEG signals, and uses a brain-computer interface device to collect them;

[0095] The data loading module can receive EEG data collected in real time during online experiments, and can also read EEG data files saved in different formats;

[0096] The data preprocessing module reduces the data volume and computational complexity by selecting a time window, performing baseline correction, filtering, and downsampling operations, and selects the P300 EEG data vector to be processed.

[0097] Specifically, the data enhancement module performs the following operations:

[0098] 1) Obtaining artificial data based on the improved generator:

[0099] like Figure 6 As shown in the figure, compared with the original generative adversarial network in which the generator receives random noise, it is improved to receive the EEG data vector of patients with consciousness disorders in the resting state to generate artificial EEG data vectors, and the original generator output layer activation function sigmoid is modified to ReLU to avoid the gradient disappearance and gradient explosion problems. The generated artificial EEG data vector Generator (D rest ):

[0100] Generator(D rest )=ReLU(BN(W3·ReLU(BN(W2·ReLU(BN(W1·D rest )))))

[0101] Where D rest represents the EEG data vector in the resting state, W1, W2, and W3 correspond to the weight matrices of each fully connected layer in the generator, BN represents the batch normalization operation, and ReLU represents the rectified linear unit activation function;

[0102] 2) The discriminator based on the double-layer LSTM calculates the discrimination result:

[0103] like Figure 6 As shown in the figure, the artificial EEG data vector generated by the generator is input into the discriminator based on the double-layer LSTM together with the P300 EEG data vector, and is processed in combination with Dropout and ReLU to obtain the discrimination result D(x):

[0104] Discriminator(x)=W dense ReLU(W lstm2 Dropout(ReLU(W lstm1 x+b lstm1 ))+b lstm2 )+b dense Where x is the P300 EEG data vector D P300 Or artificial EEG data vector Generator (D rest ), W lstm1 、W lstm2 Represents the weight matrices of the two LSTM layers in the discriminator hidden layer, W dense represents the weight matrix of the output layer, b lstm1 、b lstm2 、b dense Represent the bias vectors of the two LSTM layers and the output layer respectively;

[0105] 3) Generative Adversarial Networks Based on Wasserstein Distance:

[0106] like Figure 6 As shown in the figure, Wasserstein distance is used as a loss function to calculate the loss of the discriminator and generator during training, which is used to update the parameters of the discriminator and generator, and continuously optimize the discriminant ability of the discriminator and the generation quality of the generator. The Wasserstein distance is expressed as:

[0107]

[0108] Where, W(P P300 ,P g ) represents the distribution distance between the P300 EEG data vector and the artificial EEG data vector, P P300 Represents the distribution of P300 EEG data vector, P g Represents the distribution of artificial EEG data vector, ∏(P P300 ,P g ) is all from P P300 to P g The set of joint distributions of , γ represents P P300 to P g A joint distribution of , (x, y) represents a pair of data points sampled from the joint distribution, ||xy|| represents the distance between the data point pairs;

[0109] 4) By repeatedly iterating the above steps 1), 2) and 3), the generation ability of the generator is gradually improved, and finally a high-quality artificial EEG data vector D with a distribution similar to that of the P300 EEG data vector is obtained. g , the two EEG data vectors are spliced ​​together to obtain the synthetic EEG data vector D s , achieving the expansion of the EEG dataset:

[0110] D s =concatenate([D p300 ,D g ])

[0111] In the formula, concatenate represents the concatenation operation.

[0112] Specifically, the auxiliary communication training module performs the following operations:

[0113] 1) The shape of the synthesized EEG data vector is changed through the reshape operation to form the input of the convolutional layer:

[0114] Input=D s .reshape(C,T,1)

[0115] Where C and T represent the number of channels and data points of the synthetic EEG data vector respectively;

[0116] 2) If Figure 7 As shown in the figure, three different convolution kernels are used to process EEG data of different time periods respectively. BN, ELU and Dropout are used after each convolution to prevent overfitting, and the feature F after the first multi-scale convolution layer is obtained. (i) :

[0117] F (i) =Dropout(ELU(BN(Conv2d(Input,kernel_size=(1,s i )))))

[0118] Where i = {1, 2, 3}, corresponding to three different convolution kernels (1, 64), (1, 32) and (1, 16), respectively, F (i) Represents the features obtained by the i-th convolution kernel after the first multi-scale convolution layer. ELU is a linear unit activation function. Conv2d represents the convolution operation. (1, s i ) represents the size of the i-th convolution kernel, s i Represents the size of the i-th convolution kernel in the time dimension;

[0119] 3) If Figure 7 As shown in the figure, the different features obtained by multi-scale convolution are all learned through the channel attention mechanism to learn the information of important channels, and the feature F (i) Perform global average pooling and global maximum pooling respectively to avoid the influence of spatial information on attention, and then obtain the channel attention mask M of global average pooling through ReLU activation function and FC full connection layer. GAP and the global maximum pooled channel attention mask M GMP :

[0120] M GAP =FC(ReLU(f GAP (F (i) ))

[0121] M GMP =FC(ReLU(f GMP (F (i) ))

[0122] Where, f GAP represents the global average pooling operation, f GMP represents the global maximum pooling operation, and FC represents the fully connected operation;

[0123] The two channel attention masks obtained by different pooling operations are transformed by a variable weight matrix, and the channel weight map is output using the sigmoid activation function. The output is multiplied with the original feature to adjust the channel importance and obtain the feature after the channel attention mechanism.

[0124]

[0125] In the formula, σ represents the sigmoid activation function, α and β represent the variable weight matrices respectively;

[0126] 4) If Figure 7 As shown, the feature F obtained by merging the channel attention mechanism 4 :

[0127]

[0128] Where, and Respectively represent the features obtained by three different convolution kernels after the channel attention mechanism;

[0129] The spatial features are further extracted through a deep convolution with the same size as the number of channels of the synthetic EEG data vector, and BN, ELU and Dropout are used to prevent overfitting to obtain the feature F 5 :

[0130] F 5 =Dropout(ELU(BN(DepthwiseConv2d(F 4 ,kernel_size=(C,1)))))

[0131] Where DepthwiseConv2d represents depth convolution, and the convolution kernel size is (C, 1);

[0132] 5) If Figure 7 As shown, the features are average pooled to gradually extract the most important features. The remaining features are linearly transformed through flattening and fully connected layers, and finally the probability Prob of the predicted category distribution is obtained through the softmax activation function:

[0133] Prob=softmax(FC(flatten(f ap (F 5 )))

[0134] In the formula, flatten represents the flattening operation, and softmax represents an activation function that can obtain classification results;

[0135] Calculate the cross entropy loss Loss between the categories and labels obtained by the neural network, which is used to further train and update the parameters of the neural network and save the optimal neural network model:

[0136]

[0137] Where N represents the sample size of synthetic EEG data vector, Label j represents the original label of the jth synthetic EEG data vector, log represents the logarithmic calculation, Prob j Represents the predicted category distribution probability of the j-th synthetic EEG data vector.

[0138] Specifically, the auxiliary AC prediction module performs the following operations:

[0139] 1) Using the multi-sensory stimulation of the auditory-tactile perception paradigm design module and the question-answering process based on the brain-computer interface, combined with the EEG signal induction and acquisition module to carry out experiments. After receiving EEG signals online or loading saved EEG data files, the P300 EEG data vector is obtained through preprocessing;

[0140] 2) The P300 EEG data vector obtained in step 1) is used to generate a high-quality artificial EEG data vector through a data enhancement module, and then the synthetic EEG data vector is obtained by splicing.

[0141] 3) The synthesized EEG data vector obtained in step 2) is input into the trained neural network model to process the EEG signals collected in the online communication experiment in real time to obtain answers to the questions and realize the function of auxiliary communication. The classification accuracy of the EEG signals is subsequently calculated and the level of consciousness of the patient with impaired consciousness is evaluated based on the recorded classification accuracy.

[0142] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, any changes made based on the shape and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A brain-computer interface-assisted communication system based on auditory-tactile perception stimulation and deep learning, characterized by: include: The auditory-tactile perceptual stimulation paradigm design module designs a multi-sensory stimulation paradigm and combines it with a brain-computer interface-based question-answering process to form assisted communication. The multi-sensory stimulation includes spatial orientation hearing and muscle pulse electrotactile sensation. The brain-computer interface-based question-answering process includes question instructions, stimulation, feedback, and rest. The EEG signal induction and acquisition module uses a designed multi-sensory stimulation paradigm and a brain-computer interface-based question-answering process to induce P300 EEG signals in patients with impaired consciousness, and then uses a brain-computer interface device to collect the signals. The collected data is then pre-processed to form a P300 EEG data vector; A data augmentation module uses an improved generative adversarial network to process P300 EEG data vectors to generate high-quality artificial EEG data vectors, which are then concatenated with the P300 EEG data vectors to obtain synthetic EEG data vectors. The improved generative adversarial network generates artificial EEG data vectors using resting-state EEG data vectors from patients with impaired consciousness, improves the activation functions of the generator and discriminator output layers to linear functions, and uses Wasserstein distance as a loss function to balance the training speeds of the discriminator and generator. The communication-assisted training module designs a neural network composed of multi-scale convolution, channel attention mechanism, and deep convolution. This neural network is trained using synthetic EEG data vectors to generate a neural network model with optimal performance for assisting communication. The multi-scale convolution is used to process the synthetic EEG data vectors at different time scales, the channel attention mechanism is used to learn and amplify the weights of important channels in the synthetic EEG data vectors, and the deep convolution is used to further extract the spatial features of the synthetic EEG data vectors. The assisted communication prediction module uses the optimal neural network model obtained from the assisted communication training module to process the EEG signals of the patient with impaired consciousness in real time, obtaining answers to questions in the brain-computer interface-based question-answering process, thereby realizing the function of assisted communication and providing an assessment of the consciousness level of the patient with impaired consciousness; The auditory-tactile perception stimulation paradigm design module includes a spatial orientation auditory module, a muscle pulse electrotactile module, and a spatial-temporal consistency multi-sensory stimulation module, wherein: The spatial orientation auditory module uses the head-related transfer function and a public dataset to perform convolution processing at different angles on the audio files of the Chinese characters "yes", "no", and "empty" to generate a spatial orientation effect. When patients with impaired consciousness wear headphones in both ear canals, they can perceive the sound "yes" coming from the right front, the sound "no" coming from the left front, and the sound "empty" coming from the front. The muscle pulse electrotactile module is connected to a pulse stimulation instrument. After the positive and negative silicone electrodes that transmit pulse electrotactile sensation are attached to the skin of the patient with impaired consciousness, the intensity of the electrotactile stimulation is controlled by a computer. The electrodes are installed on the right and left wrists and abdomen of the patient with impaired consciousness, respectively, corresponding to the right, left, and front of the spatial orientation hearing. The spatial-temporal consistent multi-sensory stimulation module integrates auditory and tactile sensory stimulation and, through program regulation, achieves the delivery of auditory and tactile stimulation in the same direction to the consciousness disorder patient at a certain moment in the stimulation phase.

2. The brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning according to claim 1 is characterized in that: The auditory-tactile perception stimulation paradigm design module also includes a question-answering process module based on a brain-computer interface, wherein: The brain-computer interface-based question-answering process module includes a complete online brain-computer interface experimental process: first, the question-asking instruction link, then the multi-sensory stimulation link, followed by the brain-computer interface feedback link, and finally the rest link.

3. The brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning according to claim 2 is characterized in that: The EEG signal induction and acquisition module includes an EEG signal induction module, a data loading module and a data preprocessing module, wherein: The EEG signal induction module uses auditory and tactile perception stimulation and a designed brain-computer interface-based question-answering process to induce P300 EEG signals, and uses a brain-computer interface device to collect them; The data loading module can receive EEG data collected in real time during online experiments, and can also read EEG data files saved in different formats; The data preprocessing module reduces the data volume and computational complexity by selecting a time window, performing baseline correction, filtering, and downsampling operations, and selects the P300 EEG data vector to be processed.

4. The brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning according to claim 3 is characterized in that: The data enhancement module performs the following operations: 1) Obtain artificial EEG data vector based on the improved generator: Compared with the original generative adversarial network, the generator receives random noise, and the improvement is to receive the EEG data vector of patients with consciousness disorders in the resting state to generate artificial EEG data vectors, and the activation function sigmoid of the original generator output layer is modified to , to avoid the gradient disappearance and gradient explosion problems, the artificial EEG data vector generated in the generator part is obtained : ； Where, represents the EEG data vector in the resting state, 、 、 Corresponding to the weight matrix of each fully connected layer in the generator, represents the batch normalization operation, represents the rectified linear unit activation function; 2) The discriminator based on the double-layer LSTM calculates the discrimination result: The artificial EEG data vector generated by the generator is input into the discriminator based on the double-layer LSTM together with the P300 EEG data vector. and Process and obtain the judgment result : ； Where, is the P300 EEG data vector Or artificial EEG data vector , 、 Represent the weight matrices of the two LSTM layers in the discriminator hidden layer, represents the weight matrix of the output layer, 、 、 Represent the bias vectors of the two LSTM layers and the output layer respectively; 3) Generative Adversarial Networks Based on Wasserstein Distance: Wasserstein distance is used as a loss function to calculate the loss of the discriminator and generator during training, which is used to update the parameters of the discriminator and generator, and continuously optimize the discriminant ability of the discriminator and the generation quality of the generator. The Wasserstein distance is expressed as: ； Where, represents the distribution distance between the P300 EEG data vector and the artificial EEG data vector, represents the distribution of P300 EEG data vector, represents the distribution of artificial EEG data vectors, It's all from arrive The set of joint distributions of express arrive A joint distribution of represents a pair of data points sampled from this joint distribution, Represents the distance between pairs of data points; 4) By repeatedly iterating the above steps 1), 2) and 3), the generation ability of the generator is gradually improved, and finally a high-quality artificial EEG data vector with a distribution similar to that of the P300 EEG data vector is obtained. , the two EEG data vectors are spliced ​​together to obtain the synthetic EEG data vector , achieving the expansion of the EEG dataset: ； Where, Indicates a splicing and merging operation.

5. The brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning according to claim 4 is characterized in that: The auxiliary communication training module performs the following operations: 1) The shape of the synthesized EEG data vector is changed through the reshape operation to form the input of the convolutional layer : ； Where, Respectively represent the number of channels and data points of the synthetic EEG data vector; 2) Use three different convolution kernels to process EEG data of different time periods respectively, and use 、 and To prevent overfitting, get the features after the first multi-scale convolution layer : ； Where, , corresponding to three different convolution kernels, represents the first multi-scale convolutional layer after the first The features obtained by the convolution kernel are is a linear unit activation function, represents the convolution operation, Indicates the The size of the convolution kernel, Indicates the The size of the convolution kernel in the time dimension; 3) The different features obtained by multi-scale convolution are all learned through the channel attention mechanism to learn the information of important channels. Perform global average pooling and global maximum pooling respectively to avoid the influence of spatial information on attention, and then Activation function and Fully connected layer, respectively get the global average pooled channel attention mask and global max-pooling channel attention mask : ； ； Where, represents the global average pooling operation, represents the global maximum pooling operation, Represents a fully connected operation; The two channel attention masks obtained by different pooling operations are transformed by a variable weight matrix, and the channel weight map is output using the sigmoid activation function. The output is multiplied with the original feature to adjust the channel importance and obtain the feature after the channel attention mechanism. : ； Where, represents the sigmoid activation function, and Respectively represent the variable weight matrix; 4) Merge features obtained by channel attention mechanism : ； Where, 、 and Respectively represent the features obtained by three different convolution kernels after the channel attention mechanism; The spatial features are further extracted by a depthwise convolution with the same size as the number of channels of the synthetic EEG data vector, using 、 and To prevent overfitting, get features : ； Where, Represents depth convolution, and the convolution kernel size is ; 5) Perform average pooling on the features to gradually extract the most important features, perform linear transformation on the remaining features through flattening and fully connected layers, and finally obtain the probability of predicting the category distribution through the softmax activation function : ； Where, Represents a flattening operation, Represents an activation function that can obtain classification results; Calculate the cross entropy loss between the categories and labels obtained by the neural network , used to further train and update the parameters of the neural network and save the optimal neural network model: ； Where, represents the sample size of the synthetic EEG data vector, represents the original label of the j-th synthetic EEG data vector, represents logarithmic calculation, Represents the predicted category distribution probability of the j-th synthetic EEG data vector.

6. The brain-computer interface assisted communication system based on auditory-tactile perception stimulation and deep learning according to claim 5 is characterized in that: The auxiliary AC prediction module performs the following operations: 1) Using the multi-sensory stimulation of the auditory-tactile perception paradigm design module and the question-answering process based on the brain-computer interface, combined with the EEG signal induction and acquisition module to conduct experiments. After receiving EEG signals online or loading saved EEG data files, the P300 EEG data vector is obtained through preprocessing; 2) The P300 EEG data vector obtained in step 1) is used to generate a high-quality artificial EEG data vector through a data enhancement module, and then spliced ​​to obtain a synthetic EEG data vector; 3) The synthesized EEG data vector obtained in step 2) is input into the trained neural network model, and the EEG signals collected in the online communication experiment are processed in real time to obtain answers to the questions, thereby realizing the function of assisting communication. The classification accuracy of the EEG signals is subsequently calculated, and the level of consciousness of patients with impaired consciousness is evaluated based on the recorded classification accuracy.

Images