Brain-computer interface system based on audio-visual matching error monitoring

Abstract

The present invention discloses a brain-computer interface system based on audio-visual matching error monitoring. According to the EEG data obtained from three feedback paradigms of audio-visual matching errors, ERP analysis is applied to extract important ERP components related to audio-visual matching errors, so as to interpret the neural response within a specific time window; then, based on time-frequency analysis, the dynamic changes of EEG signals in different bands are captured, and the phase consistency ITPC of EEG activity is combined to reveal the reliability and stability of EEG activity under different feedback modes; finally, through the improved EEGNet model, automatic feature extraction and efficient classification of signals are realized, and the robustness and generalization ability of the system are improved. EEG signals are highly nonlinear and specific. By combining ERP analysis, time-frequency analysis and deep learning models, it is helpful to more effectively understand the potential neural mechanism of audio-visual matching errors, improve the performance of brain-computer interface systems and apply them to clinical research on error processing related disorders.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-computer interface technology and deep learning, and in particular to a brain-computer interface system based on audio-visual matching error monitoring. Background Art

[0002] The brain-computer interface (BCI) system decodes the brain's intentions by collecting and analyzing neural signals of brain activity, and further converts them into actionable instructions, thereby achieving barrier-free communication and interaction between humans and external devices such as computers. Its core lies in the real-time collection, processing and analysis of the brain's electrical activity signals. Electroencephalogram (EEG) is widely used because of its non-invasiveness and excellent time resolution. The BCI system decodes the brain's intentions and generates corresponding instructions through filtering, feature extraction and pattern recognition. This process relies on the spatiotemporal characteristics of the signal and the specific pattern of EEG activity, so as to achieve accurate interpretation of intentions and efficient human-computer interaction.

[0003] In recent years, BCI technology has continued to develop and has shifted from a single visual or auditory mode to a multimodal system. Audiovisual matching errors have become an important tool for neuroscience and BCI system research. The system induces event-related potential (ERP) responses in the brain by presenting inconsistent visual and auditory stimuli (such as seeing a "dog" image but hearing a "cat" sound). Such systems can directly monitor cognitive conflicts and perceptual errors and help evaluate the brain's response to environmental changes. This method shows potential in the diagnosis of consciousness disorders, cognitive function assessment, and intelligent interactive systems, especially when behavioral responses are absent or limited, providing patients with new communication channels. However, in such BCI system tasks, because visual signals suppress auditory signal processing, they mainly rely on ERP analysis, especially P300 and mismatch negativity (MMN), etc., lack multi-dimensional monitoring, resulting in poor system performance.

[0004] Based on the EEG data obtained from the three feedback paradigms of audio-visual matching errors, the present invention applies ERP analysis to extract the important ERP components related to audio-visual matching errors, thereby interpreting the neural response within a specific time window; then, based on time-frequency analysis, the dynamic changes of EEG signals in different bands are captured, and the phase consistency (Inter-Trial Phase Coherence, ITPC) of EEG activity is combined to reveal the reliability and stability of EEG activity under different feedback modes; finally, through the improved EEGNet model, automatic feature extraction and efficient classification of signals are achieved, which improves the robustness and generalization ability of the system. EEG signals are highly nonlinear and specific. By combining ERP analysis, time-frequency analysis and deep learning models, it helps us to effectively understand the potential neural mechanisms of audio-visual matching errors, improve the performance of brain-computer interface systems and apply them to clinical research on error processing related disorders.

[0005] The difficulty of the present invention lies in the selection of audio-visual stimulation, the design of experimental paradigm and the improvement of deep learning model. According to the characteristics of EEG activity caused by audio-visual matching errors, the selection of stimulus content should be common and easily recognizable objects, and the inconsistency between them should be sufficient to cause sensitive reactions of the brain. The feedback method adopted by the system should be able to fully reflect the information processing method of the brain in different environments in life. In view of the specificity of EEG signals, hyperparameter optimization is introduced in each model training to adapt to the individual differences of the subjects. In order to avoid the phenomenon of overfitting in training due to small data volume, L2 regularization and Dropout mechanism are introduced in the deep learning model to improve the robustness of the model. In order to improve the efficiency and accuracy of the model, the time window restriction is added to the input layer to make it focus on the EEG features related to audio-visual matching. Summary of the invention

[0006] The purpose of the present invention is to provide a brain-computer interface system based on audio-visual matching error monitoring to address the problems of existing BCI systems' insufficient monitoring of audio-visual matching errors, lack of comparison of EEG activity characteristics under different experimental paradigms, and complex feature extraction operations and lack of flexibility when traditional machine learning models are applied to EEG analysis. The system combines ERP analysis, time-frequency analysis and deep learning technology to form a BCI system in the audio-visual matching error paradigm that stably induces error-related potentials in subjects, and performs accurate classification and judgment through an improved deep learning model.

[0007] To achieve the above purpose, the technical solution provided by the present invention is: a brain-computer interface system based on audio-visual matching error monitoring, comprising:

[0008] The audiovisual stimulation module is used to provide the subjects with correct audiovisual matching stimulation and incorrect audiovisual matching stimulation in different feedback modes, so as to induce N400 signals and corresponding frequency band activities in their EEG; wherein, pictures of daily necessities are selected as visual stimulation, and the voices of the names of these daily necessities are recorded as auditory stimulation, and the subjects receive visual stimulation and auditory stimulation at the same time, and the correct audiovisual matching stimulation is set as the visual stimulation and auditory stimulation corresponding to the same daily necessities, and vice versa, it is the incorrect audiovisual matching stimulation; the correct audiovisual matching stimulation and the incorrect audiovisual matching stimulation are randomly composed of a stimulation sequence at a ratio equal to or less than 4:1, and each stimulation sequence includes at least 10 stimuli; each subject needs to accept the experimental paradigm of 3 feedback modes , respectively: full response, that is, pressing a button to respond to all stimuli received. This feedback mode simulates the response to information in daily life and provides a comprehensive perspective to explore the audio-visual matching process; single response, that is, when the subject receives an audio-visual matching error stimulus, the subject presses a button to respond. This response mode enables the experiment to focus on the significant enhancement of the N400 component, providing data support for understanding the brain's processing mechanism when encountering audio-visual matching errors; no response, that is, the subject does not need to respond to all stimuli. This feedback mode can better understand how the brain performs semantic processing without external interference, thereby revealing the basic cognitive process; each feedback mode contains at least 30 stimulus sequences;

[0009] The signal acquisition module is used to obtain the brain neuron electrical activities of the subjects' monitoring channels under different feedback modes, and measure the neural response signals to audiovisual stimulation in real time; wherein the channel selected according to the distribution characteristics of the EEG signal on the scalp that needs to be paid attention to is used as the monitoring channel;

[0010] A data preprocessing module is used to filter, remove electrooculographic artifacts and perform segmentation preprocessing on the original EEG signal data after the task of each feedback mode is completed, so as to obtain the preprocessed EEG signal segmentation data of the audio-visual matching correct stimulus and the audio-visual matching wrong stimulus;

[0011] The ERP monitoring module is used to monitor the millisecond-level potential response of the subject's brain after receiving the correct audio-visual matching stimulus and the incorrect audio-visual matching stimulus, draw the ERP image using the pre-processed segmented data of the correct audio-visual matching stimulus and the incorrect audio-visual matching stimulus, and monitor the appearance of the audio-visual matching error feature by comparing the difference in the amplitude of the N400 component in the ERP image under the correct audio-visual matching stimulus and the incorrect audio-visual matching stimulus in the monitoring channel;

[0012] The time-frequency monitoring module is used to monitor the dynamic changes of the frequency of the EEG signals of the subjects after they receive the correct audio-visual matching stimulation and the incorrect audio-visual matching stimulation. The pre-processed EEG signal segmented data of the correct audio-visual matching stimulation and the incorrect audio-visual matching stimulation are used to draw the event-related spectrum perturbation ERSP image and ITPC image to monitor the emergence of audio-visual matching error characteristics; among which, the ERSP image is used to reflect the changes of brain activities in different frequency bands over time, and the ITPC image is used to reveal the stability and reliability of the frequency band activities in the ERSP image;

[0013] The deep learning monitoring module uses an improved EEGNet model to cross-validate the pre-processed audio-visual matching correct stimulus and audio-visual matching incorrect stimulus EEG signal segmentation data, and judges whether the audio-visual matching error is monitored by the performance of the model on the cross-validation validation set; the improved EEGNet model includes the following improvements: ① Adding a hyperparameter optimization part, through which the model can show high performance in the monitoring of different subjects; ② Introducing L2 regularization, through which L2 regularization avoids overfitting in the cross-validation process; ③ Introducing a time window, through which the model can focus on certain target time periods in the audio-visual matching correct stimulus and audio-visual matching incorrect stimulus segmentation data, thereby improving the efficiency and accuracy of the model, where the target time period is the time period in which the N400 component in the ERP image may appear;

[0014] The comprehensive evaluation module comprehensively evaluates whether audio-visual matching errors are detected based on the ERP image obtained in the ERP monitoring module, the ERSP image and ITPC image obtained in the time-frequency monitoring module, and the average accuracy of cross-validation obtained by the deep learning monitoring module.

[0015] Furthermore, the signal acquisition module uses NuAmps amplifier and 32-channel EEG electrode cap as signal recording equipment, among which "HEOGR", "HEOGL" and "VEOGR", "VEOGL" are used as eye movement channels to monitor horizontal eye movement and vertical eye movement respectively, so as to facilitate the subsequent removal of electrooculographic artifacts in the EEG signal. All channels use the right ear protrusion as a reference to ensure signal consistency and accuracy. During the entire signal acquisition process, the sampling frequency is set to 250Hz. In order to ensure the quality of the system to obtain the signal, the contact impedance value of each channel is controlled below 5kΩ. To ensure that the surrounding environment is free of noise and electromagnetic interference, the subject is seated in front of the monitor with his or her line of sight level with the center of the monitor screen at a distance of 50-80cm. Then the subject is given an EEG electrode cap, and the electrode is filled with conductive paste to ensure that each electrode is well conductive before the subject is given headphones. Visual Studio Code software was used to write a program to play the audiovisual stimulation task. The monitor presented visual stimulation and the headphones played auditory stimulation. Before the start of each stimulation sequence, a cross mark appeared on the monitor screen to indicate entering the preparation state. While presenting visual and auditory stimulation to the patient, the raw EEG data being recorded was marked. When the correct audiovisual matching stimulation was presented, the corresponding time point in the raw EEG data was marked with "1". When the incorrect audiovisual matching stimulation was presented, the corresponding time point in the raw EEG data was marked with "2". When a cross mark appeared on the screen, the corresponding time point in the raw EEG data was marked with "81".

[0016] Furthermore, the data preprocessing module performs the following operations:

[0017] Firstly, the collected raw EEG data were bandpass filtered at 0.01-20 Hz, and the eye movement channels, namely "HEOGR", "HEOGL" and "VEOGR", "VEOGL" were marked as EOG. Then, the independent principal component analysis (ICA) was used to automatically identify, mark and remove eye movement artifacts. The raw EEG data were segmented according to the marks in the raw EEG data, and the time from a milliseconds before to b milliseconds after each mark was taken as a data segment. The data were corrected with the time from a milliseconds before the mark to the mark point as the baseline, and the EEG segmented data under the correct audio-visual matching stimulation and the incorrect audio-visual matching stimulation were obtained respectively.

[0018] Further, the ERP monitoring module performs the following operations:

[0019] Select at least one monitoring channel, use the EEG segmented data under correct audio-visual matching stimulation and incorrect audio-visual matching stimulation in each monitoring channel, superimpose and average, draw the ERP images under correct audio-visual matching stimulation and incorrect audio-visual matching stimulation, perform independent sample t-test at the same time, and use FDR correction to screen the results, compare the potential difference at each time point in the ERP image, and fill the area with significant difference in gray; when the ERP curve drawn with the EEG segmented data under correct audio-visual matching stimulation in the monitoring channel and the ERP curve drawn with the EEG segmented data under incorrect audio-visual matching stimulation show significantly different N400 waveforms, it is considered that audio-visual matching error is detected based on ERP analysis.

[0020] Further, the time-frequency monitoring module performs the following operations:

[0021] Firstly, m frequency values ​​with logarithmic uniform distribution were generated within the frequency range to be analyzed. In order to better balance the time resolution and frequency resolution, the number of wavelet cycles was set to change adaptively with the frequency. Morlet wavelet transform was performed on the EEG segmented data of all monitoring channels under correct and incorrect audiovisual matching stimulation. The power and phase consistency ITPC of each trial at different frequencies and time points were calculated, and the power value and ITPC value were converted into the logarithmic ratio of the baseline power and baseline ITPC value before drawing the ERSP image and ITPC image to more intuitively display the changes in power and ITPC values. When the theta band energy increased and the alpha band energy decreased in the ERSP image of the monitoring channel, and the ITPC value corresponding to the power change was high, it was considered that an audiovisual matching error was detected based on the time-frequency analysis.

[0022] Furthermore, the deep learning monitoring module constructs a hyperparameter space based on the improved EEGNet model, inputs the preprocessed audiovisual matching correct stimulation and audiovisual matching incorrect stimulation EEG signal segmented data under all monitoring channels, performs model training and hyperparameter tuning, and obtains the optimized convolution kernel length, number of convolution kernels, depth multiplier and dropout rate of the model; inputs the audiovisual matching correct stimulation and audiovisual matching incorrect stimulation segmented data of all monitoring channels, cross-validates the improved EEGNet model, sets the loss function to a custom F1 score loss function so that the model takes into account the classification effect of minority classes during training, outputs the single validation accuracy and the average cross-validation accuracy, and saves the model with the highest single validation accuracy as the best model for the subject under the corresponding experimental paradigm for audiovisual matching error monitoring.

[0023] Furthermore, the improved EEGNet model performs the following operations:

[0024] After preprocessing, the original EEG signal data is input into the improved EEGNet model through the input layer. A time window restriction is introduced here to ensure that the extracted features are consistent and representative within a specific time period. Then, the deep convolution layer is combined with the separable convolution layer to extract the temporal and spatial features of the data. At the same time, L2 regularization is introduced in the two convolution layers to prevent the model from overfitting, and the performance of the model is further improved through hyperparameter optimization. Subsequently, the multidimensional data is converted into a one-dimensional data vector through the flattening layer, and the global features are further learned using the fully connected layer. Softmax is used for classification, and the classification results are finally output. The model performance is optimized by calculating the loss.

[0025] Furthermore, the binomial test method was used to evaluate whether the deep learning monitoring module detected the audio-visual matching errors. In order to measure whether the accuracy was significant, the number of correct and incorrect stimulus types identified by the system was counted, and a binomial test was performed according to the following formula:

[0026]

[0027] Where X is the number of times the system correctly identifies the stimulus type, k = 1, 2, 3..., P(X = k) is the probability when the system correctly identifies the stimulus type for k times, n is the total number of stimuli, p is the probability of correct identification of a single stimulus, in order to find the critical number of correct identifications with significant accuracy, p is set to 0.5, the significance level α = 0.05, and the minimum k value that satisfies P(X = k) < α is calculated; in one feedback mode, at least 30 stimulus sequences, i.e. 300 stimuli, are included, and each verification includes 30 stimuli during ten-fold cross validation. According to the above formula, the average accuracy of the model trained with the subject data should be higher than 66.7% to be considered above the significant level.

[0028] Furthermore, the comprehensive evaluation module comprehensively measures the subject's response to the audiovisual matching error through the ERP image obtained by the ERP monitoring module, the ERSP image obtained by the time-frequency monitoring module, the ITPC image and the ten-fold cross-validation average accuracy of the deep learning monitoring module, and takes the result of deep learning monitoring as the main monitoring standard, and ERP analysis and time-frequency analysis as auxiliary monitoring standards. If the subject detects an audiovisual matching error in deep learning, and one of the ERP analysis and time-frequency analysis detects an audiovisual matching error, it can be determined that the audiovisual matching error is detected, and the subject has an obvious EEG response to the audiovisual matching error stimulus in the corresponding feedback mode; if the audiovisual matching error is detected in deep learning, but both the ERP analysis and the time-frequency analysis fail to detect the audiovisual matching error, it is determined that further observation is needed; if the deep learning monitoring module fails to detect the audiovisual matching error, it is determined that the audiovisual matching error is failed to be detected, and the subject has an obvious EEG response in the corresponding feedback mode.

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] 1. The present invention designs three feedback modes: full response mode, single response mode, and no response mode, which can respectively capture EEG data in active response, error detection, and natural state, and comprehensively analyze EEG activities in different cognitive processes. The no response mode is suitable for people with limited behavior, the single response mode is suitable for improving the sensitivity of specific tasks, and the full response mode is helpful for training actively controlled BCI systems. The combination of the three supports personalized and extensive application scenarios.

[0031] 2. The present invention performs ERP monitoring, time-frequency monitoring and deep learning model monitoring on the EEG data of the subjects, which can more comprehensively understand the brain's response in specific tasks, deeply explore the cognitive process from the three dimensions of time, frequency and features, and effectively improve the robustness and generalization ability of the system, making it suitable for complex BCI tasks.

[0032] 3. The present invention improves the generalization ability, robustness and monitoring efficiency of the system by introducing hyperparameter optimization, time window restriction and L2 regularization into the improved EEGNet model. It also overcomes the complex feature extraction operation of traditional deep learning and can be more conveniently applied to brain science research and clinical diagnosis.

[0033] 4. The present invention can not only be effectively applied to the evaluation and tracking of rehabilitation effects of patients with impaired consciousness, but also provide reliable evaluation for other new paradigms of error handling and determine the feasibility of experimental paradigms. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the system of the present invention.

[0035] Figure 2 Schematic diagram of the audiovisual stimulation process.

[0036] Figure 3 The figures are the average ERP waveform images of each monitoring channel induced by correct audio-visual matching stimulation and incorrect audio-visual matching stimulation for four subjects under three experimental paradigms; the ERP image is the curve of the change of the potential of the subjects in each monitoring channel over time after the audio-visual stimulation occurs, where the solid line is the ERP curve of the subjects under correct audio-visual matching stimulation; the dotted line is the ERP curve of the subjects under incorrect audio-visual matching stimulation; the shaded part is the time point when there is a significant difference between the ERP curve of the subjects under correct audio-visual matching stimulation and the ERP curve of the subjects under incorrect audio-visual matching stimulation; the dotted lines parallel to the vertical axis at 300ms and 600ms are the time window limits set in the deep learning monitoring module.

[0037] Figure 4These are the average ERSP and ITPC images of all monitoring channels induced by correct and incorrect audio-visual matching stimuli for four subjects under three experimental paradigms; the vertical axis on the left side of the ERSP image corresponds to different frequency values, the horizontal axis corresponds to the time after the audio-visual stimulation, and the color bar on the right reflects the logarithmic ratio of the frequency value power corresponding to different colors to the baseline power, reflecting the power changes of the brain in the two dimensions of time and frequency after the audio-visual stimulation; the vertical axis on the left side of the ITPC image corresponds to different frequency values, the horizontal axis corresponds to the time after the audio-visual stimulation, and the color bar on the right reflects the logarithmic ratio of the corresponding ITPC values ​​corresponding to different colors to the baseline ITPC values, reflecting the consistency of the phase response of the brain in the two dimensions of time and frequency after the audio-visual stimulation; the theta band corresponds to the frequency range of 4-7Hz, and the alpha band corresponds to the frequency range of 8-13Hz.

[0038] Figure 5 Schematic diagram of the structure of the improved EEGNet model. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with specific embodiments.

[0040] like Figure 1 As shown, this embodiment discloses a brain-computer interface system based on audio-visual matching error monitoring, including an audio-visual stimulation module, a signal acquisition module, a data preprocessing module, an ERP monitoring module, a time-frequency monitoring module, a deep learning monitoring module and a comprehensive evaluation module. The introduction of each functional module is as follows:

[0041] 1) Audiovisual stimulation module

[0042] It is used to provide the subjects with correct audio-visual matching stimulation and incorrect audio-visual matching stimulation in different feedback modes, so as to induce N400 signals in their EEG and corresponding frequency band activities; among them, some common pictures of daily necessities (such as mirrors, shoes, chopsticks, etc.) are selected as visual stimuli, and the voices of the names of these daily necessities are recorded as auditory stimuli. The subjects receive visual and auditory stimulation at the same time, and the correct audio-visual matching stimulation is set as the visual stimulation and auditory stimulation corresponding to the same daily necessities, and vice versa; the correct audio-visual matching stimulation and the incorrect audio-visual matching stimulation are randomly composed of a stimulation sequence in a ratio equal to or less than 4:1, and each stimulation sequence includes at least 10 stimuli; each subject needs to accept 3 experimental paradigms of feedback modes, namely: full response, that is, responding to the stimulation received The subjects responded to all stimuli by pressing a button. This feedback mode simulates the response to information in daily life and provides a comprehensive perspective to explore the audio-visual matching process. Single response, that is, when the subjects received an audio-visual matching error stimulus, they responded by pressing a button. This response mode allows the experiment to focus on the significant enhancement of the N400 component, providing data support for understanding the brain's processing mechanism when encountering an audio-visual matching error. No response, that is, the subjects do not need to respond to all stimuli. This feedback mode can better understand how the brain performs semantic processing without external interference, thereby revealing the basic cognitive process. Each feedback mode contains at least 30 stimulus sequences; in this embodiment, there are specifically 30 stimulus sequences, each stimulus sequence includes 10 stimuli, and the audio-visual stimulation process is as follows: Figure 2 shown.

[0043] 2) Signal acquisition module

[0044] It is used to obtain the brain neuron electrical activity of the subjects' monitoring channels in different feedback modes, and measure the neural response signals to audiovisual stimulation in real time; among them, the channel determined according to the distribution characteristics of the EEG signal on the scalp that needs to be paid attention to is used as the monitoring channel. The signal recording equipment uses NuAmps amplifier and 32-channel EEG electrode cap, among which the two pairs of channels "HEOGR", "HEOGL" and "VEOGR", "VEOGL" are used as eye movement channels to monitor horizontal eye movement and vertical eye movement respectively, so as to facilitate the subsequent removal of electrooculographic artifacts in the EEG signal. All channels use the right ear protrusion as a reference to ensure signal consistency and accuracy; during the entire data acquisition process, the sampling frequency is set to 250Hz. In order to ensure the quality of the system to obtain the signal, the contact impedance value of each channel should be controlled below 5kΩ. It is necessary to ensure that the surrounding environment is free of noise and electromagnetic interference. Let the subject sit in front of the monitor, keep his line of sight horizontal with the center of the monitor screen, and keep a distance of 50-80cm. Then put the EEG electrode cap on the subject, and fill the electrode with sufficient conductive paste to ensure that each electrode is well conductive before putting on the earphone; Visual Studio A program was written in Code software to play the audiovisual stimulation task. The monitor presented visual stimulation and the headphones played auditory stimulation. Before the start of each stimulation sequence, a cross mark appeared on the monitor screen to indicate entering the preparation state. While presenting visual and auditory stimulation to the patient, the original EEG signal data being recorded was marked. When the correct audiovisual matching stimulation was presented, the corresponding time point in the original EEG signal data was marked with "1". When the incorrect audiovisual matching stimulation was presented, the corresponding time point in the original EEG signal data was marked with "2". When a cross mark appeared on the screen, the corresponding time point in the original EEG signal data was marked with "81".

[0045] 3) Data preprocessing module

[0046] It is used to perform preprocessing such as filtering, removing electrooculographic artifacts and segmenting on the obtained raw EEG signal data after the task of each feedback mode is completed, so as to obtain the preprocessed audio-visual matching correct stimulation and audio-visual matching incorrect stimulation segmented data; firstly, the collected raw EEG signal data is band-pass filtered at 0.01-20Hz, and the eye movement channels (i.e. "HEOGR", "HEOGL" and "VEOGR", "VEOGL") are marked as EOG, and then the independent principal component analysis (ICA) method is used to automatically identify, mark and remove eye movement artifacts; the raw EEG signal data is segmented according to the marks in the raw EEG signal data, and the period from 100 milliseconds (ms) before each mark to 800 milliseconds (ms) after each mark is taken as a data segment, and the data is corrected with the period from 100 milliseconds (ms) before the mark to the mark point as the baseline, so as to obtain the EEG signal segmented data under the audio-visual matching correct stimulation and the audio-visual matching incorrect stimulation respectively.

[0047] 4) ERP monitoring module

[0048] Used to monitor the millisecond potential response of the subject's brain after receiving the correct audio-visual matching stimulus and the wrong audio-visual matching stimulus, draw the ERP image using the pre-processed correct audio-visual matching stimulus and the wrong audio-visual matching stimulus segmented data, and monitor the appearance of the audio-visual matching error feature by comparing the difference in the amplitude of the N400 component in the ERP image under the correct audio-visual matching stimulus and the wrong audio-visual matching stimulus in the monitoring channel. Select at least one monitoring channel, and in this embodiment, specifically select four monitoring channels ("CP4", "Cz", "CP3", "CPz"), use the EEG signal segmented data under the correct audio-visual matching stimulus and the wrong audio-visual matching stimulus under each monitoring channel, superimpose and average, and draw the ERP images under the correct audio-visual matching stimulus and the wrong audio-visual matching stimulus respectively, and perform an independent sample t test at the same time, and use FDR correction to screen the results, compare the potential difference at each time point in the ERP image, and fill the area with significant differences in gray; when the audio-visual matching error curve of the monitoring channel has an N400 waveform that is significantly different from the correct audio-visual matching curve, it is considered that the audio-visual matching error is monitored based on the ERP analysis.

[0049] 5) Time-frequency monitoring module

[0050] It is used to monitor the dynamic changes of the frequency of EEG signals over time after the subjects' brains receive correct and incorrect audio-visual matching stimuli. The pre-processed segmented data of correct and incorrect audio-visual matching stimuli are used to draw event-related spectral perturbation (ERSP) images and ITPC images to monitor the emergence of audio-visual matching error characteristics. The ERSP image can reflect the changes of brain activities in different frequency bands over time, and the ITPC image can reveal the stability and reliability of frequency band activities in the ERSP image. First, 10 frequency values ​​with logarithmic uniform distribution were generated within the frequency range to be analyzed. In order to better balance the time resolution and frequency resolution, the number of wavelet cycles was set to change adaptively with the frequency, and the number of cycles was 1 / 2 of the frequency value. Morlet wavelet transform was performed on the segmented EEG data of all monitoring channels under correct audiovisual matching stimulation and incorrect audiovisual matching stimulation. The power and phase consistency ITPC of each trial at different frequencies and time points were calculated, and the power value and ITPC value were converted to the logarithmic ratio of the baseline power before drawing the ERSP image and ITPC image to more intuitively display the changes in power and ITPC values. When the theta band energy increased and the alpha band energy decreased in the ERSP image of the monitoring channel, and the ITPC corresponding to the power change was higher than 0.5, it was considered that an audiovisual matching error was detected based on time-frequency analysis.

[0051] 6) Deep learning monitoring module

[0052] An improved EEGNet model was used to cross-validate the pre-processed EEG signal segmentation data of correct audio-visual matching stimulation and audio-visual matching error stimulation. The performance of the model on the cross-validation validation set was used to determine whether the audio-visual matching error was detected. The improved EEGNet model includes the following improvements:

[0053] ① Added hyperparameter optimization. EEG signals are specific and there may be individual differences between subjects. Through hyperparameter optimization, the model can show higher performance in monitoring different subjects.

[0054] ②Introduce L2 regularization. EEG signals often have certain noise, and the amount of data is relatively small for conventional deep learning models. L2 regularization can avoid overfitting during the cross-validation process.

[0055] ③ Introducing time windows, the model can focus on certain target time periods in the audio-visual matching correct stimulus and audio-visual matching incorrect stimulus segmented data, thereby improving the efficiency and accuracy of the model. The target time period is the time period in which the N400 component in the ERP image may appear.

[0056] The structure of the improved EEGNet model is as follows Figure 5 As shown in the figure, firstly, a hyperparameter space based on the improved EEGNet model is constructed, and the segmented data of correct audiovisual matching stimuli and incorrect audiovisual matching stimuli under all monitoring channels are input to train the model and tune the hyperparameters to obtain the optimized convolution kernel length, number of convolution kernels, depth multiplier and Dropout rate of the model; the segmented data of correct audiovisual matching stimuli and incorrect audiovisual matching stimuli of all monitoring channels are input, and the improved EEGNet model is cross-validated, and the loss function is set to a custom F1 score loss function so that the model can take into account the classification effect of minority classes during training, output the accuracy of a single validation and the average accuracy of cross-validation, and save the model with the highest single validation accuracy; the model with the highest single accuracy saved in the cross-validation is used as the best model for the subject under the corresponding experimental paradigm to monitor audiovisual matching errors.

[0057] The improved EEGNet model specifically performs the following operations:

[0058] After preprocessing, the original EEG signal data is input into the improved EEGNet model through the input layer. A time window restriction is introduced here to ensure that the extracted features are consistent and representative within a specific time period. Then, the deep convolution layer is combined with the separable convolution layer to extract the temporal and spatial features of the data. At the same time, L2 regularization is introduced in the two convolution layers to prevent the model from overfitting, and the performance of the model is further improved through hyperparameter optimization. Subsequently, the multidimensional data is converted into a one-dimensional data vector through the flattening layer, and the global features are further learned using the fully connected layer. Softmax is used for classification, and the classification results are finally output. The model performance is optimized by calculating the loss.

[0059] 7) Comprehensive evaluation module

[0060] It is used to comprehensively evaluate whether the system detects audio-visual matching errors based on the ERP images obtained by the system in the ERP monitoring module, the ERSP images obtained by the time-frequency monitoring module, the ITPC images, and the average accuracy of cross-validation obtained by the deep learning monitoring module. The binomial test method is used to evaluate whether the deep learning monitoring module detects audio-visual matching errors. In order to measure whether the accuracy is significant, the number of correct and incorrect times of the system identifying the stimulus type is counted, and the binomial test is performed according to the following formula:

[0061]

[0062] Where X is the number of times the system correctly identifies the stimulus type, k = 1, 2, 3..., P(X = k) is the probability when the system correctly identifies the stimulus type for k times, n is the total number of stimuli, p is the probability of correct identification of a single stimulus, in order to find the critical number of correct identifications with significant accuracy, p is set to 0.5, the significance level α = 0.05, and the minimum k value that satisfies P(X = k) < α is calculated; in one feedback mode, at least 30 stimulus sequences, i.e. 300 stimuli, are included, and each verification includes 30 stimuli during ten-fold cross validation. According to the above formula, the average accuracy of the model trained with the subject data should be higher than 66.7% to be considered above the significant level. The ERP images obtained by the ERP monitoring module, the ERSP images and ITPC images obtained by the time-frequency monitoring module, and the ten-fold cross-validation average accuracy of the deep learning monitoring module are used to comprehensively measure the subjects' responses to audio-visual matching errors. The results of deep learning monitoring are used as the main monitoring standard, and ERP analysis and time-frequency analysis are used as auxiliary monitoring standards. If the subject detects an audio-visual matching error in deep learning, and one of the ERP analysis and time-frequency analysis detects an audio-visual matching error, it can be determined that an audio-visual matching error is detected, and the subject has an obvious EEG response to the audio-visual matching error stimulus in the corresponding feedback mode; if an audio-visual matching error is detected in deep learning, but both the ERP analysis and the time-frequency analysis fail to detect the audio-visual matching error, it is determined that further observation is needed; if the deep learning monitoring module fails to detect the audio-visual matching error, it is determined that the audio-visual matching error is failed to be detected, and the subject's EEG response is not obvious in the corresponding feedback mode.

[0063] In this example, after four healthy subjects underwent three experimental paradigms, the system's offline classification accuracy for healthy subjects reached up to 87.1%, significantly better than the traditional method. The average ERP graph, ERSP image, and ITPC image of the four healthy subjects are shown in Figure 2. Figure 3 , Figure 4 As shown, negative waves appeared between 300-600ms in the ERP images under correct audiovisual matching stimulation and audiovisual matching error stimulation, and there were significant differences. In the ERSP images and ITPC images, it can be observed that the power of the alpha band was weakened and the power of the theta band was increased under high ITPC levels. Through the design of three experimental paradigms, it is shown that the brain-computer interface system based on audiovisual matching error monitoring proposed by the present invention can induce stable error-related EEG activities in healthy subjects, which can be successfully monitored by the system.

[0064] The embodiments described above are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A brain-computer interface system based on audio-visual matching error monitoring, characterized in that: include: The audiovisual stimulation module is used to provide the subjects with correct audiovisual matching stimulation and incorrect audiovisual matching stimulation in different feedback modes, so as to induce N400 signals and corresponding frequency band activities in their EEG; wherein, pictures of daily necessities are selected as visual stimulation, and the voices of the names of these daily necessities are recorded as auditory stimulation, and the subjects receive visual stimulation and auditory stimulation at the same time, and the correct audiovisual matching stimulation is set as the visual stimulation and auditory stimulation corresponding to the same daily necessities, and vice versa, it is the incorrect audiovisual matching stimulation; the correct audiovisual matching stimulation and the incorrect audiovisual matching stimulation are randomly composed of a stimulation sequence at a ratio equal to or less than 4:1, and each stimulation sequence includes at least 10 stimuli; each subject needs to accept the experimental paradigm of 3 feedback modes , respectively: full response, that is, pressing a button to respond to all stimuli received. This feedback mode simulates the response to information in daily life and provides a comprehensive perspective to explore the audio-visual matching process; single response, that is, when the subject receives an audio-visual matching error stimulus, the subject presses a button to respond. This response mode enables the experiment to focus on the significant enhancement of the N400 component, providing data support for understanding the brain's processing mechanism when encountering audio-visual matching errors; no response, that is, the subject does not need to respond to all stimuli. This feedback mode can better understand how the brain performs semantic processing without external interference, thereby revealing the basic cognitive process; each feedback mode contains at least 30 stimulus sequences; The signal acquisition module is used to obtain the brain neuron electrical activities of the subjects' monitoring channels under different feedback modes, and measure the neural response signals to audiovisual stimulation in real time; wherein the channel selected according to the distribution characteristics of the EEG signal on the scalp that needs to be paid attention to is used as the monitoring channel; A data preprocessing module is used to filter, remove electrooculographic artifacts and perform segmentation preprocessing on the original EEG signal data after the task of each feedback mode is completed, so as to obtain the preprocessed EEG signal segmentation data of the audio-visual matching correct stimulus and the audio-visual matching wrong stimulus; The ERP monitoring module is used to monitor the millisecond-level potential response of the subject's brain after receiving the correct audio-visual matching stimulus and the incorrect audio-visual matching stimulus, draw the ERP image using the pre-processed EEG signal segmented data of the correct audio-visual matching stimulus and the incorrect audio-visual matching stimulus, and monitor the appearance of the audio-visual matching error feature by comparing the difference in the amplitude of the N400 component in the ERP image under the correct audio-visual matching stimulus and the incorrect audio-visual matching stimulus in the monitoring channel; The time-frequency monitoring module is used to monitor the dynamic changes of the frequency of the EEG signals of the subjects after they receive the correct audio-visual matching stimulation and the incorrect audio-visual matching stimulation. The pre-processed EEG signal segmented data of the correct audio-visual matching stimulation and the incorrect audio-visual matching stimulation are used to draw the event-related spectrum perturbation ERSP image and ITPC image to monitor the emergence of audio-visual matching error characteristics; among which, the ERSP image is used to reflect the changes of brain activities in different frequency bands over time, and the ITPC image is used to reveal the stability and reliability of the frequency band activities in the ERSP image; The deep learning monitoring module uses an improved EEGNet model to cross-validate the pre-processed audio-visual matching correct stimulus and audio-visual matching incorrect stimulus EEG signal segmentation data, and judges whether the audio-visual matching error is monitored by the performance of the model on the cross-validation validation set; the improved EEGNet model includes the following improvements: ① Adding a hyperparameter optimization part, through which the model can show high performance in the monitoring of different subjects; ② Introducing L2 regularization, through which L2 regularization avoids overfitting in the cross-validation process; ③ Introducing a time window, through which the model can focus on certain target time periods in the audio-visual matching correct stimulus and audio-visual matching incorrect stimulus segmentation data, thereby improving the efficiency and accuracy of the model, where the target time period is the time period in which the N400 component in the ERP image may appear; The comprehensive evaluation module comprehensively evaluates whether audio-visual matching errors are detected based on the ERP image obtained in the ERP monitoring module, the ERSP image and ITPC image obtained in the time-frequency monitoring module, and the average accuracy of cross-validation obtained by the deep learning monitoring module.

2. The brain-computer interface system based on audio-visual matching error monitoring according to claim 1 is characterized in that: The signal acquisition module uses NuAmps amplifier and 32-channel EEG electrode cap as signal recording equipment, among which "HEOGR", "HEOGL" and "VEOGR", "VEOGL" are used as eye movement channels to monitor horizontal eye movement and vertical eye movement respectively, so as to facilitate the subsequent removal of electrooculographic artifacts in the EEG signal. All channels use the right ear protrusion as a reference to ensure signal consistency and accuracy. During the entire signal acquisition process, the sampling frequency is set to 250Hz. In order to ensure the quality of the system to obtain the signal, the contact impedance value of each channel is controlled below 5kΩ. To ensure that the surrounding environment is free of noise and electromagnetic interference, the subject is seated in front of the monitor with his or her line of sight level with the center of the monitor screen at a distance of 50-80cm. Then the subject is given an EEG electrode cap, and the electrode is filled with conductive paste to ensure that each electrode is well conductive before the subject is given headphones. VisualStudio is used Code software was used to write a program to play the audiovisual stimulation task. The monitor presented visual stimulation and the headphones played auditory stimulation. Before the start of each stimulation sequence, a cross mark appeared on the monitor screen to indicate entering the preparation state. While presenting visual and auditory stimulation to the patient, the original EEG signal data being recorded was marked. When the correct audiovisual matching stimulation was presented, the corresponding time point in the original EEG signal data was marked with "1". When the audiovisual matching stimulation was incorrect, the corresponding time point in the original EEG signal data was marked with "2". When a cross mark appeared on the screen, the corresponding time point in the original EEG signal data was marked with "81".

3. The brain-computer interface system based on audio-visual matching error monitoring according to claim 2 is characterized in that: The data preprocessing module performs the following operations: Firstly, the collected raw EEG signal data were bandpass filtered at 0.01-20Hz, and the eye movement channels, namely "HEOGR", "HEOGL" and "VEOGR", "VEOGL" were marked as EOG. Then, the independent principal component analysis (ICA) was used to automatically identify, mark and remove eye movement artifacts. The raw EEG signal data were segmented according to the marks in the raw EEG signal data. The time from a milliseconds before the mark to b milliseconds after the mark was taken as a data segment. The data were corrected with the time from a milliseconds before the mark to the mark point as the baseline, and the EEG signal segmentation data under the correct audio-visual matching stimulation and the audio-visual matching incorrect stimulation were obtained respectively.

4. The brain-computer interface system based on audio-visual matching error monitoring according to claim 3 is characterized in that: The ERP monitoring module performs the following operations: Select at least one monitoring channel, use the EEG signal segmented data under the audio-visual matching correct stimulation and audio-visual matching incorrect stimulation in each monitoring channel, superimpose and average, draw the ERP images under the audio-visual matching correct stimulation and audio-visual matching incorrect stimulation, perform independent sample t-test at the same time, and use FDR correction to screen the results, compare the potential difference at each time point in the ERP image, and fill the area with significant difference in gray; when the ERP curve drawn by the EEG signal segmented data under the audio-visual matching correct stimulation in the monitoring channel and the ERP curve drawn by the EEG signal segmented data under the audio-visual matching incorrect stimulation show a significantly different N400 waveform, it is considered that the audio-visual matching error is detected based on the ERP analysis.

5. The brain-computer interface system based on audio-visual matching error monitoring according to claim 4 is characterized in that: The time-frequency monitoring module performs the following operations: Firstly, m frequency values ​​with logarithmic uniform distribution were generated within the frequency range to be analyzed. In order to better balance the time resolution and frequency resolution, the number of wavelet cycles was set to change adaptively with the frequency. Morlet wavelet transform was performed on the EEG signal segmented data of all monitoring channels under correct audiovisual matching stimulation and incorrect audiovisual matching stimulation. The power and phase consistency ITPC of each trial at different frequencies and time points were calculated, and the power value and ITPC value were converted into the logarithmic ratio of the baseline power and baseline ITPC value before drawing the ERSP image and ITPC image to more intuitively display the changes in power and ITPC values. When the theta band energy increased and the alpha band energy decreased in the ERSP image of the monitoring channel, and the ITPC value corresponding to the power change was high, it was considered that an audiovisual matching error was detected based on the time-frequency analysis.

6. The brain-computer interface system based on audio-visual matching error monitoring according to claim 5 is characterized in that: The deep learning monitoring module constructs a hyperparameter space based on the improved EEGNet model, inputs the segmented data of the preprocessed audio-visual matching correct stimulation and audio-visual matching incorrect stimulation EEG signals under all monitoring channels, performs model training and hyperparameter tuning, and obtains the optimized convolution kernel length, number of convolution kernels, depth multiplier and dropout rate of the model; inputs the segmented data of the audio-visual matching correct stimulation and audio-visual matching incorrect stimulation of all monitoring channels, cross-validates the improved EEGNet model, sets the loss function to a custom F1 score loss function so that the model takes into account the classification effect of the minority classes during training, outputs the accuracy of a single validation and the average accuracy of cross validation, and saves the model with the highest single validation accuracy as the best model for the subject under the corresponding experimental paradigm for audio-visual matching error monitoring.

7. The brain-computer interface system based on audio-visual matching error monitoring according to claim 6 is characterized in that: The improved EEGNet model performs the following operations: After preprocessing, the original EEG signal data is input into the improved EEGNet model through the input layer. A time window restriction is introduced here to ensure that the extracted features are consistent and representative within a specific time period. Then, the deep convolution layer is combined with the separable convolution layer to extract the temporal and spatial features of the data. At the same time, L2 regularization is introduced in the two convolution layers to prevent the model from overfitting, and the performance of the model is further improved through hyperparameter optimization. Subsequently, the multidimensional data is converted into a one-dimensional data vector through the flattening layer, and the global features are further learned using the fully connected layer. Softmax is used for classification, and the classification results are finally output. The model performance is optimized by calculating the loss.

8. The brain-computer interface system based on audio-visual matching error monitoring according to claim 7 is characterized in that: The binomial test method is used to evaluate whether the deep learning monitoring module detects audio-visual matching errors. In order to measure whether the accuracy is significant, the number of correct and incorrect stimulus types identified by the system is counted, and a binomial test is performed according to the following formula: Where X is the number of times the system correctly identifies the stimulus type, k = 1, 2, 3..., P(X = k) is the probability when the system correctly identifies the stimulus type for k times, n is the total number of stimuli, p is the probability of correct identification of a single stimulus, in order to find the critical number of correct identifications with significant accuracy, p is set to 0.5, the significance level α = 0.05, and the minimum k value that satisfies P(X = k) < α is calculated; in one feedback mode, at least 30 stimulus sequences, i.e. 300 stimuli, are included, and each verification includes 30 stimuli during ten-fold cross validation. According to the above formula, the average accuracy of the model trained with the subject data should be higher than 66.7% to be considered above the significant level.

9. The brain-computer interface system based on audio-visual matching error monitoring according to claim 8, characterized in that: The comprehensive evaluation module comprehensively measures the subject's response to the audiovisual matching error through the ERP image obtained by the ERP monitoring module, the ERSP image and the ITPC image obtained by the time-frequency monitoring module, and the ten-fold cross-validation average accuracy of the deep learning monitoring module, and uses the result of deep learning monitoring as the main monitoring standard, and ERP analysis and time-frequency analysis as auxiliary monitoring standards. If the subject monitors the audiovisual matching error in deep learning, and one of the ERP analysis and time-frequency analysis monitors the audiovisual matching error, it can be determined that the audiovisual matching error is monitored, and the subject has an obvious EEG response to the audiovisual matching error stimulus in the corresponding feedback mode; if the audiovisual matching error is monitored in deep learning, but the ERP analysis and time-frequency analysis fail to monitor the audiovisual matching error, it is determined that further observation is needed; if the deep learning monitoring module fails to monitor the audiovisual matching error, it is determined that the audiovisual matching error is failed to be monitored, and the subject has an obvious EEG response in the corresponding feedback mode.

Images