Methods, devices, and systems for detecting EEG artifacts based on multimodal sensor fusion

Abstract

This disclosure provides a method, apparatus, and system for EEG artifact detection based on multimodal sensor fusion. The EEG artifact detection method includes: using a first thread to acquire raw EEG signals collected in real time by an EEG signal sensor in a head-mounted device, and IMU signals collected in real time by an IMU in the head-mounted device; using a second thread decoupled from the first thread to preprocess the raw EEG signals to obtain a EEG signal to be processed, and aligning and splicing the IMU signal with the EEG signal to be processed to obtain splicing information; using the second thread to input the EEG signal to be processed into each model in a first model set, so that each model in the first model set outputs a corresponding artifact detection result; using the second thread to input the splicing information into each model in a second model set, so that each model in the second model set outputs a corresponding artifact detection result, wherein the artifact types detected by each model in the first model set and the second model set are different from each other.

Description

Technical Field

[0001] This disclosure relates to the field of biomedical signal processing technology, and in particular to a method, apparatus and system for detecting electroencephalography (EEG) artifacts based on multimodal sensor fusion. Background Technology

[0002] EEG artifacts refer to non-neuronal interference superimposed on EEG signals. They can be divided into two main categories: physiological artifacts (generated by the subject's own physiological activities) and non-physiological artifacts (caused by external environmental or equipment factors).

[0003] Currently, the main methods for detecting brainwave artifacts include the following:

[0004] (1) Signal processing-based methods

[0005] Independent Component Analysis (ICA) is currently the most widely used method for EEG artifact removal in academia. ICA belongs to the Blind Source Separation (BSS) technique. Its basic principle is to decompose multi-channel mixed EEG signals into statistically independent components, and then manually or automatically identify and remove artifact components such as blinking, electromyography (EMG), and electrocardiogram (ECG) artifacts. Commonly used ICA algorithms include FastICA, InfomaxICA, and Second Order Blind Identification (SOBI). Among them, Infomax ICA has been integrated into the mainstream EEG analysis tool EEGLAB and is widely used. The core advantage of the ICA method is that it can theoretically completely separate artifact components from neural signals, and it does not depend on the specific waveform template of the artifact, making it suitable for the joint removal of multiple artifact types.

[0006] Wavelet transform-based methods decompose EEG signals into sub-bands of different frequency scales to identify and suppress artifact energy within specific frequency bands. Blink artifacts are mainly concentrated in the low-frequency sub-band of 1-4 Hz, while electromyography artifacts are concentrated in the high-frequency sub-band above 30 Hz. By applying soft / hard thresholding to the wavelet coefficients of the corresponding sub-bands, selective suppression of specific artifacts can be achieved in the time-frequency domain. This type of method has advantages in time-frequency resolution and is suitable for handling non-stationary, sudden artifact events.

[0007] (2) Machine learning-based methods

[0008] Shallow machine learning methods first manually extract features from EEG data, then use a classifier to detect artifacts in the feature vectors. Commonly used features include: time-domain features (peak-to-peak value, root mean square, zero-crossing rate, kurtosis, skewness), frequency-domain features (power spectral density of each frequency band, power ratio of each band), and time-frequency features (short-time Fourier transform coefficients, wavelet energy distribution), etc. Commonly used classifiers include Support Vector Machine (SVM), Random Forest, Linear Discriminant Analysis (LDA), and Naive Bayes, etc.

[0009] LDA-based blink detection algorithms were widely used in early brain-computer interface systems due to their computational simplicity and low latency. Studies on multi-class artifact classification based on SVM have shown that, in single-subject scenarios, using power spectral features combined with radial basis function (RBF) kernel SVM can achieve high classification accuracy (approximately 85%-90%). These methods generally offer some real-time performance and have low computational cost during the inference phase, making them suitable as lightweight classification schemes in computationally limited scenarios.

[0010] (3) Deep learning-based methods

[0011] In recent years, deep learning has made significant progress in the field of EEG signal analysis. Convolutional Neural Networks (CNNs) automatically extract local time-frequency features from EEG signals through multiple layers of convolutional kernels and have been widely applied to tasks such as motion visualization, emotion recognition, and artifact detection. EEGNet, a CNN architecture specifically designed for EEG signals, achieves end-to-end feature learning through a combination of depthwise convolution and separable convolution, achieving good cross-task generalization performance on multiple public datasets. Recurrent Neural Networks (RNNs) and their variant, Long Short-Term Memory (LSTM), excel at capturing long-range temporal dependencies in EEG signals, offering advantages in continuous artifact detection tasks. Furthermore, Transformer models based on attention mechanisms are increasingly being introduced into EEG analysis, modeling spatial dependencies between multiple channels through a global self-attention mechanism. The common advantage of the aforementioned deep learning models is that they do not require manual feature engineering and can directly learn discriminative features end-to-end from the raw signal. With sufficient training data, their classification performance is significantly better than that of shallow machine learning methods.

[0012] (4) Built-in methods by equipment manufacturers

[0013] Taking the Muse headband as an example, InteraXon, a Canadian company, has integrated the following three types of real-time artifact detection interfaces into the device firmware and accompanying SDK: isgood (overall signal quality assessment, outputting 0 / 1 binary values), Blink (blink event detection, triggered based on the amplitude threshold of the frontal channel), and Jaw (masseter muscle artifact detection, judged based on high-frequency power threshold). These three interfaces are output synchronously with the EEG data as additional annotation fields of the real-time data stream, requiring no additional computing resources and can be directly called by the application layer. They are currently the most representative online artifact detection solution in consumer-grade mobile EEG devices. Summary of the Invention

[0014] The inventors noted that, among related technologies, EEG artifact detection technology mainly suffers from the following defects.

[0015] 1) Insufficient detection accuracy: The detection accuracy of existing artifact detection interfaces (such as Muse's isgood, Blink, and Jaw metrics) is generally low. In real-world complex scenarios, the false detection rate and false negative rate are high, which cannot meet the requirements of high-quality data.

[0016] 2) Incomplete coverage of artifact types: The existing solution only targets a few common artifacts (blinking, masseter muscle, overall quality), and fails to cover a variety of artifact types that frequently occur in real-world scenarios, such as eye movement, raising eyebrows, frowning, power line interference, speaking, head movement, and abnormal wearing (wearing it inside out / wearing it without a case).

[0017] 3) Inability to process in real time: Offline artifact removal schemes based on traditional signal processing methods such as ICA require a long computation window (usually 30 seconds to several minutes), which cannot achieve real-time detection with a time resolution of 1 second and is not suitable for scenarios with immediate feedback.

[0018] 4) Insufficient utilization of multimodal information: Using only EEG single-modal data cannot accurately detect artifacts such as body motion-related artifacts (head movement, walking) and wearing status (wearing inside out / wearing without).

[0019] 5) Lack of real-time user feedback mechanism: Existing EEG devices do not display the types of data artifacts and quality assessment results to non-professional users in real time during the data collection process, which makes it impossible for users to detect and correct bad collection behaviors in a timely manner (such as excessive head movement, touching the charged device, etc.), forming a vicious cycle of data quality.

[0020] 6) Poor cross-subject generalization: Due to insufficient sample size and model design issues, the existing scheme has weak cross-subject generalization ability, and its performance drops significantly on new subjects that have not been seen before, which limits its practical application value.

[0021] To address at least one of the aforementioned problems, this disclosure provides a method, apparatus, and system for EEG artifact detection based on multimodal sensor fusion. By acquiring EEG and IMU signals through multimodal sensor fusion and utilizing a multi-model collaborative detection architecture, accurate artifact detection results can be obtained in real time.

[0022] In a first aspect of this disclosure, a method for detecting EEG artifacts based on multimodal sensor fusion is provided, executed by an EEG artifact detection device based on multimodal sensor fusion, comprising: using a first thread to acquire raw EEG signals collected in real time by an EEG signal sensor in a head-mounted device, and IMU signals collected in real time by an inertial measurement unit (IMU) in the head-mounted device; using a second thread decoupled from the first thread to preprocess the raw EEG signals to obtain EEG signals to be processed, and aligning and splicing the IMU signals with the EEG signals to be processed to obtain splicing information; using the second thread to input the EEG signals to be processed into each model in a first model set, so that each model in the first model set outputs a corresponding artifact detection result; using the second thread to input the splicing information into each model in a second model set, so that each model in the second model set outputs a corresponding artifact detection result, wherein the artifact types detected by the models in the first model set and the second model set are different from each other.

[0023] In some embodiments, the EEG signal sensor includes multiple electrodes; the raw EEG signal includes multi-channel EEG signals corresponding one-to-one with the multiple electrodes.

[0024] In some embodiments, the preprocessing of the original EEG signal includes: windowing the original EEG signal; performing baseline correction on each channel of the EEG signal within each time window to obtain a multi-channel corrected signal; filtering the multi-channel corrected signal to obtain a multi-channel filtered signal; and normalizing the multi-channel filtered signal to obtain the EEG signal to be processed.

[0025] In some embodiments, the filtering process for the multi-channel correction signal includes: performing high-pass filtering on each channel correction signal in the multi-channel correction signal to remove low-frequency baseline drift, to obtain a multi-channel intermediate filtered signal; and performing power frequency notch filtering on each channel intermediate filtered signal in the multi-channel intermediate filtered signal to suppress power frequency interference, to obtain the multi-channel filtered signal.

[0026] In some embodiments, the baseline correction of the EEG signals of each channel within each time window includes: subtracting the mean value of the EEG signals of the i-th channel within each time window from the EEG signal of the i-th channel to obtain the corrected signal of the i-th channel. N is the total number of channels.

[0027] In some embodiments, the plurality of electrodes includes a first electrode AF7 corresponding to the left forehead of the user wearing the head-mounted device, a second electrode AF8 corresponding to the right forehead of the user, a third electrode TP9 corresponding to the left temporal region of the user, and a fourth electrode TP10 corresponding to the right temporal region of the user.

[0028] In some embodiments, the first model set includes: a model for detecting resting state artifacts, a model for detecting blinking artifacts, a model for detecting speaking artifacts, a model for detecting chewing artifacts, a model for detecting mains interference artifacts, a model for detecting eye movement artifacts, and a model for detecting eyebrow movement artifacts; the second model set includes: a model for detecting head movement artifacts, a model for detecting head-mounted device backwards artifacts, and a model for detecting head-mounted device unmounted artifacts.

[0029] In some embodiments, all models in the first model set and the second model set are loaded into memory synchronously.

[0030] In some embodiments, each model in the first model set and the second model set includes a depthwise separable convolutional (DSC) model, an EEGNet model, a shallow convolutional network (ShallowConvNet) model, a MobileNet model, or a gradient boosting decision tree (GBDT) model.

[0031] In some embodiments, the IMU includes an accelerometer and a gyroscope; the IMU signal includes acceleration and angular velocity signals of the user's head wearing the headset.

[0032] In some embodiments, the head-mounted device includes a headband.

[0033] In some embodiments, the first thread is used to write the raw EEG signal and the IMU signal into a circular buffer; the second thread is used to read the raw EEG signal and the IMU signal from the circular buffer.

[0034] In some embodiments, a third thread, decoupled from the first thread and the second thread, is used to visualize the artifact detection results of each model in the first model set and the second model set.

[0035] In some embodiments, the third thread is used to determine whether the m-th model detects the corresponding artifact. M is the total number of models in the first model set and the second model set; if the m-th model detects the corresponding artifact, the third thread is used to send the corresponding prompt information to the user wearing the head-mounted device.

[0036] In some embodiments, determining whether the m-th model detects the corresponding artifact includes: determining whether the m-th model detects the corresponding artifact within multiple consecutive time windows; if the m-th model detects the corresponding artifact within the multiple consecutive time windows, then determining that the m-th model has detected the corresponding artifact.

[0037] In a second aspect of this disclosure, an EEG artifact detection device based on multimodal sensor fusion is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method as described in any of the above embodiments.

[0038] In a third aspect of this disclosure, an EEG artifact detection system is provided, comprising: an EEG artifact detection device as described in any of the above embodiments; and a head-mounted device including an EEG signal sensor for real-time acquisition of raw EEG signals and an IMU for real-time acquisition of inertial measurement unit (IMU) signals.

[0039] In some embodiments, the EEG signal sensor includes multiple electrodes; the raw EEG signal includes multi-channel EEG signals corresponding one-to-one with the multiple electrodes.

[0040] In some embodiments, the plurality of electrodes includes a first electrode AF7 corresponding to the left forehead of the user wearing the head-mounted device, a second electrode AF8 corresponding to the right forehead of the user, a third electrode TP9 corresponding to the left temporal region of the user, and a fourth electrode TP10 corresponding to the right temporal region of the user.

[0041] In some embodiments, the IMU includes an accelerometer and a gyroscope; the IMU signal includes acceleration and angular velocity signals of the user's head wearing the headset.

[0042] In a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0043] In a fifth aspect of this disclosure, a computer program product is provided, including computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any of the above embodiments.

[0044] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating an embodiment of the EEG artifact detection method based on multimodal sensor fusion according to this disclosure.

[0047] Figure 2 This is a schematic diagram of the electrode arrangement according to an embodiment of the present disclosure;

[0048] Figure 3 This is a schematic diagram of the structure of a single-mode signal input model according to an embodiment of the present disclosure;

[0049] Figure 4 This is a schematic diagram of the structure of a multimodal signal input model according to an embodiment of the present disclosure;

[0050] Figure 5 This is a flowchart illustrating another embodiment of the EEG artifact detection method based on multimodal sensor fusion according to the present disclosure;

[0051] Figure 6 This is a schematic diagram of the structure of an EEG artifact detection device based on multimodal sensor fusion according to an embodiment of this disclosure;

[0052] Figure 7 This is a schematic diagram of the structure of an EEG artifact detection system according to an embodiment of the present disclosure;

[0053] Figure 8 This is a schematic diagram of the detection process of an EEG artifact detection system according to an embodiment of the present disclosure. Detailed Implementation

[0054] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0055] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0056] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0057] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0058] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0059] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0060] Figure 1 This is a flowchart illustrating an embodiment of the EEG artifact detection method based on multimodal sensor fusion according to this disclosure. In some embodiments, the following EEG artifact detection method based on multimodal sensor fusion is executed by an EEG artifact detection device based on multimodal sensor fusion, including steps 11-15.

[0061] In step 11, the first thread is used to acquire the raw EEG signal collected in real time by the EEG signal sensor in the head-mounted device, and the IMU signal collected in real time by the IMU (Inertial Measurement Unit) in the head-mounted device.

[0062] It should be noted that the raw EEG and IMU signals will be collected and processed with the explicit authorization of the user, and this disclosure will not actively collect sensitive personal information such as user identification, which complies with the Personal Information Protection Law and related data security regulations.

[0063] In some embodiments, the EEG signal sensor includes multiple electrodes, and the raw EEG signal includes multi-channel EEG signals corresponding one-to-one with the multiple electrodes.

[0064] For example, the EEG signal sensor includes a first electrode AF7 corresponding to the left forehead of the user wearing the head-mounted device, a second electrode AF8 corresponding to the right forehead of the user, a third electrode TP9 corresponding to the left temporal region of the user, and a fourth electrode TP10 corresponding to the right temporal region of the user.

[0065] In this case, the raw EEG signal includes a 4-channel EEG signal corresponding to the first electrode AF7, the second electrode AF8, the third electrode TP9, and the fourth electrode TP10.

[0066] For example, the sampling frequency of the first electrode AF7, the second electrode AF8, the third electrode TP9, and the fourth electrode TP10 is 256Hz.

[0067] Figure 2 The positions of the first electrode AF7, the second electrode AF8, the third electrode TP9, and the fourth electrode TP10 relative to the subject's head are given.

[0068] For example, head-mounted devices include headbands or other devices used to collect EEG signals.

[0069] In some embodiments, the IMU includes an accelerometer and a gyroscope. Accordingly, the IMU signal includes acceleration and angular velocity signals of the user's head wearing the headset.

[0070] For example, the acceleration signal is a three-dimensional acceleration signal with a sampling rate of 52Hz. The angular velocity signal is a three-dimensional angular velocity signal with a sampling rate of 52Hz.

[0071] In some embodiments, the head-mounted device establishes a Bluetooth Low Energy (BLE) connection with the EEG artifact detection device and sends the collected raw EEG signals and IMU signals to the EEG artifact detection device.

[0072] In some embodiments, the head-mounted device also sends real-time electrode impedance data of each channel to an EEG artifact detection device so that the EEG artifact detection device can detect the contact quality of each electrode.

[0073] In step 12, the original EEG signal is preprocessed using the second thread, which is decoupled from the first thread, to obtain the EEG signal to be processed. The IMU signal is then aligned and spliced ​​with the EEG signal to be processed to obtain splicing information.

[0074] In some embodiments, a first thread writes the raw EEG signal and IMU signal into a circular buffer. Then, a second thread reads the raw EEG signal and IMU signal from the circular buffer.

[0075] It should be noted that since there is no direct calling relationship between the first thread and the second thread, they are in a completely decoupled state. Therefore, signal processing will not affect the continuous acquisition of signals, thereby effectively reducing the information processing latency.

[0076] In some embodiments, the step of preprocessing the raw EEG signal includes steps S101-S104.

[0077] S101. Perform windowing processing on the original EEG signal.

[0078] For example, by using a time window of 1 second, the end-to-end detection delay can be controlled to within 1 second.

[0079] S102. Perform baseline correction on each channel of EEG signal within each time window to obtain multi-channel corrected signals.

[0080] In some embodiments, within each time window, the average value of the EEG signal of the i-th channel within each time window is subtracted from the EEG signal of the i-th channel to obtain the correction signal of the i-th channel. N represents the total number of channels. This effectively eliminates DC drift in each channel during the acquisition process.

[0081] S103. Filter the multi-channel correction signal to obtain the multi-channel filtered signal.

[0082] In some embodiments, the filtering process may include the following steps.

[0083] 1) Perform high-pass filtering on each channel of the multi-channel correction signal to remove low-frequency baseline drift, and obtain the multi-channel intermediate filter signal.

[0084] For example, an IIR (Infinite Impulse Response Digital Filter) high-pass filter with a cutoff frequency of 0.5 Hz can be used to filter the signals in each channel to remove low-frequency baseline drift.

[0085] It should be noted that since IIR filters do not introduce future data dependencies, the preprocessing results during inference are completely consistent with those during offline training, thus avoiding model performance degradation caused by inconsistent preprocessing methods.

[0086] 2) Perform power frequency notch filtering on each channel of the multi-channel intermediate filter signal to suppress power frequency interference, and obtain the multi-channel filter signal.

[0087] For example, a 50Hz notch filter can be used to suppress power frequency interference.

[0088] It should be noted that notch filtering can effectively eliminate the 50Hz fixed interference from the power frequency, while retaining the effective signals of other frequencies, thereby improving the signal-to-noise ratio of the target signal.

[0089] S104. Normalize the multi-channel filtered signal to obtain the EEG signal to be processed.

[0090] For example, linearly normalize the amplitude of each channel signal to Intervals eliminate differences in signal amplitude between individuals and improve the model's ability to generalize across subjects.

[0091] It should be noted that EEG signals alone are insufficient for effectively detecting head motion artifacts and artifacts related to abnormal headband wearing, such as reversed wearing and unworn wearing. Since IMU information provides independent physical motion and contact state information, aligning and splicing the IMU signal with the EEG signal to be processed allows for more accurate detection of head motion artifacts and artifacts related to abnormal headband wearing, such as reversed wearing and unworn wearing.

[0092] For example, the accelerometer (3-axis) within the corresponding 1-second time window 52 points) and gyroscope (3-axis) After the 52 data points are aligned with the EEG window on the time axis, they are concatenated into a joint input tensor and fed into the corresponding model.

[0093] In step 13, the EEG signal to be processed is input into each model in the first model set using the second thread, so that each model in the first model set outputs the corresponding artifact detection result.

[0094] It should be noted that the types of artifacts detected by each model in the first model set are different.

[0095] For example, the first model set includes the following models.

[0096] 1) Model M1 for detecting resting-state artifacts.

[0097] It should be noted that resting-state artifacts are artifacts included in the EEG signal of a subject when the subject is in a resting state.

[0098] 2) Model M2 for detecting blink artifacts.

[0099] It should be noted that the blink artifact is the electro-oculogram (EOG) potential change generated when the eyelids close during blinking, which is superimposed on the forehead channel as a low-frequency waveform with a large amplitude.

[0100] 3) Model M3 for detecting speech artifacts.

[0101] It should be noted that talk artifacts are a mixture of electromyographic artifacts generated by the movement of the lips, tongue, and jaw muscles when the subject begins to speak.

[0102] 4) Model M4 for detecting chewing artifacts.

[0103] It should be noted that the chewing artifact (Jaw), also known as the masseter muscle artifact, is a high-frequency electromyography (EMG) artifact generated by the contraction of the masseter muscle during biting or chewing, which is particularly noticeable in the temporal channel.

[0104] 5) Model M5 for detecting mains interference artifacts.

[0105] It should be noted that mains interference artifacts (Line) refer to the 50Hz or 60Hz power frequency electromagnetic radiation from the AC power supply mixed into the EEG signal through capacitive coupling, which is significantly enhanced when the subject touches a charged metal object.

[0106] 6) Model M6 for detecting eye movement artifacts.

[0107] It should be noted that saccades are ocular artifacts caused by changes in corneal-retinal potentials when a subject's eyes move rapidly horizontally or vertically.

[0108] 7) Model M7 for detecting eyebrow movement artifacts.

[0109] It should be noted that eyebrow movement artifacts are electromyographic artifacts generated by the contraction of the frontal muscles when the subject raises or furrows his / her eyebrows, and they mainly affect the frontal channel.

[0110] It should also be noted that the input information for models M1-M7 is the same, which is the EEG signal to be processed, but the detection tasks of models M1-M7 are different.

[0111] In some embodiments, each model in the first model set may include a Depthwise Separable Convolution (DSC) model, an EEGNet model, a ShallowConvNet model, a MobileNet model, or a Gradient Boosting Decision Tree (GBDT) model.

[0112] It's important to note that depthwise separable convolution breaks down traditional convolution into two independent steps: depthwise convolution and pointwise convolution. Compared to standard convolution, the total number of weights in depthwise separable convolution is only 10 times that of standard convolution. ~25 The computational cost is also significantly reduced, while the feature representation capability of the convolution kernel is basically retained, and the efficiency of parameter usage is improved.

[0113] EEGNet is a lightweight CNN architecture designed specifically for EEG signals, which achieves end-to-end feature learning through depthwise separable convolutions.

[0114] ShallowConvNet is a lightweight shallow convolutional neural network specifically designed for EEG signal classification. It has advantages in temporal feature extraction and is suitable for detecting low-frequency artifacts (such as mains interference).

[0115] MobileNet is a lightweight convolutional neural network designed specifically for mobile and embedded devices. By porting the depthwise separable convolution approach from the successful MobileNetV2 in the image domain to 1D temporal signal processing, the model efficiency can be further improved.

[0116] The core idea of ​​GBDT is to sequentially iteratively train multiple CART (Classification and Regression Tree) decision trees. Each tree fits the residual (negative gradient of the loss function) of the preceding model. Finally, the prediction results of all trees are summed to obtain the final output. The prediction accuracy is improved by continuously reducing the overall loss.

[0117] For example, each model employs a lightweight temporal classification network with depthwise separable convolutions at its core. Figure 3 As shown, each model in the first model set may include layers connected in sequence: an input layer (shape 4) 256, corresponding to 4 channels 256-sample EEG frames), depthwise convolutional layers (performing independent temporal convolutions along the time axis for each EEG channel, with a kernel size of 1). 64, extracting local temporal features for each channel, with only 1 / 4 the number of parameters of an equivalent ordinary convolution), pointwise convolutional layer (1 1. Convolutional layer (linearly combines the features from each channel of the deep convolution output to achieve cross-channel feature fusion), batch normalization layer (normalizes each feature map to accelerate training convergence and improve model stability), ReLU activation layer, global average pooling layer (compresses the temporal feature map into a fixed-length feature vector while eliminating the dependence on the input temporal length), fully connected layer (maps the feature vector to a scalar), Sigmoid activation layer (compresses the output to a scalar). The range is defined as the probability of an artifact being present. The number of trainable parameters for each model is kept below 200K, and after INT8 full integer quantization, the size of a single file does not exceed 1MB.

[0118] Furthermore, in terms of model training, a binary classification model is trained independently for each type of artifact. For example, the loss function can be binary cross-entropy or other equivalent binary classification loss functions, and the optimizer can be the Adam optimizer or other optimizers based on gradient descent algorithms. The positive and negative samples in the training data come from the task segment and the clean segment (EO (Eyes Open) and EC (Eyes Closed) segments of the corresponding artifact task, respectively.

[0119] In step 14, the splicing information is input into each model in the second model set using the second thread, so that each model in the second model set outputs the corresponding artifact detection result.

[0120] It should be noted that the types of artifacts detected by the models in the first and second model sets are different.

[0121] For example, the second model set includes the following models.

[0122] 1) Model M8 for detecting head motion artifacts.

[0123] It should be noted that head motion artifacts (Nod / Shake / Walk) are changes in electrode contact state caused by the subject's overall head movements such as nodding, shaking, and walking, which produce obvious features in the IMU signal.

[0124] 2) Model M9 for detecting anti-artifacts on head-mounted devices.

[0125] It should be noted that updown artifacts occur when the headband is worn in the wrong direction, causing the spatial positions of the electrodes to be interchanged with the standard positions, resulting in a systematic abnormality in the EEG channel signal characteristics.

[0126] 3) Model M10 for detecting unworn artifacts in head-mounted devices.

[0127] It should be noted that "empty" refers to the signal state when the headband is not worn on the subject's head. At this time, the electrode impedance is extremely high, and the signal is pure noise.

[0128] As explained above, EEG signals alone are insufficient for effectively detecting head motion artifacts and artifacts related to abnormal headband wearing, such as reversed wearing and unworn wearing. Since IMU information provides independent physical motion and contact state information, aligning and splicing the IMU signal with the EEG signal to be processed, and then inputting the spliced ​​information into models M8, M9, and M10 respectively, can effectively improve the AUC (Area Under the ROC Curve) of models M8, M9, and M10.

[0129] In some embodiments, each model in the second model set includes a DSC model, an EEGNet model, a ShallowConvNet model, a MobileNet model, or a GBDT model. For example, each model employs a lightweight temporal classification network with depthwise separable convolutions at its core.

[0130] For example, the structure of each model in the second model set is as follows: Figure 4 As shown. Figure 4 and Figure 3 The difference is that, in Figure 3 In the illustrated embodiment, the model's input is the EEG signal to be processed. Figure 4 In the illustrated embodiment, the model's input is the EEG signal to be processed. IMU signal.

[0131] Figure 5 This is a flowchart illustrating another embodiment of the EEG artifact detection method based on multimodal sensor fusion. In some embodiments, the following EEG artifact detection method based on multimodal sensor fusion is executed by an EEG artifact detection device based on multimodal sensor fusion, including steps 51-56.

[0132] It should be noted here that... Figure 5 Steps 51-54 in the middle Figure 1 Steps 11-14 are the same.

[0133] In step 51, the first thread is used to acquire the raw EEG signal collected in real time by the EEG signal sensor in the head-mounted device, and the IMU signal collected in real time by the IMU in the head-mounted device.

[0134] In some embodiments, the EEG signal sensor includes multiple electrodes, and the raw EEG signal includes multi-channel EEG signals corresponding one-to-one with the multiple electrodes.

[0135] For example, the EEG signal sensor includes a first electrode AF7 corresponding to the left forehead of the user wearing the head-mounted device, a second electrode AF8 corresponding to the right forehead of the user, a third electrode TP9 corresponding to the left temporal region of the user, and a fourth electrode TP10 corresponding to the right temporal region of the user.

[0136] In this case, the raw EEG signal includes a 4-channel EEG signal corresponding to the first electrode AF7, the second electrode AF8, the third electrode TP9, and the fourth electrode TP10.

[0137] In some embodiments, the IMU includes an accelerometer and a gyroscope. Accordingly, the IMU signal includes acceleration and angular velocity signals of the user's head wearing the headset.

[0138] In step 52, the original EEG signal is preprocessed using the second thread, which is decoupled from the first thread, to obtain the EEG signal to be processed. The IMU signal is then aligned and spliced ​​with the EEG signal to be processed to obtain splicing information.

[0139] In some embodiments, a first thread writes the raw EEG signal and IMU signal into a circular buffer. Then, a second thread reads the raw EEG signal and IMU signal from the circular buffer.

[0140] It should be noted that since there is no direct calling relationship between the first thread and the second thread, they are in a completely decoupled state. Therefore, signal processing will not affect the continuous acquisition of signals, thereby effectively reducing the information processing latency.

[0141] In step 53, the EEG signal to be processed is input into each model in the first model set using the second thread, so that each model in the first model set outputs the corresponding artifact detection result.

[0142] It should be noted that the types of artifacts detected by each model in the first model set are different.

[0143] For example, the first set of models includes model M1 for detecting resting state artifacts, model M2 for detecting blinking artifacts, model M3 for detecting speaking artifacts, model M4 for detecting chewing artifacts, model M5 for detecting mains interference artifacts, model M6 for detecting eye movement artifacts, and model M7 for detecting eyebrow movement artifacts.

[0144] In some embodiments, each model in the first model set may include a DSC model, an EEGNet (electroencephalography network) model, a ShallowConvNet model, a MobileNet model, or a GBDT model.

[0145] For example, each model employs a lightweight temporal classification network with depthwise separable convolutions at its core.

[0146] In step 54, the second thread is used to input the splicing information into each model in the second model set so that each model in the second model set can output the corresponding artifact detection result.

[0147] It should be noted that the types of artifacts detected by the models in the first and second model sets are different.

[0148] For example, the second set of models includes: model M8 for detecting head motion artifacts, model M9 for detecting head-mounted device reverse artifacts, and model M10 for detecting head-mounted device unmounted artifacts.

[0149] In some embodiments, each model in the second model set includes a DSC model, an EEGNet model, a ShallowConvNet model, a MobileNet model, or a GBDT model.

[0150] For example, each model employs a lightweight temporal classification network with depthwise separable convolutions at its core.

[0151] In step 55, using a third thread decoupled from the first and second threads, the artifact detection results of each model in the first and second model sets are visualized.

[0152] It should be noted that since the third thread is decoupled from the first and second threads mentioned above, the visualization of the artifact detection results will not affect information collection and processing, thus effectively reducing information processing latency.

[0153] In some embodiments, the test results are presented in a real-time visual manner on the user interface, and a complete quality control report is generated after the session ends.

[0154] For real-time interface display, a 10-row artifact status indicator panel can be maintained, with each row corresponding to a type of artifact. When the corresponding model determines that an artifact of that type exists in the current frame, the indicator light in the corresponding row turns red; otherwise, it turns green. The panel is refreshed every 250ms with new inference results. Simultaneously, the 4-channel EEG waveform of the current 1-second window is displayed in real-time on the interface as a scrolling line graph. The top of the interface displays the cumulative session quality score, which is calculated based on the cumulative proportion of each type of artifact and weighted according to preset weights, reflecting the overall data quality of the current acquisition session in real time.

[0155] In step 56, using the third thread, it is determined whether the m-th model has detected the corresponding artifact. M represents the total number of models in the first and second model sets. If the m-th model detects a corresponding artifact, a corresponding prompt message is sent to the user wearing the head-mounted device using a third thread.

[0156] In some embodiments, it is determined whether the m-th model detects the corresponding artifact within multiple consecutive time windows. If the m-th model detects the corresponding artifact within multiple consecutive time windows, then it is determined that the m-th model has detected the corresponding artifact.

[0157] For example, a built-in rule engine can be used to trigger corresponding text prompts based on artifact state vectors: if blinking artifacts are detected for five consecutive time windows (750ms in total), the user is prompted to reduce the frequency of blinking, frowning, raising eyebrows, etc. If movement artifacts are detected for five consecutive time windows, the user is prompted to keep their head still. If mains interference is detected for five consecutive time windows, the user is immediately prompted to remove their hands from powered devices (such as the metal casing of a laptop). If the headband is detected to be worn backwards or not at all for five consecutive time windows, the user is prompted to readjust the headband's orientation and position.

[0158] Through the above processing, real-time guidance prompts constitute a closed-loop feedback mechanism for optimizing the data collection behavior, which can help reduce EEG artifacts.

[0159] It should be noted that the EEG artifact detection device employs a multi-threaded asynchronous architecture, with each functional module decoupled and running in parallel to avoid computationally intensive inference tasks blocking the data acquisition process. The multi-threaded asynchronous architecture consists of three main threads.

[0160] The EEG acquisition thread (i.e. the first thread mentioned above) is responsible for acquiring headband data streams through the BLE protocol and OSC (Open Sound Control) data packets, receiving multiple signals from EEG (256Hz) and IMU (52Hz) and storing them in a ring buffer.

[0161] The quality inspection thread (i.e. the second thread mentioned above) is responsible for retrieving sliding window data from the buffer, calling 10 classification models in parallel for inference, and outputting the artifact state vector after multiple consecutive frames of decision.

[0162] The feedback thread (i.e. the third thread mentioned above) is responsible for judging the recognition results, triggering corresponding user guidance prompts, and updating the interface display in real time.

[0163] The three threads communicate unidirectionally via two thread-safe queues, forming a pipeline structure of "acquisition, inference, and feedback." There are no direct calls between the threads, ensuring complete decoupling. By completely decoupling the computationally intensive 10-model inference task from the real-time data acquisition task, inference latency is guaranteed not to block the continuous acquisition of EEG signals. At system startup, the three threads are created by the main thread and set as daemon threads. They start with the main program and automatically terminate when the main program exits, eliminating the need for manual management of their lifecycles. After completing thread creation and model preloading, the main thread is only responsible for responding to user session start and end operations.

[0164] During system startup, the quality inspection thread first performs model initialization. Since the system needs to run 10 classification models simultaneously, and each model has a large number of parameters, a sequential loading method would significantly increase startup latency. Therefore, this disclosure adopts an asynchronous concurrent loading scheme: the weight files of the 10 models are loaded into memory concurrently to minimize system startup time. Once the models are loaded, they remain resident in memory and do not need to be reloaded during the entire acquisition session. When each frame of data arrives, the initialized model inference interface is directly called, completely eliminating the time overhead of repetitive frame-by-frame loading. Actual measurements showed that the total inference latency of the 10 detection models was 21.71ms. The inference times for each model were as follows: blinking model 2.30ms, no-wear model 2.24ms, open / closed resting state model 1.90ms, frowning / raising eyebrow model 2.23ms, teeth clenching / chewing model 2.08ms, mains interference model 2.29ms, nodding / shaking / walking model 2.14ms, eye movement model 2.40ms, speaking model 2.16ms, and reversed-wearing model 2.32ms. These metrics meet the latency constraints for real-time EEG processing. The total size of the 10 models does not exceed 10MB, allowing for simultaneous deployment and operation on a standard consumer PC without affecting normal device use.

[0165] The EEG acquisition thread establishes a connection with the headband via the BLE protocol and parses OSC data packets, distributing data using a circular buffer. The specific process is as follows: after startup, it continuously receives EEG and IMU data. Due to the different sampling rates of EEG (256Hz) and IMU (52Hz), timestamp alignment is used for multimodal data alignment, and the two signals are synchronously stored in the circular buffer according to the timeline. The EEG acquisition thread acts as a producer; when the accumulated data in the buffer reaches the sampling points required for a complete sliding window, it packages the current window data into a standardized data frame and places it in the data frame queue (frame_queue), then continues to acquire the next batch of data without waiting for inference results. The frame_queue is implemented using a thread-safe queue with a built-in mutex lock mechanism, ensuring data consistency between the acquisition thread's writes and the quality inspection thread's reads, eliminating the need for additional locking and minimizing synchronization overhead while ensuring thread safety.

[0166] The quality inspection thread acts as an inference consumer, continuously listening to the frame queue in a blocking manner. Whenever a new data frame appears in the queue, it is immediately retrieved and the inference process begins. For example, data processing can be divided into three steps executed sequentially. The first step is sliding window extraction: the raw EEG signal segment is read from the circular buffer, and the signal is processed using a sliding window mechanism. The window length is, for example, 256 sampling points (corresponding to a 1-second sampling duration), and the sliding step size is, for example, 64 sampling points (i.e., 0.25 seconds). Through 75... The high overlap design ensures that short-term transient artifacts such as blinking and electromyography can cover at least four consecutive windows without being missed due to window boundaries. The second step is parallel model inference: after each sliding window data frame arrives, the quality detection thread simultaneously calls 10 pre-loaded classification models. Each model independently performs inference on the current frame and outputs the results. The system calculates the probability of artifacts within a given interval. With 10 models executing concurrently, the total latency is only 21.71ms. The third step is temporal coherence judgment: To avoid misjudgments caused by random noise in single-frame inference, the system introduces an artifact judgment strategy based on temporal coherence. A circular judgment queue of length 5 is constructed for each type of artifact. The probability of artifact existence obtained from model inference in each frame is recorded in real time. When the number of frames with a probability higher than a preset probability threshold (e.g., the preset probability threshold is 0.75) is greater than a predetermined frame number threshold (e.g., the predetermined frame number threshold is 4 frames), a valid artifact event is determined to exist. After the 10 models complete inference and are judged, the quality detection thread puts the generated 10-dimensional artifact state vector into the result queue (result_queue) and then continues to process the next frame of data without waiting for the interface to refresh.

[0167] The feedback thread, acting as the result consumer, continuously listens to the `result_queue` in a blocking manner. Whenever a new artifact state vector appears in the queue, it is immediately retrieved. Based on the built-in rule engine, the artifact state vector is mapped to the corresponding user interface update instructions and text prompts: if five consecutive windows detect blinking or eyebrow artifacts, the user is prompted to reduce facial movements; if five consecutive windows detect movement artifacts, the user is prompted to keep their head still; if mains interference is detected, the user is immediately prompted to remove their hands from the powered device; if the headband is worn backwards or not at all, the user is prompted to readjust the headband's direction and position. The feedback thread and the quality detection thread are completely decoupled through the `result_queue`, and the time spent on UI rendering does not affect the continuous execution of the inference process.

[0168] The aforementioned multi-threaded scheduling mechanism based on a dual-queue pipeline connects three threads into a complete asynchronous data processing link through frame_queue and result_queue. It achieves thread-safe end-to-end data flow without introducing complex synchronization primitives, and has the following beneficial effects.

[0169] Firstly, the asynchronous concurrent model preloading mechanism loads 10 models concurrently during the startup phase, eliminating startup delays caused by sequential loading.

[0170] Secondly, the producer-consumer scheduling mechanism based on a dual-threaded safe queue completely decouples the three threads of acquisition, inference, and feedback, ensuring that they do not block each other.

[0171] Thirdly, 75 The overlapping rate sliding window extraction strategy ensures that short-term artifacts are not truncated by the window boundaries and that each type of artifact covers at least four consecutive windows.

[0172] Fourth, the 10-model concurrent inference mechanism allows each model to perform inference independently, with a total latency of only 21.71ms.

[0173] Fifth, a circular decision queue of length 5 is combined with a dual-threshold temporal coherence decision strategy, using a probability threshold of 0.75 and a cumulative frame count threshold of 4 frames to filter out random noise misjudgments, effectively suppressing the false alarm rate while ensuring detection sensitivity.

[0174] In the EEG artifact detection method based on multimodal sensor fusion provided in the above embodiments of this disclosure, EEG signals and IMU signals are acquired by means of multimodal sensor fusion, and accurate artifact detection results can be obtained in real time by using a multi-model collaborative detection architecture.

[0175] Figure 6 This is a schematic diagram of the structure of an EEG artifact detection device based on multimodal sensor fusion according to an embodiment of this disclosure.

[0176] like Figure 6 As shown, the EEG artifact detection device 60 can be implemented in the form of a general computing device. The EEG artifact detection device 60 includes a memory 61, a processor 62, and a bus 63 connecting different system components.

[0177] The memory 61 may include, for example, system memory, non-volatile storage media, etc. System memory may store, for example, an operating system, application programs, a boot loader, and other programs. System memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. Non-volatile storage media may store, for example, instructions for a corresponding embodiment of at least one EEG artifact detection method being executed. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0178] The processor 62 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete hardware components such as discrete gates or transistors.

[0179] For example, processor 62 is configured for memory-based instruction execution implementation such as Figure 1 , 5 The method involved in any of the embodiments.

[0180] Bus 63 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0181] The interfaces 64, 65, and 66 of the EEG artifact detection device 60, as well as the memory 61 and processor 62, can be connected via bus 63. Input / output interface 64 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. Network interface 65 provides a connection interface for various networked devices. Storage interface 66 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0182] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0183] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0184] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0185] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0186] This disclosure also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figure 1 , 5 The method involved in any of the embodiments.

[0187] This disclosure also provides a computer program product, including computer instructions, wherein the computer instructions, when executed by a processor, implement as follows: Figure 1 , 5 The method involved in any of the embodiments.

[0188] Figure 7 This is a schematic diagram of the structure of an EEG artifact detection system according to an embodiment of the present disclosure.

[0189] like Figure 7 As shown, the EEG artifact detection system includes an EEG artifact detection device 71 and a head-mounted device 72. The EEG artifact detection device 71 is... Figure 6 The EEG artifact detection device involved in any of the embodiments.

[0190] The head-mounted device 72 includes an EEG signal sensor for real-time acquisition of raw EEG signals and an IMU for real-time acquisition of IMU signals.

[0191] In some embodiments, the EEG signal sensor includes multiple electrodes, and the raw EEG signal includes multi-channel EEG signals corresponding one-to-one with the multiple electrodes.

[0192] For example, such as Figure 2As shown, the multiple electrodes include a first electrode AF7 corresponding to the left forehead of the user wearing the head-mounted device, a second electrode AF8 corresponding to the right forehead of the user, a third electrode TP9 corresponding to the left temporal region of the user, and a fourth electrode TP10 corresponding to the right temporal region of the user.

[0193] For example, head-mounted device 72 includes a headband or other devices for acquiring EEG signals.

[0194] In some embodiments, the IMU includes an accelerometer and a gyroscope. Accordingly, the IMU signal includes acceleration and angular velocity signals of the user's head wearing the headset.

[0195] For example, the acceleration signal is a three-dimensional acceleration signal with a sampling rate of 52Hz. The angular velocity signal is a three-dimensional angular velocity signal with a sampling rate of 52Hz.

[0196] Figure 8 This is a schematic diagram of the detection process of an EEG artifact detection system according to an embodiment of this disclosure. Figure 8 As shown, the EEG artifact detection system can be divided into an acquisition layer, a preprocessing layer, a parallel inference layer, and an output layer.

[0197] At the acquisition layer, raw EEG signals are acquired in real time using the EEG signal sensor in the headband, and IMU signals are acquired in real time using the IMU in the headband.

[0198] For example, the EEG signal sensor includes a first electrode AF7 corresponding to the left forehead of the user wearing the headband, a second electrode AF8 corresponding to the right forehead of the user, a third electrode TP9 corresponding to the left temporal region of the user, and a fourth electrode TP10 corresponding to the right temporal region of the user.

[0199] In this case, the raw EEG signal includes a 4-channel EEG signal corresponding to the first electrode AF7, the second electrode AF8, the third electrode TP9, and the fourth electrode TP10.

[0200] For example, an IMU includes an accelerometer and a gyroscope. Accordingly, the IMU signal includes acceleration and angular velocity signals from the user's head while wearing the headset.

[0201] Next, the acquisition layer uses the OSC protocol to send the raw EEG signals and IMU signals acquired in real time to the preprocessing layer.

[0202] In the preprocessing layer, the raw EEG signal is preprocessed to obtain the EEG signal to be processed, and the IMU signal is aligned and concatenated with the EEG signal to obtain the concatenated information. Next, the preprocessing layer sends the EEG signal to be processed and the concatenated information to the parallel inference layer.

[0203] For example, preprocessing includes windowing, baseline correction, high-pass filtering, notch filtering, normalization, and other processes.

[0204] In the parallel inference layer, a first model set and a second model set are set up. The first model set includes: model M1 for detecting resting state artifacts, model M2 for detecting blinking artifacts, model M3 for detecting speaking artifacts, model M4 for detecting chewing artifacts, model M5 for detecting mains interference artifacts, model M6 for detecting eye movement artifacts, and model M7 for detecting eyebrow movement artifacts (e.g., frowning, raising eyebrows). The second model set includes: model M8 for detecting head movement artifacts (e.g., nodding, shaking head, walking artifacts), model M9 for detecting head-mounted device reversed-wear artifacts, and model M10 for detecting head-mounted device unwearing artifacts.

[0205] It should be noted that the input of models M1, M2, M3, M4, M5, M6, and M7 in the first model set is the EEG signal to be processed. However, the input of models M8, M9, and M10 in the second model set is splicing information; that is, the input of models M8, M9, and M10 is the "EEG signal to be processed". IMU signal".

[0206] Next, models M1-M10 send their respective artifact detection results to the output layer.

[0207] In the output layer, the artifact detection results of each model are displayed in real time and saved to generate corresponding quality control reports.

[0208] Furthermore, if a model detects a corresponding artifact across multiple consecutive windows, it will provide the user with appropriate prompts. For example, if model M2 detects a blinking artifact for five consecutive time windows (750ms in total), it will prompt the user to reduce the frequency of blinking, frowning, and raising eyebrows. If model M8 detects a movement artifact for five consecutive time windows, it will prompt the user to keep their head still. If model M5 detects mains interference for five consecutive time windows, it will immediately prompt the user to remove their hands from powered devices (such as the metal casing of a laptop). If model M9 detects the headband being worn backwards for five consecutive time windows, it will prompt the user to readjust the headband's orientation. If model M10 detects the headband being worn without support for five consecutive time windows, it will prompt the user to readjust the headband's position.

[0209] By implementing the above embodiments of this disclosure, the following beneficial effects can be obtained.

[0210] 1. High-precision cross-subject artifact detection: This disclosure achieves AUC accuracy of 100% for various artifact detection models by collecting standardized datasets. 90 The high level of this technology provides a reliable guarantee for the quality assessment of EEG data.

[0211] 2. Full coverage of artifact types: This disclosure is the first to achieve full coverage detection of multiple types of mobile EEG artifacts (including resting state, blinking, chewing, eye movement, eyebrow movement, mains interference, speaking, head movement, headband worn backwards, and headband not worn) in a single system, which greatly expands the application scope of existing solutions.

[0212] 3. Ultra-low latency real-time processing: The system proposed in this disclosure achieves real-time artifact detection with a time resolution of 1 second, and the total latency of model inference is 21.71ms, which meets the timeliness requirements of real-time quality control.

[0213] 4. Efficient deployment: Due to the small size of a single model file With a capacity of 1MB, multiple models (e.g., 10 models) can be deployed and used simultaneously on the computer without affecting the normal use of the device.

[0214] 5. Effective multimodal information fusion: After introducing IMU sensor data, the detection accuracy of motion artifacts (including movement / stationary) and abnormal wearing status (wearing backwards / wearing without) is significantly improved compared with the pure EEG solution, reducing missed detections caused by insufficient information in a single modality.

[0215] 6. Closed-loop quality control ecosystem: The real-time feedback mechanism enables non-professional users to obtain intuitive data quality guidance during the collection process, effectively reducing the generation of avoidable artifacts, forming a positive feedback loop for optimizing collection behavior, and systematically improving the overall dataset quality.

[0216] In some embodiments, the functional units described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof for performing the functions described herein.

[0217] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0218] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for detecting EEG artifacts based on multimodal sensor fusion, executed by an EEG artifact detection device based on multimodal sensor fusion, comprising: Using the first thread, the raw EEG signal collected in real time by the EEG signal sensor in the head-mounted device, and the IMU signal collected in real time by the inertial measurement unit (IMU) in the head-mounted device are obtained. Using a second thread decoupled from the first thread, the original EEG signal is preprocessed to obtain the EEG signal to be processed, and the IMU signal is aligned and spliced ​​with the EEG signal to be processed to obtain splicing information. Using the second thread, the EEG signal to be processed is input into each model in the first model set, so that each model in the first model set outputs the corresponding artifact detection result; Using the second thread, the splicing information is input into each model in the second model set so that each model in the second model set outputs the corresponding artifact detection result, wherein the artifact types detected by each model in the first model set and the second model set are different from each other.

2. The method for detecting EEG artifacts according to claim 1, wherein, The EEG signal sensor includes multiple electrodes; The original EEG signal includes multi-channel EEG signals that correspond one-to-one with multiple electrodes.

3. The method for detecting EEG artifacts according to claim 2, wherein, The preprocessing of the raw EEG signal includes: The original EEG signal is then windowed. Baseline correction is performed on each channel of EEG signal within each time window to obtain a multi-channel corrected signal; The multi-channel correction signal is filtered to obtain the multi-channel filtered signal. The multi-channel filtered signal is normalized to obtain the EEG signal to be processed.

4. The method for detecting EEG artifacts according to claim 3, wherein, The filtering process for the multi-channel correction signal includes: Each channel of the multi-channel correction signal is subjected to high-pass filtering to remove low-frequency baseline drift, resulting in a multi-channel intermediate filter signal. The intermediate filter signals of each channel in the multi-channel intermediate filter signal are subjected to power frequency notch filtering for suppressing power frequency interference to obtain the multi-channel filter signal.

5. The method for detecting EEG artifacts according to claim 3, wherein, The baseline correction of each channel's EEG signal within each time window includes: Within each time window, the average value of the EEG signal of the i-th channel within each time window is subtracted from the EEG signal of the i-th channel to obtain the correction signal of the i-th channel. N is the total number of channels.

6. The method for detecting EEG artifacts according to claim 2, wherein, The plurality of electrodes includes a first electrode AF7 corresponding to the left forehead of the user wearing the head-mounted device, a second electrode AF8 corresponding to the right forehead of the user, a third electrode TP9 corresponding to the left temporal region of the user, and a fourth electrode TP10 corresponding to the right temporal region of the user.

7. The method for detecting EEG artifacts according to claim 1, wherein, The first set of models includes: a model for detecting resting artifacts, a model for detecting blinking artifacts, a model for detecting speaking artifacts, a model for detecting chewing artifacts, a model for detecting mains interference artifacts, a model for detecting eye movement artifacts, and a model for detecting eyebrow movement artifacts. The second set of models includes: a model for detecting head motion artifacts, a model for detecting head-mounted device reverse artifacts, and a model for detecting head-mounted device unmounted artifacts.

8. The method for detecting EEG artifacts according to claim 1, wherein, All models in the first model set and the second model set are loaded into memory synchronously.

9. The method for detecting EEG artifacts according to claim 1, wherein, Each model in the first model set and the second model set includes a depthwise separable convolutional (DSC) model, an EEGNet model, a shallow convolutional network (ShallowConvNet) model, a MobileNet model, or a gradient boosting decision tree (GBDT) model.

10. The method for detecting EEG artifacts according to claim 1, wherein, The IMU includes an accelerometer and a gyroscope; The IMU signal includes acceleration and angular velocity signals of the user's head while wearing the headset.

11. The method for detecting EEG artifacts according to claim 1, wherein, The head-mounted device includes a headband.

12. The method for detecting EEG artifacts according to claim 1, further comprising: Using the first thread, the original EEG signal and the IMU signal are written into a circular buffer; Using the second thread, the raw EEG signal and the IMU signal are read from the circular buffer.

13. The method for detecting EEG artifacts according to any one of claims 1-12, further comprising: Using a third thread decoupled from the first and second threads, the artifact detection results of each model in the first and second model sets are visualized.

14. The EEG artifact detection method according to claim 13 further includes: Using the third thread, determine whether the m-th model detects the corresponding artifact. M is the total number of models in the first model set and the second model set; If the m-th model detects the corresponding artifact, the third thread will be used to send a corresponding prompt message to the user wearing the head-mounted device.

15. The method for detecting EEG artifacts according to claim 14, wherein, The determination of whether the m-th model detects the corresponding artifact includes: Determine whether the m-th model detects the corresponding artifact within multiple consecutive time windows; If the m-th model detects the corresponding artifact within the consecutive multiple time windows, then the m-th model is determined to have detected the corresponding artifact.

16. A brainwave artifact detection device based on multimodal sensor fusion, comprising: Memory; A processor, coupled to a memory, configured to implement the method as described in any one of claims 1-15 based on memory-stored instruction execution.

17. A brainwave artifact detection system, comprising: The EEG artifact detection device as described in claim 16; The head-mounted device includes an EEG signal sensor for real-time acquisition of raw EEG signals and an IMU for real-time acquisition of inertial measurement unit (IMU) signals.

18. The EEG artifact detection system according to claim 17, wherein, The EEG signal sensor includes multiple electrodes; The original EEG signal includes multi-channel EEG signals that correspond one-to-one with multiple electrodes.

19. The EEG artifact detection system according to claim 18, wherein, The plurality of electrodes includes a first electrode AF7 corresponding to the left forehead of the user wearing the head-mounted device, a second electrode AF8 corresponding to the right forehead of the user, a third electrode TP9 corresponding to the left temporal region of the user, and a fourth electrode TP10 corresponding to the right temporal region of the user.

20. The EEG artifact detection system according to claim 17, wherein, The IMU includes an accelerometer and a gyroscope; The IMU signal includes acceleration and angular velocity signals of the user's head while wearing the headset.

21. A computer-readable storage medium, wherein, A computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-15.

22. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any one of claims 1-15.

Images