A multimodal physiological signal data quality detection method, system and medium

Abstract

A multi-modal physiological signal data quality detection method, system and medium, the method comprising receiving multi-channel electroencephalogram data stream and multi-axis inertial measurement unit data stream and buffering; with a fixed processing period, extract an electroencephalogram sample sequence of a time window from the buffered electroencephalogram data stream, and synchronously extract an inertial measurement sample sequence aligned with the electroencephalogram time window to form a multi-modal data window; data cleaning and standardization processing is carried out on the sample sequence in the multi-modal data window to obtain normalized multi-modal input data; the multi-modal input data is distributed to the corresponding lightweight classification model for parallel forward inference to obtain the probability distribution vector output by each lightweight classification model; compare each artifact class probability in each probability distribution vector with the pre-set confidence threshold, if any artifact class probability is greater than the pre-set confidence threshold, generate the corresponding feedback instruction.

Description

Technical Field

[0001] This invention relates to the field of electroencephalogram (EEG) signal processing technology, and in particular to a method, system, and medium for detecting the quality of multimodal physiological signal data. Background Technology

[0002] When collecting physiological signals (especially EEG signals) using portable devices (such as head-mounted EEG monitors), data quality is highly susceptible to interference from various artifacts. These artifacts mainly include physiological artifacts (such as blinking, eye movements, chewing, and electromyographic activity) and non-physiological artifacts (such as motion artifacts caused by head movements, mains power frequency interference, and poor equipment contact). These interfering signals intertwine with the target neurophysiological signals in both the time and frequency domains, significantly reducing the signal-to-noise ratio and directly impacting the reliability and accuracy of subsequent analyses, such as in applications like neurofeedback, brain-computer interfaces, or cognitive state assessment.

[0003] Currently, the industry mainly relies on the following two methods to deal with physiological signal artifacts: 1. Expert Manual Identification and Processing: Professionals manually mark and remove or correct artifacts by observing the original signal waveform or spectrogram. While this method offers some flexibility, it suffers from significant drawbacks, including low efficiency, high time and labor costs, strong subjectivity, and difficulty in large-scale application. More importantly, it is a typical "post-hoc analysis," failing to provide any quality feedback or guidance during real-time data acquisition. This can potentially cause the entire acquisition session to fail due to poor quality, wasting the time of both subjects and researchers.

[0004] 2. Offline Computational Detection Algorithms: Existing technologies include various signal processing-based artifact detection algorithms, such as Independent Component Analysis (ICA), wavelet transform, and threshold detection. However, these methods typically have high computational complexity, require multi-channel data, and assume statistical independence between artifacts and neural signals. Their effectiveness is limited in low-channel-count single-analysis scenarios, they require powerful computing platforms (such as desktop workstations or servers), and have long processing times, making them difficult to meet real-time requirements. Although some research attempts to migrate computational tasks to the cloud to reduce local load, this introduces risks such as network latency, dependence on transmission stability, and leakage of user physiological data privacy.

[0005] Furthermore, with the widespread application of mobile EEG technology, achieving high-precision, low-latency real-time artifact detection on resource-constrained mobile terminals (such as smartphones and tablets) has become a significant technical challenge.

[0006] In summary, existing technologies lack a physiological signal data quality detection method that can be deployed on mobile terminals to achieve real-time automated multimodal physiological signal quality detection and immediate feedback. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a method for detecting the quality of multimodal physiological signal data, comprising the following steps: It receives and buffers multi-channel EEG signal data streams and multi-axis inertial measurement unit data streams; With a fixed processing cycle, a sequence of EEG signal samples for a time window is extracted from the cached EEG signal data stream, and an inertial measurement sample sequence aligned with the time window of the EEG signal is extracted simultaneously to form a multimodal data window. The sample sequences in the multimodal data window are cleaned and standardized to obtain normalized multimodal input data; Based on the types of artifacts targeted by each of the multiple pre-trained lightweight classification models and their required input modalities, the multimodal input data is distributed to the corresponding lightweight classification models for parallel forward inference, resulting in a probability distribution vector output by each lightweight classification model. The probability distribution vector is used to characterize the probability that the current data window belongs to one or more preset artifact categories corresponding to the lightweight classification model. The probability of each artifact category in each probability distribution vector is compared with a preset confidence threshold. If the probability of any artifact category is greater than the preset confidence threshold, a corresponding feedback instruction is generated. The feedback instruction is used to prompt the user to correct the artifact or mark the corresponding data quality.

[0008] Optionally, the multimodal input data includes electroencephalogram (EEG) signal data, accelerometer signal data, and gyroscope signal data; Multiple pre-trained lightweight classification models include: at least one classification model that uses the EEG signal data as input data to identify preset physiological artifacts; at least one motion recognition model that uses the accelerometer signal data and gyroscope signal data as input data to identify preset motion tasks; and at least one multimodal feature fusion classification model that uses the EEG signal data, accelerometer signal data and gyroscope signal data as common input data to identify composite artifacts that depend on motion information.

[0009] Optionally, the reasoning process between the classification model and the motion recognition model includes: A one-dimensional convolutional network is used to learn and extract temporal feature representations of the input data, and a globalization layer is used to output a fixed-length feature vector. The feature vector is input into the classifier to obtain the corresponding probability distribution vector.

[0010] Optionally, the inference process of the multimodal feature fusion classification model includes: Temporal feature extraction was performed on the EEG signal data, accelerometer signal data, and gyroscope signal data respectively to obtain the feature vectors corresponding to each mode. The feature vectors corresponding to each modality are concatenated to form a fused feature vector; The fused feature vector is input into the classifier to obtain the corresponding probability distribution vector.

[0011] Optionally, the data cleaning includes: detecting whether the sample sequence in the multimodal data window contains one or more of the following: non-numerical data points, zero-value anomaly data points, infinite data points, and amplitude anomaly data points; if non-numerical data points exist, replacing them with the linear average of the preceding and following valid values; if zero-value anomaly data points exist, marking them as invalid data points and removing them; if positive or negative infinity data points exist, replacing them with the average of adjacent valid data points, or removing them directly; if amplitude anomaly data points exist, marking the artifact intervals based on dynamic thresholds or statistical methods.

[0012] Optionally, the standardization process specifically involves: normalizing the cleaned sample sequence to a preset numerical range according to a preset linear scaling formula; the maximum and minimum values ​​of the preset numerical range are respectively the global maximum and global minimum parameters obtained from the training dataset of the lightweight classification model.

[0013] Optionally, the steps for generating the corresponding feedback instructions include: Based on the probability of the artifact category that is greater than the preset confidence threshold, the corresponding artifact category is determined, and the corresponding text guidance information is retrieved from the preset correction suggestion mapping table based on it. Based on the textual guidance information and the artifact category, a corresponding feedback instruction is generated.

[0014] Optionally, the preset artifact categories covered by multiple pre-trained lightweight classification models include at least one category for judging signal quality, physiological artifacts, judging device wearing status, and external interference.

[0015] Corresponding to the aforementioned multimodal physiological signal data quality detection method, the present invention provides a multimodal physiological signal data quality detection system, comprising: The data receiving and buffering module is used to receive and buffer multi-channel EEG signal data streams and multi-axis inertial measurement unit data streams. The data window construction module is used to extract a time window of EEG signal sample sequence from the cached EEG signal data stream at a fixed processing cycle, and simultaneously extract an inertial measurement sample sequence that is time-aligned with the EEG signal time window to form a multimodal data window. The data preprocessing module is used to perform data cleaning and standardization on the sample sequences in the multimodal data window to obtain normalized multimodal input data. The parallel forward inference module is used to distribute the multimodal input data to the corresponding lightweight classification model for parallel forward inference based on the type of artifact targeted by each of the multiple pre-trained lightweight classification models and the required input modality, so as to obtain the probability distribution vector output by each lightweight classification model. The probability distribution vector is used to characterize the probability that the current data window belongs to one or more preset artifact categories corresponding to the lightweight classification model. The decision and feedback module is used to compare the probability of each artifact category in each probability distribution vector with a preset confidence threshold. If the probability of any artifact category is greater than the preset confidence threshold, a corresponding feedback instruction is generated. The feedback instruction is used to prompt the user to correct the artifact or mark the corresponding data quality.

[0016] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a multimodal physiological signal data quality detection program, wherein the multimodal physiological signal data quality detection program, when executed by a processor, implements the steps of the multimodal physiological signal data quality detection method as described above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention realizes the complete automation and real-time detection of physiological signal artifacts, fundamentally solving the problem of low efficiency caused by the reliance on manual labor in existing technologies; by receiving and caching multi-channel EEG signal data streams and multi-axis inertial measurement unit data streams, and automatically extracting time window data in combination with a fixed processing cycle, it replaces the process of manually browsing data; by performing parallel forward inference on multimodal input data through a pre-trained lightweight classification model, and automatically generating feedback instructions based on confidence thresholds, it realizes intelligent artifact detection and feedback, without the need for professional personnel intervention, and significantly improves detection efficiency and objectivity; through the mechanism of real-time data stream processing, parallel forward inference and instant decision feedback, it transforms the traditional post-analysis into online quality control during the acquisition process, enabling users to immediately obtain artifact prompts and adjust wearing methods or behaviors during the acquisition session, ensuring data quality from the source and overcoming the defect that offline analysis cannot provide real-time guidance.

[0018] (2) The multimodal input data includes three types of sensor data: EEG signals, accelerometer signals and gyroscope signals. The system is configured with three architectures: classification model, motion recognition model and multimodal fusion model. This enables the system to select the most relevant modal information for targeted detection based on the physical origin of different artifacts, avoiding false detection or missed detection caused by the limitation of single modal information, and improving the accuracy and reliability of artifact classification in complex scenarios.

[0019] (3) The classification model and motion recognition model adopt a standardized two-stage processing flow of “first extracting time-domain features and then classifying the decision” to ensure the consistency and reproducibility of different artifact detection at the feature level. At the same time, time-domain feature extraction can effectively capture the local time dynamic characteristics of the signal, providing discriminative feature representations for the subsequent classifier, thereby ensuring the stability of model inference.

[0020] (4) By independently extracting time-domain features from the EEG signal data, accelerometer signal data and gyroscope signal data, and then concatenating the feature vectors of each modality to form a fusion feature vector for classification, this architecture retains the unique information of each sensing modality while realizing cross-modal information fusion at the feature layer, enabling the multimodal feature fusion classification model to learn the correlation between motion information and EEG artifacts, effectively improving the sensitivity of recognizing composite artifacts that depend on motion state.

[0021] (5) By performing multi-dimensional detection of non-numerical points, infinite points and amplitude anomalies on the sample sequence, and by adopting a differentiated processing strategy of replacing non-numerical points with linear interpolation, eliminating and marking pseudo-trace intervals, it is possible to effectively remove or identify various abnormal data without destroying the continuity of the data window, significantly reduce the interference of outliers on model inference, and enhance the robustness and fault tolerance of the system in the actual acquisition environment.

[0022] (6) By adopting a fixed linear scaling formula based on the global maximum and minimum parameters of the training dataset for standardization, the numerical distribution of the real-time collected data is kept strictly consistent with the data distribution during model training, eliminating the distribution drift problem caused by differences in the collection equipment or environmental changes, and ensuring that the model can obtain stable and accurate inference results in different usage scenarios.

[0023] (7) By establishing a feedback instruction generation process of “probability of artifact category → determination of artifact category → retrieval of mapping table → generation of text guidance information”, the abstract probability value output by the lightweight classification model is transformed into text information that users can understand and that has clear operation suggestions, making the feedback content more targeted and executable, and helping non-professional users to quickly understand the problem and take the correct corrective measures.

[0024] (8) The preset artifact categories cover four typical scenarios: signal quality assessment, physiological artifact recognition, device wearing status detection, and external interference judgment. It has the ability to fully perceive potential problems in the entire EEG data acquisition chain, and ensures that multiple artifact sources are monitored simultaneously in a single detection cycle, thereby improving the integrity and practicality of the quality control system. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a simplified flowchart of an embodiment of the multimodal physiological signal data quality detection method of the present invention; Figure 2 This is a framework diagram of an embodiment of the multimodal physiological signal data quality detection system of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, the present invention provides a method for detecting the quality of multimodal physiological signal data, which includes the following steps: It receives and buffers multi-channel EEG signal data streams and multi-axis inertial measurement unit data streams; With a fixed processing cycle, a sequence of EEG signal samples within a time window is extracted from the cached EEG signal data stream, and an inertial measurement sample sequence aligned with the time of the EEG signal time window is extracted simultaneously to form a multimodal data window; specifically, based on the end time or start time of the EEG signal time window, the inertial measurement sample sequence with the closest timestamp is extracted simultaneously to form a multimodal data window. Data cleaning and standardization are performed on the sample sequences in the multimodal data window to obtain normalized multimodal input data; Based on the types of artifacts targeted by each of the multiple pre-trained lightweight classification models and the required input modalities, the multimodal input data is distributed to the corresponding lightweight classification models for parallel forward inference, resulting in the probability distribution vector output by each lightweight classification model. The probability distribution vector is used to characterize the probability that the current data window belongs to one or more preset artifact categories corresponding to the lightweight classification model. The probability of each artifact category in each probability distribution vector is compared with a preset confidence threshold. If the probability of any artifact category is greater than the preset confidence threshold, a corresponding feedback instruction is generated. The feedback instruction is used to prompt the user to correct the artifact or mark the corresponding data quality.

[0028] This invention achieves complete automation and real-time detection of physiological signal artifacts, fundamentally solving the inefficiency problem caused by reliance on manual labor in existing technologies. By receiving and caching multi-channel EEG signal data streams and multi-axis inertial measurement unit data streams, and automatically extracting time window data in conjunction with a fixed processing cycle, it replaces the process of manually browsing data. Through a pre-trained lightweight classification model, it performs parallel forward inference on multimodal input data and automatically generates feedback instructions based on confidence thresholds, realizing intelligent artifact detection and feedback without the need for professional personnel intervention, significantly improving detection efficiency and objectivity. Through the mechanism of real-time data stream processing, parallel forward inference, and instant decision feedback, it transforms traditional post-event analysis into online quality control during the acquisition process, enabling users to immediately receive artifact prompts and adjust their wearing methods or behaviors during the acquisition session, ensuring data quality from the source and overcoming the shortcomings of offline analysis that cannot provide real-time guidance.

[0029] In this embodiment, inference is performed at a fixed temporal resolution (preferably 1 second). When the EEG signal buffer accumulates 256 samples (corresponding to 1 second of data, assuming a sampling rate of 256Hz), the system triggers a processing cycle. Simultaneously, the system extracts the 52 consecutive samples (corresponding to approximately 1 second of data, assuming an IMU sampling rate of 52Hz) closest to the end of the current EEG signal time window from the accelerometer and gyroscope buffers, respectively. This process ensures the temporal alignment of the multimodal data, laying the foundation for subsequent fusion analysis.

[0030] In this embodiment, the multimodal input data includes electroencephalogram (EEG) signal data, accelerometer signal data, and gyroscope signal data; Multiple pre-trained lightweight classification models include: at least one classification model that uses EEG signal data as input data to identify preset physiological artifacts; at least one motion recognition model that uses accelerometer signal data and gyroscope signal data as input data to identify preset motion tasks; and at least one multimodal feature fusion classification model that uses EEG signal data, accelerometer signal data and gyroscope signal data as common input data to identify composite artifacts that depend on motion information.

[0031] Preferably, preset physiological artifacts such as blinking, chewing, etc.; composite artifacts such as nodding, shaking, wearing the device backwards, etc.; preset motor tasks such as nodding, shaking, etc.

[0032] In this embodiment, the training process of the lightweight classification model is roughly as follows: The training set utilizes a large number of precisely annotated 1-second time-window physiological signal data samples, employing time-series data augmentation techniques (such as random small-scale scaling, noise addition, and window sliding) to enhance model generalization ability. Class weights or focus loss functions are used to address the issue of imbalanced sample sizes across different artifact categories. Early stopping is employed to prevent overfitting. During training, a self-collected dataset of over 30 individuals with different artifacts and long-term, multi-point repetitions is used. A self-developed model architecture is applied to achieve the most simplified model design for both time-series (EEG) and spatial pose data (IMU) features. Hyperparameter optimization and parallel training result in a lightweight model that balances generalization and accuracy. Finally, the trained model is quantized, optimized, and converted to TFLite format for mobile integration.

[0033] The multimodal input data of this invention includes three types of sensor data: electroencephalogram (EEG) signals, accelerometer signals, and gyroscope signals. It is configured with three architectures: a classification model, a motion recognition model, and a multimodal fusion model. This enables the system to select the most relevant modal information for targeted detection based on the physical origin of different artifacts, avoiding false detections or missed detections caused by the limitations of single modal information, and improving the accuracy and reliability of artifact classification in complex scenarios.

[0034] In this embodiment, the reasoning process of the classification model and the motion recognition model includes: A one-dimensional convolutional network is used to learn and extract temporal feature representations of the input data, and a globalization layer is used to output a fixed-length feature vector. The feature vector is input into the classifier to obtain the corresponding probability distribution vector.

[0035] The classification model and motion recognition model of this invention adopt a standardized two-stage processing flow of "first extracting time-domain features, then classifying and deciding" to ensure the consistency and reproducibility of different artifact detection at the feature level. At the same time, time-domain feature extraction can effectively capture the local time dynamic characteristics of the signal, providing discriminative feature representations for the subsequent classifier, thereby ensuring the stability of model inference.

[0036] In this embodiment, the inference process of the multimodal feature fusion classification model includes: Temporal features were extracted from EEG signal data, accelerometer signal data, and gyroscope signal data respectively to obtain feature vectors corresponding to each mode. The feature vectors corresponding to each modality are concatenated to form a fused feature vector; The fused feature vector is input into the classifier to obtain the corresponding probability distribution vector.

[0037] This invention extracts temporal features independently from the EEG signal data, accelerometer signal data, and gyroscope signal data, and then concatenates the feature vectors of each modality to form a fused feature vector for classification. This architecture preserves the unique information of each sensing modality while achieving cross-modal information fusion at the feature layer. This enables the multimodal feature fusion classification model to learn the correlation between motion information and EEG artifacts, effectively improving the sensitivity of recognizing composite artifacts that depend on motion state.

[0038] Preferably, the lightweight classification model is a pre-trained model file deployed on a mobile terminal operating system (such as Android or iOS), and its format is TFLite. Its internal structure is based on a one-dimensional convolutional neural network and specifically includes: Input layer: Receives tensors of the form [C,T], where C is the number of physiological signal channels and T is the number of sampling points within a 1-second time window.

[0039] Feature extraction module: Consists of at least one one-dimensional convolutional layer. Each one-dimensional convolutional layer uses multiple convolutional kernels of predetermined length to perform sliding convolution operations along the time dimension T to extract local temporal features from the signal.

[0040] Each convolutional layer is followed by a non-linear activation function (such as the ReLU function) for transformation.

[0041] The classification decision module is connected after the feature extraction module. It first reduces the dimensionality of the extracted features through a global pooling layer (such as global average pooling) to obtain a fixed-length feature vector; then, the feature vector undergoes a nonlinear transformation through one or more fully connected layers; finally, a probability distribution vector is output through a Softmax classifier.

[0042] In this embodiment, data cleaning includes: detecting whether the sample sequence in the multimodal data window contains one or more of the following: non-numerical data points, zero-value anomalous data points, infinite data points, and amplitude anomalous data points; if non-numerical data points or infinite data points exist, the non-numerical data points are replaced with the linear average of the previous valid value and the next valid value of the non-numerical data point or infinite data point; if zero-value anomalous data points exist, they are marked as invalid data points and removed; if positive infinity data points or negative infinity data points exist, they are replaced with the average of adjacent valid data points, or directly removed; if amplitude anomalous data points exist, such as EEG signals exceeding ±500μV (outside the reasonable range of physiological signals), they are cleaned based on dynamic thresholds or statistical methods (e.g., 3...). (Principle) Mark the pseudo-trace intervals.

[0043] Preferably, zero-value outliers refer to a sequence of zero values ​​with more than three consecutive sampling points. Based on 3 The principle for marking artifacts is as follows: artifacts exceeding μ±3 are specifically marked. The data is considered an artifact. Specifically, the mean μ and standard deviation σ of the sample sequence are first calculated; the normal data range is set to [μ-3]. μ+3 Data points outside this range are marked as artifacts.

[0044] This invention performs multi-dimensional detection of non-numerical points, infinity points, and amplitude anomalies in sample sequences, and employs linear interpolation to replace non-numerical points, as well as strategies for eliminating and marking artifact intervals for differentiated processing. This effectively removes or identifies various types of abnormal data without disrupting the continuity of the data window, significantly reduces the interference of outliers on model inference, and enhances the robustness and fault tolerance of the system in actual data acquisition environments.

[0045] In this embodiment, the standardization process is specifically as follows: the sample sequence after data cleaning is normalized to a preset numerical range according to a preset linear scaling formula; the maximum and minimum values ​​of the preset numerical range are the global maximum and global minimum parameters obtained from the training dataset of the lightweight classification model, respectively.

[0046] Preferably, the preset numerical range is [-1.0, 1.0], and the preset linear scaling formula is as follows: ; in, The first part represents the standardized sample sequence; the second part represents the cleaned sample sequence. The global minimum parameter (e.g., -250μV) is obtained statistically from the training dataset of the lightweight classification model. This represents the global maximum parameter (e.g., +250μV) obtained statistically from the training dataset of the lightweight classification model. The global maximum and global minimum parameters are robust estimates obtained statistically from the training dataset after truncating extreme outliers (e.g., using percentiles).

[0047] This invention standardizes the data by using a fixed linear scaling formula based on the global maximum and minimum parameters of the training dataset. This ensures that the numerical distribution of the real-time collected data is strictly consistent with the data distribution during model training, eliminating the distribution drift problem caused by differences in collection equipment or environmental changes. This ensures that the model can obtain stable and accurate inference results in different usage scenarios.

[0048] In this embodiment, the step of generating the corresponding feedback instruction includes: Based on the probability of the artifact category exceeding the preset confidence threshold, the corresponding artifact category is determined, and the corresponding text guidance information is retrieved from the preset correction suggestion mapping table based on it. Based on the textual guidance information and the type of artifact, generate corresponding feedback instructions.

[0049] In this embodiment, the preset confidence threshold is 0.7 to 0.9.

[0050] Preferably, the text guidance information in the correction suggestion mapping table can be set according to actual needs, such as: "Check power grounding, keep away from interference sources", "Remind the user to relax the jaw and avoid chewing", "Check electrode contact and ensure normal impedance", etc.

[0051] This invention establishes a feedback instruction generation process of "artifact category probability → artifact category determination → mapping table retrieval → text guidance information generation," which transforms the abstract probability values ​​output by the lightweight classification model into text information that users can understand and that provides clear operational suggestions. This makes the feedback content more targeted and actionable, helping non-professional users quickly understand the problem and take the correct corrective measures.

[0052] In this embodiment, the preset artifact categories covered by multiple pre-trained lightweight classification models include at least one category for judging signal quality, physiological artifacts, judging device wearing status, and external interference; the preset artifact categories specifically include whether the signal quality is clean, whether there is blinking, whether there is chewing, whether the eyeballs are moving, whether there is frowning, whether it is mains power, whether there is talking, whether there is movement, whether the signal acquisition device is reversed, and whether the signal acquisition device is empty.

[0053] This invention pre-classifies artifact categories to cover four typical scenarios: signal quality assessment, physiological artifact identification, device wearing status detection, and external interference judgment. It has the ability to comprehensively perceive potential problems in the entire EEG data acquisition chain, ensuring that multiple artifact sources are monitored simultaneously within a single detection cycle, thereby improving the integrity and practicality of the quality control system.

[0054] like Figure 2 As shown, the present invention also provides a multimodal physiological signal data quality detection system, which includes: The data receiving and buffering module 10 is used to receive and buffer multi-channel EEG signal data streams and multi-axis inertial measurement unit data streams; the multi-channel EEG signal data streams can be acquired through an EEG headband; the multi-axis inertial measurement unit data streams are acquired through a triaxial accelerometer and a triaxial gyroscope to collect motion acceleration and angular velocity data of the subject's head; The data window construction module 20 is used to extract a time window of EEG signal sample sequence from the cached EEG signal data stream at a fixed processing cycle, and simultaneously extract an inertial measurement sample sequence that is time-aligned with the EEG signal time window to form a multimodal data window. The data preprocessing module 30 is used to clean and standardize the sample sequences in the multimodal data window to obtain normalized multimodal input data. The parallel forward inference module 40 is used to distribute multimodal input data to the corresponding lightweight classification model for parallel forward inference based on the type of artifact targeted by each of the multiple pre-trained lightweight classification models and the required input modality, so as to obtain the probability distribution vector output by each lightweight classification model. The probability distribution vector is used to represent the probability that the current data window belongs to one or more preset artifact categories corresponding to the lightweight classification model. The decision and feedback module 50 is used to compare the probability of each artifact category in each probability distribution vector with a preset confidence threshold. If the probability of any artifact category is greater than the preset confidence threshold, a corresponding feedback instruction is generated. The feedback instruction is used to prompt the user to correct the artifact or mark the corresponding data quality.

[0055] Furthermore, the system of the present invention also includes a display module for displaying feedback instructions. For example, one or more of the following: superimposing a bright color to mark the artifact period on the signal waveform graph, displaying a text warning (such as "High-frequency blink detection in progress") in a prominent position on the screen, and emitting a short prompt sound.

[0056] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The method for detecting the quality of multimodal physiological signal data is shown. The computer-readable storage medium may be a read-only memory, a hard disk, or an optical disk, etc.

[0057] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments and storage medium embodiments, since they are basically similar to method embodiments, the descriptions are relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0058] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for detecting the quality of multimodal physiological signal data, characterized in that, Includes the following steps: It receives and buffers multi-channel EEG signal data streams and multi-axis inertial measurement unit data streams; With a fixed processing cycle, a sequence of EEG signal samples for a time window is extracted from the cached EEG signal data stream, and an inertial measurement sample sequence aligned with the time window of the EEG signal is extracted simultaneously to form a multimodal data window. The sample sequences in the multimodal data window are cleaned and standardized to obtain normalized multimodal input data; Based on the types of artifacts targeted by each of the multiple pre-trained lightweight classification models and their required input modalities, the multimodal input data is distributed to the corresponding lightweight classification models for parallel forward inference, resulting in a probability distribution vector output by each lightweight classification model. The probability distribution vector is used to characterize the probability that the current data window belongs to one or more preset artifact categories corresponding to the lightweight classification model. The probability of each artifact category in each probability distribution vector is compared with a preset confidence threshold. If the probability of any artifact category is greater than the preset confidence threshold, a corresponding feedback instruction is generated. The feedback instruction is used to prompt the user to correct the artifact or mark the corresponding data quality.

2. The method for detecting the quality of multimodal physiological signal data according to claim 1, characterized in that, The multimodal input data includes electroencephalogram (EEG) signal data, accelerometer signal data, and gyroscope signal data; Multiple pre-trained lightweight classification models include: at least one classification model that uses the EEG signal data as input data to identify preset physiological artifacts; At least one motion recognition model that uses the accelerometer signal data and gyroscope signal data as input data to identify a preset motion task; and at least one multimodal feature fusion classification model that uses the EEG signal data, accelerometer signal data and gyroscope signal data as common input data to identify composite artifacts that depend on motion information.

3. The method for detecting the quality of multimodal physiological signal data according to claim 2, characterized in that, The reasoning process of the classification model and the motion recognition model includes: A one-dimensional convolutional network is used to learn and extract temporal feature representations of the input data, and a globalization layer is used to output a fixed-length feature vector. The feature vector is input into the classifier to obtain the corresponding probability distribution vector.

4. The method for detecting the quality of multimodal physiological signal data according to claim 2, characterized in that, The reasoning process of the multimodal feature fusion classification model includes: Temporal feature extraction was performed on the EEG signal data, accelerometer signal data, and gyroscope signal data respectively to obtain the feature vectors corresponding to each mode. The feature vectors corresponding to each modality are concatenated to form a fused feature vector; The fused feature vector is input into the classifier to obtain the corresponding probability distribution vector.

5. The method for detecting the quality of multimodal physiological signal data according to claim 1, characterized in that, The data cleaning includes: detecting whether the sample sequence in the multimodal data window contains one or more of the following: non-numerical data points, zero-value anomaly data points, infinite data points, and amplitude anomaly data points; if non-numerical data points exist, they are replaced with the linear average of the preceding and following valid values; if zero-value anomaly data points exist, they are marked as invalid data points and removed; if positive or negative infinity data points exist, they are replaced with the average of adjacent valid data points, or directly removed; if amplitude anomaly data points exist, artifact intervals are marked based on dynamic thresholds or statistical methods.

6. The method for detecting the quality of multimodal physiological signal data according to claim 5, characterized in that, The standardization process specifically involves: normalizing the cleaned sample sequence to a preset numerical range according to a preset linear scaling formula; the maximum and minimum values ​​of the preset numerical range are the global maximum and global minimum parameters obtained from the training dataset of the lightweight classification model, respectively.

7. The method for detecting the quality of multimodal physiological signal data according to claim 1, characterized in that, The steps to generate the corresponding feedback instructions include: Based on the probability of the artifact category that is greater than the preset confidence threshold, the corresponding artifact category is determined, and the corresponding text guidance information is retrieved from the preset correction suggestion mapping table based on it. Based on the textual guidance information and the artifact category, a corresponding feedback instruction is generated.

8. The method for detecting the quality of multimodal physiological signal data according to claim 1, characterized in that, The preset artifact categories covered by multiple pre-trained lightweight classification models include at least one category for judging signal quality, physiological artifacts, judging device wearing status, and external interference.

9. A multimodal physiological signal data quality detection system, characterized in that, include: The data receiving and buffering module is used to receive and buffer multi-channel EEG signal data streams and multi-axis inertial measurement unit data streams. The data window construction module is used to extract a time window of EEG signal sample sequence from the cached EEG signal data stream at a fixed processing cycle, and simultaneously extract an inertial measurement sample sequence that is time-aligned with the EEG signal time window to form a multimodal data window. The data preprocessing module is used to perform data cleaning and standardization on the sample sequences in the multimodal data window to obtain normalized multimodal input data. The parallel forward inference module is used to distribute the multimodal input data to the corresponding lightweight classification model for parallel forward inference based on the type of artifact targeted by each of the multiple pre-trained lightweight classification models and the required input modality, so as to obtain the probability distribution vector output by each lightweight classification model. The probability distribution vector is used to characterize the probability that the current data window belongs to one or more preset artifact categories corresponding to the lightweight classification model. The decision and feedback module is used to compare the probability of each artifact category in each probability distribution vector with a preset confidence threshold. If the probability of any artifact category is greater than the preset confidence threshold, a corresponding feedback instruction is generated. The feedback instruction is used to prompt the user to correct the artifact or mark the corresponding data quality.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multimodal physiological signal data quality detection program, which, when executed by a processor, implements the steps of the multimodal physiological signal data quality detection method as described in any one of claims 1 to 8.

Images