Sleep stage classification method based on zero-shot learning and contrastive learning

Abstract

The present invention provides a sleep stage classification method based on zero-shot learning and contrastive learning, comprising the following steps: S1: acquiring and processing raw EEG, EMG, and EOG physiological signals; S2: acquiring semantic information related to sleep stage classification; S3: manually matching the physiological signals and semantic information; S4: data set partitioning: dividing the matched physiological signal and semantic information groups in S3 into a training set, a validation set, and a test set; S5: pre-training: inputting the training set obtained in S4 into a zero-shot learning and contrastive learning model for pre-training, and using contrastive learning to fine-tune the zero-shot learning and contrastive learning model; S6: predicting sleep stage classification. This invention addresses the problem of low sleep stage assessment accuracy in existing technologies.

Description

Technical Field

[0001] The present invention belongs to the technical field of signal processing, and in particular relates to a sleep stage classification method based on zero-sample learning and contrastive learning. Background Art

[0002] Sleep is one of the most important physiological processes in the human body. Most people spend a third of their lives sleeping, and sleep is closely linked to both physical and mental health. Modern medical research indicates that over 80 human diseases are closely linked to long-term sleep disorders. Effective diagnosis and treatment of sleep-related disorders have become a pressing and intensive research topic in current medicine. Many doctors and researchers have long debated how best to understand this issue. In recent years, sleep has become a distinct branch of medicine, and sleep has been found to play a significant role in seemingly unrelated clinical issues.

[0003] Sleep stage classification is the first step in diagnosing sleep-related disorders. Physiologically, sleep stages are mainly divided into rapid eye movement (REM) and non-rapid eye movement (NREM). The sleep process usually cycles between REM and NREM. NREM is divided into light sleep (stages N1 and N2) and deep sleep (stage N3). Polysomnography (PSG) is mainly used as the main tool for sleep assessment, including electroencephalography (EEG), electromyography (EMG), oculoculography (EOG), etc. EEG signals are the most commonly used signals for sleep stage scoring, because clinically accepted sleep stage assessment is mainly based on reading EEG signals according to the R&K standard, which was standardized by Rechtschaffen and Kales in 1968 and further developed by the American Academy of Sleep Medicine (AASM) in 2007.

[0004] Traditional sleep stage scoring based on doctor observation is very cumbersome, time-consuming and subjective, which requires doctors to analyze the signals in the PSG recording to obtain a sleep score of about 8 hours. Therefore, many automatic sleep assessment methods have been proposed. These studies first extract various sleep-related features from EEG signals, such as time domain features, frequency features, correlation features, entropy features, etc., and then use machine learning methods (decision trees, support vector machines, etc.) to classify the extracted features. However, many potential sleep features have not been mined, and these potential sleep features may play a key role in automatic sleep stage assessment.

[0005] In recent years, an increasing number of deep learning methods have been applied to sleep classification. Convolutional neural networks (CNNs) are used to extract time-invariant local features. Recurrent neural networks (RNNs) are used to learn information related to time series, such as mining the temporal correlations between different EEG sequences within the same 30-second period. Attention mechanisms (Transformers) are used to extract global information from sequences. However, most of these models are based on unimodal EEG information and ignore other modalities provided by polysomnography (electromyography, electrooculography, etc.), which are crucial for generalization to datasets collected from other instruments. For example, the Meta-Learning model proposed by N. Banluesombatkul uses EEG, electrooculography, and electromyography to achieve state-of-the-art results on multiple sleep stage classification datasets. However, this network is relatively complex, and its multimodal processing simply involves fusion, failing to reflect the importance of a particular modality in the classification of a particular stage.

[0006] Furthermore, in sleep stage classification, the time series characteristics of light sleep stage N1 and rapid eye movement (REM), as well as light sleep stage N2 and deep sleep stage N3, are very similar, leading to misclassification as a separate class and a decrease in overall accuracy. The key issue in sleep stage classification is how to use contrastive learning to increase the distance between classes.

[0007] In addition, there is a wealth of knowledge about sleep stage classification in the R&K (Rechtschaffen and Kales) and AASM (American Academy of Sleep Medicine) guidelines, which can serve as auxiliary information for sleep stage assessment. However, there is currently no method to effectively utilize this knowledge to improve classification accuracy.

[0008] Therefore, at present, the field of sleep assessment has the following deficiencies:

[0009] (1) Traditional sleep stage assessment based on doctor observation is very cumbersome, time-consuming and subjective;

[0010] (2) Automatic sleep assessment methods based on feature preprocessing, which extract various sleep-related features by preprocessing physiological signals. Similarly, feature extraction methods are black-box, and many potential sleep features are not explored;

[0011] (3) Currently, many commonly used deep learning-based sleep stage assessment methods have achieved good performance. However, due to the single dataset used to construct the model, it is difficult to apply it to real-world clinical diagnosis scenarios. In addition, the performance of sleep stage assessment based on single-channel EEG signals is poor;

[0012] (4) Currently, in the knowledge-driven era, there is a large amount of knowledge about sleep stage classification on the Internet. This knowledge can be used as auxiliary information for sleep stage assessment, but it is difficult to utilize it. Summary of the Invention

[0013] The purpose of the embodiments of the present invention is to provide a sleep stage classification method based on zero-shot learning and contrastive learning, so as to solve the problem of low accuracy of sleep stage assessment in the prior art.

[0014] To solve the above technical problems, the technical solution adopted by the present invention is a sleep stage classification method based on zero-shot learning and contrastive learning, comprising the following steps:

[0015] S1: Obtain and process the original EEG, EMG, and EOG physiological signals;

[0016] S2: Obtain semantic information related to sleep stage classification;

[0017] S3: Manually match physiological signals and semantic information;

[0018] S4: Dataset division: Divide the matched physiological signals and semantic information groups in S3 into training set, validation set and test set;

[0019] S5: Pre-training: Input the training set obtained in S4 into the zero-shot learning and contrastive learning model for pre-training, and use contrastive learning to fine-tune the zero-shot learning and contrastive learning model;

[0020] S6: Predict sleep stage classification.

[0021] Furthermore, the S1 specifically includes: collecting original EEG, EMG, and EOG signals from a public sleep database, and down-sampling the EEG, EMG, and EOG signals so that the frequency of each signal is consistent.

[0022] Furthermore, the acquisition of semantic information related to sleep stage classification in S2 specifically includes: collecting knowledge related to sleep stage determination, integrating the knowledge into the corresponding sleep stage classification, and obtaining semantic information of the sleep stage classification.

[0023] Furthermore, the matching method in S3 is: calculating the value of the knowledge in the sleep stage determination corresponding to the physiological signal, and matching the corresponding semantic information, independently grouping them in the format of: EEG, EMG, EOG, and semantic information.

[0024] Furthermore, the zero-shot learning and contrastive learning models include:

[0025] A physiological signal encoder, configured to input a physiological signal and output a physiological signal feature vector;

[0026] A semantic information encoder, which is used to input semantic information and output a semantic information feature vector;

[0027] The physiological signal and semantic description information matching module is used to multiply the physiological information feature vector and the semantic information feature vector and output the classification result.

[0028] Furthermore, the pre-training in S5 includes the following steps: for the EEG signal:

[0029] S51: The sampling frequency used in the training set when collecting EEG signals is 100 Hz, and a cycle of 30 seconds is used to evaluate sleep stages; each data set has 3000 data points;

[0030] S52: Input the EEG signal data point into the physiological signal encoder to obtain a physiological information feature vector; input the semantic information into the semantic information encoder to obtain a semantic information feature vector;

[0031] S53: Multiply the obtained physiological signal feature vector with the semantic information feature vector, and then fine-tune the physiological signal encoder and the semantic information encoder so that the paired feature vectors are multiplied to 1 and the unpaired feature vectors are multiplied to 0. Use the validation set to track the pre-training process and observe the effect.

[0032] Furthermore, the specific process of S6 is as follows:

[0033] Input the test set data of EEG, EOG, and EMG signals, use the pre-trained zero-shot learning and contrastive learning models to obtain the physiological signal feature vector and the semantic information feature vector, and multiply the physiological signal feature vector with the semantic information feature vector to obtain the EEG signal classification results, EMG signal classification results, and EOG signal classification results. Combine the three classification results, set hyperparameters, and perform weighted fusion on the obtained classification results to obtain the final sleep stage prediction result.

[0034] The beneficial effects of the present invention are:

[0035] 1. The knowledge-driven module based on zero-shot learning incorporates traditional expert knowledge of sleep stage identification without sacrificing model performance, greatly improving the accuracy of sleep stage prediction.

[0036] 2. A two-stage approach of pre-training and prediction is adopted, and matching training is used for automatic sleep stage prediction for the first time, which solves the problem of poor generalization of traditional methods and effectively improves the performance of the method in downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 is a flow chart of a sleep stage classification method according to an embodiment of the present invention;

[0039] Figure 2 is a diagram of the pre-training network structure of an embodiment of the present invention;

[0040] Figure 3 It is a test flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] The present invention provides a knowledge-driven and contrastive learning sleep stage classification method based on the idea of ​​zero-sample learning. The method is divided into two stages: a pre-training stage and a prediction stage. The two-stage approach can better adapt to downstream tasks and has better generalization. In the pre-training stage, expert knowledge is first extracted from the standards for sleep stage classification to obtain 25 detailed semantic descriptions of the classification. The semantic information is matched with physiological signal information of multiple modalities using a manual extraction method. Secondly, an encoder obtained by pre-training multiple data sets is used in SleepTransformer (transformer-based sleep stage classification technology) to extract the corresponding features of the physiological signal information. The BERT pre-trained encoder is used to extract the semantic information feature vector, and the feature vector of the physiological signal information is matched with the semantic information feature vector in pairs to obtain the classification result. Then, the two encoders are fine-tuned by the contrastive learning method so that the multiplication of the paired physiological signal and semantic information features is 1, and the unpaired ones are 0. In the prediction phase, first, paired physiological signals and semantic descriptions are input into a pre-trained encoder, and after feature extraction, they are multiplied to obtain the classification result. Secondly, through hyperparameter setting, the classification results 1, 2, and 3 obtained from EEG, EOG, and EMG are weighted and fused to obtain the final sleep stage prediction result. The specific technical process is as follows, see Figure 1 :

[0043] S1: Acquisition of original EEG, EMG, and EOG physiological signals. Collect original EEG, EMG, and EOG signals from public sleep databases (SleepEDF, SHHS, etc.) and perform simple downsampling processing on the EEG, EMG, and EOG signals to make the frequency of each signal consistent.

[0044] S2: Acquisition of semantic information related to sleep stage classification: Knowledge related to sleep stage determination was collected from the "Handbook of Interpretation of Sleep and Related Events" published by AASM and the R&K standard. The extracted knowledge includes, for example, Alpha EEG (Alpha EEG refers to the quantitative value of the 7.5-12.5Hz EEG ratio under each working number), which represents the alpha ratio of each epoch (the epoch number is a hyperparameter that defines the number of times the learning algorithm works in the entire training dataset), and SWS EEG (SlowWave Sleep Electroencephalography) representing the K complex and spindle in each epoch, a total of 12 knowledge items. The 12 knowledge items were integrated into the corresponding sleep stage classification to obtain 25 semantic information items related to sleep stage classification (for example, the wakefulness period W can be represented by a high Alpha EEG, a high 0-30 EMG, and a low 0-4 EEG); knowledge-related digital signal processing calculation formulas were collected.

[0045] S3: Manually match the semantic information of physiological signals: Use relevant mathematical formulas in the field of digital signal processing to calculate the knowledge value of sleep stage determination corresponding to the physiological signals, and match the corresponding semantic information. Independently group them in the format of (EEG, EMG, EOG, semantic information) as input based on zero-shot learning and contrastive learning models.

[0046] S4: Dataset division: Divide the matched physiological signals and semantic information groups in S3 into training set, validation set and test set.

[0047] S5: Pre-training: The training set obtained in S4 is input into the zero-shot and contrastive learning models for pre-training. Contrastive learning is then used to fine-tune the zero-shot and contrastive learning models of the present invention. Contrastive learning increases the distribution distance between the positive class (matched physiological signals and text pairs) and the negative class (mismatched physiological signals and text pairs). Ultimately, this increases the inter-class distribution distances for the five sleep stages, ensuring the classification performance of the zero-shot and contrastive learning models.

[0048] S6: Prediction stage: Given the test set data of EEG, EOG, and EMG signals, the pre-trained zero-shot learning and contrastive learning models are used to obtain the physiological signal feature vector and the semantic information feature vector, and the physiological signal feature vector is multiplied with the semantic information feature vector to obtain the EEG classification result 1 of the EEG signal, the EMG classification result 2 of the EMG signal, and the EOG classification result 3. Combining the three classification results, by setting appropriate hyperparameters, the obtained classification result 1, classification result 2, and classification result 3 are weightedly fused to obtain the final sleep stage prediction result.

[0049] Furthermore, the pre-training process based on zero-shot learning and contrastive learning model in S5 specifically includes the following steps (in this embodiment, EEG signal is taken as an example, and the other two physiological signals are similar):

[0050] S51: Using EEG signals, this training set uses a sampling frequency of 100 Hz, with a sleep stage cycle of 30 seconds. Each data point in the dataset contains 3000 data points.

[0051] S52: These data points are paired with their corresponding semantic descriptions as input to the model. EEG signal data points are input into the SleepTransformer model (equivalent to the physiological signal encoder), which currently has the highest accuracy for sleep stage classification. The corresponding feature vector (physiological information feature vector) is obtained. Semantic information is input into the BERT (Bidirectional Encoder Representation from Transformers) model (equivalent to the semantic information encoder), which currently performs well in downstream tasks in natural language processing (NLP).

[0052] S53: Multiply the two obtained feature vectors, namely the physiological signal feature vector and the semantic information feature vector, and then fine-tune the physiological signal encoder and the semantic information encoder so that the paired feature vectors are multiplied to 1 and the unpaired feature vectors are multiplied to 0. Use the validation set to track the pre-training process and observe the effect.

[0053] Furthermore, the prediction process in S6 specifically includes the following steps:

[0054] S61: Input a polygraph (PSG), which includes three modalities: EEG, EMG, and EOG. Input them into the corresponding encoders respectively to obtain EEG, EMG, and EOG feature vectors, and then multiply them with the semantic description feature vectors to obtain the classification results of the physiological signals.

[0055] S62: The classification results of EEG, EMG and EOG are calculated according to result = αEEG + βEMG + γEOG (α, β, γ are the hyperparameters of the classification results of EEG, EMG and EOG respectively) to calculate the final confidence level of each sleep stage, and the predicted sleep stage is obtained through the softmax function.

[0056] Furthermore, the zero-shot learning and contrastive learning models include:

[0057] Physiological signal (EEG, EMG, EOG) encoder, used to input physiological signals and output physiological signal feature vectors;

[0058] A semantic information encoder, which is used to input semantic information and output a semantic information feature vector;

[0059] The physiological signal and semantic description information matching module is used to multiply the physiological information feature vector and the semantic information feature vector and output the classification result.

[0060] Example:

[0061] 1. Physiological signal collection

[0062] This paper uses the Sleep Heart Health Study (SHHS), a benchmark dataset commonly used in sleep stage assessment. These recordings were obtained from Caucasian men and women (over 40 years old) who were not taking any sleep-related medications and had sleep disorders. These recordings include horizontal electrooculograms (EOGs), electroencephalograms (EEGs) and electromyograms (EMGs) from channels C4-A1 and C3-A2, with data from each channel sampled at 256 Hz. In this dataset, the recordings were artificially divided into eight periods according to the R&K standard: W, N1, N2, N3, N4, REM, movement, and unknown. To comply with the AASM standard, this paper merged the N3 and N4 periods into a single N3 period and deleted data marked as movement and unknown. Because the dataset has a long W phase before and after sleep, this paper only retained 30 minutes of data. The dataset is labeled with artificial periodization and uses a 30-second period, so each sample has 3,000 data points. Table 1 shows the number of data points for each sleep stage in this dataset, which shows that the data is unbalanced. In this embodiment, the method of the present invention is evaluated using the C3-A2 channel as the EEG channel, the left and right channels as the EOG channel, and the chin channel as the EMG channel.

[0063] Table 1. Distribution of sleep stages in the SHHS dataset

[0064] Dataset W phase N1 N2 stage N3 REM period SHHS 46319 10304 142125 60153 65953

[0065] 2. Semantic Information Description Collection

[0066] This method uses the gold standard AASM and R&K rules used by traditional experts to assess sleep stages. It extracts 12 pieces of relevant knowledge, including Alpha EEG, 0-4 EEG, Spindle EEG, and SWS EEG, for semantic description of sleep stages. For example, stage W can be represented by elevated Alpha EEG, elevated 0-30 EMG, and elevated 0-4 EEG. The resulting semantic descriptions for each stage include one for stage W, eight for stage N1, ten for stage N2, two for stage N3, and four for REM, for a total of 25 pieces. The corresponding digital signal calculation method is used to associate physiological signals with corresponding semantic descriptions.

[0067] 3. Sleep stage assessment

[0068] like Figure 2 and Figure 3 The figure shows the specific implementation details of the pre-training and prediction stages. This paper uses zero-shot learning and contrastive learning models to derive EEG classification results 1, EMG classification results 2, and EOG classification results 3, respectively, and then performs weighted fusion to obtain the final result. Note that the best performing encoders are selected for the physiological signal and semantic information encoders in the pre-training module.

[0069] 4. Dataset Division

[0070] The present invention uses a k-fold cross-validation strategy to evaluate the model. When dividing the training set and test set, each subject in the data set is independent. In the experiments of the present invention, k of the SHHS data set is set to 20. Specifically, the present invention uses N-(N / k) subject records for training and the remaining N / k subject records for testing, where N is the number of subjects in the data set. During the training process, two subjects in the training set are randomly selected as the development set, and the development set is used to select the appropriate number of training iterations and the criteria for stopping training early. The model of the present invention needs to be trained and tested on the SHHS data set 20 times, and then the sleep stage classification results of each test data are combined to calculate the evaluation index.

[0071] 5. Results Evaluation

[0072] This embodiment uses multiple evaluation indicators to evaluate the method of the present invention, and these evaluation indicators are accuracy, recall rate, precision rate, MF1 index (average accuracy of five cycles) and kappa coefficient.

[0073] The SHHS dataset is predicted using zero-shot learning and contrastive learning models, and the resulting confusion matrix is ​​shown in Table 2 below:

[0074] Table 2. Confusion matrix of zero-shot learning and contrastive learning models for SHHS dataset classification results

[0075]

[0076]

[0077] As shown in Table 3, the method of the present invention demonstrates excellent classification performance in automated sleep stage assessment. The overall accuracy and F1 score on the SHHS dataset reached 86.47% and 82.9%, respectively, reaching the baseline of existing methods. However, various evaluation metrics for the N1 stage were very low. This is due to the close similarity between the N1 and REM stages and the fact that the N1 stage, with the least amount of data due to data imbalance, lacks discriminative information. This demonstrates the effectiveness of adding semantic information to sleep stage assessment.

[0078] Table 3. Comparison of sleep stage classification results combined with knowledge module and original model

[0079]

[0080] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A sleep stage classification method based on zero-shot learning and contrastive learning, characterized in that: The following steps are involved: S1: Obtain and process the original EEG, EMG, and EOG physiological signals; S2: Obtain semantic information related to sleep stage classification; S3: Manually match physiological signals and semantic information; S4: Dataset division: Divide the matched physiological signals and semantic information groups in S3 into training set, validation set and test set; S5: Pre-training: Input the training set obtained in S4 into the zero-shot learning and contrastive learning model for pre-training, and use contrastive learning to fine-tune the zero-shot learning and contrastive learning model; S6: Predict sleep stages; Based on zero-shot learning and contrastive learning models include: A physiological signal encoder, configured to input a physiological signal and output a physiological signal feature vector; A semantic information encoder, which is used to input semantic information and output a semantic information feature vector; The physiological signal and semantic description information matching module is used to multiply the physiological information feature vector and the semantic information feature vector and output the classification result.

2. The sleep stage classification method based on zero-shot learning and contrastive learning according to claim 1, characterized in that: Specifically, S1 includes collecting original EEG, EMG, and EOG signals from a public sleep database, and down-sampling the EEG, EMG, and EOG signals to make the frequency of each signal consistent.

3. The sleep stage classification method based on zero-shot learning and contrastive learning according to claim 1, characterized in that: The acquisition of semantic information related to sleep stage classification in S2 specifically includes: collecting knowledge related to sleep stage determination, integrating the knowledge into the corresponding sleep stage classification, and obtaining semantic information of the sleep stage classification.

4. The sleep stage classification method based on zero-shot learning and contrastive learning according to claim 1, characterized in that: The matching method in S3 is: calculating the knowledge value of sleep stage determination corresponding to the physiological signal, and matching the corresponding semantic information, independently grouping them in the format of: EEG, EMG, EOG, and semantic information.

5. The sleep stage classification method based on zero-shot learning and contrastive learning according to claim 1, characterized in that: The S5 pre-training includes the following steps: for EEG signals: S51: The sampling frequency used in the training set when collecting EEG signals is 100 Hz, and a cycle of 30 seconds is used to evaluate sleep stages; each data set has 3000 data points; S52: Input the EEG signal data point into the physiological signal encoder to obtain a physiological information feature vector; input the semantic information into the semantic information encoder to obtain a semantic information feature vector; S53: Multiply the obtained physiological signal feature vector with the semantic information feature vector, and then fine-tune the physiological signal encoder and the semantic information encoder so that the paired feature vectors are multiplied to 1 and the unpaired feature vectors are multiplied to 0. Use the validation set to track the pre-training process and observe the effect.

6. The sleep stage classification method based on zero-shot learning and contrastive learning according to claim 1, characterized in that: The specific process of S6 is as follows: Input the test set data of EEG, EOG, and EMG signals, use the pre-trained zero-shot learning and contrastive learning models to obtain the physiological signal feature vector and the semantic information feature vector, and multiply the physiological signal feature vector with the semantic information feature vector to obtain the EEG signal classification results, EMG signal classification results, and EOG signal classification results. Combine the three classification results, set hyperparameters, and perform weighted fusion on the obtained classification results to obtain the final sleep stage prediction result.

Images