An automatic sleep staging method based on hybrid neural networks

Abstract

The application discloses an automatic sleep staging method based on a hybrid neural network, which learns local features and global features within an epoch by using a convolutional transformer epoch encoder, models cross-modal context relationships by using a cross-modal context encoder, learns sequence features by using a transformer-based sequence encoder, and finally predicts the sleep stage of each epoch by using an epoch-level classifier, thereby improving the prediction performance of the automatic sleep staging model.

Description

Technical Field

[0001] This invention belongs to the field of automatic sleep staging technology, specifically relating to an automatic sleep staging method based on a hybrid neural network. Background Technology

[0002] Sleep plays a vital role in promoting physical and mental health. Sleep staging, which divides sleep into different stages, is an effective way to measure sleep quality and diagnose sleep disorders. Clinically, sleep staging is usually determined by doctors or specialists using polysomnography (PSG). PSG records electrical activity signals from sensors attached to different parts of the body. PSG consists of physiological signals such as electroencephalogram (EEG) and electrooculogram (EOG), and is typically divided into 30-second epochs. Clinically, sleep specialists manually divide these epochs into five different sleep stages according to the American Academy of Sleep Medicine (AASM) sleep standards: wakefulness (W), rapid eye movement (REM) sleep, and three types of non-rapid eye movement (N1, N2, N3) sleep. A sleep specialist may need more than two hours to completely classify a whole night's worth of PSG recordings into different sleep stages. Obviously, manually classifying a large number of sleep cycles is very time-consuming and requires a lot of human labor. Therefore, it is necessary to develop a method for automatically classifying sleep stages.

[0003] With the rapid development of artificial intelligence technology, automatic sleep staging technology has been proposed, and it has achieved good performance, but there are still several problems to be solved:

[0004] 1. The integration of local and global features within an epoch is often overlooked. Many studies focus on salient local waveforms within each epoch, neglecting global features such as the position and proportion of certain waveforms within the epoch, thus affecting sleep stage classification. While local features are certainly important in sleep staging, global features are equally crucial. Therefore, ignoring global features within an epoch will hinder the improvement of automatic sleep staging performance.

[0005] 2. Cross-modal contextual relationships between different modalities are not captured. Different sleep stages have specific EEG characteristics, which are also present in other signals. Experts usually combine EEG and other signals (such as EOG) to determine sleep stages. Blinking in EOG plays an important role in sleep staging, while SEM and REM in EOG are key to classifying two sleep cycles as N1 and REM, respectively. However, many existing deep learning methods ignore the effectiveness of multimodal data, using only EEG to classify sleep stages. Although some studies have explored the importance of different modalities, they only fuse different modalities in simple ways, such as cascading, addition, and dot product. These studies usually use epoch-level or element-level fusion methods to model the epoch-level or element-level cross-modal relationship between EEG and EOG in a one-to-one manner, ignoring the cross-epoch and cross-modal relationships between different modalities. According to the AASS standard, both salient waveforms within an epoch and information between epochs play a crucial role in sleep staging. Furthermore, most existing studies on automatic sleep staging adopt a sequence-to-sequence approach, indicating that both intra-epoch features and inter-epoch contextual features are important for sleep staging. Therefore, cross-epoch cross-modal relationships are also important for the classification of sleep data. To address the problem of neglecting cross-modal contextual information, it is essential to adopt a cross-modal contextual encoding approach to model cross-modal contextual relationships.

[0006] Many existing studies have limitations. For example, Supratak et al.'s 2017 paper, [DeepSleepNet: A Model for Automatic Sleep Stage Scoring Based on Raw Single-Channel EEG], did not consider global features within an epoch, nor the importance of multimodal data. Phan et al.'s 2022 paper, [SleepTransformer: Automatic Sleep Staging with Interpretability and Uncertainty Quantification], did not consider the importance of local features within an epoch, nor the importance of multimodal data. Jia et al.'s 2021 paper, [SalientSleepNet: Multimodal Salient Wave Detection Network for Sleep Staging], considered the importance of multimodal data but did not consider the importance of multimodal contextual relationships. Summary of the Invention

[0007] In view of the above, the present invention provides an automatic sleep staging method based on a hybrid neural network, which improves the performance of the neural network in predicting sleep stages by capturing local salient waves and global features of epochs and modeling cross-modal contextual relationships.

[0008] An automatic sleep staging method based on hybrid neural networks includes the following steps:

[0009] (1) Collect PSG data from multiple subjects, including patients with sleep disorders and healthy individuals;

[0010] (2) Perform signal selection, sleep phase labeling and signal preprocessing on PSG data;

[0011] (3) Constructing a hybrid neural network model, which includes:

[0012] The epoch encoder based on convolutional transformers is divided into two branches, which target EEG signals and EEG signals in PSG data respectively, to learn local and global features within the epoch.

[0013] The cross-modal context encoder is also divided into two branches corresponding to EEG signals and EEG signals respectively. Based on the local and global features of the two branches, the cross-modal context features of the epoch are further learned with the help of the joint attention mechanism.

[0014] The transformer-based sequence encoder learns the sequence features of PSG data by establishing the relationship between epochs in the sequence based on the cross-modal context features of the two branches and by using a self-attention mechanism.

[0015] A classifier classifies and identifies based on the sequence features, and outputs the predicted probability that each epoch belongs to each sleep stage;

[0016] (4) The collected PSG data is used to train the above hybrid neural network model to obtain a sleep staging model;

[0017] (5) By using the sleep staging model to predict the test set data, the sleep staging task of the data can be automatically realized.

[0018] Further, in step (1), PSG data is collected at a sampling frequency of 512Hz. The collected PSG data packets contain electroencephalogram (EEG) signals, electrooculogram (EOG) signals, electrocardiogram (ECG) signals, and electromyogram (EMG) signals. The EEG signals include six channels: F4-M1, C4-M1, F3-M2, C3-M2, O1-M2, and O2-M1. The EOG signals include two channels: E1-M2 and E2-M2.

[0019] Furthermore, in step (2), EEG and EEG signals are selected from the PSG data for training and testing. The signals are labeled by sleep experts according to the standards of the American Academy of Sleep Medicine, that is, the signals are divided into epochs every 30 seconds, and each epoch is classified into five different sleep stages: Wake, N1, N2, N3, and REM. The signal preprocessing includes bandpass filtering at 0.3 Hz and 35 Hz, spike filtering at 50 Hz, and resampling at 100 Hz.

[0020] Furthermore, the epoch encoder has 5 layers. The first 4 layers all use multi-scale convolutional transformer networks. The 5th layer consists of an average pooling layer, a fully connected layer, and a GELU activation function connected in sequence. The multi-scale convolutional transformer network is composed of a multi-scale convolutional network and a transformer network. The multi-scale convolutional network is used to learn salient wave features (i.e., local features) at different scales within the epoch, while the transformer network learns global features within the epoch using a self-attention mechanism.

[0021] Furthermore, the computation process of the multi-scale convolutional network is as follows:

[0022] X j =G(BN(DConv) j (X))), j∈[1,2,3]

[0023] X4 = DownSample(X)

[0024]

[0025] Where: X represents the input signal, Y represents the output signal, and DConv j () represents the j-th dilated convolution in a multi-scale convolutional network, BN() represents a Batch Normalization layer, G() represents the GELU activation function, DownSample() represents a residual connection implemented by a convolutional layer with a kernel scale of 1, LN() represents a Layer Normalization layer, and MP() represents a max pooling layer.

[0026] Furthermore, the transformer network is composed of a multi-head self-attention mechanism layer, a regularization layer L1, a feedforward neural network F1, and a regularization layer L2 connected in sequence. The input of the regularization layer L1 is the superposition of the output and input of the multi-head self-attention mechanism layer, and the input of the regularization layer L2 is the superposition of the output and input of the feedforward neural network F1.

[0027] Furthermore, the cross-modal context encoder is composed of a joint attention mechanism layer, a regularization layer L3, a feedforward neural network F2, and a regularization layer L4 connected sequentially. The input of the regularization layer L4 is the superposition of the output and input of the feedforward neural network F2. In the EEG signal branch, the query vector matrix of the joint attention mechanism layer is X o As input, the key vector matrix is ​​in the form of X e As input, the value vector matrix is ​​in the form of X e As input, the input to the regularization layer L3 is the output of the joint attention mechanism layer and X. o Superimposed; in the electrooculogram signal branch, the query vector matrix of the joint attention mechanism layer is X e As input, the key vector matrix is ​​in the form of X o As input, the value vector matrix is ​​in the form of X o As input, the input to the regularization layer L3 is the output of the joint attention mechanism layer and X. e Superimposed, X e and X o These are the output features of the epoch encoder for the EEG signal branch and the EEG signal branch, respectively.

[0028] Furthermore, the sequence encoder is implemented by a transformer network based on a multi-head self-attention mechanism layer, and its input consists of the superposition of cross-modal context features from two branches and the positional encoding of the epoch.

[0029] Furthermore, in step (4), before training the model, the collected PSG data is first divided into 10-fold cross-validation partitions for each subject. The 10-fold cross-validation partitions divide all subjects equally into 10 groups. In each fold, one group is used as the test set and the other nine groups are used as the training set. The PSG data of the same subject will not appear in both the training set and the test set at the same time. At the same time, it is necessary to ensure that there are the same number of subjects in each fold.

[0030] Furthermore, the loss function used in step (4) during model training is as follows:

[0031]

[0032] Where: Loss is the loss function, y i,j This represents the probability that the i-th epoch is predicted to be the j-th sleep stage. Let L represent the probability that the i-th epoch actually belongs to the j-th sleep stage, L be the number of epochs in the input PSG data, and N be the number of sleep stage categories.

[0033] Furthermore, in step (5), the sleep staging model is used to predict the test set data, and the predicted probability of each epoch belonging to each sleep stage is output. The sleep stage with the highest probability is taken as the sleep staging prediction result of the epoch, thereby automatically realizing the sleep staging task of the data.

[0034] This invention utilizes a convolutional transformer epoch encoder to learn local and global features within an epoch, and a cross-modal context encoder to model cross-modal context relationships. It then uses a transformer-based sequence encoder to learn sequence features, and finally uses an epoch-level classifier to predict the sleep stage of each epoch, thereby improving the predictive performance of the automatic sleep staging model. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the overall process of the automatic sleep staging method of the present invention.

[0036] Figure 2 This is a schematic diagram of the complete architecture of the hybrid neural network model of the present invention.

[0037] Figure 3 This is a schematic diagram of the overall structure of an epoch encoder based on a convolutional transformer.

[0038] Figure 4 This is a schematic diagram of the overall structure of a cross-modal context encoder.

[0039] Figure 5 This is a schematic diagram of a transformer-based sequence encoder. Detailed Implementation

[0040] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] like Figure 1 As shown, the specific steps of the automatic sleep staging method based on hybrid neural networks of this invention are as follows:

[0042] (1) Seventy-seven normal subjects and subjects with sleep disorders were recruited for PSG data collection. Wired sensors began collecting sleep signals from different parts of the subjects' bodies throughout the night at a sampling frequency of 512 Hz. The collected PSG data consisted of 6 EEG channels (F4-M1, C4-M1, F3-M2, C3-M2, O2-M1, O1-M1), 2 EOG channels (E1-M2 and E2-M2), electromyography, electrocardiography, etc.

[0043] (2) Considering that EEG and EOG data have good performance in sleep stages, EEG and EOG signals were selected for training and testing. After data collection, sleep experts manually labeled the sleep stage of each epoch according to the American Academy of Sleep Medicine standards. After the data was labeled, the data was preprocessed, including 0.3Hz and 35Hz bandpass filtering, 50Hz spike filtering and 100Hz resampling.

[0044] (3) Design as follows Figure 2 The hybrid neural network model shown learns local and global features within an epoch through a convolutional transformer-based epoch encoder, models cross-modal context relationships through a cross-modal context encoder, learns sequence features through a transformer-based sequence encoder, and finally predicts the sleep stage of each epoch through an epoch-level classifier.

[0045] The structure of an epoch encoder based on a convolution transformer is as follows: Figure 3 As shown, the model consists of a convolutional neural network (CNN) and a transformer network. The CNN is composed of a multi-scale convolutional network used to learn salient wave features at different scales within an epoch. The transformer network consists of a multi-head attention network and a feedforward network, using a self-attention mechanism to learn global features within an epoch. The epoch encoder network has 5 layers. EEG or EOG data is input into the network. The computation process of the multi-scale convolutional network in the first layer is as follows:

[0046]

[0047]

[0048]

[0049] Wherein: DConv j is the j-th dilated convolution of a multi-scale convolutional layer, BN is a Batch Normalization layer, G is the GELU activation function, DownSample represents the residual connection implemented by a convolutional layer with a kernel scale of 1, LN is a Layer Normalization layer, and MP is a max pooling layer.

[0050] The multi-head attention in transformer networks is constituted by a self-attention mechanism, the formula of which is as follows:

[0051]

[0052] Where: matrices Q, K, and V are composed of queries, keys, and values, respectively, and d k It is the dimension of the key.

[0053] After passing through the first layer of the convolutional transformer network, the result is input into the subsequent 2 to 4 layers of the convolutional transformer network. The calculation process is as follows:

[0054] X mconv(k+1) =MConv(X sk )

[0055] X s(k+1) =TransformerEncoder(X mconv(k+1) )

[0056] k∈[1,2,3]

[0057] Where: MConv is a multi-scale convolutional network, and TransformerEncoder is a transformer encoding network.

[0058] The fifth layer of the epoch encoder consists of an average pooling layer, a fully connected layer, and a GELU activation function. The calculation process is as follows:

[0059] X P =AP(X) s4 )

[0060] X e =G(FC(X) P ))

[0061] Where AP is the average pooling layer.

[0062] The structure of a cross-modal context encoder is as follows: Figure 4 As shown, the encoder consists of an EEG2EOG module and an EOG2EEG module, each composed of a joint attention network and a feedforward network. Through the joint attention mechanism, this encoder can model cross-modal contextual relationships, thereby learning cross-modal contextual features. The calculation formula for the joint attention mechanism of the EOG2EEG module is as follows:

[0063] Q = FC Q (X o ), K = FC K (X e V = FC V (X e ),

[0064]

[0065] Where: d is the feature dimension, X o2e It is the output of the joint attention of the EOG2EEG module.

[0066] Similarly, the joint attention mechanism calculation formula for the EEG2EOG module is as follows:

[0067] Q = FC Q (X e ), K = FC K (X o V = FC V (X o ),

[0068]

[0069] Where: X e2o This represents the output of joint attention in the EEG2EOG module.

[0070] Then, the output of the joint attention is fed into the corresponding Layer Normalization layer and forward propagation layer to obtain cross-modal context features, calculated as follows:

[0071] X eo =LN2(LN1(X) e2o +X o )+FC(LN1(X e2o +X o )))

[0072] X oe =LN2(LN1(X) o22 +X e )+FC(LN1(X o2e +X e )))

[0073] Where: LN1 and LN2 represent two Layer Normalization layers. Represents the cross-modal contextual features of EEG. This represents the cross-modal contextual features of EOG.

[0074] The structure of a transformer-based sequence encoder is as follows: Figure 5 As shown, this encoder establishes relationships between epochs in the sequence using a self-attention mechanism and learns transition patterns in sleep stages by modeling the relationships between epochs in the sleep sequence. Since the previous cross-modal context encoder outputs two different cross-modal context features, this encoder needs to sum these two different features along with the positional encoding before inputting them into the model. The calculation formula is as follows:

[0075] X′=TransformerEncoder(X eo +X oe +H pos )

[0076] in: It is a sequence feature.

[0077] (4) The datasets were partitioned using a subject-independent 10-fold cross-validation method on the three datasets. Subjects in each dataset were divided into 10 groups on average. In each fold, one group was used as the test set, and the other nine groups were used as the training set. The sleep sequences of the same subject never appeared simultaneously in both the training and test sets. After partitioning the datasets, the training data was input into the neural network for training in each fold. The sleep staging loss function used for training was:

[0078]

[0079] in: It is y i The j-th element represents the probability that the i-th epoch is predicted to be the j-th sleep stage; yes The j-th element represents the probability that the i-th epoch actually belongs to the j-th sleep stage.

[0080] (5) The trained automatic sleep staging model is used to predict the sleep stage on the test set data corresponding to each fold, and the prediction accuracy and Macro-F1 score are calculated. The average value of the 10-fold results is calculated and compared with other existing methods. The comparison results are shown in Table 1. It can be seen that the method of the present invention has a significant improvement in prediction effect compared with other existing prediction methods.

[0081] Table 1

[0082]

[0083]

[0084] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. It will be apparent to those skilled in the art that various modifications can be made to the above embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A hybrid neural network-based automatic sleep staging method, comprising the following steps: (1) Collecting PSG data of a plurality of subjects including patients with sleep disorders and healthy people; (2) Selecting signals from the PSG data, annotating sleep stages, and pre-processing the signals; (3) Constructing a hybrid neural network model, comprising: an epoch encoder based on a convolutional transformer, divided into two branches for electroencephalogram (EEG) signals and electrooculogram (EOG) signals in the PSG data, learning local and global features within the epoch; a cross-modal context encoder, also divided into two branches corresponding to the EEG signals and the EOG signals, further learning cross-modal context features of the epoch based on the local and global features of the two branches through a joint attention mechanism; a sequence encoder based on a transformer, establishing relationships between epochs in a sequence based on the cross-modal context features of the two branches through a self-attention mechanism, and learning sequence features of the PSG data; a classifier for classifying and identifying based on the sequence features, and outputting predicted probabilities of each epoch belonging to each sleep stage; (4) Training the above hybrid neural network model using the collected PSG data to obtain a sleep staging model; (5) Using the sleep staging model to predict the test set data, thereby automatically implementing the sleep staging task of the data.

2. The automatic sleep staging method of claim 1, wherein: In the step (1), the PSG data is collected at a sampling frequency of 512 Hz, and the collected PSG data includes EEG signals, EOG signals, electrocardiogram (ECG) signals, and electromyogram (EMG) signals. The EEG signals include F4-M1, C4-M1, F3-M2, C3-M2, O1-M2, and O2-M1, a total of 6 channels, and the EOG signals include E1-M2 and E2-M2, a total of 2 channels.

3. The automatic sleep staging method of claim 1, wherein: In the step (2), EEG signals and EOG signals are selected from the PSG data for training and testing. The signals are annotated by a sleep specialist according to the American Academy of Sleep Medicine standard, i.e., each 30s of signal is divided into an epoch, and each epoch is classified into five different sleep stages: Wake, N1, N2, N3, and REM. Signal pre-processing includes band-pass filtering of 0.3 Hz and 35 Hz, spike filtering of 50 Hz, and resampling of 100 Hz.

4. The automatic sleep staging method of claim 1, wherein: The epoch encoder has a total of 5 layers, the first 4 layers are multi-scale convolutional transformer networks, and the 5th layer is composed of an average pooling layer, a fully connected layer, and a GELU activation function connected in sequence. The multi-scale convolutional transformer network is composed of a multi-scale convolutional network and a transformer network connected in sequence. The multi-scale convolutional network is used to learn significant wave features of different scales within the epoch, and the transformer network learns global features within the epoch through a self-attention mechanism.

5. The automatic sleep staging method of claim 4, wherein: The calculation process of the multi-scale convolutional network is as follows: X j = G(BN(DConv j (X))), j e [1,2,3] X4 = DownSample(X) wherein: X represents an input signal, Y represents an output signal, DConv j () represents the jth dilated convolution of the multi-scale convolutional network, BN() represents a Batch Normalization layer, G() represents a GELU activation function, DownSample() represents a residual connection implemented by a convolutional layer with a kernel size of 1, LN() represents a Layer Normalization layer, and MP() represents a max-pooling layer.

6. The automatic sleep staging method of claim 4, wherein: The transformer network is sequentially connected by a multi-head self-attention mechanism layer, a regularization layer L1, a feedforward neural network F1, and a regularization layer L2, wherein the input of the regularization layer L1 is the superposition of the output and the input of the multi-head self-attention mechanism layer, and the input of the regularization layer L2 is the superposition of the output and the input of the feedforward neural network F1.

7. The automatic sleep staging method of claim 1, wherein: The cross-modal context encoder is composed of a joint attention mechanism layer, a regularization layer L3, a feedforward neural network F2, and a regularization layer L4 connected in sequence, and the input of the regularization layer L4 is the output of the feedforward neural network F2 superimposed with the input; in the electroencephalogram signal branch, the query vector matrix of the joint attention mechanism layer takes X o as the input, the key vector matrix takes X e as the input, the value vector matrix takes X e as the input, and the input of the regularization layer L3 is the output of the joint attention mechanism layer superimposed with X o ; in the electrooculogram signal branch, the query vector matrix of the joint attention mechanism layer takes X e as the input, the key vector matrix takes X o as the input, the value vector matrix takes X o as the input, and the input of the regularization layer L3 is the output of the joint attention mechanism layer superimposed with X e , X e , and X o are respectively the epoch encoder output features of the electroencephalogram signal branch and the electrooculogram signal branch.

8. The automatic sleep staging method of claim 1, wherein: The sequence encoder is implemented by a transformer network based on a multi-head self-attention mechanism layer, and the input thereof is composed of the superposition of the cross-modal context features of two branches and the position encoding of the epoch.

9. The automatic sleep staging method of claim 1, wherein: Before the model is trained in the step (4), the PSG data collected is subjected to subject-independent 10-fold cross-validation division, that is, all subjects are evenly divided into 10 groups, and in each fold, one group is taken as a test set in turn, and the remaining 9 groups are taken as a training set, the PSG data of the same subject cannot appear in the training set and the test set at the same time, and meanwhile, it is required to ensure that there are the same number of subjects in each fold.

10. The automatic sleep staging method of claim 1, wherein: The loss function used in the training process of the model in the step (4) is as follows: wherein: Loss is a loss function, y i,j represents the probability of the ith epoch being predicted as the jth sleep stage, represents the probability of the ith epoch actually belonging to the jth sleep stage, L is the number of epochs of input PSG data, and N is the number of classes of sleep stages.

Images