A method, apparatus, device and storage medium for automatic sleep staging

Abstract

The present application relates to a kind of automatic sleep staging method, its steps include: the EEG original data is processed to obtain EEG feature map, while the EOG original data is processed to obtain EOG feature map;EEG feature map and EOG feature map are multiplied and fused, and highlight feature map is obtained;Highlight feature map is directly added with EEG feature map and EOG feature map and fused to obtain initial fusion feature map;Initial fusion feature map is processed to obtain the most weight feature map by weight adjustment;Initial fusion feature map and the most weight feature map are multiplied and fused to obtain the final fusion feature map with weight information;According to the weight information of final fusion feature map, each sleep stage prediction probability is output.The automatic sleep staging method described in the present application is further trained and optimized, does not need too much artificial intervention, and can be simply and effectively applied in sleep staging of healthy individuals and patients with consciousness disorders.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of sleep staging, and in particular to a method and device for automatic sleep staging based on multi-modal channel attention, equipment and storage medium. BACKGROUND

[0002] Sleep staging is of great significance for capturing sleep patterns and assessing sleep quality. Sleep staging can measure sleep structure, observe sleep characteristic waves, and understand the changes in human sleep cycles. Effective and feasible sleep assessment is crucial for identifying sleep disorders and timely intervention. In order to obtain sleep data, sleep researchers use electroencephalogram (EEG) or polysomnogram (PSG) to record brain activity during sleep, which can observe and explain sleep characteristics in long-term monitoring. Currently, there are two commonly used sleep staging standards and sleep quality scales: Rechtschaffen and Kales (R&K) and American Academy of Sleep Medicine (AASM) guidelines. However, the subjectivity of manual sleep staging by sleep experts can limit the accuracy of sleep assessment, and manual sleep staging requires clinicians to have rich experience, which takes a long time to classify a whole night of 8 hours or more of sleep records. Therefore, automatic sleep staging models have become popular. In order to achieve automatic sleep staging, some related research has proposed classification methods using electroencephalogram (EEG) and electrooculogram (EOG), which rely on machine learning or deep learning. However, sleep studies on patients with consciousness disorders have not been widely combined, and there are still some challenges in automatic sleep staging methods:

[0003] During sleep, the EOG signal characteristics of N1 stage are similar to those of rapid eye movement stage, which are easily confused. N1 stage is close to wake-up stage, and the electroencephalogram oscillation of the two stages is similar, further leading to poor classification effect of N1 stage. In order to capture multi-modal sleep characteristics from PSG signals, existing automatic sleep staging methods use connection operations, ignoring the complementary effects of different modal sleep signals and the contradictory features in single modal sleep signals, which may cause partial information loss in signal conversion. Therefore, it is still necessary to add a large amount of manual intervention, which not only takes time, but also may require prior knowledge, is low in efficiency and requires high personnel quality.

[0004] The existing automatic sleep staging model also faces challenges in the application of patients with consciousness disorders. Consciousness disorders (DOC) refer to the state of consciousness loss caused by severe brain injury, such as coma, micro-consciousness state (MCS), vegetative state (VS) or unresponsive wakefulness syndrome (UWS), and sleep disorders are the main complications of patients with consciousness disorders.

[0005] For patients with consciousness disorders, sleep disorders can cause sleep cycle confusion, abnormal sleep stage proportion or stage loss. And because DOC patients suffer from severe brain damage, their EEG signal frequency slows down, similar to the light sleep brain signal of healthy individuals, but in fact the patient is not asleep and cannot be distinguished. SUMMARY

[0006] Based on this, the purpose of the present application is to provide a sleep staging method, device, equipment and storage medium, which can extract signal characteristics, learn sleep transition rules and automatically output each sleep stage based on raw electroencephalogram and electrooculogram sleep data using a hybrid method, without too much human intervention, and can be simply and effectively applied to healthy individuals and patients with consciousness disorders.

[0007] The present application provides an automatic sleep staging device, comprising:

[0008] An EEG raw data processing module for processing EEG raw data from an electroencephalograph to obtain an EEG feature map;

[0009] An EOG raw data processing module for processing EOG raw data from an electrooculograph to obtain an EOG feature map;

[0010] The EEG raw data processing module and the EOG raw data processing module are correspondingly and synchronously operated;

[0011] A multiplication fusion module for multiplying the EEG feature map and the EOG feature map to obtain a highlighted feature map;

[0012] An addition fusion module for directly adding the highlighted feature map, the EEG feature map and the EOG feature map to obtain an initial fusion feature map;

[0013] A weight adjustment module for performing weight adjustment processing on the initial fusion feature map to obtain a final weight feature map;

[0014] A weight fusion module for multiplying the initial fusion feature map and the final weight feature map to obtain a final fusion feature map with weight information;

[0015] A staging prediction module for outputting a sleep stage prediction probability according to the weight information of the final fusion feature map.

[0016] Further comprising:

[0017] A time sequence extraction module for extracting time sequence features in the final fusion feature map.

[0018] The stage prediction module is configured to output a sleep stage prediction probability according to the weight information of the final fusion feature map and the corresponding time sequence feature.

[0019] Further, the EEG raw data processing module comprises:

[0020] The EEG labeling filtering submodule is configured to label the EEG raw data according to the AAMS standard, and sample and filter the EEG raw data to obtain an EEG self-sampling dataset.

[0021] The EEG extraction submodule is configured to extract EEG shallow features from the EEG self-sampling dataset through convolution.

[0022] The first EEG compression submodule is configured to compress and select the EEG shallow features to obtain EEG information deepening features.

[0023] The EEG activation submodule is configured to activate the EEG information deepening features to obtain EEG deep features.

[0024] The second EEG compression submodule is configured to compress and select the EEG deep features to obtain an EEG feature map.

[0025] The EOG raw data processing module comprises:

[0026] The EOG labeling filtering submodule is configured to label the EOG raw data according to the AAMS standard, and sample and filter the EOG raw data to obtain an EOG self-sampling dataset.

[0027] The EOG extraction submodule is configured to extract EOG shallow features from the EOG self-sampling dataset through convolution.

[0028] The first EOG compression submodule is configured to compress and select the EOG shallow features to obtain EOG information deepening features.

[0029] The EOG activation submodule is configured to activate the EOG information deepening features to obtain EOG deep features.

[0030] The second EOG compression submodule is configured to compress and select the EOG deep features to obtain an EOG feature map.

[0031] Further, the weight adjustment module comprises the following modules:

[0032] The weight compression submodule is configured to perform global average pooling compression and reshaping on the initial fusion feature map to obtain an initial weight feature map.

[0033] The first transformation submodule is configured to perform nonlinear transformation on the initial weight feature map to obtain an intermediate weight feature map.

[0034] The second transformation submodule is configured to perform nonlinear transformation on the intermediate weight feature map to obtain the final weight feature map.

[0035] Further, the first transformation submodule adopts a ReLU activation function to perform nonlinear transformation on the initial weight feature map to obtain the intermediate weight feature map, and the second transformation submodule adopts a sigmoid activation function to perform nonlinear transformation on the intermediate weight feature map to obtain the final weight feature map.

[0036] Further, the automatic sleep staging device is optimized by the following training method:

[0037] Step R1: Obtain sleep data of healthy subjects and input the initial model of the automatic sleep staging device for training to obtain a healthy group training model;

[0038] Step R2: Replace the EEG data processing module, the EOG data processing module and the time sequence extraction module in the initial model with the corresponding modules of the healthy group training model, and freeze the EEG data processing module and the EOG data processing module to obtain an advanced model with effective health characteristics;

[0039] Step R3: Obtain sleep data of patients with consciousness disorders and input the advanced model of the automatic sleep staging device for training to obtain a multi-modal attention model.

[0040] The application also provides an automatic sleep staging device, which comprises an electroencephalogram detector, an electrooculogram detector and the automatic sleep staging device as described above.

[0041] The application also provides an automatic sleep staging method, which comprises the following steps:

[0042] Step S1a: Process the EEG raw data to obtain an EEG feature map;

[0043] Step S1b: Process the EOG raw data to obtain an EOG feature map;

[0044] The step S1b and the step S1a are performed synchronously.

[0045] Step S2: Multiply the EEG feature map and the EOG feature map to obtain a highlight feature map;

[0046] Step S3: Directly add the highlight feature map, the EEG feature map and the EOG feature map to obtain an initial fusion feature map;

[0047] Step S4: Perform weight adjustment processing on the initial fusion feature map to obtain a final weight feature map;

[0048] Step S5: multiplying the initial fusion feature map and the final weight feature map to obtain a final fusion feature map with weight information;

[0049] Step S7: outputting sleep stage prediction probabilities according to the weight information of the final fusion feature map.

[0050] Further, the method further comprises:

[0051] Step S6: extracting time sequence features in the final fusion feature map;

[0052] Step S7: outputting sleep stage prediction probabilities according to the weight information of the final fusion feature map and the corresponding time sequence features.

[0053] The application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the automatic sleep staging method.

[0054] In order to better understand and implement, the application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The steps of the automatic sleep staging method of the application are shown in the flowchart.

[0056] Figure 2 The module block diagram of the automatic sleep staging device of the application is shown.

[0057] Figure 3 The confusion matrix of the multi-modal attention model in the application is shown.

[0058] Figure 4 The comparison diagram of the multi-modal attention model in the application and five existing models is shown.

[0059] Figure 5 The comparison diagram of the multi-modal attention model in the application and two baseline models is shown. DETAILED DESCRIPTION

[0060] The automatic sleep staging device designed by the application comprises an electroencephalogram detector, an electrooculogram detector, and an automatic sleep staging device. The electroencephalogram detector is worn on the head of a subject to obtain EEG raw data of the subject. The electrooculogram detector is worn on the eyes of the subject to obtain EOG raw data of the subject. The automatic sleep staging device extracts features from the EEG raw data and the EOG data and adjusts weights to obtain sleep staging prediction results of the subject.

[0061] The following describes the model construction, model training, and effect verification in detail:

[0062] (I) Model construction:

[0063] The automatic sleep staging model provided by the application is based on original electroencephalogram and electrooculogram sleep data, adopts a multi-channel method to extract signal representation, adopts LSTM to learn sleep conversion rules, and automatically outputs each sleep stage staging through attention mechanism adjustment weight fusion of multi-modal data without too much manual intervention. The initial model is trained and optimized, which can be simply and effectively applied to healthy individuals and patients with consciousness disorders.

[0064] Please refer to Figure 1 and Figure 2 , Figure 1 is a processing flowchart of the automatic sleep staging device, Figure 2 is a module block diagram of the automatic sleep staging device. The automatic sleep staging device of the application comprises:

[0065] The EEG data processing module 10A is used to execute step S1A: processing the EEG original data to obtain the EEG feature map.

[0066] The EOG data processing module 10B is used to execute step S1B: processing the EOG original data to obtain the EOG feature map.

[0067] The EEG data processing module 10A and the EOG data processing module 10B constitute a double-flow CNN structure of the automatic sleep staging device.

[0068] The step S1A includes a series of steps of processing in the EEG channel, and the step S1B includes a series of steps of processing in the EOG channel. Each step of the two series is performed synchronously. Specifically, the EEG channel adopts C3 EEG channel, and the EOG channel adopts HEOG channel.

[0069] Correspondingly, the EEG data processing module 10A comprises:

[0070] The EEG labeling filtering submodule 11a is used to execute step S11a: labeling the EEG original data according to the AAMS standard, and sampling and filtering the EEG original data to obtain the EEG self-sampling data set.

[0071] The EEG labeling filtering submodule 11a divides the EEG raw data into sleep segments of 30 seconds per frame and classifies them into different sleep stages: wakefulness (W), non-rapid eye movement (NREM) sleep, and rapid eye movement (REM) sleep. Specifically, according to the R&K standard, non-rapid eye movement sleep can be further divided into four stages: N1, N2, N3, and N4. According to the AASM standard, N3 and N4 are combined and labeled as the N3 stage, and thus all sleep stages of the EEG raw data are labeled in the following order: {0, 1, 2, 3, 4}: W, N1, N2, N3, REM. The labeled EEG raw data is sampled and band-pass filtered to obtain an EEG self-sampling data set.

[0072] Specifically, the sampling rate is set to 100 Hz to collect the EEG raw data, and the band-pass filter wavelength range is set to 0.3-35 Hz.

[0073] The EEG extraction submodule 12a is used to perform step S12a: convolutionally extracting EEG shallow features from the EEG self-sampling data set.

[0074] The self-sampling data set can be represented as e i [Sample size, channel number, feature number], where the sample size = sampling rate x duration, the EEG extraction submodule 12a includes a first EEG convolutional layer, which is a one-dimensional convolutional layer, and the EEG self-sampling data set is convoluted to extract EEG shallow features.

[0075] Specifically, for the EEG self-sampling data set with an input shape of [3000, 1, 1], where the sample size is 3000, the channel number is 1, and the feature number is 1, the number of convolution kernels of the first EEG convolutional layer is set to 64, the convolution kernel size is 25, and the step size is 4.

[0076] The first EEG compression submodule 13a is used to perform step S13a: compressing and selecting EEG shallow features to obtain EEG information deepening features.

[0077] The first EEG compression submodule 13a includes a first EEG max-pooling layer and a first EEG dropout layer. The first EEG max-pooling layer dimensionally compresses the EEG shallow features by pooling, then takes the maximum value of each element in the compressed matrix, and retains the most important feature information in the EEG shallow features. The first EEG dropout layer randomly sets the input elements to 0 at a certain dropout rate, reduces the overfitting phenomenon, and retains the most important feature information in the EEG shallow features. The first EEG dropout layer and the first EEG max-pooling layer jointly compress and select the EEG shallow features to obtain the EEG information deepening features.

[0078] Specifically, the pooling convolution kernel size of the first EEG max-pooling layer is set to 8, and the step is 8, and the dropout rate of the first EEG dropout layer is set to 0.5.

[0079] The EEG activation submodule 14a is configured to perform step S14a: performing activation processing on the EEG information deep features to obtain EEG deep features.

[0080] The EEG activation submodule 14a includes a plurality of consecutive EEG convolution layers, and the EEG information deep features from the first EEG compression submodule 13a are input into the plurality of consecutive EEG convolution layers. Each EEG convolution layer performs convolution and batch normalization processing and then adds rectified linear unit activation function processing to obtain EEG deep features. The rectified linear unit activation function calculates the i-th feature vector a i EEG component of the i-th EEG data:

[0081]

[0082] wherein l represents a component belonging to the EEG. CNN θ represents a parallel CNN, and CNN θ The EEG component of the i-th EEG data:

[0083] Specifically, the present application adopts three EEG convolution layers with the same settings, and the number of convolution kernels is set to 128, the convolution kernel size is 8, and the step is 1.

[0084] The second EEG compression submodule 15a is configured to perform step S15a: compressing and selecting the EEG deep features to obtain an EEG feature map.

[0085] The second EEG compression submodule 15a includes a second EEG max-pooling layer and a second EEG dropout layer. The second EEG max-pooling layer compresses the dimensions of the EEG shallow features by pooling, and then takes the maximum value of each element in the compressed matrix to retain the most important feature information in the EEG deep features. The second EEG dropout layer randomly sets the input elements to 0 at a certain dropout rate to reduce overfitting while retaining the most important feature information in the EEG deep features. The second EEG dropout layer and the second EEG max-pooling layer jointly compress and select the EEG deep features to obtain an EEG feature map.

[0086] Specifically, the pooling convolution kernel size of the second EEG max-pooling layer is set to 4, and the step is 4, and the dropout rate of the second EEG dropout layer is set to 0.5.

[0087] The EOG data processing module 10B includes:

[0088] The EOG labeling and filtering submodule 11b is configured to perform step S11b: labeling the EOG raw data according to the AAMS standard, and performing sampling filtering on the EOG raw data to obtain an EOG self-sampling data set.

[0089] The EOG labeling and filtering submodule 11b is configured to perform step S11b: labeling the EOG raw data according to the AAMS standard, and performing sampling filtering on the EOG raw data to obtain an EOG self-sampling data set.

[0090] The EOG labeling and filtering submodule 11b is configured to perform step S11b: labeling the EOG raw data according to the AAMS standard, and performing sampling filtering on the EOG raw data to obtain an EOG self-sampling data set.

[0091] Specifically, the EOG raw data is collected at a sampling rate of 100 Hz, and the band-pass filtering wavelength range is set to 0.3-35 Hz.

[0092] The EOG extraction submodule 12b is configured to perform step S12b: performing convolution on the EOG self-sampling data set to extract EOG shallow features.

[0093] The EOG extraction submodule 12b is configured to perform step S12b: performing convolution on the EOG self-sampling data set to extract EOG shallow features.

[0094] The self-sampling data set can be represented as e i [Sample size, channel number, feature number], where the sample size = sampling rate x duration, the EOG extraction submodule 12b includes a first EOG convolution layer, and the first EOG convolution layer is a one-dimensional convolution layer, which is configured to perform convolution on the EOG self-sampling data set to extract EOG shallow features.

[0095] Specifically, for the EOG self-sampling data set with an input shape of [3000, 1, 1], where the sample size is 3000, the channel number is 1, and the feature number is 1, the number of convolution kernels of the first EOG convolution layer is set to 64, the convolution kernel size is 50, and the step size is 6.

[0096] The first EOG compression submodule 13b is configured to perform step S13b: compressing and selecting the EOG shallow features to obtain EOG information deepening features.

[0097] The first EOG compression submodule 13b is synchronously operated with the first EEG compression submodule 13a, and step S13b is synchronously performed with step S13a.

[0098] The first EOG compression submodule 13b comprises a first EOG max-pooling layer and a first EOG dropout layer.

[0099] Specifically, the first EOG max-pooling layer is set to have a convolution kernel size of 8 and a step size of 8, and the first EOG dropout layer is set to have a dropout rate of 0.5.

[0100] The EOG activation submodule 14b is configured to perform step S14b of performing activation processing on the EOG information deepening feature to obtain an EOG deep layer feature.

[0101] The EOG activation submodule 14b is synchronously operated with the EEG activation submodule 14a, and step S14b is synchronously performed with step S14a.

[0102] The EOG activation submodule 14b comprises a plurality of continuous EOG convolution layers. i The EOG component of the i-th EEG data is calculated as follows:

[0103]

[0104] where s represents the component belonging to EOG. θ represents a parallel CNN, and CNN θ The EOG component of CNN converts each EOG time period into a feature vector, and θ is a learnable parameter of CNN.

[0105] Specifically, three EEG convolution layers are adopted, and the number of convolution kernels is set to 128, the convolution kernel size is set to 8, and the step size is set to 1.

[0106] The second EOG compression submodule 15b is configured to perform step S15b: selecting and compressing EOG deep features to obtain an EOG feature map.

[0107] The second EOG compression submodule 15b is configured to perform step S15b: selecting and compressing EOG deep features to obtain an EOG feature map.

[0108] The second EOG compression submodule 15b includes a second EOG max pooling layer and a second EOG dropout layer. The second EOG max pooling layer performs dimension compression on EOG shallow features by pooling, and then takes the maximum value of each element in the compressed matrix to retain the most important feature information in the EOG deep features. The second EOG dropout layer randomly sets the input elements to 0 at a certain dropout rate to reduce overfitting while retaining the most important feature information in the EOG deep features. The second EOG dropout layer and the second EOG max pooling layer jointly compress and select EOG deep features to obtain an EOG feature map.

[0109] Specifically, the size of the pooling convolution kernel of the second EOG max pooling layer is set to 4, and the step is set to 4. The dropout rate of the second EOG dropout layer is set to 0.5.

[0110] Further, the automatic sleep staging device further includes the following modules:

[0111] The multiplication fusion module 20 is configured to perform step S2: multiplying the EEG feature map and the EOG feature map to obtain a highlight feature map.

[0112] The multiplication fusion module 20 can multiply the EEG feature map and the EOG feature map to make the confidence of important modal features higher and the confidence of irrelevant features lower, thereby obtaining a highlight feature map with obvious features.

[0113] The addition fusion module 30 is configured to perform step S3: directly adding the highlight feature map, the EEG feature map, and the EOG feature map to obtain an initial fusion feature map.

[0114] The addition fusion module 30 directly adds the highlight feature map, the EEG feature map, and the EOG feature map to increase the basic feature weight of the EEG feature map and the EOG feature map themselves, preventing the highlight feature map from having too high a weight on the most important features and ignoring other weaker related features, thereby obtaining an initial fusion feature map X fuse ∈R C×H×W where C represents the total number of channels, H and W represent the height and weight of the features, respectively.

[0115] Specifically, the initial fusion feature map can be represented by a feature vector a i is calculated

[0116]

[0117] wherein X fuse is the initial fusion feature map, and “·” represents element multiplication.

[0118] Specifically, the size of the initial fusion feature map is (data batch size, feature map width, feature map length, channel number), wherein the data batch size is 300, the feature map width is 16, the feature map length is 3, and the channel number is 12.

[0119] The weight adjustment module 40 is configured to perform step S4: performing a weight adjustment process on the initial fusion feature map to obtain the final weight feature map.

[0120] The weight adjustment module 40 adjusts the weight of the feature through the attention mechanism, so that the model can better learn the correlation and important features between the sleep EEG and EOG channels, and improve the fitting ability of the model.

[0121] The weight adjustment module 40 specifically includes:

[0122] The weight compression submodule 41 is configured to perform step S41: performing global average pooling compression and reshaping on the initial fusion feature map to obtain the initial weight feature map.

[0123] The weight compression submodule 41 includes a global average pooling layer, which performs global average pooling on the initial fusion feature map to obtain an initial weight feature map with a size of 1×1×C, thereby reducing the spatial dimension of the fusion feature, so that the model can focus on global information, and the weighting coefficient GAP of the global average pooling layer is calculated.

[0124] The first transformation submodule 42 is configured to perform step S42: performing nonlinear transformation on the initial weight feature map to obtain the intermediate weight feature map.

[0125] The first transformation submodule 42 includes a first fully connected layer, which performs nonlinear transformation on the initial weight feature map using an existing ReLU activation function to obtain the intermediate weight feature map, and calculates the weighting coefficient FC of the first fully connected layer. R .

[0126] Specifically, the convolution kernel size of the first fully connected layer is set to 32.

[0127] The second transformation submodule 43 is configured to perform step S43: performing nonlinear transformation on the intermediate weight feature map to obtain the final weight feature map.

[0128] The second transformation submodule 43 includes a second fully connected layer. The second fully connected layer uses a sigmoid activation function to perform a nonlinear transformation on the intermediate weight feature map to obtain the final weight feature map, and calculates the weighting coefficients FC of the second fully connected layer. S .

[0129] Specifically, the kernel size of the second fully connected layer is set to 128.

[0130] The final weighted feature map can be calculated from the initial fused feature map and the weighting coefficients of the global average pooling layer, the first fully connected layer, and the second fully connected layer.

[0131] X att =(FC S (FC R (GAP(X fuse )))

[0132] Among them, FC R The weighting coefficients of the first fully connected layer with the ReLU activation function are represented by FC. S X represents the weighting coefficient of the second fully connected layer with the sigmoid activation function, GAP represents the weighting coefficient of the global average pooling layer, and X represents the weighting coefficient of the second fully connected layer with the sigmoid activation function. att This represents the final weighted feature map.

[0133] The weight fusion module 50 is used to perform step S5: multiply and fuse the initial fusion feature map with the final weight feature map to obtain the final fusion feature map with weight information.

[0134] Compared to the salient feature map, the initial fused feature map has undergone an additive fusion step, which weakens the salientity of the features. The final weighted feature map, compared to the initial fused feature map, has undergone global average pooling, which focuses on global information, making the features even less prominent. Therefore, multiplying the initial fused feature map and the final weighted feature map can yield a final fused feature map with prominent features. At the same time, the final weighted feature map has undergone two nonlinear transformations with different weighting coefficients, so the prominent features of the final fused feature map also have corresponding weights.

[0135] Specifically, the final fused feature map can be directly calculated from the initial fused feature map and the final weighted feature map.

[0136] X′ att =X fuse ·X att

[0137] The temporal extraction module 60 is used to perform step S6: extracting temporal features from the final fused feature map.

[0138] The timing extraction module 60 includes a unidirectional LSTM layer composed of multiple LSTM units, forming an LSTM structure, which can process continuous time units forward to capture the timing features of the sleep signal.

[0139] In the unidirectional LSTM layer, the multiple LSTM units learn the time sequence information between each time unit from the EEG and EOG signals contained in the final fusion feature map, and learn the sleep transition rules of each sleep stage. In order to keep the size of the feature map unchanged, the number of neurons and the hidden size of the forward LSTM are set to the same value, W is 128. Formally, there are N fusion feature vectors {X1,…,XN} arranged in order in the final fusion feature map. n The processing process of the i-th fusion feature vector X i is as follows:

[0140] h i , c i = LSTM θ (h i-1 , c i-1 , X i )

[0141] Where LSTM θ is an LSTM with the fusion feature vector X i as input, θ represents the learnable parameters of the LSTM, h i and c i are the hidden state and cell state vectors about X i obtained by unidirectional processing of the LSTM layer, h i-1 and c i-1 are the hidden state and cell state obtained by unidirectional processing of the feature map X i-1 , h0 and c0 are initialized to 0 vectors.

[0142] Specifically, the input data of the unidirectional LSTM layer, i.e. the final fusion feature map, has a size of (data batch size, sequence length, input dimension), where the data batch size is 300, the sequence length is 20, and the input dimension is the product of the (feature map width, feature map length, channel number) dimensions of the final weight feature map, which is 2048.

[0143] The staging prediction module 70 is used to perform step S7: outputting the prediction probability of each sleep stage according to the weight information of the final fusion feature map and the corresponding timing features.

[0144] The staging prediction module 70 includes a classification layer with 5 neurons, the final fused feature map and the corresponding time sequence feature are input into the classification layer with 5 neurons, the prediction value is calculated by the layer, and the prediction probability of each sleep stage is output, which is in the shape of [5, 1], and the maximum value 0 to 4 corresponds to the sleep stage label as the final sleep stage classification.

[0145] The automatic sleep staging device of the present application adopts a CNN double-channel to directly process EEG original data and EOG original data, greatly reduces manual intervention, uses a multi-modal attention mechanism to adjust the weight of the sleep feature map, fully considers the complementary effect of different modal sleep signals and the contradictory features in the single modal sleep signal, prevents part of the information from being lost in the signal conversion process, uses a unidirectional LSTM layer as a time sequence feature extractor, greatly reduces the network parameter calculation amount, so that the model built by the present application belongs to a lightweight sleep staging model, which has more practical value in actual application scenarios.

[0146] (ii) Model training:

[0147] In order to further improve the evaluation accuracy of the above-mentioned automatic sleep staging device, it needs to be further trained and optimized. Specifically, the training method is as follows:

[0148] Step R1: Obtain sleep data of healthy subjects and input into the initial model of the automatic sleep staging device for training to obtain a healthy group training model.

[0149] In this embodiment, the existing public Sleep-EDF data set is selected as the training database, from which the sleep data of healthy subjects can be directly obtained without the need for electroencephalograph and electrooculograph. The Sleep-EDF data set is a large data set for sleep research, including 197 whole-night sleep polygraph records containing EEG, EOG and other data.

[0150] In the healthy control group of the Sleep-EDF data set, 20 subjects are set, and their EEG original data and EOG original data are analyzed offline and 20-fold cross-validated. The offline data analysis method can process large-scale data sets in an offline state without response, without affecting the model. The 20-fold cross-validation means that in each iteration experiment, a part of the data is used as the test set, and the rest of the data is divided into the training set or the validation set. The data of each round of cross-validation is different, ensuring that the test set is isolated from the training set and the validation set, and each part of the data has the opportunity to become a separate test set. In this embodiment, one subject is set as the test set, 17 subjects are set as the training set, and 2 subjects are set as the validation set in each training.

[0151] The 20 groups of data sets are respectively input into independent initial models to obtain sleep stage prediction results, the sleep stage prediction results are compared with actual sleep stage data of the 20 subjects in the Sleep-EDF data set, and finally the model corresponding to the optimal result is used as the health group training model.

[0152] In step R2, the EEG data processing module 10A, the EOG data processing module 10B and the time sequence extraction module 60 in the initial model are replaced by the corresponding modules of the health group training model, and the EEG data processing module 10A and the EOG data processing module 10B are frozen to obtain an advanced model with effective health characteristics.

[0153] In this step R2, the initial model of the automatic sleep staging process is still used as the model basis, the EEG data processing module 10A and the EOG data processing module 10B are replaced by the EEG data processing module 10A and the EOG data processing module 10B in the health group training model, and the time sequence extraction module 60 is replaced by the time sequence extraction module 60 in the health group training model. Furthermore, the EEG data processing module 10A and the EOG data processing module 10B are frozen so that they will not change in subsequent training, so that the advanced model of the automatic sleep staging process can retain the sleep characteristic knowledge learned on healthy individuals, ensuring that the subsequent model training can effectively perform automatic sleep staging on normal people. The replacement of the time sequence extraction module 60 makes the time sequence features of the subsequent model consistent with those of the previous training model when the model starts training, and the model can have the same baseline.

[0154] In step R3, sleep data of the patient with consciousness disorder is obtained and input into the advanced model of the automatic sleep staging process for training to obtain a multi-modal attention model.

[0155] In this embodiment, the sleep data of the patient with consciousness disorder is obtained by the electroencephalograph and the electrooculograph, the obtained sleep data set is divided into a training set and a test set, 80% of the data is used as the training set, 20% of the data is used as the test set, and the data in the test set is ensured not to exist in the training set, then cross-validation is performed to obtain a plurality of sleep stage prediction results, the sleep stage prediction results are compared with the actual sleep stage data of the subjects, and the model corresponding to the optimal result is used as the multi-modal attention model, i.e. the MCASleepNet model. Specifically, 11 patients with consciousness disorder are tested in this embodiment.

[0156] Through the training and optimization of the automatic sleep staging device of the present application, a multi-modal attention model which can perform automatic sleep staging on normal people and patients with consciousness disorder can be finally obtained. In order to evaluate and verify the actual effect of the multi-modal attention model built by the present application, the following processing method is used for comparison and verification.

[0157] (III) Effect verification:

[0158] Referring to Figure 3 The experimental results of the multi-modal attention model after cross-validation are visualized as a confusion matrix. Each row represents the classification results of the sleep expert, and each column represents the classification results of the proposed model. The main diagonal of the matrix represents the number of samples correctly identified by the multi-modal attention model. From Figure 3 It can be seen that the correct prediction percentage of each sleep stage is higher than other values in the row and column, and the multi-modal attention model of the present application has good accuracy. Since N1 sleep usually accounts for 5%-10% or less of the total sleep time in healthy sleep of adults, the sample of N1 is the smallest in the whole night sleep record, so the prediction of N1 by the present application has errors when the total sample size is small. This can be solved by increasing the sample size, which has little effect on the performance of the multi-modal attention model of the present application.

[0159] Referring to Figure 4 The overall performance of the multi-modal attention model is evaluated by accuracy (ACC) and macro-average F1 score (MF1). Each sleep stage category is measured by precision and MF1 score. Since different sleep stages have different proportions in the sleep cycle, accuracy cannot fully reflect the prediction ability of the model; for example, N1 stage has the smallest data amount in the entire data set, only 6.34%. MF1 will perform the same test on each class, and its value is affected by the amount of samples in each class. Therefore, the present application selects MF1 score as the index for comparison with the baseline, reducing the influence of the difference in sample size of different sleep stages on training

[0160]

[0161] where TP i is the true positive of the i-th class, F1 i is the F1 score of each i-th sleep stage, n is the number of sleep stages, and N is the total number of epochs in the test set.

[0162] The same data set is input into four existing single-channel sleep models and one existing dual-channel sleep model to obtain the corresponding sleep staging prediction results. The MF1 scores of the multi-modal attention model and the five existing models are calculated and compared. It can be seen that the MF1 score of the multi-modal attention model of the present application is the highest, with an accuracy of 89.1%, which is also higher than the five existing common models, proving that the multi-modal attention model of the present application has significantly outstanding overall performance accuracy.

[0163] Referring to Figure 5, a CNN baseline model is built, which has the same CNN structure as the multi-modal attention model, like Figure 5 The blank solid column represents the accuracy of the CNN baseline model, which uses CNN; a bidirectional LSTM baseline model is built, which adds the same bidirectional LSTM structure as the existing DeepSleepNet model to the CNN baseline model, that is, Bi-LSTM, like Figure 5 The diagonal solid column represents the accuracy of the bidirectional LSTM baseline model, which uses CNN+Bi-LSTM; as Figure 5 The horizontal solid column represents the accuracy of the multi-modal attention model of the present application, which uses MCASleepNet. The self-sleep data set of the patient with consciousness disorder is input into the CNN baseline model and the bidirectional LSTM baseline model, the accuracy of the calculation result is calculated, and the accuracy of the multi-modal attention model is compared and evaluated with the accuracy of the CNN baseline model and the accuracy of the bidirectional LSTM model. It can be seen that the accuracy of the multi-modal attention model of the present application is much higher than that of the CNN baseline model, and the whole period is also better than that of the bidirectional LSTM baseline model, and the accuracy effect is the best.

[0164] The present application uses CNN double-channel to directly process EEG original data and EOG original data, greatly reduces manual intervention, uses multi-modal attention mechanism to adjust the weight of sleep feature map, fully considers the complementary effect of different modal sleep signals and the contradictory features in single modal sleep signals, prevents part of information from being lost in the signal conversion process, uses a unidirectional LSTM layer as a time sequence feature extractor, greatly reduces the network parameter calculation amount, has more practical value in actual application scenarios, and after the model is built, multiple training and optimization are performed to obtain a multi-modal attention model which can sleep stage both normal people and patients with consciousness disorder, and solves the problem that the existing model cannot sleep stage the patients with consciousness disorder.

[0165] Based on the above-mentioned automatic sleep staging method, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the automatic sleep staging method according to any one of the above-mentioned embodiments.

[0166] The application can take the form of a computer program product implemented on one or more storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so on) including the program code for use by or in connection with an instruction execution system. The computer readable storage medium can be a permanent or non-permanent, movable or non-movable medium, and can be implemented by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0167] The above-described embodiments only express several embodiments of the present application, which are described in detail and specifically, but cannot be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, and the present application also intends to include these modifications and improvements.

Claims

1. An automatic sleep staging apparatus characterized by: The method comprises the following steps: An EEG raw data processing module is used to process EEG raw data from an electroencephalograph to obtain an EEG feature map; An EOG raw data processing module is used to process EOG raw data from an electrooculograph to obtain an EOG feature map; The EEG raw data processing module and the EOG raw data processing module are synchronously run; A multiplication fusion module is used to multiply the EEG feature map and the EOG feature map to obtain a highlight feature map; An addition fusion module is used to add the highlight feature map, the EEG feature map and the EOG feature map to obtain an initial fusion feature map; A weight adjustment module is used to adjust the weight of the initial fusion feature map to obtain a final weight feature map; A weight fusion module is used to multiply the initial fusion feature map and the final weight feature map to obtain a final fusion feature map with weight information; A stage prediction module is used to output a sleep stage prediction probability according to the weight information of the final fusion feature map.

2. The automatic sleep staging device of claim 1, wherein: Further comprising: A time sequence extraction module is used to extract time sequence features in the final fusion feature map; The stage prediction module is used to output a sleep stage prediction probability according to the weight information and the corresponding time sequence features of the final fusion feature map.

3. The automatic sleep staging device of claim 2, wherein: The EEG raw data processing module comprises: An EEG label filtering submodule is used to label the EEG raw data according to the AAMS standard, and sample and filter the EEG raw data to obtain an EEG self-sampling dataset; An EEG extraction submodule is used to extract EEG shallow features from the EEG self-sampling dataset through convolution; A first EEG compression submodule is used to compress and select EEG shallow features to obtain EEG information deepening features; an EEG activation submodule is used to activate the EEG information deepening features to obtain EEG deep features; and a second EEG compression submodule is used to compress and select the EEG deep features to obtain the EEG feature map; The EOG raw data processing module comprises the following modules: An EOG label filtering submodule is used to label the EOG raw data according to the AAMS standard, and sample and filter the EOG raw data to obtain an EOG self-sampling dataset; An EOG extraction submodule is used to extract EOG shallow features from the EOG self-sampling dataset through convolution; A first EOG compression submodule is used to compress and select EOG shallow features to obtain EOG information deepening features; an EOG activation submodule is used to activate the EOG information deepening features to obtain EOG deep features; and a second EOG compression submodule is used to compress and select the EOG deep features to obtain the EOG feature map.

4. The automatic sleep staging device of claim 3, wherein: The weight adjustment module comprises: A weight compression submodule is used to globally average pool and compress the initial fusion feature map to obtain an initial weight feature map; A first transformation submodule is used to nonlinearly transform the initial weight feature map to obtain an intermediate weight feature map; A second transformation submodule is used to nonlinearly transform the intermediate weight feature map to obtain the final weight feature map.

5. The automatic sleep staging device of claim 4, wherein: The first transformation submodule adopts a ReLU activation function to perform nonlinear transformation on the initial weight feature map to obtain an intermediate weight feature map, and the second transformation submodule adopts a sigmoid activation function to perform nonlinear transformation on the intermediate weight feature map to obtain the final weight feature map.

6. The automatic sleep staging device of claim 5, wherein: The automatic sleep staging device is optimized by the following training method: Step R1: Obtain sleep data of healthy subjects and input the initial model of the automatic sleep staging device for training to obtain a healthy group training model; Step R2: Replace the EEG data processing module, the EOG data processing module and the time sequence extraction module in the initial model with the corresponding modules of the healthy group training model, and freeze the EEG data processing module and the EOG data processing module to obtain an advanced model with effective health characteristics; Step R3: Obtain sleep data of patients with consciousness disorders and input the advanced model of the automatic sleep staging processing for training to obtain a multi-modal attention model.

7. An apparatus for automatic sleep staging, the apparatus comprising: The automatic sleep staging device comprises a brain electrical detector, an eye electrical detector and an automatic sleep staging device according to any one of claims 1 to 6.

8. A method of automatic sleep staging, characterized by: The method comprises the following steps: Step S1a: processing the EEG raw data to obtain an EEG feature map; Step S1b: processing the EOG raw data to obtain an EOG feature map; The step S1b and the step S1a are performed synchronously; Step S2: multiplying the EEG feature map and the EOG feature map to obtain a highlight feature map; Step S3: directly adding the highlight feature map, the EEG feature map and the EOG feature map to obtain an initial fusion feature map; Step S4: performing weight adjustment processing on the initial fusion feature map to obtain a final weight feature map; Step S5: multiplying the initial fusion feature map and the final weight feature map to obtain a final fusion feature map with weight information; Step S7: outputting the prediction probability of each sleep stage according to the weight information of the final fusion feature map.

9. The method of automatic sleep staging according to claim 8, wherein: Further comprising: Step S6: extracting the time sequence features in the final fusion feature map; Step S7: outputting the prediction probability of each sleep stage according to the weight information of the final fusion feature map and the corresponding time sequence features.

10. A computer readable storage medium for automatic sleep staging, the computer readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the automatic sleep staging method of claim 8 or 9.

Images