Sleep staging method based on multi-modal attention
By adopting a multi-modal attention mechanism in the sleep staging method and fusing the characteristics of ECG and EEG signals, the shortcomings of existing methods in feature robustness and fusion are solved, and higher sleep staging accuracy and Personalized identification.
Patent Information
- Application Number
- CN202411395330.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-01-24
AI Technical Summary
The existing sleep staging methods have shortcomings in terms of characteristic robustness and key feature fusion, making it difficult to effectively distinguish between N1 and N2 stages.
Adopt the method of sleep staging based on multimodalism, to extract ECG and Electronic signals through one -dimensional convolutional neural networks and multi -scale features, and use multi -modal attention fusion modules to integrate these characteristics to generate more Accurate sleep staging results.
It improves the accuracy and reliability of sleep staging, can automatically learn the feature importance of different modal signals in different sleep stages, and achieves more refined and personalized sleep state recognition.
Smart Images
Figure CN119344669A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of sleep analysis, and in particular relates to a sleep staging method based on multimodal attention. Background Art
[0002] In the field of medicine and sleep research, sleep staging is an important diagnostic process that involves breaking down sleep recordings into different sleep stages, including wakefulness, light sleep, deep sleep, and REM (rapid eye movement) sleep. This process is crucial for diagnosing sleep disorders such as sleep apnea, insomnia, and narcolepsy. Traditional sleep staging methods rely on sleep technicians to manually perform visual analysis of signals such as electroencephalogram (EEG), electrooculogram (EOG), and electromyogram (EMG). This method is not only time-consuming, but also may have consistency issues due to its reliance on the subjective judgment of technicians. With the development of technology, automatic sleep staging technology has emerged, aiming to automate this process through algorithms. These technologies usually use machine learning algorithms, such as support vector machines (SVM), neural networks, or deep learning models, to identify and classify sleep stages. Automated methods can not only improve the efficiency and accuracy of staging, but also reduce human errors and provide more consistent and reliable data. However, the single EEG signal commonly used in existing studies is often difficult to fully capture the complexity and dynamic changes of sleep states, especially for the distinction between N1 and N2 stages. Therefore, sleep staging methods based on multimodality often have higher research value. Summary of the invention
[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a sleep staging method based on multimodal attention, which solves the problems faced by the prior methods, namely, poor feature robustness and difficulty in effectively fusing key features.
[0004] In order to achieve the above-mentioned purpose of the invention, the technical solution adopted by the present invention is: a sleep staging method based on multimodal attention, comprising the following steps:
[0005] S1: acquiring original ECG data and original EEG data, and performing preprocessing to obtain preprocessed ECG data and preprocessed EEG data respectively;
[0006] S2: Based on the preprocessed ECG data, a one-dimensional convolutional neural network is used to process and extract ECG features;
[0007] S3: Based on the preprocessed EEG data, a multi-scale feature extraction module is used to process and extract EEG features;
[0008] S4: Based on the ECG features and EEG features, the multimodal attention fusion module is used to fuse them to obtain the fusion features of the ECG signal and the EEG signal;
[0009] S5: Process the fusion features of the ECG signal and the EEG signal to obtain the sleep staging result.
[0010] The beneficial effects of the present invention are as follows: the present invention establishes a sleep staging model based on a multimodal attention mechanism, fully utilizes the complementary information in electrocardiogram and electroencephalogram signals, improves the accuracy and reliability of sleep staging, and through the attention mechanism, the sleep staging model can automatically learn the feature importance of different modal signals in different sleep stages, thereby achieving more refined and personalized sleep state recognition, which is practical and reliable.
[0011] Further: the specific steps of S1 are as follows:
[0012] S101: Acquire original ECG data and original EEG data;
[0013] S102: According to the original ECG data, denoising is performed using bandpass filters with upper and lower cutoff frequencies, and baseline drift is removed using median filtering and curve fitting to obtain preprocessed ECG data;
[0014] S103: According to the original EEG data, a fourth-order Butterworth bandpass filter is used to filter out noise and artifacts to obtain preprocessed EEG data.
[0015] The beneficial effect of the above further solution is that various artifacts and noises of the original EEG data and the original ECG signal can be removed through preprocessing, thereby improving the reliability and accuracy of the data.
[0016] Further: the specific steps of S2 are as follows:
[0017] S201: Calculate the R peak position by using Hamilton path algorithm according to the preprocessed ECG data;
[0018] S202: Calculate the RR interval and the R peak amplitude according to the R peak position;
[0019] S203: using a cubic spline interpolation method to process the RR interval and the R peak amplitude to obtain the RR interval and the R peak amplitude with the same sampling rate;
[0020] S204: According to the RR interval and R peak amplitude with the same sampling rate, a one-dimensional convolutional neural network is used to extract features to obtain ECG features.
[0021] The beneficial effect of the above further scheme is that the scheme can reduce the interference of characteristic signals unrelated to sleep in the ECG data by focusing on the R peak interval and amplitude in the ECG data, so as to make the model more robust and accurate.
[0022] Further: the specific steps of S3 are as follows:
[0023] S301: extracting high-frequency features using a first one-dimensional convolution unit according to the preprocessed EEG data, and extracting low-frequency features using a second one-dimensional convolution unit;
[0024] S302: Connect the high-frequency features and the low-frequency features, and input them into the LSTM long short-term memory neural network for analysis and processing to obtain the temporal correlation of the sleep features;
[0025] S303: Based on the temporal association of the sleep characteristics, a residual network is used for processing to obtain EEG characteristics.
[0026] The beneficial effect of the above further scheme is: by extracting EEG features through two one-dimensional convolutional units, LSTM long short-term memory neural network and residual network, the interference of complex and changeable characteristics of EEG data itself can be avoided, and the effectiveness of EEG feature extraction can be ensured.
[0027] Further: the first one-dimensional convolution unit includes: a one-dimensional convolution of size 50×1, a maximum pooling layer and a Dropout layer connected in sequence, the step size of the one-dimensional convolution of size 50×1 is 6, and the number of convolution kernels is 64;
[0028] The second one-dimensional convolution unit includes: a one-dimensional convolution with a size of 400×1, a maximum pooling layer and a Dropout layer connected in sequence, the step size of the one-dimensional convolution with a size of 400×1 is 6, and the number of convolution kernels is 64.
[0029] The beneficial effect of the above further scheme is: by establishing a first one-dimensional convolution unit, it can be used to extract high-frequency features of EEG data, and by establishing a second one-dimensional convolution unit, it can be used to extract low-frequency features of EEG data.
[0030] Further: the specific steps of S4 are as follows:
[0031] S401: Based on the ECG features and the EEG features, a global average pooling layer is used to process the ECG feature map and the EEG features to obtain the ECG feature map and the EEG features with unified feature dimensions;
[0032] S402: obtaining fusion features of the ECG features and the EEG features through dot multiplication calculation according to the ECG features and the EEG features unified in feature dimensions;
[0033] S403: according to the fusion features of the electrocardiogram features and the electroencephalogram features, a feature enhancement module is used to process the fusion features to obtain enhanced features;
[0034] S404: According to the fusion features after the enhanced features, the self-attention module and the fusion module are used to process and obtain the fusion features of the ECG signal and the EEG signal.
[0035] The beneficial effects of the above further scheme are: different sleep modes have different characteristics. S4 can effectively combine the characteristics of multiple physiological signals, help realize the fusion of ECG and EEG signals, enhance the key features of different modes in sleep staging, and help improve the classification ability of different sleep stages.
[0036] Further: The expression of the multimodal attention fusion module is as follows:
[0037] Output = X EEG +X att′
[0038] X att′ =X fuse ⊙X att
[0039] X att =σ(FC2(δ(FC1(GAP(X fuse )))))
[0040] X fuse =X ECG +X EEG +(X ECG ⊙X EEG )
[0041] Among them, Output is the output of the multimodal attention fusion module, representing the fusion features of ECG and EEG signals, X att′ is the output of the self-attention module, X att X is the fusion feature after strengthening the feature. fuse is the fusion feature map of ECG features and EEG features, X ECG is the ECG feature with unified feature dimension, X EEG is the EEG feature with unified feature dimension, ⊙ is the dot product operator, σ is the ReLU activation function, δ is the sigmoid activation function, FC2 is the second fully connected layer, FC1 is the first fully connected layer, and GAP is the global average pooling operation.
[0042] The beneficial effect of the above further scheme is that through the expression, the internal mechanism of the multimodal attention fusion module can be clarified, thereby improving the practicality and scientificity of the multimodal attention fusion module. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 Flowchart of a sleep staging method based on multimodal attention. DETAILED DESCRIPTION
[0044] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0045] like Figure 1 As shown, a sleep staging method based on multimodal attention comprises the following steps:
[0046] S1: Obtaining original ECG data and original EEG data, and preprocessing them to obtain preprocessed ECG data and preprocessed EEG data respectively; the specific implementation of S1 is as follows:
[0047] S101: Acquire original ECG data and original EEG data;
[0048] S102: According to the original ECG data, denoising is performed using bandpass filters with upper and lower cutoff frequencies, and baseline drift is removed using median filtering and curve fitting to obtain preprocessed ECG data;
[0049] S103: According to the original EEG data, a fourth-order Butterworth bandpass filter is used to filter out noise and artifacts to obtain preprocessed EEG data.
[0050] In one embodiment of the present invention, in view of the different characteristics of ECG and EEG signals, the present invention adopts different preprocessing methods, and preprocesses the original ECG data and the original EEG data at the same time. The preprocessing of physiological signals is crucial for further feature extraction and can avoid the influence of electrical signals generated by other tissues on the features.
[0051] During the EEG data acquisition process, EEG is often mixed with various artifacts generated by non-brain neural tissues, such as electrooculography, electromyography, electrocardiography, biological movement, power frequency interference, etc. These interference signals overlap with the EEG signals and submerge the original waveform characteristics of the EEG signals. Since the EEG frequency range related to human sleep is between 0.5 and 35 Hz, signals beyond this range can be regarded as artifacts. Therefore, a fourth-order Butterworth bandpass filter with a frequency band range of 0.5 to 35 Hz is used to preprocess the raw EEG data to filter out noise and artifacts of the EEG signals.
[0052] During the ECG data collection process, considering that the ECG is mainly affected by high-frequency noise such as myoelectric interference and power frequency interference, a bandpass filter with upper and lower cutoff frequencies of 0.1 Hz and 35 Hz is first used to denoise the original ECG data, and then median filtering and curve fitting are used to remove the baseline drift of the original ECG data to obtain the preprocessed ECG data.
[0053] Reasonable signal time window division helps to learn the characteristics of sleep cycle changes. In the dataset, 30s of physiological signals are generally used as a time window and a sleep stage label is provided. For a total of n physiological signals and labels, a sliding window of length p is used to obtain data input = {epoch i ,epoch i+1 ,…,epoch i+p}, the label corresponding to the data is label = {true i ,true i+1 ,…,true i+p}, i is the number of data. Considering that each complete sleep cycle generally lasts 90 minutes, in order to ensure the effective learning of the changes in the sleep cycle, in this embodiment, the p value is set to 10 and a physiological signal segment is set to 30s, that is, 5 minutes of ECG data and EEG data containing 10 signal segments are selected as the original ECG data and original EEG data.
[0054] S2: Based on the preprocessed ECG data, a one-dimensional convolutional neural network is used to process and extract ECG features. The specific implementation of S2 is as follows:
[0055] S201: Calculate the R peak position by using Hamilton path algorithm according to the preprocessed ECG data;
[0056] S202: Calculate the RR interval and the R peak amplitude according to the R peak position;
[0057] S203: using a cubic spline interpolation method to process the RR interval and the R peak amplitude to obtain the RR interval and the R peak amplitude with the same sampling rate;
[0058] S204: According to the RR interval and R peak amplitude with the same sampling rate, a one-dimensional convolutional neural network is used to extract features to obtain ECG features.
[0059] In one embodiment of the present invention, the Hamilton Hamiltonian path algorithm is first used to find the R peak in the ECG data, and then the position of the R peak is used to calculate the RR interval, that is, the distance between the R peaks, and the amplitude of the R peak is extracted. Since the RR interval and the amplitude are not equal time intervals, cubic spline interpolation is used for processing. iThe fragment gets the input i = {RRI i ,RRA i}, Among them, RRI i is the RR interval extracted from the electrocardiogram, RRA i is the peak value of the R wave, l is the RRI i and RRA i The dimension of the vector, is a 2×l feature matrix; finally, a one-dimensional convolutional neural network is used to extract features and obtain ECG features; first, each input i Two consecutive convolution operations are performed with a convolution kernel of size 5. After each convolution, a 3×3 maximum pooling operation is performed with a step size of 2. At the same time, in order to reduce the risk of overfitting, a pooling layer with an 80% drop rate is introduced between the convolution layer and the fully connected layer. Finally, the ECG features are obtained.
[0060] S3: Based on the preprocessed EEG data, a multi-scale feature extraction module is used to process and extract EEG features;
[0061] The specific implementation of S3 is as follows:
[0062] S301: extracting high-frequency features using a first one-dimensional convolution unit according to the preprocessed EEG data, and extracting low-frequency features using a second one-dimensional convolution unit;
[0063] S302: Connect the high-frequency features and the low-frequency features, and input them into the LSTM long short-term memory neural network for analysis and processing to obtain the temporal correlation of the sleep features;
[0064] S303: Based on the temporal association of the sleep characteristics, a residual network is used for processing to obtain EEG characteristics.
[0065] In one embodiment of the present invention, the first one-dimensional convolution unit includes: a one-dimensional convolution of size 50×1, a maximum pooling layer and a Dropout layer connected in sequence, the step size of the one-dimensional convolution of size 50×1 is 6, the number of convolution kernels is 64, the maximum pooling layer is used to reduce the data dimension and delete redundant information, and the Dropout layer is used to prevent overfitting;
[0066] The second one-dimensional convolution unit includes: a one-dimensional convolution of size 400×1, a maximum pooling layer and a Dropout layer connected in sequence, wherein the step size of the one-dimensional convolution of size 400×1 is 6, the number of convolution kernels is 64, the maximum pooling layer is used to reduce the data dimension and delete redundant information, and the Dropout layer is used to prevent overfitting.
[0067] Connecting high-frequency features and low-frequency features can be described by the following expression:
[0068] F i =Concat(f i1 ,f i2 )
[0069] Among them, F i is the input data of LSTM long short-term memory neural network, f i1 is the high frequency feature, f i2 is a low-frequency feature, and Concat is a concatenation operation.
[0070] S4: Based on the ECG features and EEG features, the multimodal attention fusion module is used to fuse them to obtain the fusion features of the ECG signal and the EEG signal;
[0071] The specific implementation of S4 is as follows:
[0072] S401: Based on the ECG features and the EEG features, a global average pooling layer is used to process the ECG feature map and the EEG features to obtain the ECG feature map and the EEG features with unified feature dimensions;
[0073] S402: Based on the unified ECG features and EEG features of the feature dimensions, a fusion feature of the ECG features and the EEG features is obtained through dot multiplication. Through dot multiplication, the association between the ECG features and the EEG features in the sleep stages can be learned. The expression of the fusion feature of the ECG features and the EEG features is as follows:
[0074] X fuse =X ECG +X EEG +(X ECG ⊙X EEG )
[0075] Among them, X fuse is the fusion feature map of ECG features and EEG features, X ECG is the ECG feature with unified feature dimension, X EEG is the EEG feature with unified feature dimension, ⊙ is the dot product operator;
[0076] S403: Based on the fusion features of the electrocardiogram features and the electroencephalogram features, the feature enhancement module is used to process the fusion features to obtain the enhanced features. The expression of the enhanced features is as follows:
[0077] X att =σ(FC2(δ(FC1(GAP(X fuse )))))
[0078] Among them, X attis the fusion feature after strengthening the feature, σ is the ReLU activation function, δ is the sigmoid activation function, FC2 is the second fully connected layer, FC1 is the first fully connected layer, GAP is the global average pooling operation, X fuse It is a fusion feature map of ECG features and EEG features;
[0079] S404: According to the fusion features after the enhanced features, the self-attention module and the fusion module are used to process to obtain the fusion features of the ECG signal and the EEG signal. The expression of the fusion features of the ECG signal and the EEG signal is as follows:
[0080] Output = X EEG +X att′
[0081] X att′ =X fuse ⊙X att
[0082] Among them, Output is the output of the multimodal attention fusion module, representing the fusion characteristics of ECG and EEG signals, X att′ is the output result of the self-attention module;
[0083] S5: Process the fusion features of the ECG signal and the EEG signal to obtain the sleep staging result;
[0084] In one embodiment of the present invention, the fusion features of the ECG signal and the EEG signal are processed through a fully connected layer and a softmax layer to obtain a sleep staging result.
[0085] The beneficial effects of the present invention are as follows: the present invention establishes a sleep staging model based on a multimodal attention mechanism, fully utilizes the complementary information in electrocardiogram and electroencephalogram signals, improves the accuracy and reliability of sleep staging, and through the attention mechanism, the sleep staging model can automatically learn the feature importance of different modal signals in different sleep stages, thereby achieving more refined and personalized sleep state recognition, which is practical and reliable.
Claims
1. A sleep staging method based on multimodal attention, characterized in that: The following steps are involved: S1: acquiring original ECG data and original EEG data, and performing preprocessing to obtain preprocessed ECG data and preprocessed EEG data respectively; S2: Based on the preprocessed ECG data, a one-dimensional convolutional neural network is used to process and extract ECG features; S3: Based on the preprocessed EEG data, a multi-scale feature extraction module is used to process and extract EEG features; S4: Based on the ECG features and EEG features, the multimodal attention fusion module is used to fuse them to obtain the fusion features of the ECG signal and the EEG signal; S5: Process the fusion features of the ECG signal and the EEG signal to obtain the sleep staging result.
2. The sleep staging method based on multimodal attention according to claim 1, characterized in that: The specific steps of S1 are as follows: S101: Acquire original ECG data and original EEG data; S102: According to the original ECG data, denoising is performed using bandpass filters with upper and lower cutoff frequencies, and baseline drift is removed using median filtering and curve fitting to obtain preprocessed ECG data; S103: According to the original EEG data, a fourth-order Butterworth bandpass filter is used to filter out noise and artifacts to obtain preprocessed EEG data.
3. The sleep staging method based on multimodal attention according to claim 1, characterized in that: The specific steps of S2 are as follows: S201: Calculate the R peak position by using Hamilton path algorithm according to the preprocessed ECG data; S202: Calculate the RR interval and the R peak amplitude according to the R peak position; S203: using a cubic spline interpolation method to process the RR interval and the R peak amplitude to obtain the RR interval and the R peak amplitude with the same sampling rate; S204: According to the RR interval and R peak amplitude with the same sampling rate, a one-dimensional convolutional neural network is used to extract features to obtain ECG features.
4. The sleep staging method based on multimodal attention according to claim 1, characterized in that: The specific steps of S3 are as follows: S301: extracting high-frequency features using a first one-dimensional convolution unit according to the preprocessed EEG data, and extracting low-frequency features using a second one-dimensional convolution unit; S302: Connect the high-frequency features and the low-frequency features, and input them into the LSTM long short-term memory neural network for analysis and processing to obtain the temporal correlation of the sleep features; S303: Based on the temporal association of the sleep characteristics, a residual network is used for processing to obtain EEG characteristics.
5. The sleep staging method based on multimodal attention according to claim 4, characterized in that: The first one-dimensional convolution unit includes: a one-dimensional convolution with a size of 50×1, a maximum pooling layer, and a Dropout layer connected in sequence, the step size of the one-dimensional convolution with a size of 50×1 is 6, and the number of convolution kernels is 64; The second one-dimensional convolution unit includes: a one-dimensional convolution with a size of 400×1, a maximum pooling layer and a Dropout layer connected in sequence, the step size of the one-dimensional convolution with a size of 400×1 is 6, and the number of convolution kernels is 64.
6. The sleep staging method based on multimodal attention according to claim 1, characterized in that: The specific steps of S4 are as follows: S401: Based on the ECG features and the EEG features, a global average pooling layer is used to process the ECG feature map and the EEG features to obtain the ECG feature map and the EEG features with unified feature dimensions; S402: obtaining fusion features of the ECG features and the EEG features through dot multiplication calculation according to the ECG features and the EEG features unified in feature dimensions; S403: according to the fusion features of the electrocardiogram features and the electroencephalogram features, a feature enhancement module is used to process the fusion features to obtain enhanced features; S404: According to the fusion features after the enhanced features, the self-attention module and the fusion module are used to process and obtain the fusion features of the ECG signal and the EEG signal.
7. The sleep staging method based on multimodal attention according to claim 1, characterized in that: The expression of the multimodal attention fusion module is as follows: Output=X EEG +X att′ X att′ =X fuse ⊙X att X att =σ(FC2(δ(FC1(GAP(X fuse ))))) X fuse =X ECG +X EEG +(X ECG ⊙X EEG ) Among them, Output is the output of the multimodal attention fusion module, representing the fusion features of ECG and EEG signals, X att′ is the output of the self-attention module, X att X is the fusion feature after strengthening the feature. fuse is the fusion feature map of ECG features and EEG features, X ECG is the ECG feature with unified feature dimension, X EEG is the EEG feature with unified feature dimension, ⊙ is the dot product operator, σ is the ReLU activation function, δ is the sigmoid activation function, FC2 is the second fully connected layer, FC1 is the first fully connected layer, and GAP is the global average pooling operation.