Sleep stage automatic classification method based on multi-modal significant waveform analysis

Abstract

The invention discloses a sleep stage automatic classification method based on multi-modal significant waveform analysis, and belongs to the technical field of sleep medicine and artificial intelligence. According to the method, multi-mode signals such as electroencephalogram signals and electro-oculogram signals are used as main input, and through the synergistic effect of the frequency feature extraction module, the interactive convolution module and the multi-mode feature fusion module, precise extraction and classification of significant waveforms are achieved. According to the method, frequency domain features are extracted through fast Fourier transform, high-frequency noise is removed in combination with an adaptive filtering technology, and signal features are remarkably enhanced; capturing local and global dependence characteristics of the signal through interactive convolution operation; and multi-modal features are fused through an attention mechanism of a Transform encoder, so that the modeling capability of the time sequence dependency relationship is further improved. The method is excellent in classification accuracy and macro average F1 score, small in parameter quantity and high in calculation efficiency, and can be widely applied to evaluation and automatic diagnosis of clinical sleep disorders.

Description

Technical Field

[0001] The invention relates to an automatic sleep stage classification method based on multimodal significant waveform analysis, and belongs to the technical field of sleep medicine and artificial intelligence. Background Art

[0002] Sleep stage classification is a key task in sleep medicine. Its main purpose is to evaluate sleep quality and assist in the diagnosis and treatment of sleep disorders by analyzing physiological signals during individual sleep. Traditional sleep stage classification relies on polysomnography (PSG), which is a comprehensive method of recording multiple physiological signals during sleep, including electroencephalogram (EEG), electrooculogram (EOG) and electromyogram (EMG). According to the standards of the American Academy of Sleep Medicine (AASM), the sleep process is divided into five main stages: wakefulness (W), rapid eye movement (REM), non-rapid eye movement stage 1 (N1), non-rapid eye movement stage 2 (N2) and non-rapid eye movement stage 3 (N3). Each stage corresponds to specific physiological signal characteristics, such as different waveforms of EEG (delta wave, theta wave, alpha wave, etc.).

[0003] At present, sleep staging is still mainly based on manual labeling. Experts need to analyze 8 to 10 hours of polysomnography data segment by segment and classify each 30-second or 20-second segment based on signal characteristics and stage transition rules. However, this method is not only time-consuming and cumbersome, but also relies on expert experience, which may have certain subjectivity and consistency issues, and it is difficult to meet actual clinical needs.

[0004] In order to improve efficiency and accuracy, automated sleep staging methods based on machine learning and deep learning have gradually become a research hotspot in recent years. These methods can replace manual staging to a certain extent and significantly improve the efficiency of staging by automatically extracting and classifying multimodal physiological signals. However, existing methods still have the following challenges:

[0005] Complexity of multimodal signals: EEG and EOG signals often overlap and change significantly during sleep stage transitions, which makes feature extraction and stage division difficult.

[0006] Insufficient capture of significant waveform features: Key features of different sleep stages, such as sleep spindles, K-complexes, and slow waves, are difficult to accurately identify using traditional methods.

[0007] Imperfect modeling of cross-temporal dependencies: Sleep stages have strong temporal continuity, and there are long-term and short-term dependencies between signals, while existing models are insufficient in capturing such time series dependencies.

[0008] Model efficiency issue: Although some deep learning models have improved in accuracy, their large number of parameters and high computational overhead limit their application in actual clinical scenarios. Summary of the invention

[0009] In order to solve the problems of insufficient multimodal signal feature capture, imperfect time series dependency modeling, and limited classification accuracy and efficiency in the prior art of sleep stage classification, and to accurately extract and classify key waveform features in polysomnography signals, thereby improving the accuracy and efficiency of automated sleep stage classification, the present invention provides a method for automatic sleep stage classification based on multimodal significant waveform analysis, and the technical solution is as follows:

[0010] The sleep stage classification method of the present invention comprises:

[0011] Step 1: Acquire multimodal physiological signals of sleep stages and perform preprocessing;

[0012] Step 2: Use fast Fourier transform to convert the preprocessed physiological signals into the frequency domain, generate a spectrum, and remove low-frequency and high-frequency noise that is not related to the sleep stage;

[0013] Step 3: Filter the spectrum using a global weight filter and a local weight filter to obtain global features and local features respectively, and fuse the global features and local features by weighted summation to generate a unified frequency domain feature representation;

[0014] The global weight filter is used to capture significant patterns of the signal in the overall spectrum, and the local weight filter is used to capture significant patterns of the spectrum related to sleep;

[0015] Step 4: Use inverse fast Fourier transform to restore the fused frequency domain features to time domain signals;

[0016] Step 5: Perform multi-scale parallel convolution on the time domain signal obtained in step 4, respectively use two convolution kernels of different sizes to extract local and global time features, and integrate the local and global time features by element-by-element addition to form time domain features;

[0017] Step 6: Fusion of time domain features of different modalities;

[0018] Step 61: splicing the time domain features of different modes according to the channel dimension, and transforming the spliced ​​features into a unified feature space to ensure that the features of different modes have the same dimension;

[0019] Step 62: Introduce the Transformer encoder and use the multi-head self-attention mechanism to model the correlation between different modalities. Add position information to the features to capture the order information of the time series.

[0020] Step 63: The fusion feature F output by the Transformer encoder att Generate global feature representation through fully connected layers or weighted summation: F global =W fusion F att +b,W fusion and b are trainable parameters of the fusion layer; a linear projection layer is used to compress the fused high-dimensional features into a fixed-size feature representation F final , as the final multimodal fusion feature;

[0021] Step 7: Based on multimodal fusion feature F final , using the trained deep learning model to predict the sleep stage category of each signal segment.

[0022] Optionally, the physiological signals include: electroencephalogram signals and electrooculogram signals.

[0023] Optionally, step 3 further includes: introducing a frequency domain masking mechanism, weighting important frequency bands by weight adjustment, and reducing the weights of irrelevant frequency bands.

[0024] Optionally, in step 5, a 1×1 convolution kernel is used to capture local features, and a 3×3 convolution kernel is used to capture global features.

[0025] The sleep stage classification system of the present invention comprises:

[0026] A signal acquisition module, configured to acquire multimodal physiological signals of sleep stages and perform preprocessing;

[0027] A spectrum generation module is configured to convert the preprocessed physiological signal into a frequency domain using a fast Fourier transform, generate a spectrum, and remove low-frequency and high-frequency noises that are not related to the sleep stage;

[0028] A frequency domain feature generation module is configured to filter the spectrum using a global weight filter and a local weight filter to obtain global features and local features respectively, and fuse the global features and the local features by weighted summation to generate a unified frequency domain feature representation;

[0029] The global weight filter is used to capture significant patterns of the signal in the overall spectrum, and the local weight filter is used to capture significant patterns of the spectrum related to sleep;

[0030] A time domain signal calculation module is configured to restore the fused frequency domain features to a time domain signal using an inverse fast Fourier transform;

[0031] a time domain feature extraction module, configured to perform multi-scale parallel convolution on the time domain signal obtained by the time domain signal calculation module, respectively extract local and global time features using two convolution kernels of different sizes, and integrate the local and global time features using element-by-element addition to form a time domain feature;

[0032] A feature fusion module, configured to fuse time domain features of different modalities;

[0033] A prediction module, configured to predict the sleep stage category of each signal segment using a trained deep learning model based on multimodal fusion features;

[0034] The feature fusion module includes:

[0035] The splicing module is used to splice the time domain features of different modes according to the channel dimension, and transform the spliced ​​features into a unified feature space to ensure that the features of different modes have the same dimension;

[0036] Transformer encoder, which uses a multi-head self-attention mechanism to model the correlation between different modalities and adds position information to the features to capture the order information of the time series;

[0037] The multimodal fusion feature generation module generates the fusion feature F output by the Transformer encoder. att Generate global feature representation through fully connected layers or weighted summation: F global =W fusion F att +b,W fusion and b are trainable parameters of the fusion layer; a linear projection layer is used to compress the fused high-dimensional features into a fixed-size feature representation F final , as the final multimodal fusion feature;

[0038] Optionally, the physiological signals include: electroencephalogram signals and electrooculogram signals.

[0039] Optionally, the frequency domain feature generation module further includes a frequency domain masking mechanism for weighting important frequency bands by weight adjustment and reducing weights of irrelevant frequency bands.

[0040] Optionally, the time domain feature extraction module includes a 1×1 convolution kernel and a 3×3 convolution kernel, which are used to capture local features and global features, respectively.

[0041] The present invention provides a sleep stage classification device, comprising a memory and a processor;

[0042] The memory is used to store computer programs;

[0043] The processor is configured to implement the sleep stage classification method as described in any one of the above items when executing the computer program.

[0044] The present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the sleep stage classification method as described in any one of the above items is implemented.

[0045] The beneficial effects of the present invention are:

[0046] The sleep stage classification method of the present invention extracts frequency domain features through fast Fourier transform, combines adaptive filtering technology to remove high-frequency noise, and significantly enhances signal characteristics; captures local and global dependency characteristics of signals through interactive convolution operations; and fuses multimodal features through the attention mechanism of the Transformer encoder, further improving the modeling ability of time series dependencies.

[0047] Experimental results show that the present invention can effectively classify different sleep stages. The model shows a high recall rate and precision, and has high stability and reliability in different sleep states. The high recall rate of deep sleep (N3) and rapid eye movement (REM) stages shows that the classification model of the present invention has a strong ability to distinguish between these two sleep stages; under multimodal signal input, it can effectively fuse multimodal signals to improve the accuracy of sleep stage classification; through 10-fold cross-validation, the method of the present invention effectively avoids overfitting, improves the generalization ability of the system, and can play an important role in sleep monitoring and clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is an implementation flow chart of the sleep stage classification method based on multimodal significant waveform analysis of the present invention.

[0050] Figure 2 It is a structural diagram of the frequency feature extraction module in the second embodiment of the present invention.

[0051] Figure 3 It is a structural diagram of the interactive convolution module in the second embodiment of the present invention.

[0052] Figure 4 It is a structural diagram of the multimodal feature fusion module in the second embodiment of the present invention.

[0053] Figure 5 This is a schematic diagram of typical application results of the method of the present invention in the classification of sleep stages. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] Embodiment 1:

[0056] This embodiment provides a sleep stage classification method, including:

[0057] Step 1: Acquire multimodal physiological signals of sleep stages and perform preprocessing;

[0058] Step 2: Use fast Fourier transform to convert the preprocessed physiological signals into the frequency domain, generate a spectrum, and remove low-frequency and high-frequency noise that is not related to the sleep stage;

[0059] Step 3: Use global weight filter and local weight filter to filter the spectrum respectively to obtain global features and local features respectively, and fuse the global features and local features by weighted summation to generate a unified frequency domain feature representation;

[0060] The global weight filter is used to capture the significant patterns of the signal in the overall spectrum, and the local weight filter is used to capture the significant patterns of the sleep-related spectrum;

[0061] Step 4: Use inverse fast Fourier transform to restore the fused frequency domain features to time domain signals;

[0062] Step 5: Perform multi-scale parallel convolution on the time domain signal obtained in step 4, use two convolution kernels of different sizes to extract local and global time features, and use element-by-element addition to integrate the local and global time features to form time domain features;

[0063] Step 6: Fusion of time domain features of different modalities;

[0064] Step 61: splicing the time domain features of different modes according to the channel dimension, and transforming the spliced ​​features into a unified feature space to ensure that the features of different modes have the same dimension;

[0065] Step 62: Introduce the Transformer encoder and use the multi-head self-attention mechanism to model the correlation between different modalities. Add position information to the features to capture the order information of the time series.

[0066] Step 63: Fusion feature F output by Transformer encoder att Generate global feature representation through fully connected layers or weighted summation: F global =W fusion F att +b,W fusion and b are trainable parameters of the fusion layer; a linear projection layer is used to compress the fused high-dimensional features into a fixed-size feature representation F final , as the final multimodal fusion feature;

[0067] Step 7: Based on multimodal fusion feature F final , using the trained deep learning model to predict the sleep stage category of each signal segment.

[0068] The sleep stage classification method of this embodiment extracts frequency domain features through fast Fourier transform, combines adaptive filtering technology to remove high-frequency noise, and significantly enhances signal characteristics; captures local and global dependency characteristics of signals through interactive convolution operations; and fuses multimodal features through the attention mechanism of the Transformer encoder, further improving the ability to model time series dependencies.

[0069] This embodiment can effectively classify different sleep stages. The model shows a high recall rate and precision, and has high stability and reliability in different sleep states. The high recall rate of deep sleep (N3) and rapid eye movement (REM) stages shows that the classification model of the present invention has a strong ability to distinguish these two sleep stages; under multimodal signal input, it can effectively fuse multimodal signals to improve the accuracy of sleep stage classification; through 10-fold cross-validation, the method of this embodiment effectively avoids overfitting, improves the generalization ability of the system, and can play an important role in sleep monitoring and clinical diagnosis.

[0070] Embodiment 2:

[0071] This embodiment provides a sleep stage classification method based on multimodal significant waveform analysis, which includes the following steps.

[0072] Step 1: Multimodal signal acquisition.

[0073] In this embodiment, a wearable physiological signal acquisition device or a laboratory monitoring device is used to collect multimodal physiological signal data generated by the user during sleep, including but not limited to electroencephalogram (EEG) signals, electrooculogram (EOG) signals, etc. These signals respectively reflect different physiological activity states during sleep and can provide multi-dimensional feature support for sleep stage classification.

[0074] Step 1.1: Set up the signal acquisition equipment.

[0075] The equipment used for signal acquisition includes:

[0076] Electroencephalogram (EEG) acquisition equipment: used to collect brain electrical activity signals, usually using multi-channel EEG acquisition equipment, such as the electrode arrangement method defined by the international 10-20 system. Commonly used signal channels include:

[0077] EEG1: Frontal electrode, reflecting slow wave activity in EEG signals.

[0078] EEG2: Electrodes in the parietal or occipital region capture the characteristics of EEG activity related to sleep depth.

[0079] Electrooculogram (EOG) equipment: used to collect eye movement signals, usually by placing electrodes near the eye sockets to collect rapid eye movement (REM) and slow eye movement activity characteristics.

[0080] Step 1.2: Set up signal acquisition and parameters.

[0081] Sampling frequency: In order to capture high-quality signal data, the sampling frequency of EEG and EOG signals is generally set between 250Hz and 500Hz to meet the sampling requirements of high-frequency brain waves.

[0082] Signal bandwidth:

[0083] The EEG signal bandwidth is usually set to 0.1Hz to 50Hz, covering the sleep-related slow wave (Delta wave), Theta wave and Alpha wave bands.

[0084] The EOG signal bandwidth is usually set to 0.1Hz to 20Hz to capture low-frequency eye movement characteristics.

[0085] Electrode impedance: During the acquisition process, the impedance between the electrode and the skin needs to be kept below 5kΩ to reduce the impact of noise and artifacts.

[0086] Step 1.3: Signal preprocessing.

[0087] The acquired signal needs to undergo the following preprocessing steps:

[0088] Denoising:

[0089] Use a band-pass filter to remove low-frequency drift (such as breathing and body movement) and high-frequency interference (such as power supply noise).

[0090] Apply artifact removal algorithms to suppress artifacts in electrocardiogram (ECG) and electromyography (EMG) signals.

[0091] Signal Normalization:

[0092] The signal amplitude of each channel was normalized to reduce the impact of signal amplitude differences between individuals on classification.

[0093] Signal synchronization:

[0094] Ensure that the timestamps of EEG and EOG signals are aligned so that the subsequent feature extraction module can effectively fuse these signals.

[0095] Step 1.4: Data recording and annotation.

[0096] Sleep data recording: The duration of signal acquisition generally covers a complete sleep cycle (6-8 hours), and the data is stored as a multi-channel time series file.

[0097] Data annotation: Combined with the polysomnography (PSG) reference standard, professionals annotate the collected signal data with sleep stages to form an annotated data set. The annotation results include the following sleep stages:

[0098] Wake

[0099] Light sleep (N1, N2);

[0100] Deep sleep (N3);

[0101] Rapid eye movement (REM).

[0102] Through the above acquisition and processing steps, high-quality multimodal signal data is obtained, which provides reliable input for subsequent frequency feature extraction and classification modules.

[0103] Step 2: Frequency feature extraction.

[0104] After the multimodal signal acquisition is completed, the frequency feature extraction block (FEB) is used to extract significant frequency features for the signals of each channel, especially the slow wave frequency (Delta wave: 0.5-4Hz), Theta wave (4-8Hz), Alpha wave (8-12Hz) and high-frequency features that are closely related to the sleep stage. This step combines global and local frequency domain analysis methods to capture important information in the frequency features. The specific process is as follows:

[0105] Step 2.1: Perform a Fast Fourier Transform (FFT) on the signal.

[0106] The frequency domain conversion includes the following steps:

[0107] (1) The input time series signal is converted to the frequency domain through Fast Fourier Transform (FFT) to obtain the spectrum representation of the signal.

[0108] (2) The FFT result is in complex form, containing the amplitude and phase information of the frequency components.

[0109] (3) The magnitude spectrum is used for subsequent feature extraction to capture significant frequency patterns in the signal.

[0110] Spectrum range selection:

[0111] (1) Only the frequency range of interest (e.g., 0.5 Hz to 50 Hz) is retained to eliminate low-frequency and high-frequency noise that is not related to sleep stages.

[0112] (2) The energy distribution of the signal in the Delta, Theta, Alpha and other frequency bands is of great significance for classifying sleep stages.

[0113] Step 2.2: Feature filtering.

[0114] The feature filtering module combines global filters and local filters to enhance the spectrum to mine significant frequency features.

[0115] Global Filter, W G ):

[0116] (1) Designed to cover the entire range of the spectrum and act on the global characteristics of the spectrum.

[0117] (2) The global weights are automatically optimized through training and can capture significant patterns of the signal in the overall spectrum, such as changes in the overall spectrum distribution under different sleep stages.

[0118] Local Filter,W L ):

[0119] (1) Focus on the enhancement of certain specific frequency bands (e.g., Delta, Theta, and Alpha waves).

[0120] (2) The filter uses trainable weights that can adaptively highlight key features in different frequency ranges, especially slow wave frequencies related to sleep stages.

[0121] Frequency Domain Masking:

[0122] (1) A frequency domain masking mechanism is introduced into the filter to weight important frequency bands and reduce the weights of irrelevant frequency bands.

[0123] (2) The mask can effectively suppress the influence of artifact interference (such as electrocardiogram and electromyography signals) on the spectrum.

[0124] Step 2.3: Feature fusion.

[0125] Weighted Fusion:

[0126] The global features and local features are fused by weighted summation to generate a unified frequency domain feature representation. The fusion strategy balances the contribution of global and local features through adaptive adjustment.

[0127] Spectral Regularization:

[0128] During the feature fusion process, regularization operations (such as L2 norm) are used to smooth the frequency domain feature distribution and reduce the interference of noise on the classification results.

[0129] Step 2.4: Inverse Fast Fourier Transform (IFFT).

[0130] Time domain signal reconstruction: The fused frequency domain features are restored to time domain signals through Inverse Fast Fourier Transform (IFFT).

[0131] Time domain signal optimization: During the time domain reconstruction process, the time domain signal is further optimized through filters to eliminate possible frequency domain information loss.

[0132] The output signal retains the time domain representation after frequency feature enhancement and provides input for the subsequent time domain feature extraction module.

[0133] Step 2.5: Output features.

[0134] After being processed by the frequency feature extraction module, each signal channel generates a time domain signal feature that integrates frequency information, which is defined as:

[0135] F f =[f1,f2,...,f n ]

[0136] Among them, F f Represents the frequency enhancement characteristics of the signal, f i represents the spectrum feature vector, and n is the length of the signal segment.

[0137] Step 3: Time domain feature extraction.

[0138] After the frequency feature extraction is completed, the time domain feature extraction module is used to further explore the time series characteristics of the signal. The time domain feature extraction module adopts a multi-scale convolution structure to capture patterns at different time scales, comprehensively characterize the dynamic change characteristics of the signal, and provide high-quality input for subsequent multimodal feature fusion.

[0139] The temporal feature extraction module is designed based on the multi-scale parallel convolution module (Inception Convolution Block, ICB), which can capture local and global temporal features at the same time. The specific structure is as follows:

[0140] (1) Input signal: The signal Ft enhanced in the frequency domain and converted back to the time domain in step 2 is input into the ICB module. The shape of Ft is (C, T), where C represents the number of signal channels and T represents the length of the time series.

[0141] (2) Multi-scale convolution kernels: The ICB module introduces two convolution kernels of different sizes to extract local and global temporal features respectively:

[0142] Small-scale convolution kernel (Conv1D_small): The convolution kernel size is 1×1, which is used to capture local features within a short time range, such as fast-fluctuating change patterns.

[0143] Large-scale convolution kernel (Conv1D_big): The convolution kernel size is 3×3, which is used to capture global features over a long period of time, such as periodic slow wave activity.

[0144] (3) Activation function: After the convolution operation, the features are processed by the GELU (Gaussian Error Linear Unit) activation function to improve the model’s nonlinear feature learning ability.

[0145] (4) Feature fusion: The outputs of small-scale and large-scale convolution kernels are fused to integrate local and global features through element-by-element addition.

[0146] The steps of time domain feature extraction are as follows:

[0147] Step 3.1: Small-scale temporal feature extraction.

[0148] The input signal is passed through a small-scale convolution kernel (1×1) to extract features in short time series. The small convolution kernel can efficiently capture the fine-grained features of the time series, such as instantaneous signal fluctuations or local signal mutations. The convolution result is batch normalized and GELU activated to generate a local time feature representation:

[0149] F small =GRLU(BN(Conv1D_small(F t ))

[0150] Among them, F small Represents local time characteristics.

[0151] Step 3.2: Large-scale temporal feature extraction.

[0152] The input signal is passed through a large-scale convolution kernel (3×3) to extract features in the long time series. The large convolution kernel can capture the macroscopic patterns of the time series, such as periodic activities or slow wave characteristics; the convolution result is also batch normalized and GELU activated to generate a global time feature representation:

[0153] F big =GRLU(BN(Conv1D_big(F t ))

[0154] Among them, F big Represents the global time feature.

[0155] Step 3.3: Feature fusion.

[0156] The local feature F small and the global feature F big Perform element-by-element addition fusion to form comprehensive time features:

[0157] F fusion =F small +F big

[0158] The fused features are compressed through an additional convolution kernel (Conv1D) to ensure that the output time domain features have high expression efficiency while reducing the computational complexity: F out =conv1D(F fusion )

[0159] Step 3.4: Feature output.

[0160] The output of the time domain feature extraction module is an efficient time series feature representation with a shape of (C′, T′), where: C′ represents the number of feature channels after compression, which is usually less than the number of input channels C; T′ represents the length of the time series, which is consistent with the input length T.

[0161] The output time domain feature F out It will be passed to the multimodal feature fusion module in the next stage for further processing.

[0162] Step 4: Multimodal feature fusion.

[0163] The multimodal feature fusion block (MFFB) is one of the core components of this method. It is mainly used to fuse features from different modalities (such as EEG1, EEG2 and EOG) to generate a unified cross-modal feature representation. Through this module, the complementarity of different modal signals can be fully utilized to improve the accuracy of sleep stage classification.

[0164] Step 4.1: Feature input.

[0165] The input of the multimodal feature fusion module is the multimodal time domain feature F extracted from the previous stage. EEG1 、F EEG2 、F EOG , its shape is: F i ∈R C×T .

[0166] Where: i represents the signal channel or modality (such as EEG1, EEG2 or EOG), C represents the number of feature channels, and T represents the length of the time series.

[0167] Features of different modalities are generated through independent feature extraction modules, so they are relatively independent in the feature space and need to be processed uniformly through a fusion module.

[0168] Step 4.2: Multimodal feature fusion.

[0169] Multimodal feature fusion is divided into three main stages: preliminary fusion, cross-modal correlation modeling, and global feature generation.

[0170] Step 4.2.1: Initial fusion.

[0171] (1) Feature concatenation:

[0172] First, the time domain features of different modes are concatenated according to the channel dimension:

[0173] F cat =Concat(F EEG1 ,F EEG2 ,F EOG )

[0174] Among them, F cat The shape is R (3C)×T ,The spliced ​​features retain the independence of different modes, facilitating subsequent modeling.

[0175] (2) Feature Transformation (Linear Transformation):

[0176] Use a 1D convolution or linear projection layer to transform the concatenated features into a unified feature space, ensuring that features of different modalities have the same dimension C′:

[0177] F trans =W linear F cat +b

[0178] Among them, W linearand b are trainable parameters.

[0179] Step 4.2.2: Modeling cross-modal correlations.

[0180] (1) Introducing the Transformer encoder:

[0181] Use the Multi-Head Attention mechanism to model the correlation between different modalities, with the input being F trans , the output is the fused feature representation F att : F att = Transformer(F trans ).

[0182] The specific mechanisms of Transformer include:

[0183] Multi-Head Attention: Captures the interdependence between modalities by calculating the correlation matrix between the features of each modality.

[0184] Residual Connection: Retains the original information of the input features and enhances the stability of the network.

[0185] Feed-Forward Network (FFN): further extracts deep features between modalities.

[0186] The core equation of the Transformer encoder is:

[0187]

[0188] Where Q, K, and V represent query, key, and value matrices, respectively. k is the dimension of the key.

[0189] (2) Positional Encoding:

[0190] The feature adds position information to capture the order information of the time series. The position encoding formula is:

[0191]

[0192] Among them, t is the time step and i is the dimension index.

[0193] Step 4.2.3: Global feature generation.

[0194] (1) Feature weighting:

[0195] The fusion feature F output by the Transformer encoder att Generate global feature representation through fully connected layers or weighted summation: F global =W fusion F att +b,W fusion and b are trainable parameters of the fusion layer, and the weighted approach can dynamically adjust the contribution of different modalities to the final classification result.

[0196] (2) Feature compression:

[0197] A linear projection layer is used to compress the fused high-dimensional features into a fixed-size feature representation F. final , the shape is R C″×T′ , this feature is represented as the final multimodal fusion feature and is used for the next classification task.

[0198] Step 4.3: Output features.

[0199] The output of the multimodal feature fusion module is a unified feature representation F final , F final ∈R C″×T′ , where: C″ represents the number of feature channels after fusion; T′ represents the length of the time series, which is consistent with the input time dimension T.

[0200] This feature representation integrates the time domain and frequency domain information of all input modalities, providing global semantic feature support for the classifier module.

[0201] Step 5: Classify sleep stages.

[0202] After completing the multimodal feature fusion, the classification module inputs the extracted high-dimensional global features into the fully connected layer and the classification layer, and maps the features to different sleep stage categories through the Softmax activation function. The goal of the classification module is to classify the input signal of each time period and output the corresponding sleep stage label (such as light sleep, deep sleep, rapid eye movement sleep (REM) and awake state).

[0203] The input of the classification module is the output feature F of the multimodal feature fusion module final , its shape is: F final ∈R C ″×T′ , where: C″ represents the number of feature channels after fusion; T′ represents the length of the time series. In order to achieve the classification task, F final It is input into the classification module, which includes feature compression, classification mapping and Softmax activation steps.

[0204] The composition and processing steps of the classification module are as follows:

[0205] Step 5.1: Feature compression.

[0206] F final Through a fully connected layer, the high-dimensional features are compressed into a low-dimensional feature representation that matches the classification task:

[0207] F compressed =W FC F final +b FC

[0208] Where: W FC ∈R N×C″ is the weight matrix of the fully connected layer; b FC∈ R N is the bias vector; N represents the number of categories of the classification target (that is, the number of categories of sleep stages, for example, 5 categories: awake, N1, N2, N3, REM).

[0209] Output F after feature compression compressed The shape is R N×T′ .

[0210] Use nonlinear activation functions (such as ReLU or LeakyReLU) to further improve feature expression capabilities and avoid over-linearization of features; enhanced =ReLU(F compressed ), non-linear activation functions can effectively capture complex patterns in input features.

[0211] Step 5.2: Classification mapping.

[0212] Step 5.2.1: Classification score calculation.

[0213] Enhanced feature F enhanced Mapped to the classification score space through a fully connected layer: S = W cls F enhanced +b cls .

[0214] Where: W cls ∈R K×N is the weight matrix of the classification layer; b cls ∈R K is the bias vector; K represents the specific number of sleep stage classifications (for example: 5 stages - awake, N1, N2, N3, REM).

[0215] Step 5.2.2: Softmax activation.

[0216] In order to map the classification scores to probability distribution, the Softmax activation function is used:

[0217]

[0218] Where: S i represents the score of the i-th category in the classification score space. P(y i |x) represents the probability that the input signal belongs to the i-th category.

[0219] Step 5.2.3: Output classification results.

[0220] The signal of each time period is finally generated through the Softmax layer to generate a category probability distribution, and the category with the largest probability value is taken as the classification result of the current time period: Label = arg max P (y i ∣x).

[0221] The format of the classification results includes the following:

[0222] (1) Classification label:

[0223] The output classification label is the sleep stage identifier corresponding to each time period, for example: Output = {Wake, N1, N2, N3, REM}.

[0224] Each label corresponds to a fixed-length window in the time series.

[0225] (2) Probability distribution:

[0226] The classification module can also output the probability distribution of each stage for further analysis of classification confidence or error correction. For example: Probability Output = {P(Wake), P(N1), P(N2), P(N3), P(REM)}.

[0227] Step 5.3: Training of the classification module.

[0228] The classification module is trained using a supervised learning method, using the cross entropy loss function as the optimization objective:

[0229]

[0230] Where: M is the number of training samples, K is the number of classification categories; y i,k represents the one-hot encoding of the true label of the i-th sample; P(yi,k) is the predicted probability of the k-th class.

[0231] The parameters of the classification module are updated using the Adam optimizer to ensure fast convergence and robustness.

[0232] The learning rate is dynamically adjusted through a scheduling strategy, such as using Learning Rate Decay.

[0233] Step 6: Model validation.

[0234] The present invention performs performance verification on the Sleep-EDF-20 and Sleep-EDF-78 datasets to ensure the robustness and generalization ability of the model on different datasets.

[0235] The Sleep-EDF-20 dataset consists of sleep data from 20 healthy adult volunteers. The dataset mainly contains records from a variety of physiological signals (such as electroencephalogram (EEG), electrooculogram (EOG), and electromyogram (EMG). The sleep data of each participant is recorded in a standard multi-channel sleep monitoring instrument.

[0236] The Sleep-EDF-78 dataset is a larger sleep research dataset that contains sleep monitoring records of 78 healthy adults. Similar to the Sleep-EDF-20 dataset, the Sleep-EDF-78 dataset contains multi-channel physiological signals and has detailed sleep stage annotations. This dataset covers a wider range than the Sleep-EDF-20 dataset, with participants ranging in age from 20 to 80 years old, covering a wider range of sleep patterns and individual differences.

[0237] In order to avoid overfitting and fully evaluate the performance of the model, the k-fold cross-validation method is used. The specific steps are as follows:

[0238] Dataset division:

[0239] Both Sleep-EDF-20 and Sleep-EDF-78 datasets are preprocessed to remove artifacts, perform signal normalization, and divide the continuous sleep signals into 30-second segments for training and testing.

[0240] For each data set, the k-fold cross-validation method is used, k = 10, that is, the data set is randomly divided into 10 subsets, 9 subsets are used for training, and the remaining 1 subset is used for validation. This is repeated 10 times to ensure that each subset participates in training and validation.

[0241] Training and validation process:

[0242] In each round of cross-validation, the time domain and frequency domain features from EEG and EOG signals are used, processed by the frequency feature extraction module, the interactive convolution module, and modeled using the multimodal feature fusion module.

[0243] During the training process, the Adam optimizer (β1=0.9, β2=0.999) was used, the initial learning rate was set to 0.001, and the learning rate decay strategy was set to decay by 0.0001 every 100 epochs.

[0244] Performance indicators:

[0245] The performance of the model is evaluated using metrics such as accuracy, recall, precision, and F1 score.

[0246] Accuracy: Evaluate the classification accuracy of all test samples.

[0247] Recall rate: The recall ability of each sleep stage reflects the model's ability to recognize each sleep stage.

[0248] Precision: The proportion of samples classified as a certain sleep stage that actually belong to that stage.

[0249] F1 score: The harmonic average of precision and recall, a comprehensive measure of classification performance.

[0250] Performance Results:

[0251] Performance on the Sleep-EDF-20 dataset:

[0252] Using 10-fold cross validation, the average result in each fold is:

[0253] Accuracy: 87.9%.

[0254] Recall rate: It is higher in the deep sleep stage (N3) and rapid eye movement (REM) stage, at 91% and 88% respectively.

[0255] Accuracy: The accuracy for the light sleep (N1) stage is 85%, while the accuracy for deep sleep (N3) is higher at 89%.

[0256] F1 score: It performs better in the light sleep (N2) stage, reaching 0.87.

[0257] Performance on the Sleep-EDF-78 dataset:

[0258] Using 10-fold cross validation, the average results are as follows:

[0259] Accuracy: 85.7%.

[0260] Recall rate: The recall rate in the rapid eye movement (REM) stage is higher, reaching 88%; the recall rate in the deep sleep (N3) stage is 85%.

[0261] Accuracy: The accuracy for the light sleep (N1) stage is higher, reaching 87%.

[0262] F1 score: The F1 scores of deep sleep (N3) and REM stages are 0.86 and 0.84 respectively.

[0263] Cross-validation results on the Sleep-EDF-20 and Sleep-EDF-78 datasets show that the proposed method can effectively classify different sleep stages, especially in deep sleep and rapid eye movement stages. The model exhibits high recall and precision, verifying the stability and reliability of the system under different sleep states.

[0264] The high recall rates in deep sleep (N3) and rapid eye movement (REM) stages indicate that the model has strong discrimination ability in these two sleep stages.

[0265] The high accuracy and F1 score prove that the system can effectively fuse EEG and EOG signals under multimodal signal input and improve the accuracy of sleep stage classification.

[0266] Through 10-fold cross validation, the method of the present invention effectively avoids overfitting and improves the generalization ability of the system. The experimental results show that the automated sleep stage classification system based on multimodal significant waveform analysis can play an important role in sleep monitoring and clinical diagnosis.

[0267] also, Figure 5 This is a schematic diagram of a typical application result of the method of the present invention in sleep stage classification. The figure shows the comparison between the sleep stages predicted by the model and the actual labels, illustrating the high-precision performance of the method of the present invention in the sleep stage classification task.

[0268] Embodiment three:

[0269] This embodiment provides a sleep stage classification system, including:

[0270] A signal acquisition module, configured to acquire multimodal physiological signals of sleep stages and perform preprocessing;

[0271] A spectrum generation module is configured to convert the preprocessed physiological signal into a frequency domain using a fast Fourier transform, generate a spectrum, and remove low-frequency and high-frequency noises that are not related to the sleep stage;

[0272] The frequency domain feature generation module is configured to filter the spectrum using a global weight filter and a local weight filter to obtain global features and local features respectively, and fuse the global features and the local features by weighted summation to generate a unified frequency domain feature representation;

[0273] The global weight filter is used to capture the significant patterns of the signal in the overall spectrum, and the local weight filter is used to capture the significant patterns of the sleep-related spectrum;

[0274] A time domain signal calculation module is configured to restore the fused frequency domain features to a time domain signal using an inverse fast Fourier transform;

[0275] The time domain feature extraction module is configured to perform multi-scale parallel convolution on the time domain signal obtained by the time domain signal calculation module, respectively extract local and global time features using two different sizes of convolution kernels, and integrate the local and global time features by element-by-element addition to form time domain features;

[0276] A feature fusion module, configured to fuse time domain features of different modalities;

[0277] A prediction module, configured to predict the sleep stage category of each signal segment using a trained deep learning model based on multimodal fusion features;

[0278] The feature fusion module includes:

[0279] The splicing module is used to splice the time domain features of different modes according to the channel dimension, and transform the spliced ​​features into a unified feature space to ensure that the features of different modes have the same dimension;

[0280] Transformer encoder, which uses a multi-head self-attention mechanism to model the correlation between different modalities and adds position information to the features to capture the order information of the time series;

[0281] The multimodal fusion feature generation module generates the fusion feature F output by the Transformer encoder. att Generate global feature representation through fully connected layers or weighted summation: F global =W fusion F att +b,W fusion and b are trainable parameters of the fusion layer; a linear projection layer is used to compress the fused high-dimensional features into a fixed-size feature representation F final , as the final multimodal fusion feature.

[0282] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0283] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A sleep stage classification method, characterized in that: The method comprises: Step 1: Acquire multimodal physiological signals of sleep stages and perform preprocessing; Step 2: Use fast Fourier transform to convert the preprocessed physiological signals into the frequency domain, generate a spectrum, and remove low-frequency and high-frequency noise that is not related to the sleep stage; Step 3: Filter the spectrum using a global weight filter and a local weight filter to obtain global features and local features respectively, and fuse the global features and local features by weighted summation to generate a unified frequency domain feature representation; The global weight filter is used to capture significant patterns of the signal in the overall spectrum, and the local weight filter is used to capture significant patterns of the spectrum related to sleep; Step 4: Use inverse fast Fourier transform to restore the fused frequency domain features to time domain signals; Step 5: Perform multi-scale parallel convolution on the time domain signal obtained in step 4, respectively use two convolution kernels of different sizes to extract local and global time features, and integrate the local and global time features by element-by-element addition to form time domain features; Step 6: Fusion of time domain features of different modalities; Step 61: splicing the time domain features of different modes according to the channel dimension, and transforming the spliced ​​features into a unified feature space to ensure that the features of different modes have the same dimension; Step 62: Introduce the Transformer encoder and use the multi-head self-attention mechanism to model the correlation between different modalities. Add position information to the features to capture the order information of the time series. Step 63: The fusion feature F output by the Transformer encoder att Generate global feature representation through fully connected layers or weighted summation: F global =W fusion F att +b,W fusion and b are trainable parameters of the fusion layer; a linear projection layer is used to compress the fused high-dimensional features into a fixed-size feature representation F final , as the final multimodal fusion feature; Step 7: Based on multimodal fusion feature F final , using the trained deep learning model to predict the sleep stage category of each signal segment.

2. The sleep stage classification method according to claim 1, characterized in that: The physiological signals include: electroencephalogram signals and electrooculogram signals.

3. The sleep stage classification method according to claim 1, characterized in that: The step 3 also includes: introducing a frequency domain masking mechanism, weighting important frequency bands by weight adjustment, and reducing the weights of irrelevant frequency bands.

4. The sleep stage classification method according to claim 1, characterized in that: The step 5 uses a 1×1 convolution kernel to capture local features and a 3×3 convolution kernel to capture global features.

5. A sleep stage classification system, characterized in that: The system comprises: A signal acquisition module, configured to acquire multimodal physiological signals of sleep stages and perform preprocessing; A spectrum generation module is configured to convert the preprocessed physiological signal into a frequency domain using a fast Fourier transform, generate a spectrum, and remove low-frequency and high-frequency noises that are not related to the sleep stage; A frequency domain feature generation module is configured to filter the spectrum using a global weight filter and a local weight filter to obtain global features and local features respectively, and fuse the global features and the local features by weighted summation to generate a unified frequency domain feature representation; The global weight filter is used to capture significant patterns of the signal in the overall spectrum, and the local weight filter is used to capture significant patterns of the spectrum related to sleep; A time domain signal calculation module is configured to restore the fused frequency domain features to a time domain signal using an inverse fast Fourier transform; a time domain feature extraction module, configured to perform multi-scale parallel convolution on the time domain signal obtained by the time domain signal calculation module, respectively extract local and global time features using two convolution kernels of different sizes, and integrate the local and global time features using element-by-element addition to form a time domain feature; A feature fusion module, configured to fuse time domain features of different modalities; A prediction module, configured to predict the sleep stage category of each signal segment using a trained deep learning model based on multimodal fusion features; The feature fusion module includes: The splicing module is used to splice the time domain features of different modes according to the channel dimension, and transform the spliced ​​features into a unified feature space to ensure that the features of different modes have the same dimension; Transformer encoder, which uses a multi-head self-attention mechanism to model the correlation between different modalities and adds position information to the features to capture the order information of the time series; The multimodal fusion feature generation module generates the fusion feature F output by the Transformer encoder. att Generate global feature representation through fully connected layers or weighted summation: F global =W fusion F att +b,W fusion and b are trainable parameters of the fusion layer; a linear projection layer is used to compress the fused high-dimensional features into a fixed-size feature representation F final , as the final multimodal fusion feature.

6. The sleep stage classification system according to claim 5, characterized in that: The physiological signals include: electroencephalogram signals and electrooculogram signals.

7. The sleep stage classification system according to claim 5, characterized in that: The frequency domain feature generation module also includes a frequency domain masking mechanism for weighting important frequency bands through weight adjustment and reducing the weights of irrelevant frequency bands.

8. The sleep stage classification system according to claim 5, characterized in that: The time domain feature extraction module includes a 1×1 convolution kernel and a 3×3 convolution kernel, which are used to capture local features and global features respectively.

9. A sleep stage classification device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the sleep stage classification method according to any one of claims 1 to 4 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the sleep stage classification method according to any one of claims 1 to 4 is implemented.

Images