Sleep stage detection method based on multi-mode and visual transformation network

Abstract

The invention provides a sleep stage detection method based on a multi-modal and visual transformation network, and the method comprises the steps: extracting EEG channel signals and EOG channel signals in PSG, carrying out the feature extraction of different frequency bands through a multi-scale convolutional network, and fusing the multi-modal features to enhance the classification accuracy. Thirdly, the time-space relevance of network modeling signals is transformed through vision, and then different stages of sleep are effectively recognized; the method also provides confidence evaluation of the classification result, and the reliability of the detection result is enhanced. The automatic detection method can be used for clinical sleep staging to assist doctors in evaluating sleep quality and diagnosing sleep disorders.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to a sleep stage detection method based on multimodality and visual transformation network. Background Art

[0002] Sleep staging, also known as sleep classification, refers to the process of classifying different stages of sleep based on the analysis of physiological channels recorded during sleep. The sleep staging system divides sleep into several different stages, each of which is characterized by specific patterns of brain activity, eye movements, muscle tension, and other physiological parameters. According to the American Academy of Sleep Medicine (AASM), the five major categories of sleep stages are W, N1, N2, N3, and REM, and each category segment is a 30S data segment, among which W is the wakefulness stage; N1 is the light sleep stage, which is the transition stage from wakefulness to sleep; N2 is the light sleep stage, which is the main stage of sleep; N3 is the deep sleep stage; REM is the rapid eye movement sleep stage, which is the main stage of dreaming. In the clinical field, the results of sleep staging are considered to be related to brain health status such as sleep quality and memory consolidation. The characteristics of each stage of sleep staging (duration, stage transition frequency) can be used as an important marker for evaluating sleep patterns and diagnosing sleep disorders. Distinguishing sleep stages from a full-night polysomnogram is the basis for obtaining the above characteristics, so it has important clinical research value. However, currently the diagnosis and differentiation of sleep stages in clinical practice still mainly relies on visual inspection of polysomnograms by experts. This manual calibration method is not only highly professional and costly, but also inefficient.

[0003] In recent years, many scholars have explored this field, and automated sleep staging algorithms have emerged and achieved certain success. They can be mainly divided into two categories: classifier-based methods and deep learning-based methods. Classifier-based methods mainly use machine learning classifiers (such as decision trees, support vector machines, and random forests) to directly classify the time-frequency representations and nonlinear features extracted from electroencephalogram (EEG) signals; deep learning-based methods usually use neural network models to automatically extract the morphological features of signal feature waves to determine the category of sleep stages, but these methods basically only focus on EEG signals, and there are few studies that take into account other modal signals (such as electro-oculogram (EOG)). In addition, these automated detection methods still face certain challenges in accuracy, and all studies only have one classification result, and there is no confidence in the research results.

[0004] Therefore, an accurate and efficient automated sleep stage detection method is urgently needed to solve the above problems. Summary of the invention

[0005] The present invention provides a sleep stage detection method based on multimodality and visual transformation network, so as to solve the defect of lack of efficient and accurate technical means for sleep stage detection in the prior art.

[0006] In a first aspect, the present invention provides a sleep stage detection method based on multimodal and visual transformation network, comprising: Extracting EEG channel signals and EOG channel signals from the polysomnogram to obtain initial training data; Preprocessing the initial training data to obtain processed training data; Inputting the processed training data into a classification network constructed based on multimodal feature fusion and visual transformation network for training to obtain a sleep stage detection model; The sleep stage detection model is trained multiple times to obtain result confidence and obtain a final sleep stage detection result.

[0007] According to a sleep stage detection method based on multimodal and visual transformation network provided by the present invention, EEG channel signals and EOG channel signals are extracted from a polysomnogram to obtain initial training data, including: Cutting the EEG channel signal and the EOG channel signal into segment signals of preset duration respectively; The cut signal is filtered based on a preset bandpass filter to obtain an EEG channel signal and an EOG filter signal, and the EEG channel signal and the EOG filter signal are concatenated to form a model input vector as the initial training data.

[0008] According to a sleep stage detection method based on a multimodal and visual transformation network provided by the present invention, initial training data is preprocessed to obtain processed training data, including: Downsampling each channel data in the initial training data to a preset frequency to obtain downsampled channel data; The downsampled channel data is standardized using a Z-score standardization method to obtain standardized data; The standardized data is spliced ​​three-in-one to obtain spliced ​​data, which includes any three fragment data, the number of channels and the sampling frequency in sequence. The label of the spliced ​​data is consistent with the label of the spliced ​​intermediate data, and the splicing moving step is 1. The spliced ​​data is used as the processed training data.

[0009] According to a sleep stage detection method based on multimodal and visual transformation network provided by the present invention, the processed training data is input into a classification network constructed based on multimodal feature fusion and visual transformation network for training to obtain a sleep stage detection model, including: Construct a classification network including a feature extraction module, a feature fusion module and a spatiotemporal information fusion module; Inputting the processed training data into the feature extraction module, and extracting feature maps of different frequency bands using a multi-scale convolutional network; Inputting the feature maps of different frequency bands into the feature fusion module for deep fusion to obtain fusion features; The fused features are input into the spatiotemporal information fusion module for spatiotemporal association, and the sleep stage detection model is output.

[0010] According to a sleep stage detection method based on a multimodal and visual transformation network provided by the present invention, the processed training data is input into the feature extraction module, and a multi-scale convolutional network is used to extract feature maps of different frequency bands, including: Determining that the feature extraction module includes a first-layer encoding module, a second-layer encoding module, and a third-layer encoding module, wherein the first-layer encoding module, the second-layer encoding module, and the third-layer encoding module each include a first convolution block, a first pooling layer, a second convolution block, a third convolution block, and a second pooling layer connected in sequence; The first convolution block, the second convolution block and the third convolution block each include a convolution layer, a batch normalization layer and a ReLU activation function; The first pooling layer and the second pooling layer include a maximum pooling layer and a random dropout layer; Wherein, the first convolution block, the second convolution block and the third convolution block have convolution kernels and strides of different sizes; The processed training data are respectively input into the first layer encoding module, the second layer encoding module and the third layer encoding module to obtain the feature maps of different frequency bands.

[0011] According to a sleep stage detection method based on multimodal and visual transformation network provided by the present invention, the feature maps of different frequency bands are input into the feature fusion module for deep fusion to obtain fusion features, including: Splicing the EEG feature map and the EOG feature map in the feature maps of different frequency bands in the channel dimension to obtain a spliced ​​feature; Using a local attention layer to extract local features from the spliced ​​features to obtain local features; Using a global attention layer to extract global features from the spliced ​​features to obtain global features; The local feature and the global feature are added together to obtain a weighted value through a sigmoid function. The EEG feature map and the EOG feature map are fused and weighted multiple times using the weighted value to output the fused feature.

[0012] According to a sleep stage detection method based on multimodal and visual transformation network provided by the present invention, the fusion feature is input into the spatiotemporal information fusion module for spatiotemporal association, and the sleep stage detection model is output, including: Inputting any three continuous time segment feature maps of the fused features into an embedding layer, segmenting the any three continuous time segment feature maps into a plurality of patches, and performing a convolution operation on the plurality of patches to generate a patch embedding representation; Creating a learnable position embedding parameter from the number of patches and the embedding dimension, adding a classification tag before the patch embedding representation, and adding it to the position embedding parameter to obtain the patch space information; Through the multi-head attention of Transformer, multiple patch spatial information after layer normalization is concatenated, the obtained attention output is added to the input to form a residual connection, the added output is normalized in the second layer, the linear layer of the feedforward network is used for connection and the ReLU activation function is added to add the output of the feedforward network to the input; The output of the last Transformer block is subjected to layer normalization, a vector corresponding to the classification label is extracted from the processed output, and the final result is mapped to the classification space through a linear layer to obtain the sleep stage detection model.

[0013] According to a sleep stage detection method based on a multimodal and visual transformation network provided by the present invention, the sleep stage detection model is trained multiple times to obtain result confidence and obtain a final sleep stage detection result, including: Obtaining the predicted category probability of each module in the sleep stage detection model predicting the same data segment; Accumulate all predicted category probabilities and select the category with the highest probability in the accumulated results as the final predicted category of the data segment; Calculate the standard deviation of each module for the final predicted category; Mapping the standard deviation to a preset interval to obtain the result confidence; The final sleep stage detection result is determined according to the result confidence.

[0014] In a second aspect, the present invention further provides a sleep stage detection system based on a multimodal and visual transformation network, comprising: Constructing a module for extracting EEG channel signals and EOG channel signals from a polysomnogram to obtain initial training data; A preprocessing module is used to preprocess the initial training data to obtain processed training data; A training module, used for inputting the processed training data into a classification network constructed based on multimodal feature fusion and visual transformation network for training to obtain a sleep stage detection model; The confidence module is used to train the sleep stage detection model multiple times to obtain the result confidence and obtain the final sleep stage detection result.

[0015] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a sleep stage detection method based on a multimodal and visual transformation network as described above.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention uses the Fpz-Cz and EOG channel data in the selected PSG as source data, which only needs simple preprocessing to be input into the model to obtain the final sleep stage classification result, without the need for expert prior knowledge assistance; (2) The present invention can provide experts with the confidence level of the model results, so as to facilitate the experts to control the model results, and is expected to assist doctors in sleep staging in the clinical field; (3) Since the N1 stage in the sleep stage category is difficult to distinguish from other categories, the present invention uses multimodal data to assist in the distinction and designs a feature fusion module between modalities to allow the model to pay more attention to the modal features that are more important for sleep stage classification; (4) In order to be more in line with the labeling method of experts (classifying a segment by combining the stages of the previous and next segments), the present invention bundles the data of three consecutive segments and inputs them into the VitNet network. The attention mechanism is used to combine the features of different segments, which greatly improves the accuracy of sleep stage classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is one of the flow charts of the sleep stage detection method based on multimodality and visual transformation network provided by the present invention; Figure 2This is the second flow chart of the sleep stage detection method based on multimodality and visual transformation network provided by the present invention; Figure 3 It is a feature extraction structure diagram provided by the present invention; Figure 4 It is a convolution block structure diagram provided by the present invention; Figure 5 It is a weight matrix block structure diagram provided by the present invention; Figure 6 It is a structural diagram of a feature addition block provided by the present invention; Figure 7 It is a structural schematic diagram of a sleep stage detection system based on a multimodal and visual transformation network provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In view of the problems existing in the prior art, the present invention proposes an automatic detection method for sleep stages based on multimodality and visual transformation network to achieve accurate recognition and classification of sleep stages.

[0021] Figure 1 is one of the flow charts of the sleep stage detection method based on multimodality and visual transformation network provided by an embodiment of the present invention, such as Figure 1 As shown, including: Step 100: extracting EEG channel signals and EOG channel signals from the polysomnogram to obtain initial training data; Step 200: pre-processing the initial training data to obtain processed training data; Step 300: inputting the processed training data into a classification network constructed based on multimodal feature fusion and visual transformation network for training to obtain a sleep stage detection model; Step 400: training the sleep stage detection model multiple times to obtain result confidence and obtain a final sleep stage detection result.

[0022] The embodiment of the present invention extracts the Fpz-Cz channel signal and the EOG channel signal from the polysomnogram, and then cuts them into data of each segment lasting 30 seconds according to the staging rule as the original signal X. 11 and X 12 , and then the original signal X 11 and X 12 Perform bandpass filtering to obtain the filtered signal X 21 and X 22 , and finally the corresponding filtered signal segment X 21 and X 22 Splice into a vector X0 = [X 21 , X 22 ] and after preprocessing, it is used as the input source signal of the model. The label of the W stage in the sleep stage is defined as 0, the label of the N1 stage is defined as 1, the label of the N2 stage is defined as 2, the label of the N3 stage is defined as 3, and the label of the REM stage is defined as 4. Then the training samples are balanced using the sample balancing method. Finally, a classification network based on multimodal feature fusion and visual transformation is designed and the training data set is input into the model for training. The specific steps include: Step 1: Extract the Fpz-Cz channel signal X that is strongly correlated with sleep stage classification from the polysomnogram 11 and EOG channel signal X 12 Considering that the frequency band of the sleep stage is mainly concentrated in the range of 0.5 to 49.5 Hz, the original signal X 11 and X 12 Perform bandpass filtering to obtain a filtered signal. Then convert the filtered signal segment X 21 and X 22 Splice into a vector X0 = [X 21 , X 22 ] as the input source data of the model.

[0023] Step 2: Preprocessing: X in the concatenated vector X0 obtained in step 1 21 and X 22 Then, in order to combine the data before and after the sleep stage, the vector before and after vector X0 are concatenated to obtain a three-in-one vector X = [X -1 , X0, X1], the label of this vector is the label of the intermediate vector.

[0024] Step 3: Input the data processed in step 2 into the classification network based on multimodal feature fusion and visual transformation for training. The classification network based on multimodal feature fusion and visual transformation mainly includes: feature extraction module, feature fusion module and spatiotemporal information fusion module.

[0025] Step 4: Train the previously designed model multiple times to obtain the standard deviation between the model results, then map it to between 0 and 1 to obtain the result confidence, and obtain the final detection result.

[0026] In one embodiment, step 1 includes: Step 1.1: In polysomnography (PSG) recordings, blood oxygen saturation, electroencephalogram (EEG), electrooculogram (EOG), electromyogram (EMG) and electrocardiogram (ECG) are usually collected through electrodes placed on the test subject. When collecting EEG data, a 10-20 international standard lead system is usually used to cover the EEG activity of different parts of the scalp. This system is widely used in sleep research to ensure that EEG signals from multiple brain regions such as the frontal lobe, central region, and occipital lobe are collected. Fpz is located at the frontal pole, and Cz is the vertex of the central region. This channel can capture EEG activity from the frontal lobe to the central region, and is suitable for detecting the transition from the non-ventilated sleep stage; EOG data records the horizontal and vertical movements of the eyeballs through electrodes placed around the eyes. It is mainly used to monitor eye movement information, which can help distinguish rapid eye movement (REM) sleep from non-rapid eye movement (NREM) sleep. When experts are labeling sleep stages, they mainly refer to the EEG data of the central area and combine it with the corresponding electrooculogram data. Therefore, the Fpz-Cz and EOG channel signals are extracted as data sources. According to the sleep staging rules, each segment lasts for 30 seconds, so the Fpz-Cz and EOG channel signals are cut into 30-second segments.

[0027] Step 1.2: The brain waves in the sleep stage are mainly alpha waves (8-13 Hz), beta waves (13-30 Hz), theta waves (4-7 Hz), delta waves (0.5-4 Hz) and gamma waves (30-45 Hz). In order to enhance the ability of the present invention to distinguish the sleep stages, the raw data after cutting is filtered, that is, a filter is used for bandpass filtering, and the cut-off frequencies are set to 0.5 Hz and 49.5 Hz, respectively, to obtain the filtered signal, and then the EEG filtered signal and the EOG filtered signal are spliced ​​into a vector X0= [X 21 ,X 22 ] as the input source data of the model, where X 21 represents the EEG filtered signal, X 22 Represents the EOG filtered signal.

[0028] In one embodiment, step 2 includes preprocessing and assigning sleep stage labels: Step 2.1: For each channel i in vector X, the data corresponding to Downsample to 100 Hz, and then use the Z-score standardization method to standardize the channel data. The specific operation of standardization is shown in formula (1).

[0029] (1) in Representation data The mean of Representation data The standard deviation of It is a channel The results obtained after standardization.

[0030] Step 2.2: Perform three-in-one splicing on the standardized data, the label of the spliced ​​data is consistent with the label of the intermediate data, and the step length of each splicing movement is 1. Therefore, the shape of the model input data is (3, 2, T×fs), where 3 represents three fragments of data, 2 represents two channels, T represents the time length, and fs represents the sampling frequency. In the embodiment of the present invention, T is 30 and fs is 100.

[0031] In one embodiment, step 3 includes designing a classification network based on multimodal feature fusion and visual transformation, specifically including: Step 3.1: Feature extraction module. The classification network designed by the present invention is first a feature extraction module, which is responsible for multi-scale feature extraction of the input signal to capture features at different levels. By using large-scale, medium-scale and small-scale convolution kernels, feature representations in the data are extracted from local to global levels.

[0032] Step 3.1.1: First layer encoding module. The input data first passes through a convolution layer with a kernel size of 1*11, a stride of 2, and an output channel of 64. After the convolution layer, a batch normalization layer (BN), a ReLU activation function, a maximum pooling layer (Max Pooling) and a random dropout layer (Dropout) are used. After that, it passes through two convolution layers with a kernel size of 1*9 and output channels of 128 and 30. After each convolution layer, a batch normalization layer and a ReLU activation function are used. The specific operation is shown in formula (2).

[0033] (2) in Indicates Layer Output feature maps; Indicates In the layer, The output channels are connected to the The convolution kernel weights between the input channels; Indicates In the layer, Bias for each output channel; Represents a convolution operation.

[0034] The parameter of the random dropout layer is set to 0.5, that is, the model sets the output of some neurons to zero with a probability of 0.5 during training, which can improve the generalization ability of the model and prevent overfitting. The convolution kernel of the maximum pooling layer is set to 3, and the stride is set to 3, which can reduce the data dimension while retaining the maximum value of every three adjacent elements and retain its main features. The specific operation is shown in formula (3).

[0035] (3) in Represent three adjacent elements in the input sequence, Represents the output of the max pooling layer.

[0036] Step 3.1.2: Second layer encoding module. The second encoding module is similar to the first encoding module, except that the initial convolution kernel size is 1*50 and the stride is 6.

[0037] Step 3.1.3: The third layer encoding module adopts a similar composition to the previous two, except that the initial convolution kernel size is 1*400 and the stride is 50.

[0038] After passing through the feature extraction module, the data is concatenated in the last dimension.

[0039] Step 3.2: Feature fusion module. The determination of sleep stages does not only rely on EEG data, but experts also refer to electrooculographic data for assistance. This is mainly because EOG can provide information about eye movements and open and closed states, which is particularly important for identifying rapid eye movement (REM) sleep. Combining these two types of data can improve the accuracy and reliability of judgment and reduce misjudgment, especially in the transition stage of the sleep cycle. In addition, multi-channel monitoring (including EEG and EOG) can provide more comprehensive physiological signals, which helps to adapt to individual differences and provide personalized sleep assessment. Therefore, the present invention combines EEG and EOG signals and deeply fuses the features of the two through a feature fusion module. This module aims to dynamically adjust the weights of input features through local and global attention mechanisms, so as to better fuse feature information from different sources. The module includes a local attention layer, a global attention layer, and attention weight calculation.

[0040] Step 3.2.1: Concatenate the input features EEG and EOG in the channel dimension to form new features .

[0041] Step 3.2.2: Local attention. Use the local attention layer to focus on the features Local feature extraction is performed Specifically, a convolution sequence is defined to extract local features, including a 1D convolution layer (mapping the input channel to the intermediate channel, the convolution kernel size is 1*1, and the step size is 1), a batch normalization layer, a ReLu activation function, followed by a 1D convolution layer (mapping the number of intermediate channels back to the original number of channels) and a batch normalization layer.

[0042] Step 3.2.3: Global Attention. Use the global attention layer to focus on features Perform global feature extraction to obtain Specifically, global features are extracted through global average pooling, including adaptive average pooling layer, convolution layer (mapped to the number of intermediate channels), batch normalization layer, ReLu activation function, and finally the number of intermediate channels is mapped back to the original number of channels through the convolution layer and normalized. Compared with the local attention layer, it adds an adaptive average pooling.

[0043] Step 3.2.4: Add local features and global features, calculate the weighted value w through sigmoid, and then fuse the input features according to the weight; repeat the previous steps, use local and global attention to further specialize and fuse features, and finally output features . It is a fused feature that combines the information of the original features EEG and EOG, and dynamically adjusts their weights through the attention mechanism, so that the model can better focus on important feature information.

[0044] (4) (5) (6) in, is local attention, is global attention, is the result of adding the two together. It is calculated by sigmoid The weighted value obtained is is the input feature of EEG, is the input feature of EOG, It is the feature result after adjusting the weighted value. By executing the above process once, you can get the final output features. .

[0045] Step 3.3: Spatiotemporal information fusion module. The EEG data and EOG data in PSG are typical time series information with non-stationary characteristics. In order to accurately identify and detect sleep stages, the spatiotemporal correlation information contained in these signals is crucial. To this end, the present invention specially designs a spatiotemporal information fusion module to model the spatiotemporal information within and between signal features, helping the model to more effectively understand and capture the spatiotemporal features within and between signal features.

[0046] The embodiment of the present invention uses feature maps from three consecutive time segments as input, and captures dynamic changes across time segments through a finely designed feature extraction pipeline. In the input, each channel represents a feature map at a different time step, which enables the model to perceive the temporal dependencies between consecutive frames. On this basis, we use VisionTransformer (ViT) for deep spatial feature modeling. ViT uses its self-attention mechanism to capture the dependencies between image patches globally and provide a fine-grained spatial representation for each time segment. In this way, ViT can not only learn the spatial information within each time step, but also construct a rich spatiotemporal joint representation by integrating features between different time steps. This model architecture that combines multi-temporal input with self-attention mechanism can capture spatiotemporal relationships globally, thereby adaptively modeling the dynamic changes and spatial layout of time segments. Unlike traditional single-frame or frame-by-frame analysis methods, this architecture can achieve more detailed cross-temporal dependency modeling and provide a higher level of understanding of complex spatiotemporal patterns.

[0047] Step 3.3.1: In the embedding layer, the input image is split into multiple patches and the embedded representation of the patches is generated through convolution operations. Specifically, the size of the convolution kernel is set to (15, 10) and the number of output channels is set to embed_dim, which is 1020 in this case. This is done by creating a convolution layer using nn.Conv2d with parameters in_channels=3 (number of input channels) and out_channels=1020 (output embedding dimension), and the stride is the same as the convolution kernel size. After the convolution operation, the shape of the output tensor is (batch_size, 1020, H_out, W_out), where H_out and W_out are the height and width calculated by the convolution and stride.

[0048] Step 3.3.2: In the position embedding step, in order to let the model know the position information of the patch in the image, a learnable parameter pos_embeddings is created with the dimension (1, 1 + num_patches, embed_dim). Here, num_patches is calculated as: (7) A classification token (cls_token) with shape (batch_size, 1, embed_dim) is added before the patch sequence. Subsequently, the classification token is concatenated with the patch sequence and added with the position embedding to provide spatial information of the patch.

[0049] Step 3.3.3: In each Transformer block, the processing flow is as follows. First, the input is layer normalized using nn.LayerNorm(embed_dim) where embed_dim is 1020. Then, the attention score is calculated using a multi-head self-attention mechanism using num_heads=5, and the dimensions of each head are: (8) The output of each head is then concatenated to obtain an attention output of shape (batch_size, 1 + num_patches, embed_dim). Next, the attention output is added to the input to form a residual connection to keep the information flowing. The added output is then normalized for the second layer. Next, a feedforward network FeedForward is used, where the hidden layer dimension hidden_dim is set to 512, connected through two linear layers and a ReLU activation function is added, and the shape of the final output is maintained as (batch_size, 1 + num_patches, embed_dim). Finally, the output of the feedforward network is added to the input to ensure information preservation.

[0050] Step 3.3.4: In the output layer, perform the following operations. First, perform layer normalization on the output of the last Transformer block. Next, extract the vector corresponding to the classification label from the processed output, with shape (batch_size, embed_dim). Then, map the final result to the classification space through a linear layer to_out with input dimension embed_dim and output dimension num_classes (set to the specific number of categories, such as 5). The final output tensor shape is (batch_size, num_classes) and is used for prediction of classification tasks.

[0051] In one embodiment, step 4 obtains confidence through multiple predictions. Because the Dropout layer randomly "discards" some neurons in each forward propagation, the same model structure will also show different prediction capabilities during multiple trainings, thereby providing diversity for the final confidence calculation, including: Step 4.1: Train the models constructed by the three aforementioned embodiments.

[0052] Step 4.2: Predict the probability distribution of the same data segment. Input the same data segment to each model, and the model will output a probability distribution vector , indicating the probability that this data segment belongs to each category. Among them, It is The output of a model represents the probability value for each class it predicts.

[0053] Step 4.3: Calculate the final predicted category. Add the prediction results of the three models together, that is, add up the probabilities of each category to get the cumulative result = + + , then select The category with the highest probability is taken as the final predicted category of the data segment.

[0054] Step 4.4: Calculate the standard deviation of the three models' predictions for this category. The standard deviation is used to measure the consistency of the model's prediction results for this category. The calculation process is as follows: First, calculate the average of the three models' predicted probabilities for this category: (9) in, Indicates The predicted probability of the model for this category is is the number of models, here it is 3. Next, we use the standard deviation formula to measure the degree of deviation of these predicted probabilities from the mean, the formula is: (10) This formula calculates the square of the difference between the predicted probability of each model and the mean, takes the average and then squares it. The smaller the standard deviation, the more consistent the predictions of the three models for that category; the larger the standard deviation, the greater the difference between the model predictions.

[0055] Step 4.5: Map the standard deviation to a range between 0 and 1 to obtain the model's confidence in the final prediction result.

[0056] In one embodiment, the following is described in conjunction with specific drawings and specific implementations: Part 1: PSG data extraction and preprocessing.

[0057] The data of Fpz-Cz and EOG channels in PSG were extracted, and then band-pass filtered with a range of 0.5-49.5. The data were then cut into 30-s segments according to the staging rule, and the Fpz-Cz and EOG channels were spliced; In order to reduce the impact of noise and improve the analysis speed, this embodiment resamples the above signal with a sampling rate of 100 Hz, and then performs standardization to ensure the consistency and comparability of the data, and splices the three consecutive segments, so the data shape of the model input in this embodiment is [n, 3, 2, 3000]. Where n represents batch_size, 3 represents 3 consecutive data segments, 2 represents 2 modal data (EEG and EOG), and 3000 represents 3000 data points for each modal data.

[0058] The second part is model structure design.

[0059] Design a multi-scale feature extraction module, such as Figure 3 and Figure 4 As shown in the figure, the feature extraction module is mainly composed of convolution blocks and pooling layers. There are three small modules in total, each of which consists of three convolution blocks and two pooling layers: the convolution block consists of a one-dimensional convolution layer, a batchnorm layer and a ReLu layer. The convolution kernel sizes of the convolution blocks of the first small module are 11, 9, 9 from top to bottom; the convolution kernel sizes of the convolution blocks of the second small module are 50, 8, 8 from top to bottom. The convolution kernel sizes of the convolution blocks of the third small module are 400, 7, 7 from top to bottom. The number of output channels in these small modules changes from 1 to 64, then to 128, and finally to 30. The pooling layer is the maximum pooling, the convolution kernel size is 3, and the stride is 3. The data of Fpz-Cz and EOG enter different feature extraction modules respectively, and the output dimensions are [n, 30, 80], where n represents the batch size, which is 128 here.

[0060] A feature fusion module is designed, such as Figure 2 , Figure 5 and Figure 6As shown in the figure, first, EEG and EOG features are concatenated and then feature fusion is performed. Feature fusion mainly uses local attention and global attention: the local attention is mainly composed of a convolution block followed by a convolution layer and a batchnorm layer, which are all 1*1 convolution kernels, that is, point convolution; global attention is based on local attention and adds a global average pooling layer. After the results obtained by local attention and global attention are concatenated, the sigmoid layer is connected to obtain the weights of different features. After the weights are combined with the corresponding features and added, the module is continued to obtain new weights, and then the fused features are output. The output fusion feature dimension is [n,30,80]. Figure 2 The fragment concatenation in is to concatenate the feature results obtained from three consecutive data fragments in the channel dimension to obtain [n, 3, 30, 80].

[0061] The ViTNet network structure is used. The patch size of the feature map is (15, 10), the number of input channels is 3, and the embedding dimension is 1020. In the Transformer attention mechanism, multi-head attention is used, with 5 heads, embed_dim is 1020, and hidden_dim is 512. The results of the 5 heads are then concatenated, and the final output is obtained through the linear layer and the ReLu layer. That is, the feature results obtained above are input into the VITNet network to obtain the final classification result [n,5].

[0062] The third part is model training and testing.

[0063] In this embodiment, the model is trained in an end-to-end manner. All sample data are divided into training set, validation set and test set, and the training and testing are performed using the 20-fold cross validation method. Specifically, the stochastic gradient descent optimization algorithm is used for training, a total of 50 epochs, and the early stopping method is used during the training process to prevent the model from overfitting, and the learning rate is set to 0.001.

[0064] Use the above trained models to test the test data. Train three models and use the dropout layer in the model to expand the comprehensive field of view of the three models. Through the comprehensive evaluation of the three models, obtain the confidence of the model test results.

[0065] The sleep stage detection system based on multimodal and visual transformation network provided by the present invention is described below. The sleep stage detection system based on multimodal and visual transformation network described below and the sleep stage detection method based on multimodal and visual transformation network described above can be referenced to each other.

[0066] Figure 7is a structural diagram of a sleep stage detection system based on a multimodal and visual transformation network provided by an embodiment of the present invention, such as Figure 7 As shown, it includes: a construction module 71, a preprocessing module 72, a training module 73 and a confidence module 74, wherein: The construction module 71 is used to extract EEG channel signals and EOG channel signals from the polysomnogram to obtain initial training data; the preprocessing module 72 is used to preprocess the initial training data to obtain processed training data; the training module 73 is used to input the processed training data into a classification network constructed based on multimodal feature fusion and visual transformation network for training to obtain a sleep stage detection model; the confidence module 74 is used to train the sleep stage detection model multiple times to obtain result confidence and obtain the final sleep stage detection result.

[0067] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the sleep stage detection method based on the multimodal and visual transformation network, the method comprising: extracting EEG channel signals and EOG channel signals from the polysomnogram to obtain initial training data; preprocessing the initial training data to obtain processed training data; inputting the processed training data into a classification network constructed based on multimodal feature fusion and visual transformation network for training to obtain a sleep stage detection model; training the sleep stage detection model multiple times to obtain result confidence and obtain the final sleep stage detection result.

[0068] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0069] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sleep stage detection method based on multimodal and visual transformation network, characterized in that: include: Extract EEG channel signals and EOG channel signals from the polysomnogram to obtain initial training data; Preprocessing the initial training data to obtain processed training data; Inputting the processed training data into a classification network constructed based on multimodal feature fusion and visual transformation network for training to obtain a sleep stage detection model; The sleep stage detection model is trained multiple times to obtain result confidence and obtain a final sleep stage detection result.

2. The sleep stage detection method based on multimodal and visual transformation network according to claim 1, characterized in that: Extract EEG channel signals and EOG channel signals from the polysomnogram to obtain initial training data, including: Cutting the EEG channel signal and the EOG channel signal into segment signals of preset duration respectively; The cut signal is filtered based on a preset bandpass filter to obtain an EEG channel signal and an EOG filter signal, and the EEG channel signal and the EOG filter signal are concatenated to form a model input vector as the initial training data.

3. The sleep stage detection method based on multimodal and visual transformation network according to claim 1, characterized in that: Preprocess the initial training data to obtain processed training data, including: Downsampling each channel data in the initial training data to a preset frequency to obtain downsampled channel data; The downsampled channel data is standardized using a Z-score standardization method to obtain standardized data; The standardized data is spliced ​​three-in-one to obtain spliced ​​data, which includes any three fragment data, the number of channels and the sampling frequency in sequence. The label of the spliced ​​data is consistent with the label of the spliced ​​intermediate data, and the splicing moving step is 1. The spliced ​​data is used as the processed training data.

4. The sleep stage detection method based on multimodal and visual transformation network according to claim 1, characterized in that: The processed training data is input into a classification network constructed based on multimodal feature fusion and visual transformation network for training to obtain a sleep stage detection model, including: Construct a classification network including a feature extraction module, a feature fusion module and a spatiotemporal information fusion module; Inputting the processed training data into the feature extraction module, and extracting feature maps of different frequency bands using a multi-scale convolutional network; Inputting the feature maps of different frequency bands into the feature fusion module for deep fusion to obtain fusion features; The fused features are input into the spatiotemporal information fusion module for spatiotemporal association, and the sleep stage detection model is output.

5. The sleep stage detection method based on multimodal and visual transformation network according to claim 4, characterized in that: The processed training data is input into the feature extraction module, and feature maps of different frequency bands are extracted using a multi-scale convolutional network, including: Determining that the feature extraction module includes a first-layer encoding module, a second-layer encoding module, and a third-layer encoding module, wherein the first-layer encoding module, the second-layer encoding module, and the third-layer encoding module each include a first convolution block, a first pooling layer, a second convolution block, a third convolution block, and a second pooling layer connected in sequence; The first convolution block, the second convolution block and the third convolution block each include a convolution layer, a batch normalization layer and a ReLU activation function; The first pooling layer and the second pooling layer include a maximum pooling layer and a random dropout layer; Wherein, the first convolution block, the second convolution block and the third convolution block have convolution kernels and strides of different sizes; The processed training data are respectively input into the first layer encoding module, the second layer encoding module and the third layer encoding module to obtain the feature maps of different frequency bands.

6. The sleep stage detection method based on multimodal and visual transformation network according to claim 4, characterized in that: Inputting the feature maps of different frequency bands into the feature fusion module for deep fusion to obtain fusion features, including: Splicing the EEG feature map and the EOG feature map in the feature maps of different frequency bands in the channel dimension to obtain a spliced ​​feature; Using a local attention layer to extract local features from the spliced ​​features to obtain local features; Using a global attention layer to extract global features from the spliced ​​features to obtain global features; The local feature and the global feature are added together to obtain a weighted value through a sigmoid function. The EEG feature map and the EOG feature map are fused and weighted multiple times using the weighted value to output the fused feature.

7. The sleep stage detection method based on multimodal and visual transformation network according to claim 4, characterized in that: Inputting the fusion features into the spatiotemporal information fusion module for spatiotemporal association and outputting the sleep stage detection model includes: Inputting any three continuous time segment feature maps of the fused features into an embedding layer, segmenting the any three continuous time segment feature maps into a plurality of patches, and performing a convolution operation on the plurality of patches to generate a patch embedding representation; Creating a learnable position embedding parameter from the number of patches and the embedding dimension, adding a classification tag before the patch embedding representation, and adding it to the position embedding parameter to obtain the patch space information; Through the multi-head attention of Transformer, multiple patch spatial information after layer normalization is concatenated, the obtained attention output is added to the input to form a residual connection, the added output is normalized in the second layer, the linear layer of the feedforward network is used for connection and the ReLU activation function is added to add the output of the feedforward network to the input; The output of the last Transformer block is subjected to layer normalization, a vector corresponding to the classification label is extracted from the processed output, and the final result is mapped to the classification space through a linear layer to obtain the sleep stage detection model.

8. The sleep stage detection method based on multimodal and visual transformation network according to claim 1, characterized in that: The sleep stage detection model is trained multiple times to obtain result confidence and obtain the final sleep stage detection result, including: Obtaining the predicted category probability of each module in the sleep stage detection model predicting the same data segment; Accumulate all predicted category probabilities and select the category with the highest probability in the accumulated results as the final predicted category of the data segment; Calculate the standard deviation of each module for the final predicted category; Mapping the standard deviation to a preset interval to obtain the result confidence; The final sleep stage detection result is determined according to the result confidence.

9. A sleep stage detection system based on multimodal and visual transformation network, characterized in that: include: Constructing a module for extracting EEG channel signals and EOG channel signals from a polysomnogram to obtain initial training data; A preprocessing module is used to preprocess the initial training data to obtain processed training data; A training module, used for inputting the processed training data into a classification network constructed based on multimodal feature fusion and visual transformation network for training to obtain a sleep stage detection model; The confidence module is used to train the sleep stage detection model multiple times to obtain the result confidence and obtain the final sleep stage detection result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the sleep stage detection method based on multimodality and visual transformation network as described in any one of claims 1 to 8 is implemented.

Images