A multi-modal data multi-view sleep staging method

Abstract

The application discloses a multi-modal data multi-view sleep staging method. First, the electroencephalogram (EEG) channel, the electrooculogram (EOG) channel and the electromyogram (EMG) channel data of polysomnography (PSG) are processed and analyzed, and the original multi-modal PSG data is processed into three different view signals, namely, one-dimensional signals, two-dimensional time-frequency diagrams and graph topological structures. Then, the combination of the signals is taken as an independent variable, a sleep staging judgment result is taken as a dependent variable, a multi-view fusion model is constructed by testing model performance indexes through a leave-one-out validation method, and finally, the whole-night PSG data of a person is input into the above multi-dimensional fusion model to learn the model, so that the sleep staging judgment result is obtained. The automatic sleep staging method is efficient, suitable for clinical use and robust in result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of sleep analysis, in particular, relates to a multi-modal data multi-view sleep staging method. BACKGROUND

[0002] Healthy sleep is an important guarantee for people to carry out normal production activities. In recent years, with the acceleration of social rhythm, more and more people suffer from sleep problems of different degrees due to various reasons, which greatly interferes with people's normal production activities. Sleep medicine can improve the sleep quality of people by studying the sleep of people and giving different treatment intervention means according to the conditions of different people. Sleep staging is an important diagnostic basis in sleep medicine research, which can effectively monitor and evaluate sleep quality. According to the recommended standard AASM of the American Sleep Medicine Society, sleep staging includes the following five stages: wakefulness (W), non-rapid eye movement 1 (N1), non-rapid eye movement 2 (N2), non-rapid eye movement 3 (N3), and rapid eye movement (REM).

[0003] Traditional sleep staging is mainly performed by manual sleep stage labeling. Before manual labeling, a lot of time is spent on learning the method of sleep stage labeling. In addition, such labeling is evaluated by experts through visual analysis, and due to the repeated and large amount of work, the experts are also prone to make wrong judgments after a long time of work.

[0004] In recent years, a new and efficient technology is emerging, which is gradually replacing the traditional manual method with low efficiency and prone to errors. Using artificial intelligence (AI) technology to realize automatic sleep staging, compared with the traditional manual labeling method which takes nearly 1 hour, AI can complete sleep stage labeling in a few minutes, and the average accuracy rate can reach more than 85%. Therefore, automatic sleep staging labeling through computer analysis of sleep data has broad prospects.

[0005] Most of the existing sleep staging methods (for example, CN CN114398932A, CN114190897A) extract single-view features of sleep data, such as one-dimensional time domain features or two-dimensional time-frequency domain features. Generally, the input is only one of one-dimensional signal, two-dimensional time-frequency diagram and graph topology structure, or a combination of the first two, without combination with graph topology structure or combination of the three. And the selection of the verification set is a simple proportional random division, without data verification for a single person.

[0006] In summary, the deficiencies of the existing technology are reflected in several aspects:

[0007] 1. The traditional manual labeling method is low in efficiency and high in cost. The sleep stage labeling is performed by doctors manually, and the artificial

[0008] The training time cost is high and the labeling process is time-consuming and labor-intensive.

[0009] 2. Existing deep learning methods generally use the method of leaving out a small proportion of data to evaluate the model. When evaluating the model, they do not adopt the leave-one-out validation evaluation method for a single person, and cannot objectively reflect the effect of the model on a single person.

[0010] 3. Existing deep learning methods do not fully utilize data, and the sleep staging results for a single person are not robust. Existing technologies primarily use one of three methods: one-dimensional signals, two-dimensional time-frequency graphs, or graph topology to construct a single-view model. Summary of the Invention

[0011] To address the high labor costs and low efficiency of traditional manual methods, as well as the inability of existing computer-based automated staging algorithms to effectively reflect individual patient outcomes and produce robust results, the present invention aims to provide an efficient, clinically applicable, and robust automated sleep staging method. This invention provides a systemic solution for clinically applicable automated sleep staging that utilizes overnight polysomnographic data, combined with deep learning methods, and integrates multimodal and multi-view data.

[0012] This invention utilizes a multi-perspective data processing and fusion algorithm. Regarding model input selection, this approach processes raw data into multi-perspective data and fuses several of these perspectives. This algorithmic model fully utilizes the raw data and integrates the diverse feature information from multiple perspectives, improving the robustness of sleep staging results compared to single-perspective models. This invention employs a leave-one-out validation method for model training and verification, addressing the issue of existing approaches that fail to reflect individual sleep stage interpretation results in clinical scenarios. Through this leave-one-out validation approach, sleep staging results for each individual are obtained and their mean is used as a standard for evaluating model performance.

[0013] The present invention processes and analyzes data from the electroencephalogram (EEG), electrooculogram (EOG), and electromyogram (EMG) channels of polysomnography (PSG), converting the raw multimodal PSG data into three different perspectives: a one-dimensional signal, a two-dimensional time-frequency graph, and a graph topology. The signals are then input into a neural network for feature extraction and fusion. Finally, the system automatically outputs sleep staging results, providing reliable diagnostic evidence for sleep medicine. The technical solutions of the present invention are described in detail below.

[0014] A method for constructing a multi-view fusion model for obtaining individual sleep staging results based on multimodal and multi-view data includes the following steps:

[0015] Step 1: Process the individual's overnight PSG data containing multimodal EEG, EOG, and EMG, and process the PSG data into multi-view data: one-dimensional signal, two-dimensional time-frequency graph, and graph topology structure;

[0016] Step 2: Use the multi-head self-attention module to fuse the features of multi-view data extracted by the neural network;

[0017] The multi-view data are respectively input into a one-dimensional signal feature extraction network for extracting one-dimensional signal features, a two-dimensional time-frequency graph feature extraction network for extracting two-dimensional time-frequency graph features, and a graph topology structure feature extraction network for extracting graph topology structure features to extract features from each view;

[0018] Balance the data after feature extraction from each perspective;

[0019] Different splicing operations are performed on the equalized feature data of each view, which are used as the input of the multi-head self-attention network. After multi-head splicing and linear mapping in a single fully connected layer, the softmax function is used to output the sleep stage probability;

[0020] Step 3: Use the leave-one-out method to verify the performance indicators of the fusion model, perform model optimization, and obtain a multi-perspective fusion model for predicting individual sleep staging results.

[0021] In the present invention, in step 1, a one-dimensional signal is obtained by filtering, downsampling, and channel merging the original PSG data;

[0022] The two-dimensional time-frequency diagram is obtained by performing a short-time Fourier transform on the multimodal one-dimensional signal to obtain a time-frequency diagram, and then performing a logarithmic transformation;

[0023] The graph topology is defined as G = (V, E, A), where V represents the vertices of the graph, which refers to the channels of PSG, E represents the edges between vertices, and A is the adjacency matrix of the graph G, which represents the relationship between vertices and edges. The resulting graph topology is satisfy:

[0024]

[0025] F is the number of vertices of the graph, here the number of channels of PSG, and L is the length of the spliced ​​frame.

[0026] In the application, in step two, the one-dimensional signal extracts the time domain features through the double-branch convolution, and then extracts the time sequence features through the long short-term memory recursive network (LSTM); the two-dimensional time-frequency diagram signal extracts the time sequence information within a single frame of the time-frequency diagram through the short-term LSTM, and then extracts the time sequence information within L frames through the long-term LSTM; the graph topology structure extracts the correlation between channels through the brain network adjacency matrix, and then sends it to the graph convolution network (GCN) to extract the features of the graph topology structure.

[0027] In the application, in step two, the input of the multi-head self-attention network is two or more different combinations of the one-dimensional signal extraction features, the two-dimensional time-frequency diagram extraction features and the graph topology structure extraction features; the multi-head self-attention mechanism is used to realize the adaptive fusion of the features between the one-dimensional signal extraction network, the two-dimensional time-frequency diagram extraction network and the graph topology structure extraction network, and a plurality of multi-view fusion models are obtained.

[0028] In the application, in step three, the multi-view fusion model is selected from any one of the one-dimensional signal and two-dimensional time-frequency diagram fusion model, the one-dimensional signal and graph topology structure fusion model, the two-dimensional signal and graph topology structure fusion model, or the one-dimensional signal, two-dimensional time-frequency diagram and graph topology structure three-view fusion model.

[0029] In the application, in step three, the performance index of the test model is verified by the leave-one-out method.

[0030] Suppose that N people have annotated sleep staging labels of whole-night PSG data, and the first and second persons are taken as the validation subset and the test subset in order, and the remaining N-2 data are taken as the training subset, at this time, the training set, the validation set and the test set are denoted as set S1, and a total of N different sets S1, S2, …, S N ;

[0031] N different models of the same kind are trained by the leave-one-out method, and the evaluation results suitable for the clinical scene are obtained by taking the average of the performance evaluation of the N models, and the evaluation results mainly include the accuracy, the F1 value and the K value.

[0032] Compared with the prior art, the application has the following advantages:

[0033] 1. The application uses a multi-view method to process the original PSG data, including but not limited to two or three view data combinations (mainly the combination of the graph topology structure and other views), which fuses the time domain information of the one-dimensional signal, the frequency domain information of the two-dimensional time-frequency diagram and the electrode correlation information of the graph topology structure, and the angle is more comprehensive and the result is more accurate.

[0034] 2. The application can simultaneously process the one-dimensional signal, the two-dimensional time-frequency diagram and the graph topology structure and take them as the input of the neural network model, and the processing is more efficient.

[0035] 3. The application uses a multi-head self-attention structure for multi-view fusion, and the multi-view model established has better robustness than the single-view model. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a whole system framework diagram.

[0037] Figure 2 is a multi-view preprocessing flowchart.

[0038] Figure 3 is a multi-view feature extraction diagram; a, one-dimensional signal feature extraction network; b, two-dimensional time-frequency diagram feature extraction network; c, graph topology structure feature extraction network.

[0039] Figure 4 . Multi-head self-attention structure diagram. DETAILED DESCRIPTION

[0040] The technical solutions of the application will be described in detail below in combination with the drawings and examples.

[0041] The application provides an automatic sleep staging scheme based on multi-lead sleep instrument data. The prediction scheme includes two parts, namely a training test part and a practical part. In training and testing the machine learning model, preprocessed multi-modal multi-view data is used, further fusion of several kinds of multi-view data is used, and the leave-one-out validation method is used to train and verify the test model. When actually using the trained model, the automatic sleep staging task can be completed according to the whole night polysomnogram data. The overall framework of the system is shown in Figure 1 .

[0042] I. Training test part:

[0043] This part belongs to the prediction model establishment part before the practical part

[0044] 1. Preprocess the original PSG data.

[0045] The original PSG data generally has multiple channels, and different channels collect electrical signals at different positions of the head. Taking a single lead signal as an example, there are generally the following channels (M1, M2, C3, C4, O1, O2, F3, F4, T3, T4, E1, E2, Lchin, Rchin, Cchin, etc.). Among them, the EEG channel is generally “C4 - M1”, the EOG channel is generally “E1 – M2”, and the EMG channel is generally “Lchin-Rchin”.

[0046] First, obtain the multimodal one-dimensional signal V1. Process the original signal to filter, downsample, and merge channels. Divide the original signal into 30s frames, and then perform filtering and downsampling operations. Taking the Butterworth filter as an example, EEG and EOG filters can be set to 2nd order 0.3~35Hz bandpass filtering, and EMG filters can be set to 2nd order 10Hz high-pass filtering, then downsample to 100Hz, and finally splice every 20 frames. At this time

[0047]

[0048] L is the length of the concatenated frame, here L = 20. C is the number of channels, here C = 3, i.e., there are three channels: EEG, EOG, and EMG. N is the length of the downsampled signal, here N = 1000.

[0049] Then obtain the multimodal two-dimensional time-frequency diagram. Continue to process the multimodal one-dimensional signal V1, first perform short-time Fourier transform on V1 to obtain the time-frequency diagram, and then perform logarithmic transformation to obtain the time-frequency diagram in logarithmic form. Taking the multimodal one-dimensional signal V1 obtained by the above processing as an example, set the window length of the short-time Fourier transform to 200, the window function overlap number to 100, and the FFT point number to 256. Finally, the multimodal time-frequency diagram V2 is obtained. At this time

[0050]

[0051] L is the length of the concatenated frame, here L = 20. C is the number of channels, here C = 3, meaning there are three channels: EEG, EOG, and EMG. W is the width of the time-frequency plot, here W = 129. H is the height of the time-frequency plot, here H = 29.

[0052] Finally, the graph topology is processed. The graph topology is generally defined as G= (V, E, A), where V represents the vertices of the graph, which refers to the channels of PSG. E represents the edges between vertices. A is the adjacency matrix of the graph G, which represents the relationship between the vertices and edges of the graph. There are correlations between the channels of multi-channel EEG signals. According to different correlation methods, the adjacency matrix A can be generated in many ways, such as Pearson correlation coefficient method, mutual information method, differential entropy or adaptive method, etc., to extract the intrinsic network features of multi-channel EEG signals. Regardless of the method, the final generated graph topology satisfies

[0053]

[0054] Here, A is the adjacency matrix of the graph G, F is the number of vertices in the graph, and is the number of channels in the PSG. L is the length of the concatenated frame, which is set to 20 here. F is the number of channels, which can be 15 here. N is the signal length of a single channel, which is set to 1000 here.

[0055] Then the multi-view data is input into the multi-view fusion model for training.

[0056] 2. Multi-view data fusion.

[0057] The present invention uses a multi-head self-attention module to fuse the features of multi-view data extracted by the neural network. Figure 3 As shown, it is divided into three steps:

[0058] 1) Select the sleep staging network, retain the feature extraction part, and delete the fully connected layer;

[0059] 2) Equalize the data after feature extraction from each perspective;

[0060] 3) Multi-head self-attention feature fusion.

[0061] Here, the input data is described using multimodal one-dimensional signals, multimodal time-frequency graph signals, and graph topology structures as examples, and finally the features are fused using the method of the present invention.

[0062] The single-view network retains the feature extraction part, that is, the fully connected layer and the last normalization layer are deleted. The one-dimensional signal feature extraction network design is as follows Figure 3 As shown in a, the two-dimensional time-frequency graph feature extraction network design is as follows Figure 3 As shown in b, the graph topology feature extraction network design is as follows Figure 4 c. The multimodal one-dimensional signal V1 uses a two-branch convolution to extract time-domain features, followed by a long short-term memory (LSTM) recurrent network to extract temporal features. The multimodal two-dimensional time-frequency graph signal V2 uses a short-term LSTM to extract temporal information within a single frame of the time-frequency graph, followed by a long-term LSTM to extract temporal information within L frames. The graph topology V3 uses the brain network adjacency matrix A to extract the correlation between each channel, which is then fed into a graph convolutional network (GCN) to extract graph topology features. The attention mechanism then fuses the information from L frames.

[0063] The dimension of the multimodal one-dimensional signal V1 changes from [L, C, N] to [L, T1] after it enters the one-dimensional signal network feature extraction part. The data at this time is recorded as The dimension of the multimodal two-dimensional time-frequency graph V2 changes from [L, C, W, H] to [L, T2] after inputting the two-dimensional time-frequency graph network feature extraction part. The data at this time is recorded as , the graph topology structure V3 changes from graph topology structure to [L, T3] after inputting the two-dimensional time-frequency graph network feature extraction part. The data at this time is recorded as At this time, balance is needed The proportion of Will The feature dimension is reduced to T, that is, [L, T1], [L, T2], [L, T3] all become [L, T], then .

[0064] Then, the above-mentioned perspective features are combined in different ways to achieve multi-perspective fusion; taking three-perspective fusion as an example, through the concat operation ( ), get the input of the multi-head self-attention structure , input X into the structure for feature fusion and finally output the fused sleep stage results. The multi-head self-attention structure is as follows Accuracy As shown in the figure, the number of heads can be multiple. A multi-head self-attention network structure is applied to the multi-view data to learn the fusion features. Then, the multiple heads are concatenated and linearly mapped to a single fully connected layer, and the softmax function is used to output the sleep stage probability.

[0065] 3. Divide the dataset into training and testing subsets.

[0066] This example uses the leave-one-out method to train the model. Assume that there are N people with full-night PSG data with sleep stage labels. Sequentially take the first and second people as the validation subset and test subset respectively, and the remaining N-2 cases as the training subset. The training set, validation set, and test set are denoted as set S1. A total of N different sets S1, S2, ..., S N .

[0067] 4. Train the model on the training subset.

[0068] For a single subset S t , t∈[1,N], train the multimodal multi-view fusion model M t The early stopping method is used during training to avoid overfitting of the model, that is, the subset S is used for each fixed number of training rounds E. t The validation set verifies the loss of the model. Set the threshold T, stop training when E> T, and save the model.

[0069] 5. Test subset test model and multi-view model selection.

[0070] Through the leave-one-out training method, N models of the same type with different parameters will be obtained. By evaluating the performance of the N models separately and taking the average, the model evaluation results suitable for clinical scenarios can be obtained. The evaluation results here mainly include accuracy, F1 value, and K value.

[0071] By combining data from different perspectives, the performance of different perspective fusion models can be evaluated, such as a one-dimensional signal and two-dimensional time-frequency graph fusion model, a one-dimensional signal and graph topology structure fusion model, a two-dimensional signal and graph topology result fusion model, and a three-perspective fusion model of one-dimensional signal, two-dimensional time-frequency graph, and graph topology structure. Compared to single-perspective models, multi-perspective fusion models can improve model robustness. As shown in Table 1, these are the sleep staging results of 68 full-night PSG data from a hospital using single-perspective and multi-perspective fusion models for leave-one-out validation. It can be seen that the model fused using the multi-perspective fusion method of the present invention surpasses the single-perspective model in terms of accuracy, standard deviation, minimum value, and maximum value.

[0072] Table 1. Sleep staging results of a hospital's whole-night PSG data using single-view and multi-view fusion models

[0073] One-dimensional signal (single view) 83.6 ± 6.1% (min 65.4%, max 92.8%) Two-dimensional time-frequency map (single view) 86.1 ± 4.2% (min 70.8%, max 93.6%) One-dimensional + two-dimensional (multi-view fusion) 87.3 ± 3.7% (min 75.1%, max 94.1%) ​

[0074] 2. Practical part:

[0075] This section describes the actual use of the established multi-view fusion model. By inputting a full night of PSG data into the model, sleep staging results can be obtained to assist doctors in sleep-related diagnosis.

[0076] The present invention processes the original multimodal PSG signal containing EEG, EOG, and EMG into multimodal and multi-perspective one-dimensional signals, two-dimensional time-frequency graphs, and graph topology structures, and selects several combinations before inputting them into a neural network.

[0077] The present invention uses the leave-one-out validation method to test model performance indicators, which is closer to medical clinical usage scenarios.

[0078] In this invention, multi-view fusion is performed using a multi-head self-attention structure.

Claims

1. A method for constructing a multi-view fusion model for obtaining individual sleep staging results based on multimodal and multi-view data; comprising the following steps: Step 1: Process the individual's overnight PSG data containing multimodal EEG, EOG, and EMG, and process the PSG data into multi-view data: one-dimensional signal, two-dimensional time-frequency graph, and graph topology structure; Step 2: Use the multi-head self-attention module to fuse the features of multi-view data extracted by the neural network; The multi-view data are respectively input into a one-dimensional signal feature extraction network for extracting one-dimensional signal features, a two-dimensional time-frequency graph feature extraction network for extracting two-dimensional time-frequency graph features, and a graph topology structure feature extraction network for extracting graph topology structure features to extract features from each view; Balance the data after feature extraction from each perspective; Different splicing operations are performed on the equalized feature data of each view, which are used as the input of the multi-head self-attention network. After multi-head splicing and linear mapping in a single fully connected layer, the softmax function is used to output the sleep stage probability. Step 3: Validate and test the fusion model performance indicators using the leave-one-out method, perform model optimization, and obtain a multi-view fusion model for predicting individual sleep staging results; where: In step 1, the one-dimensional signal is obtained by filtering, downsampling, and channel merging the original PSG data; The two-dimensional time-frequency diagram is obtained by performing a short-time Fourier transform on the multimodal one-dimensional signal to obtain a time-frequency diagram, and then performing a logarithmic transformation; The graph topology is defined as G = (V, E, A), where V represents the vertices of the graph, which refers to the channels of PSG, E represents the edges between vertices, and A is the adjacency matrix of the graph G, which represents the relationship between vertices and edges. The resulting graph topology is satisfy: ； F is the number of vertices of the graph, here the number of channels of PSG, and L is the length of the spliced ​​frame.

2. The construction method according to claim 1, characterized in that In step 2, the one-dimensional signal is subjected to a dual-branch convolution to extract the time domain features of the signal, and then the timing features are extracted through the long short-term memory recursive network LSTM; the two-dimensional time-frequency graph signal is subjected to a short-term LSTM to extract the timing information within a single frame of the time-frequency graph, and then the timing information within L frames is extracted through a long-term LSTM; the graph topology structure is subjected to the brain network adjacency matrix to extract the correlation between each channel, and then it is sent to the graph convolutional network GCN to extract the features of the graph topology structure.

3. The construction method according to claim 1, characterized in that In step three, the multi-perspective fusion model is selected from any one of a one-dimensional signal and two-dimensional time-frequency graph fusion model, a one-dimensional signal and graph topology structure fusion model, a two-dimensional signal and graph topology result fusion model, or a one-dimensional signal, two-dimensional time-frequency graph, and graph topology structure three-perspective fusion model.

4. The construction method according to claim 1, wherein In step 3, the performance indicators of the test model are verified by the leave-one-out method; Assume that there are N people with full-night PSG data with sleep stage labels. Take the first and second people in sequence as the validation subset and test subset respectively, and the remaining N-2 cases as the training subset. At this time, the training set, validation set, and test set are recorded as set S1. A total of N different sets S1, S2, ..., S N ; Through the leave-one-out training method, N models of the same type with different parameters are obtained. By performing performance evaluation on the N models separately and taking the average, the model evaluation results suitable for clinical scenarios are obtained. The evaluation results mainly include accuracy, F1 value, and K value.

Images