Sleep stage recognition method and apparatus based on multi-modal signals, device, and medium

Abstract

The application discloses a sleep stage recognition method and device based on a multi-modal signal, equipment and a medium. The method comprises the following steps: extracting target lead signals including electroencephalogram signals and electrooculogram signals from a multi-lead sleep monitoring signal of a target object; performing data preprocessing on the target lead signals to obtain reference lead signals; inputting the reference lead signals into a sleep stage recognition model to obtain a sleep stage prediction result of the target object; wherein the sleep stage recognition model comprises a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit and a classification unit, the deep feature extraction unit is constructed based on a one-dimensional focus modulation network and adopts a multi-scale one-dimensional deep convolution, and the two deep feature extraction units are respectively used for deep feature extraction of the electroencephalogram signals and the electrooculogram signals. The scheme can improve the accuracy and robustness of sleep stage recognition by using the sleep stage recognition model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal processing, and in particular to a sleep stage recognition method and device based on multi-modal signals, equipment and medium. BACKGROUND

[0002] Polysomnography (PSG) is the gold standard method for diagnosing various sleep disorders in clinical practice, which widely collects multi-modal physiological signals such as electroencephalogram (EEG), electrooculogram (EOG), electromyogram (EMG), respiratory airflow, chest and abdominal belts, oxygen saturation (SpO2), electrocardiogram (ECG), etc. Although these data provide rich pathological information, the existing PSG analysis process still highly depends on manual segment-by-segment interpretation, which not only consumes a long time and has high labor intensity, but also has the problems of poor result consistency and large subjective bias.

[0003] In recent years, deep learning methods have been gradually applied to sleep staging and respiratory event detection, such as DeepSleepNet, SleepTransformer, etc. However, most of them only process a single signal (such as EEG), ignoring the complementary information between multi-modal signals, resulting in poor model robustness and difficulty in processing signal missing or artifact interference. Existing multi-modal methods mostly use static fusion (such as direct concatenation), which cannot adaptively adjust the importance of different modalities according to different time points, limiting the modeling ability of the model to state mutations or modality dominance. In addition, the structure design of most models does not consider both temporal and global modeling capabilities, making it difficult to capture multi-scale features from local changes to global trends, and relying mostly on short segment input (such as 30s), which loses the context information across cycles, affecting the integrity and accuracy of the detection. SUMMARY

[0004] The present application provides a sleep stage recognition method, device, equipment and medium based on multi-modal signals, which constructs a sleep stage recognition model based on multi-modal signals for dynamic modeling, considering both local and global feature expression, and having the ability to process long-term sleep data. The sleep stage recognition model can be used to improve the accuracy and robustness of sleep stage recognition.

[0005] According to an aspect of the present application, a sleep stage recognition method based on multi-modal signals is provided, the method comprising:

[0006] extracting a target lead signal from a polysomnography signal of a target object; wherein the target lead signal includes an electroencephalogram signal and an electrooculogram signal;

[0007] performing data preprocessing on the target lead signal to obtain a reference lead signal; wherein the preprocessing includes filtering, resampling, standardization and segmentation;

[0008] inputting the reference lead signal into a pre-trained sleep stage recognition model, and determining a sleep stage prediction result of the target object based on an output result of the sleep stage recognition model;

[0009] The sleep stage recognition model comprises a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction unit is constructed based on a one-dimensional focal modulation network. The deep feature extraction unit adopts multi-scale one-dimensional deep convolution. The two deep feature extraction units are respectively used for deep feature extraction of the electroencephalogram signal and the electrooculogram signal. The dynamic gate fusion unit is used for modal adaptive fusion of outputs of the two deep feature extraction units.

[0010] According to another aspect of the present application, a sleep stage recognition device based on multi-modal signals is provided, and the device comprises:

[0011] A signal extraction module is configured to extract a target lead signal from a multi-lead sleep monitoring signal of a target object. The target lead signal comprises an electroencephalogram signal and an electrooculogram signal.

[0012] A signal processing module is configured to perform data preprocessing on the target lead signal to obtain a reference lead signal. The preprocessing comprises filtering, resampling, standardization, and segmentation.

[0013] A sleep stage prediction module is configured to input the reference lead signal into a pre-trained sleep stage recognition model, and determine a sleep stage prediction result of the target object based on an output result of the sleep stage recognition model.

[0014] The sleep stage recognition model comprises a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction unit is constructed based on a one-dimensional focal modulation network. The deep feature extraction unit adopts multi-scale one-dimensional deep convolution. The two deep feature extraction units are respectively used for deep feature extraction of the electroencephalogram signal and the electrooculogram signal. The dynamic gate fusion unit is used for modal adaptive fusion of outputs of the two deep feature extraction units.

[0015] According to another aspect of the present application, an electronic device is provided, and the electronic device comprises:

[0016] At least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the sleep stage recognition method based on the multi-modal signal according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the sleep stage recognition method based on the multi-modal signal according to any one of the embodiments of the present application when executed by the processor.

[0018] The technical scheme of the embodiments of the present application firstly extracts a target lead signal from a multi-lead sleep monitoring signal of a target object; wherein the target lead signal includes an electroencephalogram signal and an electrooculogram signal; then performs data preprocessing on the target lead signal to obtain a reference lead signal; wherein the preprocessing includes filtering, resampling, standardization and fragmentation; further inputs the reference lead signal into a pre-trained sleep stage recognition model, and determines a sleep stage prediction result of the target object based on an output result of the sleep stage recognition model; wherein the sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit and a classification unit, the deep feature extraction unit is constructed based on a one-dimensional focus modulation network, the deep feature extraction unit adopts a multi-scale one-dimensional deep convolution, and the two deep feature extraction units are respectively used for deep feature extraction on the electroencephalogram signal and the electrooculogram signal, and the dynamic gate fusion unit is used for modality adaptive fusion on the outputs of the two deep feature extraction units. The technical scheme constructs a sleep stage recognition model based on multi-modal signals for dynamic modeling, taking into account local and global feature expression, and having the ability to process long-time sleep data, and can improve the accuracy and robustness of sleep stage recognition by using the sleep stage recognition model.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0021] Figure 1is a flow chart of a sleep stage recognition method based on multi-modal signals according to an embodiment of the present application;

[0022] Figure 2 is a working flow diagram of a deep feature extraction unit according to an embodiment of the present application;

[0023] Figure 3 is a flow chart of a sleep stage recognition method based on multi-modal signals according to an embodiment of the present application;

[0024] Figure 4 is a flow chart of another sleep stage recognition method based on multi-modal signals according to an embodiment of the present application;

[0025] Figure 5 is a structural diagram of a sleep stage recognition device based on multi-modal signals according to an embodiment of the present application;

[0026] Figure 6 is a structural diagram of an electronic device implementing a sleep stage recognition method based on multi-modal signals according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", "target" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment one

[0030] Figure 1A flowchart of a sleep stage recognition method based on multi-modal signals is provided for Embodiment One of the present application. This embodiment can be applied to high-precision recognition of sleep stages based on multi-modal signals. The method can be executed by a sleep stage recognition device based on multi-modal signals. The sleep stage recognition device based on multi-modal signals can be realized in the form of hardware and / or software and can be configured in an electronic device with data processing capability. As shown in FIG. 10, the method comprises the following steps. Figure 1

[0031] S110, target lead signals are extracted from multi-lead sleep monitoring signals of a target object; wherein the target lead signals include electroencephalogram signals and electrooculogram signals.

[0032] In this embodiment, target lead signals (such as C4A1 channel of EEG and EOGL channel of EOG) including electroencephalogram signals and electrooculogram signals are first extracted from PSG record files (such as.edf format) of a target object and can be saved as.npy files as an intermediate form for subsequent processing. The target object can refer to a measured object that needs to be recognized for sleep stages, and the target lead signals can be used as basic monitoring multi-modal data for sleep stage recognition. It should be noted that this embodiment adopts a Many-to-Many (many-to-many) prediction mode. For input signals of any length (such as 1 hour or all night), the model outputs the full sequence sleep stage prediction results corresponding to the input length at one time, maintaining the continuity of the time sequence.

[0033] S120, reference lead signals are obtained by data preprocessing of the target lead signals; wherein the preprocessing includes filtering, resampling, standardization and fragmentation.

[0034] ​In this embodiment, after obtaining the target lead signal, data preprocessing is needed to convert it into a unified format tensor data (.pt file) that can be directly processed by the deep learning model. Specifically, the EEG (such as C4A1.npy) and EOG (such as EOGL.npy) signals of the target object are loaded from the file, and read using the memory mapping mode of NumPy, which can effectively reduce the memory occupation of high-resolution, long-time sequence data during loading, thereby achieving efficient data loading. To suppress the DC drift and high-frequency noise, first pass the EEG signal and EOG signal through a band-pass filter (such as a 4th order Butterworth filter), and set the filter range of the EEG signal to 0.5-40 Hz and the filter range of the EOG signal to 0.5-10 Hz. Since the sampling rates of the original EEG signal and EOG signal are not consistent (e.g., EEG is 125 Hz and EOG is 50 Hz), it is necessary to uniformly resample each channel signal to the target sampling rate (e.g., 100 Hz) to achieve accurate alignment of multi-modal data in the time dimension and meet the input format requirements of the neural network.

[0035] Next, each signal channel is standardized to enhance the consistency of signal distribution between different individuals and improve the training stability and generalization ability of the model. For example, Z-score standardization (zero mean, unit variance) can be used. Then the standardized EEG signal and EOG signal are segmented into fixed-length segments to obtain the reference lead signal. Set a target length (e.g., 30s), if the original data is too long, it will be cut to the target length; if the original data is insufficient, it will be padded with zeros at the end. For example, 5 hours of EEG / EOG signal corresponds to 5x3600x100=1800000 sampling points; 6 hours of EEG / EOG signal corresponds to 6x3600x100=2160000 sampling points. Finally, it can be saved as a structured PyTorch.pt file, including the processed EEG tensor and EOG tensor, type float32, dimension [1, length x 100Hz]. Thus, the complete processing flow from the original PSG multi-modal signal to the structured tensor is completed, which is efficient, modular and expandable, and can support preprocessing of signals of different lengths, providing flexible and standardized basic data support for multi-length sleep modeling tasks.

[0036] S130, inputting the reference lead signal into the pre-trained sleep stage recognition model, and determining the sleep stage prediction result of the target object based on the output result of the sleep stage recognition model.

[0037] The sleep stage recognition model can be a deep learning network model capable of sleep stage recognition. Specifically, the sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction units are constructed based on a one-dimensional focus modulation network, and the deep feature extraction units use multi-scale one-dimensional deep convolution. The two deep feature extraction units are respectively used for deep feature extraction of the electroencephalogram signal and the electrooculogram signal. The dynamic gate fusion unit is used for modal adaptive fusion of the outputs of the two deep feature extraction units.

[0038] In this embodiment, a sleep stage recognition model including a two-level feature extraction structure is constructed in advance in view of the characteristics of the sleep EEG / EOG signal, such as high sampling rate, long sequence, strong local features, and significant modal differences. The constructed sleep stage recognition model is trained based on a training data set created based on reference PSG data. The two-level feature extraction structure can capture long-range dependencies across cycles while preserving local sleep microstructure. The creation of the training data set can refer to the processing method of the PSG signal of the target object. On this basis, the sleep staging labels are parsed from the.rml label file matched with the PSG recording file and saved as.npy files, and then the labels are aligned and filled. For example, an artificial label can be generated every 30 seconds, where N hours correspond to Nx120 labels. The label sequence is automatically aligned with the signal length. If the target number is not enough, the tail is filled with a preset label value to represent an invalid label. The model training can automatically ignore the invalid label to avoid interference with loss calculation. Finally, the sleep label sequence (type int64, dimension [lengthx120]) can be added to the corresponding structured PyTorch.pt file. Thus, a training data set including the processed EEG tensor and EOG tensor and the corresponding sleep label sequence can be obtained.

[0039] In the model training stage, an end-to-end learning strategy is adopted, combined with a cross-entropy loss function, an Adam optimizer, mixed precision training, and gradient accumulation technology, and the model is comprehensively evaluated by accuracy and macro F1 value. The calculation formula of the loss is as follows: , wherein, n represents the total number of categories, represents a specific category, represents the probability of the category being . The Adam optimizer is an adaptive gradient optimization algorithm that dynamically adjusts the learning rate by using first-order moment estimation and second-order moment estimation. Specifically, the Adam optimizer can update the model parameters by the following update rule: (1) update the first-order moment estimation and the second-order moment estimation according to the current gradient and the historical value: , . where, is the first moment estimation, representing the weighted average of the gradient; is the second moment estimation, representing the weighted average of the square of the gradient; is the current gradient. (2) Bias correction: since in the initial stage, and will be biased, and the following formula can be used for bias correction: , . where, is the corrected gradient first moment; is the corrected gradient second moment. (3) Parameter update: the parameters of the final model are updated as: . where, is the learning rate, controlling the step size of each update; is a small constant to avoid division by zero errors.

[0040] Assume represents the number of samples of class that are correctly predicted as , represents the number of samples other than class that are correctly predicted as non , represents the number of samples other than class that are incorrectly predicted as , represents the number of samples of class that are incorrectly predicted as samples of other classes. Thus, the accuracy can be represented as: . The precision corresponding to class can be represented as: , and the macro-averaged precision can be represented as . For class , the corresponding recall can be represented as: , and further, the macro-averaged recall can be represented as: . For class , the corresponding F1 value can be represented as: , and the macro-averaged F1 value can be represented as: .

[0041] In this embodiment, after the sleep stage recognition model is trained, it can be put into use to realize sleep stage prediction. Optionally, the preliminary feature extraction unit includes a multi-scale one-dimensional convolution layer and a pooling layer; correspondingly, the reference lead signal is input into the pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model, including: inputting the reference lead signal into the preliminary feature extraction unit of the sleep stage recognition model, extracting local features of the reference lead signal through the multi-scale one-dimensional convolution layer, and compressing the extracted local features in space size through the pooling layer to obtain initial features.

[0042] Specifically, to avoid noise accumulation, gradient instability and computational burden caused by the direct input of original sampling point level EEG / EOG into the deep network, the present application first adopts a preliminary feature extraction unit composed of a multi-scale one-dimensional convolution layer and a pooling layer to preliminarily extract features from the reference lead signal. Among them, the shape of the EEG and EOG signal tensor in the reference lead signal is [B, 1, T], B is the batch size (Batch Size), and T is the original sequence length. The multi-scale one-dimensional convolution layer uses one-dimensional convolution with multiple different kernel sizes to capture short-time key waveform structures in EEG, such as sleep spindle, K-complex, rapid eye movement, etc., and the output features have higher feature aggregation than the original signal. The pooling layer is used to reduce the time resolution of the original EEG / EOG sequence (such as T→T / 2→T / 4), reduce noise and reduce subsequent computational complexity, while increasing the feature channel dimension, so that the model can express multiple local waveform features from a higher dimensional space, thereby obtaining a more stable structure and more semantic time sequence feature representation. The output of the preliminary feature extraction unit is a feature sequence with dimensions [B, L, C], L is the length of the signal after convolution and pooling, and C is the number of feature channels. The feature sequence serves as the basis for the subsequent deep feature extraction unit.

[0043] In this embodiment, optionally, the deep feature extraction unit includes a local context modeling layer, a global context modeling layer, and a single-modal fusion output layer; accordingly, the reference lead signal is input to the pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model, and further includes: obtaining first features by performing multi-scale one-dimensional deep convolution and local multi-scale aggregation on the initial features corresponding to the electroencephalogram signal through the local context modeling layer of the first deep feature extraction unit; obtaining second features by performing pooling compression and linear mapping on the initial features corresponding to the electroencephalogram signal through the global context modeling layer of the first deep feature extraction unit; generating a first context modulator according to the initial features, the first features, and the second features corresponding to the electroencephalogram signal through the single-modal fusion output layer of the first deep feature extraction unit, and performing element-level modulation on the initial features corresponding to the electroencephalogram signal by using the first context modulator, and obtaining deep features corresponding to the electroencephalogram signal by normalizing the modulated features; obtaining third features by performing multi-scale one-dimensional deep convolution and local multi-scale aggregation on the initial features corresponding to the electrooculogram signal through the local context modeling layer of the second deep feature extraction unit; obtaining fourth features by performing pooling compression and linear mapping on the initial features corresponding to the electrooculogram signal through the global context modeling layer of the second deep feature extraction unit; generating a second context modulator according to the initial features, the third features, and the fourth features corresponding to the electrooculogram signal through the single-modal fusion output layer of the second deep feature extraction unit, and performing element-level modulation on the initial features corresponding to the electrooculogram signal by using the second context modulator, and obtaining deep features corresponding to the electrooculogram signal by normalizing the modulated features.

[0044] It should be noted that in order to efficiently capture local sleep microstructure (such as spindle, K-complex) and long-range periodicity (such as sleep stage transition) on long-time series EEG / EOG, the present application has carried out structural modification based on FocalNet architecture, and constructed a one-dimensional focal modulation network (1D-FocalNet) suitable for physiological signals as a deep feature extraction unit. Among them, the traditional FocalNet is mainly used for image / video processing (2D / 3D), and cannot be directly applied to one-dimensional physiological signals such as sleep EEG / EOG. The present application reconstructs the overall structure from two dimensions to one-dimensional convolution structure, so that the network can directly capture the local waveform and global trend of EEG / EOG in the time dimension. Since one-dimensional physiological signals do not have two-dimensional spatial dimensions, the spatial window division mechanism in the original FocalNet is removed, and only the local convolution and global modulation framework along the time dimension are retained, so that it adapts to the structural characteristics of time series, and significantly reduces the computational complexity. Compared with the traditional Transformer self-attention, 1D-FocalNet has the following advantages: 1. No need to construct attention matrix, computational complexity from O (N2) to O (N), and the time complexity from O (N) to O (N), which is 2-3 orders of magnitude lower than the traditional Transformer. Fall to ; 2. More sensitive to local structure, since key physiological events of sleep EEG (e.g. spindles, K-complexes) have well-defined local time windows, the convolutional structure achieves pattern detection within a fixed receptive field in a short time window, which has better local sensitivity and computational advantage in several physiological event detection scenarios. 3. Adapt to ultra-long sequences (hundreds to thousands of time slices), which can be deployed to clinical devices with limited memory.

[0045] The present application combines the typical event duration of EEG / EOG, and designs a set of convolution kernels at a sampling rate of 100 Hz. The designed convolution kernels (15 / 21 / 27 / 33 / 39 points) correspond to a receptive field of 150-390 ms at a sampling rate of 100 Hz, which can cover the high-frequency oscillation component of sleep spindles (oscillation period 100-300 ms), the key burst component of K-complexes (200-600 ms, but the most discriminative burst peak component and rapid turning section usually concentrate in the first 200-400 ms), and the transient waveform of micro-awakening (100-300 ms). The above convolution kernel size is only a preferred embodiment, and those skilled in the art can adjust the convolution receptive field according to the needs. These time scales all belong to the key local structure window of sleep events. Aligning the convolution receptive field with the clinical characteristics enables the model to extract local waveform features based on physiological time scales, and naturally has the ability to recognize sleep events. It should be noted that the above convolution kernel size design is a structure optimization driven by clinical prior knowledge, which is different from the arbitrary setting of convolution size in the prior art, and has a clear technical effect.

[0046] In this embodiment, independent 1D-FocalNet branches are constructed for EEG and EOG signals with different frequency bands and structural characteristics. 1D-FocalNet aims to efficiently capture local microstructure and global periodic patterns in the input sequence, replacing traditional attention mechanisms to achieve context-aware modeling. Among them, the first deep feature extraction unit is used for deep feature extraction of EEG signals, focusing on high-frequency local rhythm changes; the second deep feature extraction unit is used for deep feature extraction of EOG signals, focusing on low-frequency eye movement trends.

[0047] Figure 2 A workflow diagram of a deep feature extraction unit is shown. Specifically, as shown in Figure 2 , taking deep feature extraction of EEG signals as an example, assuming that the initial feature representation of the EEG signal is: , first pass the local context modeling layer of the first deep feature extraction unit to extract local context features under different receptive fields through multi-scale one-dimensional deep convolution in parallel, which can be specifically represented as: . Wherein, represents the first local context features of one scale, denotes the size of the convolution kernel; denotes the one-dimensional depthwise convolution using the convolution kernel ; The one-dimensional depthwise convolution (Depthwise Convolution, see DWConv in Figure 2 ) is a group convolution structure, and is specifically set to a group number equal to the number of input channels, so that each convolution kernel independently extracts features from a time series of one channel. This structure significantly reduces the parameter quantity and computational complexity of the model while ensuring that long-time sequence features are captured by a large receptive field (such as a convolution kernel size of 31 or more). Then, the first feature is obtained by local context modeling layer for local multi-scale aggregation, which can be expressed as: . Wherein, denotes the first feature (see local context feature in Figure 2 ), denotes the number of multi-scale convolutions. Next, the initial feature corresponding to the EEG signal is compressed and linearly mapped by the global context modeling layer of the first deep feature extraction unit to obtain the second feature, thereby modeling the periodic global context, which can be expressed as: , . Wherein, denotes the global feature after pooling (see global pooling feature in Figure 2 ), denotes the time dimension average pooling, denotes the global modulation linear weight, denotes the bias, denotes the second feature (see global context feature in Figure 2 ). Then, the first context modulator is generated according to , and by the single-modal fusion output layer of the first deep feature extraction unit, and the initial feature corresponding to the electroencephalogram signal is element-level modulated using the first context modulator, and the modulated feature is normalized, see the following formula: . Wherein, denotes the generated first context modulator (see Modulation in Figure 2 ); denotes element-level multiplication operation; denotes the deep feature corresponding to the electroencephalogram signal; Representation layer normalization, specifically normalization along the feature channel dimension, that is, calculating the mean and variance of the feature vector of each time step, is usually used in neural networks to accelerate training and improve model stability. It should be noted that the feature fusion in the embodiment is not ordinary linear superposition, but element-by-element multiplication modulation of the initial features by the context modulator to achieve adaptive enhancement of key signal patterns, which is significantly different from the traditional residual addition structure. The deep feature extraction process of the EOG signal can refer to the above process and will not be described here.

[0048] In the embodiment, the reference lead signal is input to the pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model, further comprising: for each time step after fragmentation, the deep features corresponding to the electroencephalogram signal and the electrooculogram signal are spliced by the dynamic gating fusion unit of the sleep stage recognition model, and the target gating coefficient is determined based on the spliced features; wherein the target gating coefficient is used to represent the relative contribution degree of the deep features corresponding to the electroencephalogram signal and the electrooculogram signal; the deep features corresponding to the electroencephalogram signal are weighted using the target gating coefficient, the deep features corresponding to the electrooculogram signal are weighted using the difference between 1 and the target gating coefficient, and the weighted deep features are added to obtain the multi-modal fusion features.

[0049] In the embodiment, the fusion ratio of the two modalities is dynamically adjusted according to the signal-to-noise ratio and feature contribution of EEG / EOG in different time periods, to enhance the robustness of the model to single-modality degradation. Specifically, the deep features corresponding to the EEG signal are denoted as , and the deep features corresponding to the EOG signal are denoted as . For each time step , the deep features corresponding to the two modalities are spliced by the dynamic gating fusion unit of the sleep stage recognition model, and the target gating coefficient is determined based on the spliced features, which is specifically represented as: , . Wherein, represents the spliced deep features, represents the gating weight matrix; represents the gating bias; is a Sigmoid activation function, so that ; represents the target gating coefficient, decides the relative contribution degree of EEG and EOG in time step . This gating mechanism can adaptively adjust the modality weight according to the physiological characteristics of the sleep stage, for example, automatically increasing the weight of the electrooculogram signal in the rapid eye movement (REM) stage, and increasing the weight of the electroencephalogram signal in the deep sleep (N3) stage, thereby improving the recognition performance of the aliasing state. Then based on fusing the deep features corresponding to the two modalities at each time step to obtain multi-modal fusion features corresponding to each time step , which can be expressed as: + (1- ) . Then, all the time steps are concatenated in order to obtain , which is taken as the final output of the dynamic gated fusion unit.

[0050] In this embodiment, the reference lead signal is input to the pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model, which further includes: performing forward long short-term memory network calculation and backward long short-term memory network calculation on the multi-modal fusion features corresponding to each time step through the bidirectional long short-term memory network unit of the sleep stage recognition model, and splicing the forward and backward calculation results to obtain the bidirectional output features of each time step; determining the attention score corresponding to the bidirectional output features at each time step through the attention unit of the sleep stage recognition model, normalizing the attention score to obtain the attention weight, and weighting the bidirectional output features according to the attention weight of each time step to obtain the target context features with the time sequence dimension reserved; and mapping the target context features to the sleep stage type of each time step through the classification unit of the sleep stage recognition model as the sleep stage prediction result of the target object.

[0051] The bidirectional long short-term memory network unit can be used to capture long-range dependencies and model the transition rules of sleep stages. Through the Bi-directional Long Short-Term Memory (BiLSTM), the model can simultaneously integrate the forward and backward context information, thereby more accurately depicting the transition characteristics and time sequence structure between sleep stages. It should be noted that, since the length of the sleep sequence is much longer than the natural language task, directly using self-attention will bring huge computational overhead, therefore, the BiLSTM is used instead of the Transformer structure to model long-range dependencies. Specifically, the BiLSTM performs forward and backward LSTM calculations on the multi-modal fusion features corresponding to each time step The forward and backward LSTM calculations are performed simultaneously, which can be seen from the formula: , . Then, the forward and backward calculation results are spliced, i.e. , wherein is the LSTM hidden state dimension, thereby obtaining the bidirectional output features of each time step . Then, all the time steps are concatenated in order to obtain and take it as the final output of the bidirectional long short-term memory network unit. Wherein, the BiLSTM can correct the current judgment by using future information, for example, a certain segment looks like "awake", but before and after it is "deep sleep", at this time the model will correct it to deep sleep, which is a context encoding, not a feedback loop, but can achieve functional global correction, and essentially introduces the context information of the future segment through the hidden state of the backward LSTM. On the basis of inter-block sequence modeling, the application further proposes a global trend correction mechanism, which uses the global context information integrated by the bidirectional long short-term memory network in the hidden space to model the trend consistency of local features in the semantic scale, thereby improving the continuity of prediction. The mechanism is not another independent structure, but based on the context encoding ability of the bidirectional long short-term memory network before and after the whole sequence, the local features are re-modulated by the hidden state distribution in the whole night range, so that the local prediction can automatically align with the overall law of the whole night sleep structure, thereby improving the stability of long-term sequence prediction.

[0052] The "re-modulation" described in the application is not a structural feedback loop, but uses the integration of the whole sequence context by the bidirectional long short-term memory network in forward and backward calculation to constrain the global trend of local features in the hidden space. That is, the model corrects the local representation in the semantic scale by the global distribution of the bidirectional hidden state in the reasoning process, so that the prediction of each time step remains consistent under the overall sleep stage evolution rule.

[0053] In the embodiment, in order to enhance the perception ability of the model to the key segments (such as rapid eye movement stage) in the long sequence, an attention mechanism is introduced to construct an attention unit for weighted aggregation of the time sequence representation output by the BiLSTM unit. By automatically learning the importance weight of each time step, the model can focus on the information segment that contributes most to the current sleep state judgment. Specifically, for each time step The corresponding bidirectional output feature calculates the attention score, as shown in the following formula: . Wherein, indicates the attention score, is the weight matrix of the attention linear transformation, is the linear transformation bias, is the internal projection dimension of the attention mechanism, is a nonlinear activation function; is the attention query vector, which is used to map the projected vector to a scalar. Then the attention score is normalized to obtain the attention weight, as shown in the following formula: . According to the attention weight of each time step, the target context feature with the reserved time sequence dimension is obtained by element-wise weighting of the bidirectional output feature, as shown in the following formula: . Wherein, indicates the Target context features at each time step.

[0054] Optionally, before inputting the target context features into the classification unit, the present invention may also set a temporal refine head to perform further temporal modeling and smoothing on the attention-weighted feature sequence, so as to enhance the continuity and stability of the prediction results of adjacent time steps.

[0055] In a preferred embodiment of the present invention, the temporal refining head can be constructed using a one-dimensional temporal convolutional network (TCN) based on a dilated convolution architecture or a bidirectional long short-term memory network (BiLSTM), and can be combined with residual connections to achieve secondary extraction of local context and error suppression.

[0056] Through this time-series refinement head, the model can further integrate short- and medium-range time dependencies after attention is focused, thereby reducing fragment-level misjudgments caused by noise interference and improving the temporal consistency and physiological rationality of sleep stage recognition.

[0057] The classification unit receives attention-weighted target context features and maps them to the sleep stage category at each time step. Optionally, sleep stage types include wakefulness, N1, N2, N3, and REM sleep. The classification unit undertakes the final prediction task and is the information convergence output layer of the entire model; its design must balance expressive power and parameter efficiency. For example, the classification unit can be constructed using the classification head of a feedforward network. Specifically, a linear layer first maps the high-order features of each time step to the hidden space, as shown in the following formula: Building upon this, to suppress class imbalance caused by long-tail distribution, a Logit adjustment mechanism is introduced. Assume that the prior probabilities of each class in the pre-calculated training set are... Using adjustment coefficient Features of hidden space Log-probability adjustment is performed to obtain The calculation formula is: Then, a Softmax layer is used to adjust the features. Normalization is performed, thus outputting the class probability at each time step, as shown in the following formula: ftmax( From this, we can obtain the predicted probability distribution for each time step, whose tensor dimension is... ,in, This represents the number of categories (set to 5 here), and the category with the highest probability is used as the sleep stage prediction result for the corresponding time step.

[0058] Figure 3 A flowchart of a sleep stage recognition method based on multi-modal signals is shown. Figure 3 As shown, after the acquired EEG signals and EOG signals are subjected to preliminary feature extraction by a preliminary feature extraction unit (not embodied in the Figure 3 ), they are respectively input into a first deep feature extraction unit and a second deep feature extraction unit for deep feature extraction, and then the output features of the two deep feature extraction units are dynamically and adaptively fused by a dynamic gating fusion unit, and then the long-range dependency is captured by a bidirectional long short-term memory network unit, and the output features of the BiLSTM are weighted and aggregated by an attention unit (not embodied in the Figure 3 ), and finally the sleep stage recognition (i.e., sleep staging) is performed by a classification unit (i.e., the classification head in the Figure 3 ).

[0059] The present application proposes a three-layer sequence modeling structure in the field of sleep staging: (1) Intra-block: The initial features of each time step are input into FocalNet to capture spindle, K-complex, slow wave, and other local microstructures. (2) Inter-block: The feature sequence of all time steps is input into a BiLSTM lightweight timing network to learn the sleep cycle regularity of the whole night. (3) Global trend correction: The whole night sleep cycle trend obtained by the inter-block timing network is used as a correction factor to re-modulate the local features of each time step, so that the local staging results are consistent with the whole night sleep structure, thereby improving the prediction continuity and physiological rationality. This structure is completely different from the planar modeling of Transformer, and is a hierarchical sequence modeling method that adapts to physiological laws. In addition, compared with traditional modal splicing, the cross-modal dynamic modulation mechanism (i.e., the dynamic gating fusion unit) proposed by the present application can adaptively adjust the importance of different modalities according to the sleep stage, and the effect is particularly prominent in REM, N1, and other sleep stages. Through verification, the present application achieves 87% accuracy and 80% F1 value on the SHHS1 public data set, proving the effectiveness of the structure of the present application.

[0060] The technical scheme of the embodiment of the present application firstly extracts a target lead signal from a multi-lead sleep monitoring signal of a target object; wherein the target lead signal comprises an electroencephalogram signal and an electrooculogram signal; then performs data preprocessing on the target lead signal to obtain a reference lead signal; wherein the preprocessing comprises filtering, resampling, standardization and fragmentation; further, the reference lead signal is input into a pre-trained sleep stage recognition model, and a sleep stage prediction result of the target object is determined based on an output result of the sleep stage recognition model; wherein the sleep stage recognition model comprises a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit and a classification unit, the deep feature extraction unit is constructed based on a one-dimensional focus modulation network, the deep feature extraction unit adopts a multi-scale one-dimensional deep convolution, the two deep feature extraction units are respectively used for deep feature extraction of the electroencephalogram signal and the electrooculogram signal, and the dynamic gate fusion unit is used for modal adaptive fusion of the outputs of the two deep feature extraction units. The technical scheme constructs a sleep stage recognition model based on multi-modal signals, which dynamically models, considers local and global feature expression, and has the ability to process long-time sleep data, and can improve the accuracy and robustness of sleep stage recognition by using the sleep stage recognition model.

[0061] Embodiment two

[0062] Figure 4 A flowchart of a sleep stage recognition method based on multi-modal signals is provided for the second embodiment of the present application, and the present embodiment is optimized based on the above-mentioned embodiment. The specific optimization is that the target lead signal is preprocessed to obtain the reference lead signal, which comprises: band-pass filtering the target lead signal based on a Butterworth filter, and resampling the filtered signal based on a target sampling rate; each channel of the resampled signal is respectively subjected to Z-score standardization, and the standardized signal is subjected to fragmentation processing based on a target signal length to obtain the reference lead signal.

[0063] As shown in Figure 4 , the method of the present embodiment specifically comprises the following steps:

[0064] S210, extracting a target lead signal from a multi-lead sleep monitoring signal of a target object; wherein the target lead signal comprises an electroencephalogram signal and an electrooculogram signal.

[0065] S220, band-pass filtering the target lead signal based on a Butterworth filter, and resampling the filtered signal based on a target sampling rate.

[0066] S230, respectively performing Z-score standardization on each channel of the resampled signal, and performing fragmentation processing on the standardized signal based on a target signal length to obtain a reference lead signal.

[0067] S240, input the reference lead signal to the pre-trained sleep stage recognition model, and determine the sleep stage prediction result of the target object based on the output result of the sleep stage recognition model.

[0068] The sleep stage recognition model comprises a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit and a classification unit, the deep feature extraction unit is constructed based on a one-dimensional focus modulation network, the deep feature extraction unit adopts a multi-scale one-dimensional deep convolution, and the two deep feature extraction units are respectively used for deep feature extraction of the electroencephalogram signal and the electrooculogram signal.

[0069] The technical scheme of the embodiment of the application constructs a sleep stage recognition model based on multi-modal signals, which dynamically models, considers local and global feature expression, and has the ability to process long-time sleep data, so that the sleep stage recognition model can be used to improve the accuracy and robustness of sleep stage recognition.

[0070] Embodiment three

[0071] Figure 5 A structure schematic diagram of a sleep stage recognition device based on multi-modal signals is provided for the third embodiment of the application, the device can execute the sleep stage recognition method based on multi-modal signals provided by any embodiment of the application, and has the corresponding function modules and beneficial effects of the execution method. Figure 5 As shown in the figure, the device comprises:

[0072] The signal extraction module 310 is configured to extract a target lead signal from the multi-lead sleep monitoring signal of the target object, wherein the target lead signal comprises an electroencephalogram signal and an electrooculogram signal.

[0073] The signal processing module 320 is configured to perform data preprocessing on the target lead signal to obtain a reference lead signal, wherein the preprocessing comprises filtering, resampling, standardization and fragmentation.

[0074] The sleep stage prediction module 330 is configured to input the reference lead signal to a pre-trained sleep stage recognition model, and determine the sleep stage prediction result of the target object based on the output result of the sleep stage recognition model.

[0075] The sleep stage recognition model comprises a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit and a classification unit, the deep feature extraction unit is constructed based on a one-dimensional focus modulation network, the deep feature extraction unit adopts multi-scale one-dimensional deep convolution, and the two deep feature extraction units are respectively used for deep feature extraction of the electroencephalogram signal and the electrooculogram signal, and the dynamic gate fusion unit is used for modal adaptive fusion of outputs of the two deep feature extraction units.

[0076] Optionally, the preliminary feature extraction unit comprises a multi-scale one-dimensional convolution layer and a pooling layer.

[0077] Correspondingly, the sleep stage prediction module 330 is used for:

[0078] The reference lead signal is input into the preliminary feature extraction unit of the sleep stage recognition model, local feature extraction of the reference lead signal is performed through the multi-scale one-dimensional convolution layer, and initial features are obtained by performing spatial size compression on the extracted local features through the pooling layer.

[0079] Optionally, the deep feature extraction unit comprises a local context modeling layer, a global context modeling layer and a single-modal fusion output layer.

[0080] Correspondingly, the sleep stage prediction module 330 is further used for:

[0081] The initial features corresponding to the electroencephalogram signal are subjected to multi-scale one-dimensional deep convolution and local multi-scale aggregation through the local context modeling layer of the first deep feature extraction unit to obtain first features;

[0082] The initial features corresponding to the electroencephalogram signal are subjected to pooling compression and linear mapping through the global context modeling layer of the first deep feature extraction unit to obtain second features;

[0083] The first context modulator is generated according to the initial features, the first features and the second features corresponding to the electroencephalogram signal through the single-modal fusion output layer of the first deep feature extraction unit, the initial features corresponding to the electroencephalogram signal are subjected to element-level modulation by using the first context modulator, the modulated features are normalized to obtain deep features corresponding to the electroencephalogram signal;

[0084] The initial features corresponding to the electrooculogram signal are subjected to multi-scale one-dimensional deep convolution and local multi-scale aggregation through the local context modeling layer of the second deep feature extraction unit to obtain third features;

[0085] pooling and linear mapping of the initial features corresponding to the electrooculogram signal by a global context modeling layer of the second deep feature extraction unit to obtain fourth features;

[0086] generating a second context modulator according to the initial features, the third features and the fourth features corresponding to the electrooculogram signal by a single-modal fusion output layer of the second deep feature extraction unit, and performing element-level modulation on the initial features corresponding to the electrooculogram signal by using the second context modulator, and performing normalization on the modulated features to obtain deep features corresponding to the electrooculogram signal.

[0087] Optionally, the sleep stage prediction module 330 is further configured to:

[0088] performing feature splicing on the deep features corresponding to the electroencephalogram signal and the electrooculogram signal by a dynamic gate fusion unit of the sleep stage identification model for each time step after fragmentation, and determining a target gate coefficient based on the spliced features; wherein the target gate coefficient is used to represent the relative contribution degree of the deep features corresponding to the electroencephalogram signal and the electrooculogram signal.

[0089] performing weighting on the deep features corresponding to the electroencephalogram signal by using the target gate coefficient, performing weighting on the deep features corresponding to the electrooculogram signal by using the difference between the unit 1 and the target gate coefficient, and adding the weighted deep features to obtain multi-modal fusion features.

[0090] Optionally, the sleep stage prediction module 330 is further configured to:

[0091] performing forward long short-term memory network calculation and backward long short-term memory network calculation on the multi-modal fusion features corresponding to each time step by a bidirectional long short-term memory network unit of the sleep stage identification model, and splicing the forward and backward calculation results to obtain bidirectional output features of each time step;

[0092] determining an attention score corresponding to the bidirectional output features of each time step by an attention unit of the sleep stage identification model, performing normalization on the attention score to obtain an attention weight, and performing element weighting on the bidirectional output features according to the attention weight of each time step to obtain a target context feature with a reserved time dimension;

[0093] mapping the target context feature to a sleep stage type of each time step by a classification unit of the sleep stage identification model as a sleep stage prediction result of the target object.

[0094] Optionally, the sleep stage type includes a wake period, an N1 period, an N2 period, an N3 period and a REM period.

[0095] Optionally, the signal processing module 320 is configured to:

[0096] perform band-pass filtering on the target lead signal based on a Butterworth filter, and resample the filtered signal based on a target sampling rate;

[0097] respectively perform Z-score standardization on each channel of the resampled signal, and perform fragmentation processing on the standardized signal based on a target signal length to obtain a reference lead signal.

[0098] The sleep stage recognition device based on the multi-modal signal provided in the embodiments of the present application can perform the sleep stage recognition method based on the multi-modal signal provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0099] Embodiment four

[0100] Figure 6 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0101] As shown in Figure 6 The electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which are communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0102] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0103] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the sleep stage identification method based on multi-modal signals.

[0104] In some embodiments, the sleep stage identification method based on multi-modal signals can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the sleep stage identification method based on multi-modal signals described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the sleep stage identification method based on multi-modal signals by any other appropriate means, such as by means of firmware.

[0105] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0106] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0107] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0109] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0110] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0111] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, and the present disclosure is not limited in this regard.

[0112] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalents, alternatives, and / or improvements made to the specific embodiments described above are intended to fall within the scope of the present disclosure.

Claims

1. A sleep stage identification method based on multimodal signals, characterized in that, The method includes: Extract target lead signals from the polysomnography signals of the target subject; wherein, the target lead signals include electroencephalogram (EEG) signals and electrooculogram (EOG) signals; The target lead signal is preprocessed to obtain a reference lead signal; wherein the preprocessing includes filtering, resampling, normalization and fragmentation. The reference lead signal is input into a pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model. The sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction unit is constructed based on a one-dimensional focus modulation network and employs multi-scale one-dimensional depth convolution. The two deep feature extraction units are used to extract deep features from the EEG signal and the EEG signal, respectively. The dynamic gating fusion unit is used to perform modality adaptive fusion of the outputs of the two deep feature extraction units.

2. The method according to claim 1, characterized in that, The preliminary feature extraction unit includes a multi-scale one-dimensional convolutional layer and a pooling layer; Accordingly, the reference lead signal is input to a pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model, including: The reference lead signal is input into the preliminary feature extraction unit of the sleep stage recognition model. Local features are extracted from the reference lead signal through the multi-scale one-dimensional convolutional layer, and the extracted local features are spatially compressed through the pooling layer to obtain initial features.

3. The method according to claim 2, characterized in that, The deep feature extraction unit includes a local context modeling layer, a global context modeling layer, and a single-modal fusion output layer; Accordingly, the reference lead signal is input into a pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model, which further includes: The first feature is obtained by performing multi-scale one-dimensional deep convolution and local multi-scale aggregation on the initial features corresponding to the EEG signal through the local context modeling layer of the first deep feature extraction unit; The second feature is obtained by pooling, compressing and linearly mapping the initial features corresponding to the EEG signal through the global context modeling layer of the first deep feature extraction unit; The first deep feature extraction unit generates a first context modulator based on the initial features, first features and second features corresponding to the EEG signal through the single-modality fusion output layer. The first context modulator is then used to perform element-level modulation on the initial features corresponding to the EEG signal, and the modulated features are normalized to obtain the deep features corresponding to the EEG signal. The third feature is obtained by performing multi-scale one-dimensional deep convolution and local multi-scale aggregation on the initial features corresponding to the electrooculogram signal through the local context modeling layer of the second deep feature extraction unit; The fourth feature is obtained by pooling, compressing and linearly mapping the initial features corresponding to the electrooculogram signal through the global context modeling layer of the second deep feature extraction unit; The single-modal fusion output layer of the second deep feature extraction unit generates a second context modulator based on the initial, third, and fourth features corresponding to the electrooculogram (EOG) signal. The second context modulator is then used to perform element-level modulation on the initial features corresponding to the EOG signal, and the modulated features are normalized to obtain the deep features corresponding to the EOG signal.

4. The method according to claim 3, characterized in that, The process further includes inputting the reference lead signal into a pre-trained sleep stage recognition model, determining the sleep stage prediction result of the target object based on the output of the sleep stage recognition model, and including: For each time step after fragmentation, the deep features corresponding to the EEG and EEG signals are spliced ​​together by the dynamic gating fusion unit of the sleep stage recognition model, and the target gating coefficient is determined based on the spliced ​​features; wherein, the target gating coefficient is used to characterize the relative contribution of the deep features corresponding to the EEG and EEG signals; The deep features corresponding to the EEG signal are weighted using the target gating coefficient, and the deep features corresponding to the EEG signal are weighted using the difference between unit 1 and the target gating coefficient. The weighted deep features are then added together to obtain the multimodal fusion features.

5. The method according to claim 4, characterized in that, The process further includes inputting the reference lead signal into a pre-trained sleep stage recognition model, determining the sleep stage prediction result of the target object based on the output of the sleep stage recognition model, and including: The bidirectional long short-term memory network unit of the sleep stage recognition model performs forward long short-term memory network calculation and backward long short-term memory network calculation on the multimodal fusion features corresponding to each time step, and concatenates the forward and backward calculation results to obtain the bidirectional output features of each time step. The attention score corresponding to the bidirectional output feature at each time step is determined by the attention unit of the sleep stage recognition model. The attention score is normalized to obtain the attention weight. The bidirectional output feature is element-wise weighted according to the attention weight at each time step to obtain the target context feature that retains the temporal dimension. The sleep stage identification model uses its classification unit to map the target context features to the sleep stage type at each time step, which is then used as the sleep stage prediction result for the target object.

6. The method according to claim 5, characterized in that, The sleep stages include wakefulness, N1, N2, N3, and REM sleep.

7. The method according to any one of claims 1-6, characterized in that, The target lead signal is preprocessed to obtain a reference lead signal, including: The target lead signal is bandpass filtered based on a Butterworth filter, and the filtered signal is resampled based on the target sampling rate. Z-score normalization is performed on each channel of the resampled signal, and the normalized signal is fragmented based on the target signal length to obtain the reference lead signal.

8. A sleep stage recognition device based on multimodal signals, characterized in that, The device includes: A signal extraction module is used to extract target lead signals from the polysomnography signals of the target object; wherein, the target lead signals include electroencephalogram (EEG) signals and electrooculogram (EOG) signals; The signal processing module is used to perform data preprocessing on the target lead signal to obtain a reference lead signal; wherein the preprocessing includes filtering, resampling, normalization and fragmentation. The sleep stage prediction module is used to input the reference lead signal into a pre-trained sleep stage recognition model, and determine the sleep stage prediction result of the target object based on the output result of the sleep stage recognition model. The sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction unit is constructed based on a one-dimensional focus modulation network and employs multi-scale one-dimensional depth convolution. The two deep feature extraction units are used to extract deep features from the EEG signal and the EEG signal, respectively. The dynamic gating fusion unit is used to perform modality adaptive fusion of the outputs of the two deep feature extraction units.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the sleep stage identification method based on multimodal signals according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the sleep stage recognition method based on multimodal signals as described in any one of claims 1-7.

Images