Sleep stage identification method, device and equipment based on multi-mode signal and medium

Abstract

The invention discloses a sleep stage identification method and device based on a multi-mode signal, equipment and a medium. The method comprises the following steps: extracting a target lead signal comprising an electroencephalogram signal and an electro-oculogram signal from a polysleep monitoring signal of a target object; performing data preprocessing on the target lead signal to obtain a reference lead signal; inputting the reference lead signal into a sleep stage recognition model to obtain a sleep stage prediction result of the target object; wherein the sleep stage recognition model comprises a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long-short-term memory network unit, an attention unit and a classification unit, the deep feature extraction units are constructed based on a focus adjustment mechanism, and multi-scale one-dimensional deep convolution is adopted; and the two deep feature extraction units are used for performing deep feature extraction on the electroencephalogram signal and the electro-oculogram signal respectively. According to the scheme, the sleep stage recognition precision and robustness can be improved by using the sleep stage recognition model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal processing, and in particular to a sleep stage recognition method and device based on multi-modal signals, equipment and medium. BACKGROUND

[0002] Polysomnography (PSG) is the gold standard method for diagnosing various sleep disorders in clinical practice, which widely collects multi-modal physiological signals such as electroencephalogram (EEG), electrooculogram (EOG), electromyogram (EMG), respiratory airflow, chest and abdominal belts, oxygen saturation (SpO2), electrocardiogram (ECG), etc. Although these data provide rich pathological information, the existing PSG analysis process still highly depends on manual segment-by-segment interpretation, which not only consumes a long time and has high labor intensity, but also has the problems of poor result consistency and large subjective bias.

[0003] In recent years, deep learning methods have been gradually applied to sleep staging and respiratory event detection, such as DeepSleepNet, SleepTransformer, etc. However, most of them only process a single signal (such as EEG), ignoring the complementary information between multi-modal signals, resulting in poor model robustness and difficulty in processing signal missing or artifact interference. Existing multi-modal methods mostly use static fusion (such as direct concatenation), which cannot adaptively adjust the importance of different modalities according to different time points, limiting the modeling ability of the model to state mutations or modality dominance. In addition, the structure design of most models does not consider both temporal and global modeling capabilities, making it difficult to capture multi-scale features from local changes to global trends, and relying mostly on short segment input (such as 30s), which loses the context information across cycles, affecting the integrity and accuracy of the detection. SUMMARY

[0004] The present application provides a sleep stage recognition method, device, equipment and medium based on multi-modal signals, which constructs a sleep stage recognition model based on multi-modal signals for dynamic modeling, considering both local and global feature expression, and having the ability to process long-term sleep data. The sleep stage recognition model can be used to improve the accuracy and robustness of sleep stage recognition.

[0005] According to an aspect of the present application, a sleep stage recognition method based on multi-modal signals is provided, the method comprising:

[0006] extracting a target lead signal from a polysomnography signal of a target object; wherein the target lead signal includes an electroencephalogram signal and an electrooculogram signal;

[0007] performing data preprocessing on the target lead signal to obtain a reference lead signal; wherein the preprocessing includes filtering, resampling, standardization and segmentation;

[0008] The reference lead signal is input into a pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model.

[0009] The sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction unit is constructed based on a focus adjustment mechanism and employs multi-scale one-dimensional deep convolution. The two deep feature extraction units are used to extract deep features from the EEG signal and the EEG signal, respectively. The dynamic gating fusion unit is used to perform modality adaptive fusion of the outputs of the two deep feature extraction units.

[0010] According to another aspect of the present invention, a sleep stage identification device based on multimodal signals is provided, the device comprising:

[0011] A signal extraction module is used to extract target lead signals from the polysomnography signals of the target object; wherein, the target lead signals include electroencephalogram (EEG) signals and electrooculogram (EOG) signals;

[0012] The signal processing module is used to perform data preprocessing on the target lead signal to obtain a reference lead signal; wherein the preprocessing includes filtering, resampling, normalization, and fragmentation.

[0013] The sleep stage prediction module is used to input the reference lead signal into a pre-trained sleep stage recognition model, and determine the sleep stage prediction result of the target object based on the output result of the sleep stage recognition model.

[0014] The sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction unit is constructed based on a focus adjustment mechanism and employs multi-scale one-dimensional deep convolution. The two deep feature extraction units are used to extract deep features from the EEG signal and the EEG signal, respectively. The dynamic gating fusion unit is used to perform modality adaptive fusion of the outputs of the two deep feature extraction units.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the sleep stage recognition method based on multimodal signals according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the sleep stage identification method based on multimodal signals according to any embodiment of the present invention.

[0018] The technical solution of this invention first extracts target lead signals from the multichannel sleep monitoring signals of the target object; wherein, the target lead signals include EEG signals and EEG signals; then, the target lead signals are preprocessed to obtain reference lead signals; wherein, the preprocessing includes filtering, resampling, standardization, and fragmentation; then, the reference lead signals are input into a pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model; wherein, the sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction units are constructed based on a focus adjustment mechanism and employ multi-scale one-dimensional deep convolution. The two deep feature extraction units are used to extract deep features from the EEG signals and EEG signals, respectively, and the dynamic gating fusion unit is used to perform modal adaptive fusion of the outputs of the two deep feature extraction units. This technical solution constructs a sleep stage recognition model based on multimodal signals for dynamic modeling, taking into account both local and global feature representation, and capable of processing long-term sleep data. The accuracy and robustness of sleep stage recognition can be improved by utilizing the sleep stage recognition model.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a flowchart of a sleep stage identification method based on multimodal signals according to an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the workflow of a deep feature extraction unit according to an embodiment of the present invention;

[0023] Figure 3 This is a flowchart illustrating a sleep stage identification method based on multimodal signals according to an embodiment of the present invention.

[0024] Figure 4 This is a flowchart of another sleep stage identification method based on multimodal signals provided by an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of a sleep stage recognition device based on multimodal signals according to an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device that implements a sleep stage recognition method based on multimodal signals according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1This is a flowchart of a sleep stage identification method based on multimodal signals provided in Embodiment 1 of the present invention. This embodiment is applicable to situations requiring high-precision identification of sleep stages based on multimodal signals. The method can be executed by a sleep stage identification device based on multimodal signals, which can be implemented in hardware and / or software. This device can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0031] S110, extract target lead signals from the polysomnography signals of the target subject; wherein, the target lead signals include electroencephalogram (EEG) signals and electrooculogram (EOG) signals.

[0032] In this embodiment, target lead signals (such as the C4A1 channel of EEG and the EOGL channel of EOG) including EEG and EOG signals are first extracted from the target object's PSG recording file (e.g., .edf format). This extracted lead signal can be saved as a .npy file as an intermediate form for subsequent processing. The target object can refer to the subject undergoing sleep stage identification, and the target lead signal can serve as the basic monitoring multimodal data for sleep stage identification. It should be noted that this embodiment employs a Many-to-Many prediction mode. For input signals of any duration (e.g., 1 hour or an entire night), the model outputs the full-sequence sleep stage prediction results corresponding to the input duration in a single step, maintaining temporal continuity.

[0033] S120, the target lead signal is preprocessed to obtain the reference lead signal; the preprocessing includes filtering, resampling, normalization and fragmentation.

[0034] In this embodiment, after obtaining the target lead signal, it needs to be preprocessed to convert it into tensor data (.pt file) in a unified format that can be directly processed by the deep learning model. Specifically, the EEG (e.g., C4A1.npy) and EOG (e.g., EOGL.npy) signals of the target object are loaded from the file using NumPy's memory mapping mode, which effectively reduces the memory usage of high-resolution, long-term sequence data during loading, thereby achieving efficient data loading. To suppress DC drift and high-frequency noise, the EEG and EOG signals are first filtered using bandpass filters (e.g., a 4th-order Butterworth filter). The filtering range for the EEG signal can be set to 0.5-40 Hz, and the filtering range for the EOG signal can be set to 0.5-10 Hz. Since the original EEG and EOG signals have different sampling rates (e.g., EEG is 125Hz, EOG is 50Hz), it is necessary to resample each channel signal to the target sampling rate (e.g., 100Hz) to achieve accurate alignment of multimodal data in the time dimension and meet the requirements of the neural network input format.

[0035] Next, each signal channel is standardized to enhance the consistency of signal distribution among different individuals, thereby improving the training stability and generalization ability of the model. For example, Z-score standardization (zero mean, unit variance) can be used. Then, the standardized EEG and EOG signals are processed into fixed-length segments to obtain reference lead signals. A target length (e.g., 30 seconds) is set; if the original data is too long, it is truncated to the target length; if the original data is insufficient, zeros are padded at the end. For example, a 5-hour EEG / EOG signal corresponds to 5 × 3600 × 100 = 1,800,000 sampling points; a 6-hour EEG / EOG signal corresponds to 6 × 3600 × 100 = 2,160,000 sampling points. Finally, it can be saved as a structured PyTorch.pt file, including the processed EEG and EOG tensors, of type float32, with dimensions [1, duration × 100Hz]. This completes the entire processing flow from raw PSG multimodal signals to structured tensors, combining high efficiency, modularity, and scalability. It can support the preprocessing of signals of different lengths, providing flexible and standardized basic data support for multi-duration sleep modeling tasks.

[0036] S130, the reference lead signal is input into the pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model.

[0037] The sleep stage recognition model refers to a deep learning network model capable of identifying sleep stages. Specifically, the sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction units are constructed based on a focus adjustment mechanism and employ multi-scale one-dimensional deep convolution. The two deep feature extraction units are used to extract deep features from EEG and EEG signals, respectively. The dynamic gating fusion unit is used to perform modality adaptive fusion of the outputs of the two deep feature extraction units.

[0038] In this embodiment, considering the characteristics of sleep EEG / EOG signals, such as high sampling rate, long sequence, strong local features, and significant modal differences, a sleep stage recognition model with a two-level feature extraction structure is pre-constructed. A training dataset is then created based on reference PSG data to train the constructed sleep stage recognition model. The two-level feature extraction structure can capture long-range dependencies across cycles while preserving local sleep microstructures. The creation of the training dataset can refer to the processing method of the target PSG signal. Furthermore, sleep stage labels need to be parsed from the .rml tag file accompanying the PSG record file and saved as a .npy file, followed by label alignment and padding. For example, a manual label can be generated every 30 seconds, where N hours correspond to N×120 labels. The label sequence is automatically aligned with the signal length; if the number is insufficient, a preset label value is added to the end to indicate an invalid label. Invalid labels can be automatically ignored during model training to avoid interference with loss calculation. Finally, the sleep label sequence (type int64, dimension [duration × 120]) can be added to the corresponding structured PyTorch.pt file. This yields a training dataset that includes the processed EEG tensor and EOG tensor, as well as the corresponding sleep label sequences.

[0039] During the model training phase, an end-to-end learning strategy is employed, combining cross-entropy loss function, Adam optimizer, mixed precision training, and gradient accumulation techniques. Accuracy and macro-average F1 score are used to comprehensively evaluate the model. Among these, the loss... The calculation formula is: , Indicates the total number of categories. Indicates a specific category, Indicates category as The probability. The Adam optimizer is an adaptive gradient optimization algorithm that uses both first-order moment estimation and second-order moment estimation to dynamically adjust the learning rate. Specifically, the Adam optimizer can update the model parameters through the following update rules: (1) Update the first-order moment estimation and second-order moment estimation according to the current gradient and historical values: , .in, It is a first-order moment estimate, representing a weighted average of the gradients; It is a second-moment estimate, representing the weighted average of the squares of the gradients; This is the current gradient. (2) Deviation correction: Due to the initial stage, and There may be deviations, which can be corrected using the following formula: , .in, It is the corrected first moment of the gradient; It is the corrected second moment of the gradient. (3) Parameter update: The parameters of the final model Updated to: .in, It is the learning rate, which controls the step size of each update; It is a small constant, avoiding division by zero errors.

[0040] Assumption Indicates category The sample was correctly predicted as The number of samples, Indicates category Samples other than those specified were correctly predicted as non-standard. The number of samples, Indicates category Samples other than those listed were incorrectly predicted as The number of samples, Indicates category The number of samples that were incorrectly predicted as other classes. Therefore, the accuracy can be... Represented as: .category The corresponding precision can be expressed as: The macro average precision can be expressed as For categories The corresponding recall rate can be expressed as: Furthermore, the macro average recall rate can be expressed as: For categories Its corresponding F1 value can be expressed as: The macro average F1 score can be expressed as: .

[0041] In this embodiment, the sleep stage recognition model can be put into use for sleep stage prediction after training. Optionally, the preliminary feature extraction unit includes a multi-scale one-dimensional convolutional layer and a pooling layer; correspondingly, the reference lead signal is input into the pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model, including: inputting the reference lead signal into the preliminary feature extraction unit of the sleep stage recognition model, extracting local features from the reference lead signal through a multi-scale one-dimensional convolutional layer, and compressing the spatial size of the extracted local features through a pooling layer to obtain initial features.

[0042] Specifically, to avoid noise accumulation, gradient instability, and computational burden caused by directly inputting raw sampled-point EEG / EOG data into deep networks, this invention first employs a preliminary feature extraction unit composed of multi-scale one-dimensional convolutional layers and pooling layers to perform preliminary feature extraction on the reference lead signal. The shape of the EEG and EOG signal tensors in the reference lead signal is [B, 1, T], where B is the batch size and T is the original sequence length. The multi-scale one-dimensional convolutional layers utilize multiple one-dimensional convolutions with different kernel sizes to capture short-term key waveform structures in the EEG, such as sleep spindle waves, K-complex waves, and rapid eye movements, resulting in output features with higher feature aggregation than the original signal. Pooling layers are used to reduce the temporal resolution of the original EEG / EOG sequence (e.g., T→T / 2→T / 4), reducing noise and subsequent computational complexity, while simultaneously increasing the feature channel dimension. This allows the model to express various local waveform features from a higher-dimensional space, thereby obtaining a more structurally stable and semantically rich temporal feature representation. The output of the initial feature extraction unit is a feature sequence of dimension [B,L,C], where L is the length of the signal after convolution and pooling, and C is the number of feature channels. This feature sequence serves as the basic representation for subsequent deep feature extraction units.

[0043] In this embodiment, optionally, the deep feature extraction unit includes a local context modeling layer, a global context modeling layer, and a single-modal fusion output layer; correspondingly, the reference lead signal is input to a pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model. The method further includes: performing multi-scale one-dimensional deep convolution and local multi-scale aggregation on the initial features corresponding to the EEG signal through the local context modeling layer of the first deep feature extraction unit to obtain a first feature; performing pooling compression and linear mapping on the initial features corresponding to the EEG signal through the global context modeling layer of the first deep feature extraction unit to obtain a second feature; and generating a first context modulator based on the initial features, the first feature, and the second feature corresponding to the EEG signal through the single-modal fusion output layer of the first deep feature extraction unit. The first context modulator is used to perform element-wise modulation on the initial features corresponding to the EEG signal, and the modulated features are normalized to obtain the deep features corresponding to the EEG signal. The local context modeling layer of the second deep feature extraction unit performs multi-scale one-dimensional depth convolution and local multi-scale aggregation on the initial features corresponding to the EOS signal to obtain the third feature. The global context modeling layer of the second deep feature extraction unit performs pooling compression and linear mapping on the initial features corresponding to the EOS signal to obtain the fourth feature. The single-modality fusion output layer of the second deep feature extraction unit generates the second context modulator based on the initial features, the third feature and the fourth feature corresponding to the EOS signal, and uses the second context modulator to perform element-wise modulation on the initial features corresponding to the EOS signal, and the modulated features are normalized to obtain the deep features corresponding to the EOS signal.

[0044] It should be noted that, in order to efficiently capture local sleep microstructures (such as spindles and K-complexes) and long-range periodic patterns (such as sleep stage transitions) in long-term EEG / EOG, this invention structurally modulates the FocalNet architecture, constructing a one-dimensional focus modulation network (1D-FocalNet) suitable for physiological signals as a deep feature extraction unit. Traditional FocalNet is mainly used for image / video processing (2D / 3D) and cannot be directly applied to one-dimensional physiological signals such as sleep EEG / EOG. This invention specifically reconstructs its overall structure from two-dimensional to a one-dimensional convolutional structure, enabling the network to directly capture the local waveforms and global trends of EEG / EOG in the time dimension. Since one-dimensional physiological signals do not have a two-dimensional spatial dimension, this invention removes the spatial window partitioning mechanism in the original FocalNet, retaining only the local convolution and global modulation framework along the time dimension, making it adaptable to the structural characteristics of time series and significantly reducing computational complexity. Compared with traditional Transformer self-attention, 1D-FocalNet has the following advantages: 1. No need to construct... Attention matrix, computational complexity from Down to 2. More sensitive to local structures: Because key physiological events in sleep EEG (such as spindle waves and K-complexes) have defined local time windows, convolutional structures achieve pattern detection within a short time window by fixing the receptive field, exhibiting better local sensitivity and computational advantages in several physiological event detection scenarios. 3. Adaptable to ultra-long sequences (hundreds to thousands of time slices), allowing deployment in clinical devices with limited memory.

[0045] This invention combines the typical event durations of EEG / EOG to design a set of convolutional kernels at a 100Hz sampling rate. The designed convolutional kernels (15 / 21 / 27 / 33 / 39 points) correspond to a receptive field of 150–390ms at 100Hz sampling, covering the high-frequency oscillation components of sleep spindles (oscillation period 100-300ms), the key burst components of K-complexes (200-600ms, but the most discriminative burst peak components and rapid transition segments are usually concentrated in the first 200-400ms), and the transient waveforms of micro-arousals (100-300ms). The above-mentioned kernel size is only a preferred embodiment; those skilled in the art can adjust the convolutional receptive field according to their needs. These timescales all belong to the key local structural windows of sleep events. By fully aligning the convolutional receptive field with clinical features, the model can extract local waveform features based on physiological timescales, naturally possessing the ability to recognize sleep events. It should be noted that the above-mentioned convolution kernel size design is a structure optimization driven by clinical prior knowledge, which is different from the existing technology of arbitrarily setting the convolution size and has a clear technical effect.

[0046] In this embodiment, since EEG and EOG signals have different frequency bands and structural characteristics, independent 1D-FocalNet branches are constructed for each. 1D-FocalNet aims to efficiently capture local microstructures and global periodic patterns in the input sequence, replacing traditional attention mechanisms to achieve context-aware modeling. Specifically, the first deep feature extraction unit is used for deep feature extraction of EEG signals, focusing on high-frequency local rhythmic changes; the second deep feature extraction unit is used for deep feature extraction of EOG signals, focusing on low-frequency eye-movement trends.

[0047] Figure 2 This diagram illustrates the workflow of a deep feature extraction unit. Specifically, for example... Figure 2 As shown, deep feature extraction of EEG signals is illustrated as an example. Assume the initial feature representation of the EEG signal is as follows: First, local context features under different receptive fields are extracted through parallel multi-scale one-dimensional deep convolution in the local context modeling layer of the first deep feature extraction unit. Specifically, this can be represented as follows: .in, Indicates the first Local contextual features at each scale, Indicates the first The size of the convolutional kernel; Indicates the use of convolution kernel One-dimensional depthwise convolution excels at capturing local oscillations, such as spindle waves and K-complexes. Among these, multi-scale one-dimensional depthwise convolution (see...) Figure 2 The `DWConv` in the model is a grouped convolutional structure, specifically configured so that the number of groups equals the number of input channels, allowing each convolutional kernel to independently extract features from the time series of one channel. This structure significantly reduces the number of model parameters and computational complexity while ensuring a large receptive field (e.g., a kernel size of 31 or larger) to capture long-term temporal features. Then, the first feature is obtained through local multi-scale aggregation via a local context modeling layer, which can be represented as: .in, Indicates the first feature (see Figure 2 (local context features) This represents the number of multi-scale convolutions. Next, the initial features corresponding to the EEG signal are pooled, compressed, and linearly mapped using the global context modeling layer of the first deep feature extraction unit to obtain the second feature, thereby modeling the periodic global context, as shown below: , .in, Represents the global features after pooling (see...) Figure 2 Global pooling features in (the context of pooling). This indicates average pooling over the time dimension. Represents the global modulation linear weights. Indicates bias. Indicates the second feature (see Figure 2 (Global context features in the text). Then, the single-modal fusion output layer of the first deep feature extraction unit is based on... , and A first context modulator is generated, and the initial features corresponding to the EEG signal are element-wise modulated using the first context modulator. The modulated features are then normalized, as shown in the following formula: .in, This represents the first context modulator generated (see [link]). Figure 2 Modulation in (the text) This represents element-wise multiplication. This indicates the deep features corresponding to the EEG signal; Representation layer normalization, specifically, is performed along the feature channel dimension, i.e., calculating the mean and variance of the feature vector at each time step. This is commonly used in neural networks to accelerate training and improve model stability. It should be noted that the feature fusion in this embodiment is not a simple linear superposition, but rather uses a context modulator to perform element-wise multiplication modulation on the initial features, achieving adaptive enhancement of key signal patterns. Its feature processing method is significantly different from the traditional residual addition structure. The deep feature extraction process for EOG signals can refer to the above process and will not be elaborated further.

[0048] In this embodiment, optionally, the reference lead signal is input to a pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model. The method further includes: for each fragmented time step, the deep features corresponding to the EEG signal and the EEG signal are spliced ​​together by the dynamic gating fusion unit of the sleep stage recognition model, and a target gating coefficient is determined based on the spliced ​​features; wherein the target gating coefficient is used to characterize the relative contribution of the deep features corresponding to the EEG signal and the EEG signal; the deep features corresponding to the EEG signal are weighted using the target gating coefficient, and the deep features corresponding to the EEG signal are weighted using the difference between unit 1 and the target gating coefficient, and the weighted deep features are added together to obtain a multimodal fusion feature.

[0049] In this embodiment, considering the varying signal-to-noise ratio and feature contributions of EEG / EOG at different time periods, the fusion ratio of the two modalities is dynamically adjusted to enhance the model's robustness against single-modal degradation. Specifically, the deep features corresponding to the EEG signal are denoted as... The deep features corresponding to the EOG signal are denoted as For each time step The sleep stage recognition model uses a dynamic gating fusion unit to concatenate deep features corresponding to two modalities, and determines the target gating coefficient based on the concatenated features, as specifically expressed as: , .in, This represents the deep features after splicing. Represents the gate weight matrix; Indicates gating bias; It is the Sigmoid activation function, which makes ; Indicates the target gating coefficient. Decide on the time step The relative contributions of EEG and EOG are considered. This gating mechanism can adaptively adjust modal weights according to the physiological characteristics of sleep stages. For example, it automatically increases the weight of electrooculogram (EOG) signals during REM sleep and increases the weight of electroencephalogram (EEG) signals during deep sleep (N3), thereby improving the recognition performance of aliasing states. Then, based on... The deep features corresponding to the two modalities at each time step are fused to obtain the multimodal fused features for each time step. , can be represented as: +(1- ) Then concatenate all the time steps in sequence to obtain... This is used as the final output of the dynamic gating fusion unit.

[0050] In this embodiment, optionally, the reference lead signal is input into a pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model. This further includes: performing forward and backward long short-term memory network calculations on the multimodal fusion features corresponding to each time step using the bidirectional long short-term memory network unit of the sleep stage recognition model, and concatenating the forward and backward calculation results to obtain the bidirectional output features for each time step; determining the attention score corresponding to the bidirectional output features at each time step using the attention unit of the sleep stage recognition model, normalizing the attention score to obtain the attention weight, and element-wise weighting the bidirectional output features according to the attention weight at each time step to obtain the target context features that retain the temporal dimension; and mapping the target context features to the sleep stage type at each time step using the classification unit of the sleep stage recognition model as the sleep stage prediction result for the target object.

[0051] The bidirectional long short-term memory (BiLSTM) network unit can be used to capture long-range dependencies and model the transition patterns of sleep stages. Through BiLSTM, the model can simultaneously integrate forward and backward contextual information, thus more accurately characterizing the transitional features and temporal structure between sleep stages. It should be noted that since the length of sleep sequences far exceeds that of natural language tasks, directly using self-attention would incur huge computational overhead. Therefore, this invention uses BiLSTM instead of the Transformer structure for long-range dependency modeling. Specifically, BiLSTM performs long-range dependency modeling at each time step... Simultaneous computation of forward LSTM and backward LSTM can be performed; see the formula: , Then, the forward and backward calculation results are concatenated, i.e. ,in The hidden state dimension of the LSTM is used to obtain the bidirectional output features at each time step. Then concatenate all the time steps in sequence to obtain... This is then used as the final output of the Bidirectional Long Short-Term Memory (BiLSTM) network unit. The BiLSTM can use future information to correct current judgments. For example, if a segment appears to be "awake" but is preceded by "deep sleep," the model will correct it to "deep sleep." This is a contextual encoding, not a feedback loop, but it achieves functional global correction. Essentially, it introduces contextual information of future segments through the hidden states of the backward LSTM. Building upon inter-block sequence modeling, this invention further proposes a global trend correction mechanism. It utilizes the global contextual information integrated in the latent space by the BiLSTM network to model the trend consistency of local features at the semantic scale, thereby improving prediction continuity. This mechanism does not use a separate independent structure but rather leverages the BiLSTM network's ability to encode the context of the entire sequence. By remodulating local features through the distribution of hidden states over the entire night, local predictions can automatically align with the overall pattern of the night's sleep structure, thus improving the stability of long-term predictions.

[0052] The "remodulation" described in this invention does not form a structural feedback loop, but rather utilizes a bidirectional long short-term memory network to integrate the entire sequence context in forward and backward computations, enabling local features to obtain global trend constraints in the latent space. That is, during inference, the model uses the global distribution of bidirectional latent states to perform semantic-scale correction on local representations, thereby ensuring consistency in predictions at each time step within the overall sleep stage evolution pattern.

[0053] In this embodiment, to enhance the model's ability to perceive key segments (such as the REM sleep phase) in long time series, an attention mechanism is introduced to construct attention units, which are used to weight and aggregate the time series representations output by the BiLSTM units. By automatically learning the importance weights for each time step, the model can focus on the information segments that contribute most to the judgment of the current sleep state. Specifically, for each time step... The attention score is calculated based on the corresponding bidirectional output features, as shown in the following formula: .in, Indicates attention score, Let be the weight matrix of the attention linear transformation. For linear transformation bias, For the internal projection dimension of the attention mechanism, It is a non-linear activation function; The attention query vector is used to map the projected vector to a scalar. Then, the attention score is normalized to obtain the attention weights, as shown in the following formula: Then, based on the attention weights at each time step, the bidirectional output features are element-wise weighted to obtain the target context features that retain the temporal dimension, as shown in the following formula: .in, Indicates the first Target context features at each time step.

[0054] Optionally, before inputting the target context features into the classification unit, the present invention may also set a temporal refine head to perform further temporal modeling and smoothing on the attention-weighted feature sequence, so as to enhance the continuity and stability of the prediction results of adjacent time steps.

[0055] In a preferred embodiment of the present invention, the temporal refining head can be constructed using a one-dimensional temporal convolutional network (TCN) based on a dilated convolution architecture or a bidirectional long short-term memory network (BiLSTM), and can be combined with residual connections to achieve secondary extraction of local context and error suppression.

[0056] Through this time-series refinement head, the model can further integrate short- and medium-range time dependencies after attention is focused, thereby reducing fragment-level misjudgments caused by noise interference and improving the temporal consistency and physiological rationality of sleep stage recognition.

[0057] The classification unit receives attention-weighted target context features and maps them to the sleep stage category at each time step. Optionally, sleep stage types include wakefulness, N1, N2, N3, and REM sleep. The classification unit undertakes the final prediction task and is the information convergence output layer of the entire model; its design must balance expressive power and parameter efficiency. For example, the classification unit can be constructed using the classification head of a feedforward network. Specifically, a linear layer first maps the high-order features of each time step to the hidden space, as shown in the following formula: Building upon this, to suppress class imbalance caused by long-tail distribution, a Logit adjustment mechanism is introduced. Assume that the prior probabilities of each class in the pre-calculated training set are... Using adjustment coefficient Features of hidden space Log-probability adjustment is performed to obtain The calculation formula is: Then, a Softmax layer is used to adjust the features. Normalization is performed, thus outputting the class probability at each time step, as shown in the following formula: ftmax( From this, we can obtain the predicted probability distribution for each time step, whose tensor dimension is... ,in, This represents the number of categories (set to 5 here), and the category with the highest probability is used as the sleep stage prediction result for the corresponding time step.

[0058] Figure 3 A flowchart illustrating a sleep stage identification method based on multimodal signals is shown. Figure 3 As shown, the acquired EEG and EOG signals are processed by a preliminary feature extraction unit (not yet in the initial feature extraction stage). Figure 3 After initial feature extraction (as shown in the diagram), the features are processed by the first and second deep feature extraction units respectively. Then, a dynamic gating fusion unit dynamically and adaptively fuses the output features of the two deep feature extraction units. Following this, a bidirectional long short-term memory network unit captures long-range dependencies, and finally, an attention unit (not shown in the diagram) performs the final fusion. Figure 3 The output features of BiLSTM are weighted and aggregated (as shown in the diagram), and finally processed by a classification unit (i.e., ...). Figure 3 The classification head in the middle is used to identify sleep stages (i.e., sleep staging).

[0059] This invention proposes a three-layer sequence modeling structure in the field of sleep staging: (1) Intra-block: The initial features of each time step are input into FocalNet to capture local microstructures such as spindle, K-complex, and slow wave. (2) Inter-block: The feature sequences of all time steps are input into a lightweight BiLSTM temporal network to learn the sleep cycle pattern of the whole night. (3) Global trend correction: The sleep cycle trend of the whole night obtained by the inter-block temporal network is used as a correction factor to remodulate the local features of each time step, so that the local staging results are consistent with the sleep structure of the whole night, thereby improving the continuity of prediction and physiological rationality. This structure is completely different from the planar modeling of Transformer and is a hierarchical sequence modeling method that adapts to physiological patterns. In addition, compared with traditional modal splicing, the cross-modal dynamic modulation mechanism (i.e., dynamic gating fusion unit) proposed in this invention can adaptively adjust the importance of different modalities according to the sleep stage, and the effect is particularly prominent in sleep stages such as REM and N1. It has been verified that this invention achieves 87% accuracy and 80% F1 score on the SHHS1 public dataset, proving the effectiveness of the structure of this invention.

[0060] The technical solution of this invention first extracts target lead signals from the multichannel sleep monitoring signals of the target object; wherein, the target lead signals include EEG signals and EEG signals; then, the target lead signals are preprocessed to obtain reference lead signals; wherein, the preprocessing includes filtering, resampling, standardization, and fragmentation; then, the reference lead signals are input into a pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model; wherein, the sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction units are constructed based on a focus adjustment mechanism and employ multi-scale one-dimensional deep convolution. The two deep feature extraction units are used to extract deep features from the EEG signals and EEG signals, respectively, and the dynamic gating fusion unit is used to perform modal adaptive fusion of the outputs of the two deep feature extraction units. This technical solution constructs a sleep stage recognition model based on multimodal signals for dynamic modeling, taking into account both local and global feature representation, and capable of processing long-term sleep data. The accuracy and robustness of sleep stage recognition can be improved by utilizing the sleep stage recognition model.

[0061] Example 2

[0062] Figure 4 This is a flowchart of a sleep stage identification method based on multimodal signals provided in Embodiment 2 of the present invention. This embodiment is an optimization based on the above embodiment. Specifically, the optimization is as follows: data preprocessing is performed on the target lead signal to obtain a reference lead signal, including: bandpass filtering of the target lead signal based on a Butterworth filter, and resampling of the filtered signal based on the target sampling rate; Z-score normalization is performed on each channel of the resampled signal, and the normalized signal is fragmented based on the target signal length to obtain the reference lead signal.

[0063] like Figure 4 As shown, the method in this embodiment specifically includes the following steps:

[0064] S210, extract target lead signals from the polysomnography signals of the target subject; wherein, the target lead signals include electroencephalogram (EEG) signals and electrooculogram (EOG) signals.

[0065] S220 performs bandpass filtering on the target lead signal based on a Butterworth filter and resamples the filtered signal based on the target sampling rate.

[0066] S230 performs Z-score normalization on each channel of the resampled signal and fragments the normalized signal based on the target signal length to obtain the reference lead signal.

[0067] S240, the reference lead signal is input into the pre-trained sleep stage recognition model, and the sleep stage prediction result of the target object is determined based on the output result of the sleep stage recognition model.

[0068] The sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction unit is constructed based on a focus adjustment mechanism and employs multi-scale one-dimensional depth convolution. The two deep feature extraction units are used to extract deep features from EEG and EEG signals, respectively. The dynamic gating fusion unit is used to perform modal adaptive fusion of the outputs of the two deep feature extraction units.

[0069] The technical solution of this invention constructs a sleep stage recognition model based on multimodal signals for dynamic modeling, taking into account both local and global feature representation, and capable of processing long-term sleep data. The sleep stage recognition model can be used to improve the accuracy and robustness of sleep stage recognition.

[0070] Example 3

[0071] Figure 5 This is a schematic diagram of a sleep stage recognition device based on multimodal signals provided in Embodiment 3 of the present invention. This device can execute the sleep stage recognition method based on multimodal signals provided in any embodiment of the present invention, and possesses the corresponding functional modules and beneficial effects of the method. For example... Figure 5 As shown, the device includes:

[0072] The signal extraction module 310 is used to extract target lead signals from the polysomnography signals of the target object; wherein, the target lead signals include electroencephalogram (EEG) signals and electrooculogram (EOG) signals;

[0073] The signal processing module 320 is used to perform data preprocessing on the target lead signal to obtain a reference lead signal; wherein, the preprocessing includes filtering, resampling, normalization and fragmentation;

[0074] The sleep stage prediction module 330 is used to input the reference lead signal into a pre-trained sleep stage recognition model, and determine the sleep stage prediction result of the target object based on the output result of the sleep stage recognition model.

[0075] The sleep stage recognition model includes a preliminary feature extraction unit, two deep feature extraction units, a dynamic gating fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction unit is constructed based on a focus adjustment mechanism and employs multi-scale one-dimensional deep convolution. The two deep feature extraction units are used to extract deep features from the EEG signal and the EEG signal, respectively. The dynamic gating fusion unit is used to perform modality adaptive fusion of the outputs of the two deep feature extraction units.

[0076] Optionally, the preliminary feature extraction unit includes a multi-scale one-dimensional convolutional layer and a pooling layer;

[0077] Accordingly, the sleep stage prediction module 330 is used for:

[0078] The reference lead signal is input into the preliminary feature extraction unit of the sleep stage recognition model. Local features are extracted from the reference lead signal through the multi-scale one-dimensional convolutional layer, and the extracted local features are spatially compressed through the pooling layer to obtain initial features.

[0079] Optionally, the deep feature extraction unit includes a local context modeling layer, a global context modeling layer, and a single-modal fusion output layer;

[0080] Accordingly, the sleep stage prediction module 330 is also used for:

[0081] The first feature is obtained by performing multi-scale one-dimensional deep convolution and local multi-scale aggregation on the initial features corresponding to the EEG signal through the local context modeling layer of the first deep feature extraction unit;

[0082] The second feature is obtained by pooling, compressing and linearly mapping the initial features corresponding to the EEG signal through the global context modeling layer of the first deep feature extraction unit;

[0083] The first deep feature extraction unit generates a first context modulator based on the initial features, first features and second features corresponding to the EEG signal through the single-modality fusion output layer. The first context modulator is then used to perform element-level modulation on the initial features corresponding to the EEG signal, and the modulated features are normalized to obtain the deep features corresponding to the EEG signal.

[0084] The third feature is obtained by performing multi-scale one-dimensional deep convolution and local multi-scale aggregation on the initial features corresponding to the electrooculogram signal through the local context modeling layer of the second deep feature extraction unit;

[0085] The fourth feature is obtained by pooling, compressing and linearly mapping the initial features corresponding to the electrooculogram signal through the global context modeling layer of the second deep feature extraction unit;

[0086] The single-modal fusion output layer of the second deep feature extraction unit generates a second context modulator based on the initial, third, and fourth features corresponding to the electrooculogram (EOG) signal. The second context modulator is then used to perform element-level modulation on the initial features corresponding to the EOG signal, and the modulated features are normalized to obtain the deep features corresponding to the EOG signal.

[0087] Optionally, the sleep stage prediction module 330 is further configured to:

[0088] For each time step after fragmentation, the deep features corresponding to the EEG and EEG signals are spliced ​​together by the dynamic gating fusion unit of the sleep stage recognition model, and the target gating coefficient is determined based on the spliced ​​features; wherein, the target gating coefficient is used to characterize the relative contribution of the deep features corresponding to the EEG and EEG signals;

[0089] The deep features corresponding to the EEG signal are weighted using the target gating coefficient, and the deep features corresponding to the EEG signal are weighted using the difference between unit 1 and the target gating coefficient. The weighted deep features are then added together to obtain the multimodal fusion features.

[0090] Optionally, the sleep stage prediction module 330 is further configured to:

[0091] The bidirectional long short-term memory network unit of the sleep stage recognition model performs forward long short-term memory network calculation and backward long short-term memory network calculation on the multimodal fusion features corresponding to each time step, and concatenates the forward and backward calculation results to obtain the bidirectional output features of each time step.

[0092] The attention score corresponding to the bidirectional output feature at each time step is determined by the attention unit of the sleep stage recognition model. The attention score is normalized to obtain the attention weight. The bidirectional output feature is element-wise weighted according to the attention weight at each time step to obtain the target context feature that retains the temporal dimension.

[0093] The sleep stage identification model uses its classification unit to map the target context features to the sleep stage type at each time step, which is then used as the sleep stage prediction result for the target object.

[0094] Optionally, the sleep stage types include wakefulness, N1, N2, N3 and REM sleep.

[0095] Optionally, the signal processing module 320 is used for:

[0096] The target lead signal is bandpass filtered based on a Butterworth filter, and the filtered signal is resampled based on the target sampling rate.

[0097] Z-score normalization is performed on each channel of the resampled signal, and the normalized signal is fragmented based on the target signal length to obtain the reference lead signal.

[0098] The sleep stage recognition device based on multimodal signals provided in this embodiment of the invention can execute the sleep stage recognition method based on multimodal signals provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0099] Example 4

[0100] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0101] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0102] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a sleep stage recognition method based on multimodal signals.

[0104] In some embodiments, the sleep stage identification method based on multimodal signals can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the sleep stage identification method based on multimodal signals described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the sleep stage identification method based on multimodal signals by any other suitable means (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0109] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0110] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0111] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A sleep stage recognition method based on multi-modal signals, characterized in that, The method comprises: extracting a target lead signal from a polysomnography signal of a target object; wherein the target lead signal comprises an electroencephalogram signal and an electrooculogram signal; data preprocessing the target lead signal to obtain a reference lead signal; wherein the preprocessing comprises filtering, resampling, standardization, and fragmentation; inputting the reference lead signal into a pre-trained sleep stage recognition model, and determining a sleep stage prediction result of the target object based on an output result of the sleep stage recognition model; wherein the sleep stage recognition model comprises a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit, the deep feature extraction unit is constructed based on a focal adjustment mechanism, the deep feature extraction unit adopts a multi-scale one-dimensional deep convolution, the two deep feature extraction units are respectively used for deep feature extraction of the electroencephalogram signal and the electrooculogram signal, and the dynamic gate fusion unit is used for modal adaptive fusion of outputs of the two deep feature extraction units.

2. The method of claim 1, wherein, The preliminary feature extraction unit comprises a multi-scale one-dimensional convolution layer and a pooling layer. Correspondingly, inputting the reference lead signal into a pre-trained sleep stage recognition model, and determining a sleep stage prediction result of the target object based on an output result of the sleep stage recognition model, comprises: inputting the reference lead signal into the preliminary feature extraction unit of the sleep stage recognition model, performing local feature extraction on the reference lead signal through the multi-scale one-dimensional convolution layer, and performing spatial size compression on the extracted local features through the pooling layer to obtain initial features.

3. The method of claim 2, wherein, The deep feature extraction unit comprises a local context modeling layer, a global context modeling layer, and a single-modal fusion output layer. Correspondingly, inputting the reference lead signal into a pre-trained sleep stage recognition model, and determining a sleep stage prediction result of the target object based on an output result of the sleep stage recognition model, further comprises: performing multi-scale one-dimensional deep convolution and local multi-scale aggregation on the initial features corresponding to the electroencephalogram signal through the local context modeling layer of the first deep feature extraction unit to obtain first features; performing pooling compression and linear mapping on the initial features corresponding to the electroencephalogram signal through the global context modeling layer of the first deep feature extraction unit to obtain second features; generating a first context modulator according to the initial features, the first features, and the second features corresponding to the electroencephalogram signal through the single-modal fusion output layer of the first deep feature extraction unit, and performing element-level modulation on the initial features corresponding to the electroencephalogram signal by using the first context modulator, and normalizing the modulated features to obtain deep features corresponding to the electroencephalogram signal; performing multi-scale one-dimensional deep convolution and local multi-scale aggregation on the initial features corresponding to the electrooculogram signal through the local context modeling layer of the second deep feature extraction unit to obtain third features; performing pooling compression and linear mapping on the initial features corresponding to the electrooculogram signal through the global context modeling layer of the second deep feature extraction unit to obtain fourth features; The single-modal fusion output layer of the second deep feature extraction unit generates a second context modulator according to the initial feature, the third feature and the fourth feature corresponding to the electro-oculogram signal, and modulates the initial feature corresponding to the electro-oculogram signal at an element level by using the second context modulator, and normalizes the modulated feature to obtain the deep feature corresponding to the electro-oculogram signal.

4. The method of claim 3, wherein, The reference lead signal is input into a pre-trained sleep stage recognition model, and a sleep stage prediction result of the target object is determined based on an output result of the sleep stage recognition model, and the method further includes: For each time step after fragmentation, the deep features corresponding to the electroencephalogram signal and the electro-oculogram signal are spliced by a dynamic gate fusion unit of the sleep stage recognition model, and a target gate coefficient is determined based on the spliced features; wherein the target gate coefficient is used to represent the relative contribution degree of the deep features corresponding to the electroencephalogram signal and the electro-oculogram signal; The deep features corresponding to the electroencephalogram signal are weighted by using the target gate coefficient, the deep features corresponding to the electro-oculogram signal are weighted by using the difference between the unit 1 and the target gate coefficient, and the weighted deep features are added to obtain a multi-modal fusion feature.

5. The method of claim 4, wherein, The reference lead signal is input into a pre-trained sleep stage recognition model, and a sleep stage prediction result of the target object is determined based on an output result of the sleep stage recognition model, and the method further includes: The multi-modal fusion feature corresponding to each time step is calculated by a forward long short-term memory network and a backward long short-term memory network through a bidirectional long short-term memory network unit of the sleep stage recognition model, and the forward and backward calculation results are spliced to obtain a bidirectional output feature of each time step; The attention score corresponding to the bidirectional output feature of each time step is determined by an attention unit of the sleep stage recognition model, the attention score is normalized to obtain an attention weight, and the bidirectional output feature is weighted element by element according to the attention weight of each time step to obtain a target context feature with a reserved time dimension; The target context feature is mapped to a sleep stage type of each time step by a classification unit of the sleep stage recognition model as the sleep stage prediction result of the target object.

6. The method of claim 5, wherein, The sleep stage type includes a wake period, an N1 period, an N2 period, an N3 period and a REM period.

7. The method according to any one of claims 1 to 6, characterized in that, The reference lead signal is obtained by data preprocessing of the target lead signal, including: The target lead signal is band-pass filtered based on a Butterworth filter, and the filtered signal is resampled based on a target sampling rate; Each channel of the resampled signal is respectively subjected to Z-score standardization, and the standardized signal is fragmented based on a target signal length to obtain the reference lead signal.

8. A sleep stage recognition device based on multimodal signals, characterized in that, The device includes: A signal extraction module is configured to extract a target lead signal from a multi-lead sleep monitoring signal of a target object; wherein the target lead signal includes an electroencephalogram signal and an electro-oculogram signal; A signal processing module is configured to obtain a reference lead signal by data preprocessing of the target lead signal; wherein the preprocessing includes filtering, resampling, standardization and fragmentation. The sleep stage prediction module is configured to input the reference lead signal into a pre-trained sleep stage recognition model, and determine a sleep stage prediction result of the target subject based on an output result of the sleep stage recognition model. The sleep stage recognition model comprises a preliminary feature extraction unit, two deep feature extraction units, a dynamic gate fusion unit, a bidirectional long short-term memory network unit, an attention unit, and a classification unit. The deep feature extraction units are constructed based on a focal adjustment mechanism. The deep feature extraction units adopt multi-scale one-dimensional deep convolution. The two deep feature extraction units are respectively configured to perform deep feature extraction on the electroencephalogram signal and the electrooculogram signal. The dynamic gate fusion unit is configured to perform modal adaptive fusion on outputs of the two deep feature extraction units.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to execute the sleep stage recognition method based on a multi-modal signal according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the sleep stage recognition method based on a multi-modal signal according to any one of claims 1-7 when executed.

Images