Sleep monitoring method and system based on electroencephalogram signal

Abstract

This application provides a sleep monitoring method and system based on electroencephalogram (EEG) signals. The method includes: acquiring EEG signals of a user during sleep; processing the EEG signals to obtain multiple input signals of a set time length, wherein each input signal contains alpha waves, beta waves, delta waves, theta waves, spindle waves, and sawtooth waves of the EEG signal within the same time period; for each input signal: inputting the input signal into a preset sleep monitoring model to obtain the sleep stage category corresponding to the input signal output by the sleep monitoring model, wherein the sleep monitoring model includes a signal feature extraction module and a sleep stage classification module, the signal feature extraction module is used to extract features from the input signal and use the extracted signal features as the input vector of the sleep stage classification module, the sleep stage classification module is used to classify and identify the sleep stage of the input vector to obtain the sleep stage category corresponding to the input signal.

Description

Technical Field

[0001] The present application relates to the technical field of sleep monitoring, and in particular to a sleep monitoring method and system based on electroencephalogram signals. Background Art

[0002] Sleep is one of the important factors that affect human physical and mental health. With the acceleration of the pace of modern life, more and more people suffer from sleep disorders, which has an adverse impact on people's life, study, work, etc. At present, medical research on sleep problems mainly focuses on the diagnosis and evaluation of sleep stages, and professional doctors determine the patient's sleep status. This method relies heavily on the experience of doctors, is inefficient, and is prone to overlooking factors and leading to misjudgment.

[0003] EEG signals are a kind of tiny biological signals with a certain rhythm that are spontaneously generated by nerve cells. They have the characteristics of high noise, relatively weak effective signals, complex composition, and strong randomness. These characteristics make it difficult to extract the features of EEG signals. Although the characteristics of EEG signals vary from individual to individual and at different times, the EEG waveforms generated by nerve cell activity have certain regularity. Therefore, artificial intelligence methods can be used to extract and classify EEG signals.

[0004] The four stages of sleep are drowsiness, light sleep, deep sleep, and rapid eye movement sleep. The brainwave activity in the drowsiness stage is mainly 4-7.5Hz theta rhythm, the characteristics of the brainwaves in the light sleep stage are sleep spindles and K-complex waves, the characteristics of the brainwaves in the deep sleep stage are delta waves, and the brainwaves in the rapid eye movement sleep stage show mixed frequency desynchronized low amplitude. The characteristics of sleep stages are the basis for the classification and identification of sleep stages.

[0005] Existing EEG signal feature extraction usually uses FFT (Fast Fourier Transform), wavelet transform and Hermite transform to extract the features of EEG signals in the time domain and frequency domain. After extracting the EEG signal features, classification methods such as SVM, random forest, K-means clustering algorithm can be used to classify the sleep stages of EEG signals, but the classification and recognition effect still needs to be further improved. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a sleep monitoring method and system based on EEG signals to improve the sleep stage classification effect based on EEG signals.

[0007] In order to achieve the above purpose, the embodiments of the present application are implemented in the following manner:

[0008] In a first aspect, an embodiment of the present application provides a sleep monitoring method based on EEG signals, comprising: obtaining EEG signals of a user in a sleeping state; processing the EEG signals to obtain multiple input signals of a set time length, wherein each input signal comprises α wave, β wave, δ wave, θ wave, shuttle wave, and sawtooth wave of the EEG signals in the same time period; for each input signal: inputting the input signal into a preset sleep monitoring model to obtain a sleep stage category corresponding to the input signal output by the sleep monitoring model, wherein the sleep monitoring model comprises a signal feature extraction module and a sleep stage classification module, the signal feature extraction module being used to perform feature extraction on the input signal, and using the extracted signal features as input vectors of the sleep stage classification module, the sleep stage classification module being used to perform sleep stage classification and identification on the input vector to obtain the sleep stage category corresponding to the input signal.

[0009] In combination with the first aspect, in a first possible implementation of the first aspect, the EEG signal contains signal components of m signal channels, and the EEG signal is processed to obtain multiple input signals of a set time length, including: for each signal component of the EEG signal, performing wavelet decomposition on the signal component to obtain corresponding 6 waveform signals: α wave, β wave, δ wave, θ wave, shuttle wave, and sawtooth wave; for each waveform signal, using a sliding window with a time scale of τ and a step size of s to extract waveform signal segments to obtain g waveform signal segments, thereby obtaining g input signals, each input signal containing 6 waveform signals of m signal components, totaling 6m waveform signal segments, where s≤τ.

[0010] In combination with the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the signal feature extraction module of the sleep monitoring model includes an input layer, m convolution channels and a feature fusion layer, wherein the input layer is connected to the m convolution channels, for receiving the input signal, and inputting each signal component in the input signal into the corresponding convolution channel, wherein the m convolution channels correspond one-to-one to the m signal components in the input signal, and each signal component has 6 waveform signal segments; each convolution channel is used to perform convolution and pooling processing on the signal components input into the convolution channel, and then flatten them to obtain a channel vector; the feature fusion layer is connected to the channel connection layer of each convolution channel, for receiving the channel vector output by each channel connection layer, and fusing the m channel vectors to obtain a feature vector corresponding to the input signal.

[0011] In combination with the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the input signal is a vector of [m, M, 6], where m is the number of signal channels, M is the number of sampling data points of each waveform signal segment, M=fτ, f is the sampling frequency, 6 is the number of waveform signal segments, and each convolution channel Ch i Contains convolutional layer C i1 , Pooling layer P i1 , convolutional layer C i2 , Pooling layer P i2 , convolutional layer C i3 , Pooling layer P i3 , channel connection layer FC i , i = 1, 2, ..., m, for each convolution channel Ch i :The convolution layer C i1 , used to input the signal component u of this convolution channel i Convolution processing is performed; the pooling layer P i1 , used for the convolutional layer C i1 The output features are downsampled; the convolution layer C i2 , used for the pooling layer P i1 The output features are convolutionally processed; the pooling layer P i2 , used for the convolutional layer C i2 The output features are downsampled; the convolution layer C i3 , used for the pooling layer P i2 The output features are convolutionally processed; the pooling layer P i3 , used for the convolutional layer C i3 The output features are downsampled; the channel connection layer FC i , used for the pooling layer P i3 The output features are flattened to obtain the channel vector v of this convolution channel i .

[0012] In combination with the first aspect, in a fourth possible implementation of the first aspect, the sleep stage classification module of the sleep monitoring model includes a first LSTM structure and a second LSTM structure, wherein the first LSTM structure is used to process a feature vector sequence of a forward input, and the second LSTM structure is used to process a feature vector sequence of a backward input, wherein the feature vector sequence includes n feature vectors, and the first LSTM structure includes n LSTM units, respectively denoted as LSTM j1 , n LSTM units LSTM j1 They are connected to the feature fusion layer respectively, and each LSTM unit LSTM j1 Connect sequentially, LSTM j1Used to receive the jth feature vector in the feature vector sequence as input, or receive the jth feature vector, the j-1th feature vector and the LSTM (j-1)1 The output of is taken as input, j = 1, 2, ..., n; the second LSTM structure includes n LSTM units, respectively denoted as LSTM j2 , n LSTM units LSTM j2 They are connected to the feature fusion layer respectively, and each LSTM unit LSTM j2 Connect sequentially, LSTM j2 Used to receive the jth feature vector in the feature vector sequence as input, or receive the jth feature vector, LSTM j1 The output of LSTM (j+1)2 The output of is taken as input, j = 1, 2, ..., n; the LSTM in the first LSTM structure j1 The LSTM in the second LSTM structure j2 Construct a bidirectional LSTM group, denoted as LSTM j , for LSTM-based j1 The output and LSTM j2 The output of determines the final output of the bidirectional LSTM group.

[0013] In combination with the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, for the LSTM unit LSTM in the first LSTM structure j1 :

[0014] The input gate satisfies:

[0015]

[0016] Among them, j Representation of LSTM j1 The input gate, σ is the sigmoid function, W inh , All are LSTM j1 The input gate weight, b in For LSTM j1 The input gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input;

[0017] The forget gate satisfies:

[0018]

[0019] Among them, f j Representation of LSTM j1 The forget gate, σ is the sigmoid function, W fh , All are LSTM j1 The forget gate weight, b f For LSTM j1 The forget gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and LLSTM at time j j1 Input;

[0020] The output gate satisfies:

[0021]

[0022] Among them, out j Representation of LSTM j1 The output gate of , σ is the sigmoid function, W outh , All are LSTM j1 The output gate weight, b out For LSTM j1 The output gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input.

[0023] In combination with the fifth possible implementation of the first aspect, in a sixth possible implementation of the first aspect, for the LSTM unit LSTM in the first LSTM structure j1 :

[0024] The forward output is:

[0025]

[0026] in, For LSTM j1 The forward output of , σ is the sigmoid function, W out For LSTM j1 The comprehensive weight of h j For LSTM j1The output of the hidden layer at time j;

[0027] h j =out j ⊙tanh(y′ j ),

[0028] Among them, tanh is the hyperbolic tangent function, y′ j Representation of LSTM j1 The output at time j, ⊙, represents the element-wise product;

[0029]

[0030] Among them, f j For LSTM j1 The forget gate of y′ j-1 Representation of LSTM j1 The output at time j-1, in j Representation of LSTM j1 The input gate, Represents LSTM at time j j1 Candidate vector update value of ;

[0031]

[0032] Among them, W ch , All are LSTM j1 The candidate vector weight, b c For LSTM j1 The candidate vector bias of h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input.

[0033] In combination with the fourth possible implementation of the first aspect, in a seventh possible implementation of the first aspect, for the LSTM unit LSTM in the second LSTM structure j2 :

[0034] The input gate satisfies:

[0035]

[0036] Among them, in′ j Representation of LSTM j2 The input gate, σ is the sigmoid function, All are LSTM j2The input gate weight, b′ in For LSTM j2 The input gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input;

[0037] The forget gate satisfies:

[0038]

[0039] Among them, f′ j Representation of LSTM j2 The forget gate, σ is the sigmoid function, All are LSTM j2 The forget gate weight, b′ f For LSTM j2 The forget gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input;

[0040] The output gate satisfies:

[0041]

[0042] Among them, out′ j Representation of LSTM j2 The output gate of , σ is the sigmoid function, All are LSTM j2 The output gate weight, b′ out For LSTM j2 The output gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input.

[0043] In combination with the seventh possible implementation of the first aspect, in an eighth possible implementation of the first aspect, for the LSTM unit LSTM in the second LSTM structurej2 :

[0044] The backward output is:

[0045]

[0046] in, For LSTM j2 The backward output, σ is the sigmoid function, W′ out For LSTM j2 The comprehensive weight of h′ j For LSTM j2 The output of the hidden layer at time j;

[0047] h′ j =out′ j ⊙tanh(y″ j ),

[0048] Among them, tanh is the hyperbolic tangent function, y″ j Representation of LSTM j2 The output at time j, ⊙, represents the element-wise product;

[0049]

[0050] Among them, f′ j For LSTM j2 The forget gate of j+1 Representation of LSTM j2 The output at time j+1, in′ j Representation of LSTM j2 The input gate, Represents LSTM at time j j2 Candidate vector update value of ;

[0051]

[0052] Among them, W′ ch , All are LSTM j2 The candidate vector weight, b′ c For LSTM j2 The candidate vector bias of h′ j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input.

[0053] In a second aspect, an embodiment of the present application provides a sleep monitoring system based on EEG signals, comprising: an EEG signal acquisition unit, used to acquire EEG signals of a user in a sleeping state; an EEG signal processing unit, used to process the EEG signals to obtain multiple input signals of a set time length, wherein each input signal includes α wave, β wave, δ wave, θ wave, shuttle wave, and sawtooth wave of the EEG signals in the same time period; a sleep stage monitoring unit, used to: for each input signal: input the input signal into a preset sleep monitoring model to obtain a sleep stage category corresponding to the input signal output by the sleep monitoring model, wherein the sleep monitoring model includes a signal feature extraction module and a sleep stage classification module, the signal feature extraction module is used to extract features from the input signal, and use the extracted signal features as input vectors of the sleep stage classification module, the sleep stage classification module is used to classify and identify the input vectors into sleep stages to obtain the sleep stage category corresponding to the input signal.

[0054] Beneficial effects: By processing the EEG signals of the user in the sleeping state, multiple input signals of set time length are obtained (each input signal contains α wave, β wave, δ wave, θ wave, shuttle wave, sawtooth wave in the same time period), and the input signal is input into the preset sleep monitoring model. The signal feature extraction module is used to extract the features of the input signal, and the extracted signal features are used as the input vector of the sleep stage classification module. The sleep stage classification module is used to classify and identify the sleep stage of the input vector to obtain the sleep stage category corresponding to the input signal. 1. In this way, the signals of different waveforms of the input signal can be used to enrich the information dimension of the input signal, which is conducive to improving the richness of the extracted features, thereby improving the accuracy of sleep stage classification and identification. 2. The EEG signal is cut by sliding window to obtain multiple input signals that are continuous in time. Each input signal contains 6m waveform signal segments, which can make the input signal have diversified information and meet the standards. The signal feature extraction module is designed as a multi-channel feature extraction model, which uses convolution-pooling technology to realize feature extraction, extracts the signal component of each channel as a one-dimensional vector, and uses the feature fusion layer to perform feature fusion (such as splicing) to obtain the corresponding feature vector. It can consider the characteristics of various waveforms and various signal channels, improve the information richness of the feature vector, and help ensure the classification and recognition effect of sleep stages. 3. Design a double-loop LSTM, the first LSTM structure and the second LSTM structure respectively process the feature vector sequence of the forward input and the feature vector sequence of the backward input (the sequence composed of the feature vectors extracted by the signal feature extraction module from the input signal at multiple consecutive time nodes), and can consider the input between adjacent LSTM units (the input at one time node is used as the input of two adjacent LSTMs in the first LSTM structure). j1 and LSTM (j+1)1The input of the second LSTM structure in the bidirectional LSTM group at this time node j2 Input; and, the LSTM of the second LSTM structure j2 , and at the same time, the input of this time node and the next time node LSTM (j+1)2 The intermediate output and the next time node of LSTM (j+1)1 The forward output of j2 The final output of a bidirectional LSTM group is obtained by comprehensively combining the input of the LSTM unit, the intermediate output (the output of the hidden layer in the LSTM unit), the forward output and the backward output. This can take into account the correlation on the time scale and help reduce the long-term dependence on data, thereby improving the model's effect on sleep stage classification and recognition, and lightweight sleep monitoring model.

[0055] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 A flowchart of a sleep monitoring method based on EEG signals provided in an embodiment of the present application.

[0058] Figure 2 Schematic diagram of the structural design of the signal feature extraction module in the sleep monitoring model of an embodiment of the present application.

[0059] Figure 3 Schematic diagram of the structural design of the sleep stage classification module in the sleep monitoring model of an embodiment of the present application.

[0060] Figure 4 A structural block diagram of a sleep monitoring system based on EEG signals provided in an embodiment of the present application.

[0061] Icons: 10-sleep monitoring system; 11-EEG signal acquisition unit; 12-EEG signal processing unit; 13-sleep stage monitoring unit. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0063] In order to monitor the user's sleep, an electronic device can be used to perform a sleep monitoring method based on EEG signals to classify and identify the sleep stages so as to monitor the user's sleep state. The electronic device here can be a smart terminal (such as a personal computer, tablet computer, etc.) or a server (such as a cloud server, network server, etc.), which is not limited here.

[0064] See also Figure 1 , Figure 1 A flowchart of a sleep monitoring method based on EEG signals provided in an embodiment of the present application. The sleep monitoring method based on EEG signals may include step S10, step S20 and step S30.

[0065] First, the electronic device may execute step S10.

[0066] Step S10: Acquire the EEG signal of the user in the sleeping state.

[0067] In this embodiment, the electronic device can obtain the EEG signal of the user in the sleep state, and the EEG signal here can be the EEG signal of the user in the sleep state collected by the EEG signal acquisition device, and the EEG signal here includes signal components of m signal channels. For example, the common signal channels of the EEG acquisition device are 16 channels or 32 channels (this embodiment is described by taking 16 channels as an example, but not limited thereto), and the collected EEG signal also includes signal components of the corresponding number of channels, such as signal components of 16 signal channels or signal components of 32 signal channels.

[0068] After obtaining the EEG signal of the user in the sleeping state, the electronic device may execute step S20.

[0069] Step S20: Processing the EEG signal to obtain multiple input signals of set time length, wherein each input signal includes α wave, β wave, δ wave, θ wave, spindle wave, and sawtooth wave of the EEG signal in the same time period.

[0070] In this embodiment, the electronic device can process the EEG signal as follows:

[0071] Since the EEG signal contains signal components of m signal channels, for each signal component of the EEG signal, the electronic device can perform wavelet decomposition (for example, using 4-layer wavelet decomposition) on the signal component (i.e., the signal component currently being processed), and decompose to obtain the corresponding 6 waveform signals: α wave, β wave, δ wave, θ wave, shuttle wave, and sawtooth wave. Since the application of such methods in the prior art is relatively mature, they will not be described in detail here.

[0072] After decomposing the six waveform signals contained in each signal component of the EEG signal, the electronic device can use a sliding window with a time scale of τ and a step size of s to intercept waveform signal segments to obtain g waveform signal segments, where s≤τ. Considering the effectiveness of sleep stage classification and recognition, τ can be set to 4 seconds (or 2 seconds), and the step size can be set to 1 second, 2 seconds, 4 seconds, etc., which can be determined based on the recognition accuracy requirements when the sleep stage is switched. This embodiment is illustrated by taking τ=4 and s=4 as an example. If other schemes intend to control the recognition accuracy of the sleep stage within 1 second or 2 seconds, the step size s can be set to 1 second or 2 seconds.

[0073] Then, the electronic device can merge the waveform signal segments of the 6 waveform signals of the m signal components corresponding to each time period (a total of 6m) into the input signal corresponding to this time period (including m signal components, 6 waveforms of each signal component), thereby obtaining g input signals corresponding to the EEG signal. Of course, the acquisition of EEG signals can also be multiple continuous in time, and each EEG signal of the same length is obtained each time, and each EEG signal is processed into g input signals, thereby obtaining several continuous input signals on a time scale. In addition, the form of each input signal satisfies [m, M, 6], where m is the number of signal channels, M is the number of sampling data points for each waveform signal segment, M = fτ, f is the sampling frequency (for example, 50 or 60 times per second), and 6 is the number of waveform signal segments.

[0074] After obtaining the input signal, the electronic device may execute step S30 for each input signal.

[0075] Step S30: Input the input signal into a preset sleep monitoring model to obtain the sleep stage category corresponding to the input signal output by the sleep monitoring model, wherein the sleep monitoring model includes a signal feature extraction module and a sleep stage classification module, the signal feature extraction module is used to extract features from the input signal, and use the extracted signal features as input vectors of the sleep stage classification module, and the sleep stage classification module is used to perform sleep stage classification and identification on the input vector to obtain the sleep stage category corresponding to the input signal.

[0076] In order to facilitate the understanding of this solution, the sleep monitoring model provided in this solution is introduced here. In this embodiment, the sleep monitoring model may include a signal feature extraction module and a sleep stage classification module. The signal feature extraction module is used to extract features from the input signal and use the extracted signal features as the input vector of the sleep stage classification module, while the sleep stage classification module is used to classify and identify the sleep stage of the input vector to obtain the sleep stage category corresponding to the input signal.

[0077] See also Figure 2 , Figure 2 Schematic diagram of the structural design of the signal feature extraction module in the sleep monitoring model of an embodiment of the present application.

[0078] In this embodiment, the signal feature extraction module is based on the design architecture of CNN, including an input layer, m convolution channels and a feature fusion layer.

[0079] Exemplarily, the input layer is connected to m convolution channels for receiving an input signal and inputting each signal component in the input signal into a corresponding convolution channel, wherein the m convolution channels correspond one-to-one to the m signal components in the input signal, and each signal component has 6 waveform signal segments.

[0080] Exemplarily, each convolution channel can be used to perform convolution processing and pooling processing on the signal components input into the convolution channel, and then flatten the signal components to obtain a channel vector.

[0081] Specifically, each convolution channel Ch i Contains convolutional layer C i1 , Pooling layer P i1 , convolutional layer C i2 , Pooling layer P i2 , convolutional layer C i3 , Pooling layer P i3 , channel connection layer FC i , i=1,2,…,m.

[0082] For each convolution channel Ch i : Convolutional layer C i1 The signal component u of the input convolution channel can be i Perform convolution processing; pooling layer P i1 For the convolution layer C i1 The output features are downsampled; convolution layer C i2 For the pooling layer P i1 The output features are convolved; the pooling layer P i2 For the convolution layer C i2 The output features are downsampled; convolution layer C i3 For the pooling layer P i2 The output features are convolved; the pooling layer P i3 For the convolution layer C i3 The output features are downsampled; channel connection layer FC i For the pooling layer P i3 The output features are flattened to obtain the channel vector v of this convolution channel i .

[0083] For example, the input convolution channel Ch i The signal component u iis a two-dimensional vector of [240,6], which can be written as a 240×6 matrix, and then the convolution layer C is used i1 Perform convolution, convolution layer C i1 There are 8 groups of convolution kernels with a size of 10×2. Each convolution kernel performs convolution with a step size of 5×1, and obtains 8 groups of two-dimensional vectors of [47,5]. The pooling layer P i1 The pooling window is 3×1 and the step size is 2×1. The two-dimensional vector of [47,5] is downsampled to obtain 8 sets of two-dimensional vectors of size [23, 5]. Then, the convolution layer C is used to i2 Perform convolution, convolution layer C i2 There are 8 groups of convolution kernels of size 3×2, each convolution kernel is convolved in the form of step size 2×1, and 8 groups of two-dimensional vectors of size [11, 4] are obtained. The pooling layer P i2 The pooling window is 2×1 and the step size is 1×1. The two-dimensional vector of [11, 4] is downsampled to obtain 8 sets of two-dimensional vectors of size [10, 4]. Then, the convolution layer C is used to i3 Perform convolution, convolution layer C i3 There are 8 groups of convolution kernels of size 2×2, each convolution kernel is convolved in the form of step size 1×1, and 8 groups of two-dimensional vectors of size [9, 3] are obtained. The pooling layer P i3 The pooling window is 3×1, the step size is 1×1, and the two-dimensional vector of [9, 3] is downsampled to obtain 8 groups of two-dimensional vectors of size [7, 3]. Using the channel connection layer FC i Flatten the 8 groups of [7, 3] two-dimensional vectors and arrange them in sequence to obtain a one-dimensional feature vector of [168, 1] as the channel vector v of this convolution channel i .

[0084] The feature fusion layer is connected to the channel connection layer of each convolution channel to receive the channel vector output by each channel connection layer and fuse the m channel vectors to obtain the feature vector corresponding to the input signal.

[0085] For example, the feature fusion layer can transform the 16 channel vectors v of [168, 1] into i The [2688, 1] feature vector corresponding to the input signal is obtained by splicing according to the channel number, which serves as the input of the sleep stage classification module at that moment.

[0086] See also Figure 3 , Figure 3 Schematic diagram of the structural design of the sleep stage classification module in the sleep monitoring model of an embodiment of the present application.

[0087] In this embodiment, the sleep stage classification module of the sleep monitoring model may include a first LSTM structure (located at Figure 3The lower middle row of LSTMs) and the second LSTM structure (located at Figure 3 The first LSTM structure is used to process the feature vector sequence of the forward input, and the second LSTM structure is used to process the feature vector sequence of the backward input. The feature vector sequence contains n feature vectors, i.e., the n feature vectors output by the signal feature extraction module.

[0088] For example, the first LSTM structure includes n LSTM units, respectively denoted as LSTM j1 , n LSTM units LSTM j1 They are connected to the feature fusion layer respectively, and each LSTM unit LSTM j1 Connect sequentially, LSTM j1 Used to receive the jth feature vector in the feature vector sequence as input, or receive the jth feature vector, the j-1th feature vector and the LSTM (j-1)1 The output of is taken as input, j=1,2,…,n.

[0089] For the LSTM unit LSTM in the first LSTM structure j1 :

[0090] The input gate satisfies the formula:

[0091]

[0092] Among them, j Representation of LSTM j1 The input gate, σ is the sigmoid function, W inh , All are LSTM j1 The input gate weight, b in For LSTM j1 The input gate bias, h j-1 For LSTM j1 The output of the hidden layer (each LSTM unit also contains input layer, hidden layer and output layer) at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input.

[0093] The forget gate satisfies the formula:

[0094]

[0095] Among them, f j Representation of LSTM j1 The forget gate, σ is the sigmoid function, W fh , All are LSTM j1 The forget gate weight, b f For LSTM j1 The forget gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input.

[0096] The output gate satisfies the formula:

[0097]

[0098] Among them, out j Representation of LSTM j1 The output gate of , τ is the sigmoid function, W outh , All are LSTM j1 The output gate weight, b out For LSTM j1 The output gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input.

[0099] Then, the LSTM unit LSTM in the first LSTM structure j1 The forward output is:

[0100]

[0101] in, For LSTM j1 The forward output of , σ is the sigmoid function, W out For LSTM j1 The comprehensive weight of h j For LSTM j1 The output of the hidden layer at time j. j satisfy:

[0102] h j =out j ⊙tanh(y′ j ), (5)

[0103] Among them, tanh is the hyperbolic tangent function, y′j Representation of LSTM j1 The output at time j, ⊙ represents the element-wise product. j satisfy:

[0104]

[0105] Among them, f j For LSTM j1 The forget gate of y′ j-1 Representation of LSTM j1 The output at time j-1, in j Representation of LSTM j1 The input gate, Represents LSTM at time j j1 The candidate vector update value of . satisfy:

[0106]

[0107] Among them, W ch , All are LSTM j1 The candidate vector weight, b c For LSTM j1 The candidate vector bias of h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input.

[0108] Exemplarily, the second LSTM structure includes n LSTM units, respectively denoted as LSTM j2 , n LSTM units LSTM j2 They are connected to the feature fusion layer respectively, and each LSTM unit LSTM j2 Connect sequentially, LSTM j2 Used to receive the jth feature vector in the feature vector sequence as input, or receive the jth feature vector, LSTM j1 The output of LSTM (j+1)2 The output of is taken as input, j=1,2,…,n.

[0109] For the LSTM unit LSTM in the second LSTM structure j2 :

[0110] The input gate satisfies the formula:

[0111]

[0112] Among them, in′ j Representation of LSTM j2 The input gate, σ is the sigmoid function, All are LSTM j2 The input gate weight, b′ in For LSTM j2 The input gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input.

[0113] The forget gate satisfies the formula:

[0114]

[0115] Among them, f′ j Representation of LSTM j2 The forget gate, σ is the sigmoid function, All are LSTM j2 The forget gate weight, b′ f For LSTM j2 The forget gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input.

[0116] The output gate satisfies the formula:

[0117]

[0118] Among them, out′ j Representation of LSTM j2 The output gate of , σ is the sigmoid function, All are LSTM j2 The output gate weight, b′ out For LSTM j2 The output gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time jj2 Input.

[0119] Then, the LSTM unit LSTM in the second LSTM structure j2 The backward output is:

[0120]

[0121] in, For LSTM j2 The backward output, σ is the sigmoid function, W′ out For LSTM j2 The comprehensive weight of h′ j For LSTM j2 The output of the hidden layer at time j. h′ j satisfy:

[0122] h′ j =out′ j ⊙tanh(y″ j ), (12)

[0123] Among them, tanh is the hyperbolic tangent function, y″ j Representation of LSTM j2 The output at time j, ⊙ represents the element-wise product. y″ j satisfy:

[0124]

[0125] Among them, f′ j For LSTM j2 The forget gate of j+1 Representation of LSTM j2 The output at time j+1, in′ j Representation of LSTM j2 The input gate, Represents LSTM at time j j2 The candidate vector update value of . satisfy:

[0126]

[0127] Among them, W′ ch , All are LSTM j2 The candidate vector weight, b′ c For LSTM j2 The candidate vector bias of h′ j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1The forward output at time j+1, x j Represents LSTM at time j j2 Input.

[0128] Based on this, the LSTM in the first LSTM structure j1 The LSTM in the second LSTM structure j2 Construct a bidirectional LSTM group, denoted as LSTM j , for LSTM-based j1 The output and LSTM j2 The output of determines the final output of the bidirectional LSTM group.

[0129] Then, the bidirectional LSTM group LSTM j The final output is:

[0130]

[0131] Among them, y j LSTM for bidirectional LSTM group j The final output of can be mapped to sleep stage categories.

[0132] For the LSTM with a double loop structure designed in this embodiment, a cross entropy loss function can be used during training:

[0133]

[0134] Among them, L loss is the loss function, N represents the number of sleep cycle categories, Represents the true sample category.

[0135] In addition, in order to improve the calculation efficiency, σ (sigmoid function) in this embodiment can be designed as follows:

[0136]

[0137] Among them, σ(z) is the sigmoid function and z is the variable.

[0138] By inputting the input signal into the sleep monitoring model, the sleep stage category output by the sleep monitoring model can be obtained, thereby realizing the sleep stage classification of the EEG signal.

[0139] Electronic devices can further organize and record the patient's sleep stage categories at each time point during sleep into a sleep cycle diagram for doctors to make judgments. They can even further design deep learning models to evaluate the sleep state of the sleep cycle diagram in order to give targeted improvement suggestions. This is a direction that can be further developed based on this solution.

[0140] See also Figure 4 Based on the same inventive concept, the present application also provides a sleep monitoring system 10 based on EEG signals, including:

[0141] The EEG signal acquisition unit 11 is used to acquire the EEG signal of the user in the sleeping state.

[0142] The EEG signal processing unit 12 is used to process the EEG signal to obtain multiple input signals of a set time length, wherein each input signal includes α wave, β wave, δ wave, θ wave, spindle wave, and sawtooth wave of the EEG signal in the same time period.

[0143] The sleep stage monitoring unit 13 is used for each input signal: inputting the input signal into a preset sleep monitoring model to obtain the sleep stage category corresponding to the input signal output by the sleep monitoring model, wherein the sleep monitoring model includes a signal feature extraction module and a sleep stage classification module, the signal feature extraction module is used to extract features from the input signal, and use the extracted signal features as input vectors of the sleep stage classification module, and the sleep stage classification module is used to perform sleep stage classification and identification on the input vector to obtain the sleep stage category corresponding to the input signal.

[0144] In this embodiment, the EEG signal processing unit 12 is specifically used to: for each signal component of the EEG signal, perform wavelet decomposition on the signal component to obtain corresponding 6 waveform signals: α wave, β wave, δ wave, θ wave, shuttle wave, and sawtooth wave; for each waveform signal, use a sliding window with a time scale of τ and a step size of s to intercept waveform signal segments to obtain g waveform signal segments, thereby obtaining g input signals, each input signal contains 6 waveform signals of m signal components, totaling 6m waveform signal segments, where s≤τ.

[0145] In addition, for the sleep monitoring model, please refer to the previous introduction, which will not be repeated here.

[0146] In summary, the embodiment of the present application provides a sleep monitoring method and system based on EEG signals. By processing the EEG signals of the user in the sleep state, multiple input signals of a set time length are obtained (each input signal contains α waves, β waves, δ waves, θ waves, shuttle waves, and sawtooth waves in the same time period), and the input signals are input into a preset sleep monitoring model. The input signals are extracted by the signal feature extraction module, and the extracted signal features are used as the input vectors of the sleep stage classification module. The sleep stage classification module is used to classify and identify the sleep stages of the input vectors to obtain the sleep stage category corresponding to the input signal. 1. In this way, the signals of different waveforms of the input signal can be used to enrich the information dimension of the input signal, which is conducive to improving the richness of the extracted features, thereby improving the accuracy of sleep stage classification and identification. 2. The EEG signal is cut by a sliding window to obtain multiple input signals that are continuous in time, each of which contains 6m waveform signal segments, so that the input signal has diversified information and meets the standard. The signal feature extraction module is designed as a multi-channel feature extraction model, which uses convolution-pooling technology to realize feature extraction, extracts the signal component of each channel as a one-dimensional vector, and uses the feature fusion layer to perform feature fusion (such as splicing) to obtain the corresponding feature vector. It can consider the characteristics of various waveforms and various signal channels, improve the information richness of the feature vector, and help ensure the classification and recognition effect of sleep stages. 3. Design a double-loop LSTM, the first LSTM structure and the second LSTM structure respectively process the feature vector sequence of the forward input and the feature vector sequence of the backward input (the sequence composed of the feature vectors extracted by the signal feature extraction module from the input signal at multiple consecutive time nodes), and can consider the input between adjacent LSTM units (the input at one time node is used as the input of two adjacent LSTMs in the first LSTM structure). j1 and LSTM (j+1)1 The input of the second LSTM structure in the bidirectional LSTM group at this time node j2 Input; and, the LSTM of the second LSTM structure j2 , and at the same time, the input of this time node and the next time node LSTM (j+1)2 The intermediate output and the next time node of LSTM (j+1)1 The forward output is used as the input of LSTMj2 at this time node), the intermediate output (the output of the hidden layer in the LSTM unit), the forward output and the backward output, and the final output of a bidirectional LSTM group is obtained comprehensively, which can take into account the correlation on the time scale and help reduce the long-term dependence on data, thereby improving the model's effect on sleep stage classification and recognition, and lightweight sleep monitoring model.

[0147] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0148] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A sleep monitoring method based on electroencephalogram signals, characterized in that: include: Obtain the user's EEG signal when sleeping; Processing the EEG signal to obtain multiple input signals of a set time length, wherein each input signal includes an alpha wave, a beta wave, a delta wave, a theta wave, a spindle wave, and a sawtooth wave of the EEG signal in the same time period; For each input signal: input the input signal into a preset sleep monitoring model to obtain the sleep stage category corresponding to the input signal output by the sleep monitoring model, wherein the sleep monitoring model includes a signal feature extraction module and a sleep stage classification module, the signal feature extraction module is used to extract features from the input signal, and use the extracted signal features as input vectors of the sleep stage classification module, the sleep stage classification module is used to perform sleep stage classification and identification on the input vector to obtain the sleep stage category corresponding to the input signal; The sleep stage classification module of the sleep monitoring model includes a first LSTM structure and a second LSTM structure, wherein the first LSTM structure is used to process a feature vector sequence of a forward input, and the second LSTM structure is used to process a feature vector sequence of a backward input, wherein the feature vector sequence includes n feature vectors. The first LSTM structure includes n LSTM units, respectively denoted as LSTM j1 , n LSTM units LSTM j1 They are connected to the feature fusion layer respectively, and each LSTM unit LSTM j1 Connect sequentially, LSTM j1 Used to receive the jth feature vector in the feature vector sequence as input, or receive the jth feature vector, the j-1th feature vector and the LSTM (j-1)1 The output of is taken as input, j = 1, 2, ..., n; The second LSTM structure includes n LSTM units, respectively denoted as LSTM j2 , n LSTM units LSTM j2 They are connected to the feature fusion layer respectively, and each LSTM unit LSTM j2 Connect sequentially, LSTM j2 Used to receive the jth feature vector in the feature vector sequence as input, or receive the jth feature vector, LSTM j1 The output of LSTM (j+1)2 The output of is taken as input, j = 1, 2, ..., n; The LSTM in the first LSTM structure j1 The LSTM in the second LSTM structure j2 Construct a bidirectional LSTM group, denoted as LSTM j , for LSTM-based j1 The output and LSTM j2 The output of determines the final output of the bidirectional LSTM group; For the LSTM unit LSTM in the first LSTM structure j1 : The input gate satisfies: Among them, j Representation of LSTM j1 The input gate, σ is the sigmoid function, W inh , All are LSTM j1 The input gate weight, b in For LSTM j1 The input gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input; The forget gate satisfies: Among them, f j Representation of LSTM j1 The forget gate, σ is the sigmoid function, W fh , All are LSTM j1 The forget gate weight, b f For LSTM j1 The forget gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j -1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input; The output gate satisfies: Among them, out j Representation of LSTM j1 The output gate of , σ is the sigmoid function, W outh , All are LSTM j1 The output gate weight, b out For LSTM j1 The output gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input; For the LSTM unit LSTM in the second LSTM structure j2 : The input gate satisfies: Among them, in′ j Representation of LSTM j2 The input gate, σ is the sigmoid function, All are LSTM j2 The input gate weight, b′ in For LSTM j2 The input gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input; The forget gate satisfies: Among them, f′ j Representation of LSTM j2 The forget gate, σ is the sigmoid function, All are LSTM j2 The forget gate weight, b′ f For LSTM j2 The forget gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input; The output gate satisfies: Among them, out′ j Representation of LSTM j2 The output gate of , σ is the sigmoid function, All are LSTM j2 The output gate weight, b′ out For LSTM j2 The output gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input.

2. The sleep monitoring method based on EEG signals according to claim 1, characterized in that: The EEG signal includes signal components of m signal channels. The EEG signal is processed to obtain a plurality of input signals of a set time length, including: For each signal component of the EEG signal, the signal component is decomposed by wavelet to obtain corresponding 6 waveform signals: α wave, β wave, δ wave, θ wave, shuttle wave, and sawtooth wave; For each waveform signal, a sliding window with a time scale of τ and a step size of s is used to extract waveform signal fragments to obtain g waveform signal fragments, thereby obtaining g input signals. Each input signal contains 6 waveform signals with m signal components, totaling 6m waveform signal fragments, where s≤τ.

3. The sleep monitoring method based on EEG signals according to claim 2, characterized in that: The signal feature extraction module of the sleep monitoring model includes an input layer, m convolution channels and a feature fusion layer. The input layer is connected to the m convolution channels, and is used to receive the input signal and input each signal component in the input signal into the corresponding convolution channel, wherein the m convolution channels correspond one-to-one to the m signal components in the input signal, and each signal component has 6 waveform signal segments; Each convolution channel is used to perform convolution and pooling processing on the signal components input into the convolution channel, and then flatten them to obtain a channel vector; The feature fusion layer is connected to the channel connection layer of each convolution channel, and is used to receive the channel vector output by each channel connection layer, and fuse the m channel vectors to obtain the feature vector corresponding to the input signal.

4. The sleep monitoring method based on EEG signals according to claim 3, characterized in that: The input signal is a vector of [m, M, 6], where m is the number of signal channels, M is the number of sampling data points for each waveform signal segment, M=fτ, f is the sampling frequency, 6 is the number of waveform signal segments, and each convolution channel Ch i Contains convolutional layer C i1 , Pooling layer P i1 , convolutional layer C i2 , Pooling layer P i2 , convolutional layer C i3 , Pooling layer P i3 , channel connection layer FC i , i = 1, 2, ..., m, For each convolution channel Ch i : The convolutional layer C i1 , used to input the signal component u of this convolution channel i Perform convolution processing; The pooling layer P i1 , used for the convolutional layer C i1 The output features are downsampled; The convolutional layer C i2 , used to perform convolution processing on the features output by the pooling layer Pi1; The pooling layer Pi2 is used to pool the convolutional layer C i2 The output features are downsampled; The convolutional layer C i3 , used for the pooling layer P i2 The output features are convolved; The pooling layer P i3 , used for the convolutional layer C i3 The output features are downsampled; The channel connection layer FC i , used for the pooling layer P i3 The output features are flattened to obtain the channel vector v of this convolution channel i .

5. The sleep monitoring method based on EEG signals according to claim 1, characterized in that: For the LSTM unit LSTM in the first LSTM structure j1 : The forward output is: in, For LSTM j1 The forward output of , σ is the sigmoid function, W out For LSTM j1 The comprehensive weight of h j For LSTM j1 The output of the hidden layer at time j; h j =out j ⊙tanh(y′ j ), Among them, tanh is the hyperbolic tangent function, y′ j Representation of LSTM j1 The output at time j, ⊙, represents the element-wise product; Among them, f j For LSTM j1 The forget gate of y′ j-1 Representation of LSTM j1 The output at time j-1, in j Representation of LSTM j1 The input gate, Represents LSTM at time j j1 Candidate vector update value of ; Among them, W ch , All are LSTM j1 The candidate vector weight, b c For LSTM j1 The candidate vector bias of h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input.

6. The sleep monitoring method based on EEG signals according to claim 1, characterized in that: For the LSTM unit LSTM in the second LSTM structure j2 : The backward output is: in, For LSTM j2 The backward output, σ is the sigmoid function, W′ out For LSTM j2 The comprehensive weight of h′ j For LSTM j2 The output of the hidden layer at time j; h′ j =out′ j ⊙tanh(y′ j ), Among them, tanh is the hyperbolic tangent function, y″ j Representation of LSTM j2 The output at time j, ⊙, represents the element-wise product; Among them, f′ j For LSTM j2 The forget gate, y″ j+1 Representation of LSTM j2 The output at time j+1, in′ j Representation of LSTM j2 The input gate, Represents LSTM at time j j2 Candidate vector update value of ; Among them, W′ ch , All are LSTM j2 The candidate vector weight, b′ c For LSTM j2 The candidate vector bias of h′ j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input.

7. A sleep monitoring system based on EEG signals, characterized in that: include: An electroencephalogram signal acquisition unit, used to acquire the electroencephalogram signal of the user in a sleeping state; An electroencephalogram signal processing unit is used to process the electroencephalogram signal to obtain a plurality of input signals of a set time length, wherein each input signal includes an alpha wave, a beta wave, a delta wave, a theta wave, a spindle wave, and a sawtooth wave of the electroencephalogram signal in the same time period; A sleep stage monitoring unit, for each input signal: inputting the input signal into a preset sleep monitoring model to obtain a sleep stage category corresponding to the input signal output by the sleep monitoring model, wherein the sleep monitoring model includes a signal feature extraction module and a sleep stage classification module, the signal feature extraction module is used to extract features from the input signal, and use the extracted signal features as input vectors of the sleep stage classification module, the sleep stage classification module is used to perform sleep stage classification and identification on the input vector to obtain the sleep stage category corresponding to the input signal; The sleep stage classification module of the sleep monitoring model includes a first LSTM structure and a second LSTM structure, wherein the first LSTM structure is used to process a feature vector sequence of a forward input, and the second LSTM structure is used to process a feature vector sequence of a backward input, wherein the feature vector sequence includes n feature vectors. The first LSTM structure includes n LSTM units, respectively denoted as LSTM j1 , n LSTM units LSTM j1 They are connected to the feature fusion layer respectively, and each LSTM unit LSTM j1 Connect sequentially, LSTM j1 Used to receive the jth feature vector in the feature vector sequence as input, or receive the jth feature vector, the j-1th feature vector and the LSTM (j-1)1 The output of is taken as input, j = 1, 2, ..., n; The second LSTM structure includes n LSTM units, respectively denoted as LSTM j2 , n LSTM units LSTM j2 They are connected to the feature fusion layer respectively, and each LSTM unit LSTM j2 Connect sequentially, LSTM j2 Used to receive the jth feature vector in the feature vector sequence as input, or receive the jth feature vector, LSTM j1 The output of LSTM (j+1)2 The output of is taken as input, j = 1, 2, ..., n; The LSTM in the first LSTM structure j1 The LSTM in the second LSTM structure j2 Construct a bidirectional LSTM group, denoted as LSTM j , for LSTM-based j1 The output and LSTM j2 The output of determines the final output of the bidirectional LSTM group; For the LSTM unit LSTM in the first LSTM structure j1 : The input gate satisfies: Among them, j Representation of LSTM j1 The input gate, σ is the sigmoid function, W inh , All are LSTM j1 The input gate weight, b in For LSTM j1 The input gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input; The forget gate satisfies: Among them, f j Representation of LSTM j1 The forget gate, σ is the sigmoid function, W fh , All are LSTM j1 The forget gate weight, b f For LSTM j1 The forget gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input; The output gate satisfies: Among them, out j Representation of LSTM j1 The output gate of , σ is the sigmoid function, W outh , All are LSTM j1 The output gate weight, b out For LSTM j1 The output gate bias, h j-1 For LSTM j1 The output of the hidden layer at time j-1, x j-1 and x j Respectively represent the j-1 time LSTM j1 The input and j-time LSTM j1 Input; For the LSTM unit LSTM in the second LSTM structure j2 : The input gate satisfies: Among them, in′ j Representation of LSTM j2 The input gate, σ is the sigmoid function, All are LSTM j2 The input gate weight, b′ in For LSTM j2 The input gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input; The forget gate satisfies: Among them, f′ j Representation of LSTM j2 The forget gate, σ is the sigmoid function, All are LSTM j2 The forget gate weight, b′ f For LSTM j2 The forget gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input; The output gate satisfies: Among them, out′ j Representation of LSTM j2 The output gate of , σ is the sigmoid function, All are LSTM j2 The output gate weight, b′ out For LSTM j2 The output gate bias, h j+1 For LSTM j2 The output of the hidden layer at time j+1, For LSTM j1 The forward output at time j+1, x j Represents LSTM at time j j2 Input.

Images