Speech motor imagery decoding system and method
By processing EEG signals using residual networks, deep convolutional neural networks, and vector quantization variational autoencoders, syllable labels, MEL spectrograms, and vocalization trajectories were obtained, solving the accuracy problem of speech motor imagery decoding and restoring the communication ability of aphasia patients.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- AFFILIATED HUSN HOSPITAL OF FUDAN UNIV
- Filing Date
- 2026-04-15
- Publication Date
- 2026-05-12
AI Technical Summary
Current technology makes it difficult to accurately decode brainwave signals to achieve verbal motor imagery, resulting in aphasia patients being unable to communicate effectively.
A syllable classifier with a residual network architecture, a speech synthesizer with a deep convolutional neural network, and a speech motor synthesizer with a vector quantization variational autoencoder are used to process EEG signals to obtain syllable labels, MEL spectrograms, and speech motor trajectories. This information is then combined to perform speech motor imagery decoding.
It achieves accurate decoding of brainwave signals, effectively reconstructs the results of speech motor imagery, and restores the communication ability of aphasia patients.
Smart Images
Figure CN122024697A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of brain-computer interface technology, and in particular to a speech-motor imagery decoding system and method. Background Technology
[0002] Language is the most basic form of human communication, a natural and intuitive way for people to express their thoughts and ideas. However, in clinical practice, major neurological diseases such as stroke, amyotrophic lateral sclerosis (ALS), brain tumors, and locked-in syndrome can all cause severe dysarthria or aphasia. Although patients may retain full consciousness and thinking ability, they lose the important daily communication ability of speech output.
[0003] Speech motor imagery is often considered a truncation of the speech articulation process. How to obtain accurate decoding results of speech motor imagery based on EEG signals has become an urgent technical problem to be solved. Summary of the Invention
[0004] The purpose of this invention is at least to provide a speech motor imagery decoding system and method that can obtain accurate speech motor imagery decoding results based on electroencephalogram (EEG) signals.
[0005] In a first aspect, the present invention provides a speech motor imagery decoding system, comprising: a syllable classifier for acquiring a target EEG signal to be processed and predicting the target EEG signal to obtain a target syllable label; a speech synthesizer for acquiring a MEL spectrogram corresponding to the target EEG signal, the MEL spectrogram being used for speech synthesis; and a vocalization motor synthesizer for acquiring a 13-dimensional vocalization motor trajectory corresponding to the target EEG signal; wherein the target syllable label, the MEL spectrogram, and the 13-dimensional vocalization motor trajectory are used to determine the speech motor imagery decoding result corresponding to the target EEG signal.
[0006] Optionally, the syllable classifier employs a residual network architecture, comprising, in sequence, a two-dimensional convolutional layer, a batch normalization layer, a modified linear unit layer, a two-dimensional max pooling layer, four consecutive residual block groups, a global averaging layer, a flattening layer, a fully connected layer, a loss layer, and a softmax layer, wherein: the target EEG signal is input to the two-dimensional convolutional layer, which is used to extract initial features from the target EEG signal; the batch normalization layer is used to perform batch normalization processing on the output of the two-dimensional convolutional layer; the modified linear unit layer is used to perform nonlinear transformation on the output of the batch normalization layer; and the two-dimensional max pooling layer is used to further process the output of the modified linear unit layer. The process involves: downsampling; four consecutive residual block groups used to learn the output of the two-dimensional max pooling layer to obtain the target features corresponding to the target EEG signal; a global averaging layer used to perform global average pooling on the target features, compressing the spatial dimension to 1; a flattening layer used to flatten the global averaging layer; a fully connected layer used to map the output of the flattening layer to the category space based on the number of categories; a loss layer used to randomly lose a predetermined number of neurons; and a softmax layer used to output the probability distribution corresponding to each category of syllable label, with the sum of the probabilities of each category of syllable label being 1; the target syllable label being the syllable label with the highest probability.
[0007] Optionally, the four consecutive residual block groups include a first residual block group, a second residual block group, a third residual block group, and a fourth residual block group, wherein: the first residual block group includes three residual blocks, which learn fine-grained spatiotemporal features based on the output of the two-dimensional max pooling layer; the second residual block group includes four residual blocks, which are used to learn abstract features; the third residual block group includes six residual blocks, which are used to extract high-level semantic features; and the fourth residual block group includes three residual blocks, which are used to obtain the target features.
[0008] Optionally, the speech synthesizer adopts a deep convolutional neural network architecture, including four consecutive modules arranged in series and a fully connected layer, wherein: the target EEG signal is input to the first consecutive module among the four consecutive modules, and after passing through the four consecutive modules, the MEL spectrogram is output through the fully connected layer.
[0009] Optionally, any consecutive module includes a two-dimensional convolutional layer, a residual block, and a two-dimensional max-pooling layer arranged sequentially; wherein: the two-dimensional convolutional layer of the first consecutive module is used to perform initial feature extraction on the target EEG signal; the residual block of the first consecutive module is used for residual learning; the two-dimensional max-pooling layer of the first consecutive module is used for the first spatial downsampling; the two-dimensional convolutional layer of the second consecutive module is used for further feature extraction on the output of the first consecutive module; the residual block of the second consecutive module is used for residual learning; the two-dimensional max-pooling layer of the second consecutive module is used for the second spatial downsampling; the two-dimensional convolutional layer of the third consecutive module is used to extract features from the output of the second consecutive module to obtain high-level features; the third... The residual blocks of the first continuous module are used for residual learning; the two-dimensional max pooling layer of the third continuous module is used for third spatial downsampling; the two-dimensional convolutional layer of the fourth continuous module is used for feature extraction from the output of the third continuous module to obtain target features; the residual blocks of the fourth continuous module are used for residual learning; the two-dimensional max pooling layer of the fourth continuous module is used for fourth spatial downsampling; the fully connected layer includes a flattening layer, a fully connected layer, and a reshaping layer arranged sequentially, wherein: the flattening layer is used to flatten the result of the fourth spatial downsampling; the fully connected layer is used to map the output of the flattening layer to the MEL spectrum; the reshaping layer is used to reshape the output of the fully connected layer to obtain the MEL spectrum map.
[0010] Optionally, the speech motion synthesizer includes an encoder, a speech codebook, and a decoder arranged in sequence, wherein: the encoder includes a two-dimensional convolutional layer, five consecutive residual blocks, a nonlocal block, a residual block, a group normalization layer, a swish layer, and two cascaded two-dimensional convolutional layers arranged in sequence; the decoder includes a two-dimensional convolutional layer, a residual block, a nonlocal block, five consecutive residual blocks, a group normalization layer, a swish layer, and two cascaded two-dimensional convolutional layers arranged in sequence.
[0011] Optionally, the two-dimensional convolutional layer in the encoder is used to extract features from the target EEG signal; the five consecutive residual blocks in the encoder correspond to downsampling layers, where the input of any residual block is the output of the downsampling layer corresponding to the previous residual block; the nonlocal block is used to obtain the dependency relationship of the outputs of the five consecutive residual blocks; the residual block is used to perform post-attention residual processing on the output of the nonlocal block; the group normalization layer is used to perform group normalization processing on the residual block; the swish layer is used to perform self-gated activation on the output of the group normalization layer; in the two cascaded two-dimensional convolutional layers, the first two-dimensional convolutional layer is used to map the output of the swish layer to the latent space dimension, and the second two-dimensional convolutional layer is used to compress the output of the first two-dimensional convolutional layer.
[0012] Optionally, the two-dimensional convolutional layer in the decoder is used to recover the codebook vector output by the pronunciation codebook from the latent space to high-dimensional features; the residual block is used to perform residual learning on the output of the two-dimensional convolutional layer; the non-local block is used to obtain the dependencies of the output of the residual block; the five consecutive residual blocks are used to perform deep feature processing and upsampling processing on the output of the non-local block; the group normalization layer is used to perform group normalization processing on the output of the five consecutive residual blocks; the swish layer is used to perform self-gated activation on the output of the group normalization layer; in the two cascaded two-dimensional convolutional layers, the first two-dimensional convolutional layer is used to map the output of the swish layer to the AKT dimension, and the second two-dimensional convolutional layer is used to adjust the output of the first two-dimensional convolutional layer in the time dimension.
[0013] Optionally, the target EEG signal is acquired in the following manner: acquiring the raw EEG signal collected by cortical EEG electrode pads; downsampling the raw EEG signal, and acquiring the EEG signal with a frequency in the beta1 band from the downsampled signal as the target EEG signal.
[0014] Secondly, the present invention also provides a speech motor imagery decoding method, comprising: predicting a target EEG signal to be processed to obtain a target syllable label; acquiring a mel spectrogram corresponding to the target EEG signal, the mel spectrogram being used for speech synthesis; acquiring a 13-dimensional articulation trajectory corresponding to the target EEG signal; and determining a speech motor imagery decoding result corresponding to the target EEG signal based on the target syllable label, the mel spectrogram, and the 13-dimensional articulation trajectory.
[0015] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects: A syllable classifier acquires target syllable labels corresponding to the target EEG signal; a speech synthesizer acquires the corresponding MEL spectrogram of the target EEG; and a vocalization motor synthesizer acquires the corresponding 13-dimensional vocalization motor trajectory. Then, a processor processes the target syllable labels, MEL spectrogram, and 13-dimensional vocalization motor trajectory to obtain the speech motor imagery decoding result corresponding to the target EEG signal. Thus, the target EEG signal can be accurately decoded, yielding accurate speech motor imagery decoding results. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the structure of a speech motor imagery decoding system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a syllable classifier in an embodiment of the present invention; Figure 3 This is a schematic diagram of a speech synthesizer according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a speech motion synthesizer according to an embodiment of the present invention; Figure 5 This is a flowchart of a speech motor imagery decoding method according to an embodiment of the present invention. Detailed Implementation
[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0018] Reference Figure 1 The present invention provides a schematic diagram of the structure of a speech motor imagery decoding system according to an embodiment of the present invention.
[0019] In this embodiment of the invention, the speech motion imagery decoding system may include: a syllable classifier 110, a speech synthesizer 120, and a vocalization motion synthesizer 130.
[0020] In a specific implementation, the syllable classifier 110 acquires the target EEG signal to be processed and predicts the target EEG signal to obtain a target syllable label; the speech synthesizer 120 acquires the MEL spectrogram corresponding to the target EEG signal, and the MEL spectrogram is used for speech synthesis; the articulation motion synthesizer 130 acquires the 13-dimensional articulation motion trajectory corresponding to the target EEG signal; wherein, the target syllable label, the MEL spectrogram, and the 13-dimensional articulation motion trajectory are used to determine the speech motor imagery decoding result corresponding to the target EEG signal.
[0021] In practice, the correspondence between the target syllable label, the MEL spectrogram, and the 13-dimensional articulation trajectory can be determined. If all three correspond to the same speech, the corresponding speech motor imagery decoding result can be determined. If any two of the three correspond to the first speech and the other corresponds to the second speech, the first speech can be determined as the corresponding speech motor imagery decoding result.
[0022] In some embodiments, the speech corresponding to the 13-dimensional speech motion trajectory can be given the highest priority. When the 13-dimensional speech motion trajectory and the MEL spectrogram correspond to a first speech and the target syllable label corresponds to a second speech, the first speech is determined as the corresponding speech motion imagery decoding result.
[0023] In other embodiments, when the target syllable label and the mel spectrogram both correspond to the first language, and the 13-dimensional articulation trajectory corresponds to the second language, it can be determined that an accurate speech motion imagery decoding result cannot be obtained.
[0024] In specific implementation, the aforementioned target syllable labels can be used to characterize whether different syllables can be obtained from the target EEG signal; the speech synthesized based on the aforementioned MEL spectrogram can be used to characterize whether intelligible speech can be reconstructed from the target EEG signal; and the aforementioned 13-dimensional articulation motion trajectory can be used to characterize whether the target EEG signal truly encodes articulation actions.
[0025] In this embodiment of the invention, the target brainwave signal can be obtained by processing the original brainwave signal accordingly.
[0026] In practice, electrocorticometry (ECOG) electrodes can be used to collect the subject's brainwave signals to obtain raw brainwave signals. Then, the raw brainwave signals undergo a series of processing steps to obtain a downsampled signal with a frequency of 100Hz. From this downsampled signal, brainwave signals with frequencies in the beta1 band (12Hz~24Hz) are extracted as the target brainwave signal.
[0027] In some embodiments, a series of processing steps performed on the raw EEG signal may include channel checking (to exclude channels with no signal or channels with continuous epileptic waves), artifact removal, notch filtering, bandpass filtering, downsampling, etc. The specific process of processing the raw EEG signal to obtain the target EEG signal with a frequency in the beta1 band can be referred to in the prior art for obtaining EEG signals in the beta1 band, and will not be elaborated here.
[0028] The syllable classifier 110, speech synthesizer 120, and articulation motor synthesizer 130 described above will be explained below.
[0029] Reference Figure 2 A schematic diagram of a syllable classifier in an embodiment of the present invention is provided.
[0030] In practical implementation, the syllable classifier can adopt a 34-layer residual network (ResNet-34) architecture. The syllable classifier includes, in sequence, a two-dimensional convolutional (Conv2D) layer, a batch normalization (BNU) layer, a rectified linear unit (ReLU), a two-dimensional max pooling (2D) layer, four residual blocks (ResBlocks1~ResBlocks4 in sequence), a global averaging layer, a flattening layer, a fully connected (dense) layer, a dropout layer, and a softmax layer.
[0031] In some embodiments, the aforementioned two-dimensional convolutional layer may employ a 7×7 kernel, use rectified linear units as the activation function, and employ a 40% loss layer to mitigate overfitting. The network is trained using a stochastic gradient descent (SGD) optimizer to minimize the joint cross-entropy loss, and a 10-fold cross-validation strategy is used to determine the average performance.
[0032] In the specific implementation, in the syllable classifier, the target EEG signal is input into a two-dimensional convolutional layer, which is used to extract initial features from the target EEG signal; a batch normalization layer is used to perform batch normalization processing on the output of the two-dimensional convolutional layer; a modified linear unit layer is used to perform nonlinear transformation on the output of the batch normalization layer; a two-dimensional max pooling layer is used to downsample the output of the modified linear unit layer; four consecutive residual block groups are used to learn the output of the two-dimensional max pooling layer to obtain the target features corresponding to the target EEG signal; a global averaging layer is used to perform global average pooling on the target features to compress the spatial dimension to 1; a flattening layer is used to flatten the global averaging layer; a fully connected layer is used to map the output of the flattening layer to the category space based on the number of categories; a loss layer is used to randomly lose a predetermined number of neurons; a softmax layer is used to output the probability distribution corresponding to each category of syllable label, and the sum of the probabilities of each category of syllable label is 1; the target syllable label is the syllable label with the highest probability.
[0033] In some embodiments, the four consecutive residual block groups mentioned above include residual block group 1, residual block group 2, residual block group 3, and residual block group 4, wherein: residual block group 1 includes three residual blocks connected in series, which learn fine-grained spatiotemporal features based on the output of the two-dimensional max pooling layer; residual block group 2 includes four residual blocks connected in series, which are used to learn abstract features; residual block group 3 includes six residual blocks connected in series, which are used to extract high-level semantic features; and residual block group 4 includes three residual blocks connected in series, which are used to obtain target features.
[0034] In practical implementation, the speech synthesizer can employ a deep convolutional neural network architecture, consisting of four consecutive modules. Each module includes a 2D convolutional (Conv2D) layer, a residual block (ResBlock), and a 2D maxpooling (maxpooling 2D) layer. These four modules are concatenated, with the input of each module being the corresponding 2D convolutional layer and the output being the corresponding 2D maxpooling layer. A fully connected layer (FClayer) is connected to the output of the fourth consecutive module.
[0035] Reference Figure 3 A schematic diagram of the structure of a speech synthesizer according to an embodiment of the present invention is provided.
[0036] In practical implementation, Mel-cepstral distortion (MCD) coefficients are used as an objective metric for speech synthesizers. MCD quantifies the error in Mel-Frequency Cepstral Coefficients (MFCCs), as follows:
[0037] Where d represents each MFCC dimension (0 < d < 25), y is the synthesized speech, and y is the actual sound signal produced by the subject. Performance evaluation used a 10-fold cross-validation strategy to determine the average MCD for each fold.
[0038] Figure 3 In this model, connected two-dimensional convolutional layers, residual blocks, and two-dimensional max pooling layers form a continuous module. The two-dimensional max pooling layer of the first continuous module is connected to the two-dimensional convolutional layer of the second continuous module, and so on.
[0039] In specific implementation, any consecutive module includes a two-dimensional convolutional layer, a residual block, and a two-dimensional max pooling layer arranged sequentially; wherein: The first continuous module's two-dimensional convolutional layer is used to extract initial features from the target EEG signal; the first continuous module's residual block is used for residual learning; and the first continuous module's two-dimensional max pooling layer is used for the first spatial downsampling. The second continuous module's two-dimensional convolutional layer is used to further extract features from the output of the first continuous module; the residual block of the second continuous module is used for residual learning; and the two-dimensional max pooling layer of the second continuous module is used for a second spatial downsampling. The three-dimensional convolutional layer of the third continuous module is used to extract features from the output of the second continuous module to obtain high-level features; the residual block of the third continuous module is used for residual learning; and the two-dimensional max pooling layer of the third continuous module is used for third spatial downsampling. The four consecutive modules' two-dimensional convolutional layers are used to extract features from the output of the third consecutive module to obtain the target features; The residual block of the fourth consecutive module is used for residual learning; the two-dimensional max pooling layer of the fourth consecutive module is used for the fourth spatial downsampling. The fully connected layer consists of a flattening layer, a fully connected layer, and a reshaping layer arranged sequentially. The flattening layer is used to flatten the result of the fourth spatial downsampling. The fully connected layer is used to map the output of the flattening layer to the MEL spectrum. The reshaping layer is used to reshape the output of the fully connected layer to obtain the MEL spectrum.
[0040] In practical implementation, the speech motion synthesizer can employ a vector-quantized variational autoencoder (VQ-VAE). The speech motion synthesizer may consist of an encoder, a speech codebook, and a decoder. The encoder is composed of residual blocks and non-local blocks, and the decoder can also be composed of residual blocks and non-local blocks.
[0041] In this embodiment of the invention, the speech motion synthesizer may include an encoder, a speech codebook, and a decoder arranged in sequence, wherein: the encoder includes a two-dimensional convolutional layer, five consecutive residual blocks, a nonlocal block, a residual block, a group normalization layer, a swish layer, and two cascaded two-dimensional convolutional layers arranged in sequence; the decoder includes a two-dimensional convolutional layer, a residual block, a nonlocal block, five consecutive residual blocks, a group normalization layer, a swish layer, and two cascaded two-dimensional convolutional layers arranged in sequence.
[0042] In practice, the number of consecutive residual blocks can also be other values, such as 3, 4, 6, 7, etc. The specific number of consecutive residual blocks can be set according to the actual application scenario.
[0043] In specific implementation, the two-dimensional convolutional layer in the encoder is used to extract features from the target EEG signal; the encoder has five consecutive residual blocks corresponding to downsampling layers, where the input of any residual block is the output of the downsampling layer corresponding to the previous residual block; the nonlocal block is used to obtain the dependency relationship of the outputs of the five consecutive residual blocks (residual blocks 1~5); the residual block is used to perform post-attention residual processing on the output of the nonlocal block; the group normalization layer is used to perform group normalization processing on the residual blocks; the swish layer is used to perform self-gated activation on the output of the group normalization layer; in the two cascaded two-dimensional convolutional layers, the first two-dimensional convolutional layer is used to map the output of the swish layer to the latent space dimension, and the second two-dimensional convolutional layer is used to compress the output of the first two-dimensional convolutional layer.
[0044] The decoder uses two-dimensional convolutional layers to recover high-dimensional features from the latent space of the codebook output from the pronunciation codebook; residual blocks are used to perform residual learning on the output of the two-dimensional convolutional layers; non-local blocks are used to obtain the dependencies of the output of the residual blocks; five consecutive residual blocks (residual blocks 1-5) are used to perform deep feature processing and upsampling on the output of the non-local blocks; group normalization layers are used to perform group normalization on the output of the five consecutive residual blocks; swish layers are used to perform self-gated activation on the output of the group normalization layers; in the two cascaded two-dimensional convolutional layers, the first two-dimensional convolutional layer is used to map the output of the swish layer to the AKT dimension, and the second two-dimensional convolutional layer is used to adjust the output of the first two-dimensional convolutional layer in the time dimension.
[0045] Reference Figure 4 A schematic diagram of the structure of a speech motion synthesizer according to an embodiment of the present invention is provided.
[0046] Combination Figure 4 The target EEG signal is input to the encoder, which extracts features from the target EEG signal. The features extracted by the encoder are mapped to a 256-dimensional pronunciation codebook, thereby generating a discretized codebook. The discretized codebook is input to the decoder, which reconstructs the 13-dimensional pronunciation motion trajectory.
[0047] In practice, the vocalization trajectory can be represented by a matrix, where the matrix represents the vocalization trajectory and the columns represent the time steps.
[0048] In some embodiments, the articulation motion trajectory can be a 13×192 matrix, where 13 represents 13 articulation motion trajectories and 192 is the time step. The 13 articulation motion trajectories include: a 2D motion trajectory of the upper lip, a 2D motion trajectory of the lower lip, a 2D motion trajectory of the mandible, a 2D motion trajectory of the tip of the tongue, a 2D motion trajectory of the surface of the tongue, a 2D motion trajectory of the root of the tongue, and a 1D motion trajectory of the larynx.
[0049] In some embodiments, the Adam optimizer can also be used to optimize the speech motion synthesizer to minimize the mean squared error loss. The initial learning rate for the Adam optimizer is 2.25 × 10⁻⁶. -5 The first-order moment decay rate β1 = 0.5, the second-order moment decay rate β2 = 0.5, and the stability constant = ε = 1×10 -8 Performance was evaluated using a five-fold cross-validation strategy. The dataset was randomly divided into ten subsets: eight for training, one for validation, and one for testing. For each subset, performance was evaluated based on the average Pearson correlation coefficient (r) of the 13 vocalization trajectories, with the final performance being the average of the results from the five-fold cross-validation strategy.
[0050] Reference Figure 5 This invention provides a speech motor imagery decoding method according to an embodiment of the invention, which will be described in detail below through specific steps.
[0051] Step 501: Predict the target EEG signal to be processed to obtain the target syllable label.
[0052] Step 502: Obtain the MEL spectrogram corresponding to the target EEG signal.
[0053] Step 503: Obtain the 13-dimensional vocalization motion trajectory corresponding to the target EEG signal.
[0054] Step 504: Based on the target syllable label, mel spectrogram, and 13-dimensional articulation trajectory, determine the speech motor imagery decoding result corresponding to the target EEG signal.
[0055] In specific implementation, the execution process of step 501 can be referred to the syllable classifier provided in the above embodiments, the execution of step 502 can be referred to the speech synthesizer provided in the above embodiments, the execution of step 503 can be referred to the articulation motion synthesizer provided in the above embodiments, and the execution of step 504 can be referred to the processor provided in the above embodiments.
[0056] In specific implementation, the modules / units included in the various devices and products described in the above embodiments can be software modules / units, hardware modules / units, or a combination of both.
[0057] For example, for various devices and products applied to or integrated into a chip, each module / unit can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs that run on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits; for various devices and products applied to or integrated into a chip module, each module / unit can be implemented using hardware methods such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware methods such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.
[0058] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.
[0059] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.
Claims
1. A speech motor imagery decoding system, characterized in that, include: A syllable classifier acquires the target EEG signal to be processed and predicts the target EEG signal to obtain the target syllable label; A speech synthesizer acquires a mel spectrogram corresponding to the target EEG signal, and the mel spectrogram is used for speech synthesis. A speech motion synthesizer acquires the 13-dimensional speech motion trajectory corresponding to the target EEG signal; The target syllable label, the mel spectrogram, and the 13-dimensional articulation trajectory are used to determine the speech motor imagery decoding result corresponding to the target EEG signal.
2. The speech motor imagery decoding system as described in claim 1, characterized in that, The syllable classifier employs a residual network architecture, comprising, in sequence, a two-dimensional convolutional layer, a batch normalization layer, a corrected linear unit layer, a two-dimensional max pooling layer, four consecutive residual block groups, a global averaging layer, a flattening layer, a fully connected layer, a loss layer, and a softmax layer, wherein: The target EEG signal is input to the two-dimensional convolutional layer, which is used to perform initial feature extraction on the target EEG signal. The batch normalization layer is used to perform batch normalization processing on the output of the two-dimensional convolutional layer; The modified linear unit layer is used to perform a nonlinear transformation on the output of the batch normalization layer; The two-dimensional max pooling layer is used to downsample the output of the modified linear unit layer. The four consecutive residual block groups are used to learn the output of the two-dimensional maximum pooling layer to obtain the target features corresponding to the target EEG signal; The global averaging layer is used to perform global average pooling on the target features, compressing the spatial dimension to 1. The flattening layer is used to flatten the global averaging layer. The fully connected layer is used to map the output of the flattening layer to the category space based on the number of categories; The loss layer is used to randomly lose a predetermined number of neurons; The softmax layer is used to output the probability distribution of each category of syllable label, and the sum of the probabilities of each category of syllable label is 1; the target syllable label is the syllable label with the highest probability.
3. The speech motor imagery decoding system as described in claim 2, characterized in that, The four consecutive residual block groups include a first residual block group, a second residual block group, a third residual block group, and a fourth residual block group, wherein: The first residual block group includes three residual blocks, which learn fine-grained spatiotemporal features based on the output of the two-dimensional max pooling layer; The second residual block group comprises four residual blocks for learning abstract features; The third residual block group comprises six residual blocks for extracting high-level semantic features; The fourth residual block group includes three residual blocks for acquiring the target features.
4. The speech motor imagery decoding system as described in claim 1, characterized in that, The speech synthesizer employs a deep convolutional neural network architecture, comprising four consecutive modules arranged in series and a fully connected layer, wherein: The target EEG signal is input to the first of the four consecutive modules, and after passing through the four consecutive modules, the MEL spectrogram is output through the fully connected layer.
5. The speech motor imagery decoding system as described in claim 4, characterized in that, Any consecutive module comprises a two-dimensional convolutional layer, a residual block, and a two-dimensional max-pooling layer arranged sequentially; wherein: The first continuous module's two-dimensional convolutional layer is used to extract initial features from the target EEG signal; the first continuous module's residual block is used for residual learning; and the first continuous module's two-dimensional max pooling layer is used for the first spatial downsampling. The second continuous module's two-dimensional convolutional layer is used to further extract features from the output of the first continuous module; the residual block of the second continuous module is used for residual learning; the two-dimensional max pooling layer of the second continuous module is used for second spatial downsampling. The three-dimensional convolutional layer of the third continuous module is used to extract features from the output of the second continuous module to obtain high-level features; the residual block of the third continuous module is used for residual learning; and the two-dimensional max pooling layer of the third continuous module is used for third spatial downsampling. The four consecutive modules have a two-dimensional convolutional layer for extracting features from the output of the three consecutive modules to obtain target features; the four consecutive modules have a residual block for residual learning; and the four consecutive modules have a two-dimensional max pooling layer for performing a fourth spatial downsampling. The fully connected layer includes a flattening layer, a fully connected layer, and a reshaping layer arranged sequentially, wherein: the flattening layer is used to flatten the result of the fourth spatial downsampling; the fully connected layer is used to map the output of the flattening layer to the mel spectrum; and the reshaping layer is used to reshape the output of the fully connected layer to obtain the mel spectrum.
6. The speech motor imagery decoding system as described in claim 1, characterized in that, The speech motion synthesizer includes an encoder, a speech codebook, and a decoder arranged sequentially, wherein: The encoder includes a two-dimensional convolutional layer, five consecutive residual blocks, a nonlocal block, a residual block, a group normalization layer, a swish layer, and two cascaded two-dimensional convolutional layers arranged sequentially. The decoder includes a two-dimensional convolutional layer, a residual block, a non-local block, five consecutive residual blocks, a group normalization layer, a swish layer, and two cascaded two-dimensional convolutional layers arranged in sequence.
7. The speech motor imagery decoding system as described in claim 6, characterized in that, The two-dimensional convolutional layer in the encoder is used to extract features from the target EEG signal; the five consecutive residual blocks in the encoder correspond to downsampling layers, where the input of any residual block is the output of the downsampling layer corresponding to the previous residual block; the nonlocal block is used to obtain the dependency relationship of the outputs of the five consecutive residual blocks; the residual block is used to perform post-attention residual processing on the output of the nonlocal block. The group normalization layer is used to perform group normalization processing on the residual block; the swish layer is used to perform self-gated activation on the output of the group normalization layer; in the two concatenated two-dimensional convolutional layers, the first two-dimensional convolutional layer is used to map the output of the swish layer to the latent spatial dimension, and the second two-dimensional convolutional layer is used to compress the output of the first two-dimensional convolutional layer.
8. The speech motor imagery decoding system as described in claim 7, characterized in that, The decoder uses a two-dimensional convolutional layer to recover high-dimensional features from the codebook vector output by the pronunciation codebook; a residual block is used to perform residual learning on the output of the two-dimensional convolutional layer; a non-local block is used to obtain the dependencies of the output of the residual block; five consecutive residual blocks are used to perform deep feature processing and upsampling processing on the output of the non-local block; a group normalization layer is used to perform group normalization processing on the output of the five consecutive residual blocks; a swish layer is used to perform self-gated activation on the output of the group normalization layer; in the two cascaded two-dimensional convolutional layers, the first two-dimensional convolutional layer is used to map the output of the swish layer to the AKT dimension, and the second two-dimensional convolutional layer is used to adjust the output of the first two-dimensional convolutional layer in the time dimension.
9. The speech motor imagery decoding system as described in any one of claims 1 to 8, characterized in that, The target EEG signal was obtained using the following method: Acquire raw electroencephalogram (EEG) signals from cortical EEG electrodes; The original EEG signal is downsampled, and the EEG signal with a frequency in the beta1 band is obtained from the downsampled signal as the target EEG signal.
10. A method for decoding verbal motor imagery, characterized in that, include: Predict the target EEG signal to be processed to obtain target syllable labels; The MEL spectrogram corresponding to the target EEG signal is obtained, and the MEL spectrogram is used for speech synthesis. Obtain the 13-dimensional vocalization motion trajectory corresponding to the target EEG signal; Based on the target syllable label, the mel spectrogram, and the 13-dimensional articulation trajectory, the speech motor imagery decoding result corresponding to the target EEG signal is determined.