A speech brain-computer interface decoding method based on multimodal feature fusion

Abstract

The present invention discloses a speech brain-computer interface decoding method based on multimodal feature fusion, comprising: (1) obtaining original neural signals and facial electromyographic signals, and synchronously obtaining corresponding text sequences; (2) extracting neural signal features and electromyographic signal features respectively; for text sequences, extracting text features and phoneme features after extracting phoneme sequences; (3) constructing a decoding model based on multimodal feature fusion, comprising an encoder and a decoder; (4) for neural signal features, electromyographic signal features and phoneme features, after passing through an embedding layer in the encoder, combining them in pairs to calculate feature similarity, and training the encoder based on a contrast loss function; (5) training the decoder based on a CTC loss function; (5) in the application process, inputting the neural signal and electromyographic signal to be decoded into the decoding model, and finally obtaining the predicted text. The present invention can solve the influence of neural signal instability and noise interference on decoding performance, and enhance the robustness of the decoding model.

Description

Technical Field

[0001] The present invention relates to the field of neural signal decoding, and in particular to a speech brain-computer interface decoding method based on multimodal feature fusion. Background Art

[0002] Brain-computer interfaces decode text information from neural signals, which is of great significance for revealing the brain's language processing mechanism and helping patients with language disorders restore their communication ability. By implanting a microelectrode array in the cerebral cortex, neural data from the language expression area is extracted to train the model, and neural signals are decoded based on the model.

[0003] Machine learning technology plays an important role in neural decoding. By training the model to establish a mapping relationship between neural signals and text, the automatic conversion of neural signals to text is realized. For example, the Chinese patent document with publication number CN111973178A discloses an EEG signal recognition system and method, which decodes the encoded information according to the decoder in the trained recognition model to obtain the text information and emotional information corresponding to the EEG signal.

[0004] The machine learning model methods currently used for decoding include recurrent neural networks, Kalman filters, Bayesian decoders, integrated learning methods, etc. Among these methods, recurrent neural networks and their variants are very suitable for processing time series data, capturing the long-term dependencies of time data, and achieving good results in neural signal decoding.

[0005] However, the instability of neural signals and noise interference are challenges faced in the decoding process. During the neural decoding process, the physiological state of the participants, such as fatigue, emotional changes, distraction, etc., may affect the activity pattern of neurons, thereby affecting the stability of neural signals. In addition, the neural signal acquisition process may be interfered by other equipment, resulting in noise mixed in the signal, affecting the signal quality.

[0006] Currently, most intracortical brain-computer interfaces only use neural signals to decode text information. However, due to the instability and noise of the above-mentioned neural signals, the robustness of the decoding model is often not strong. Currently, an ideal solution is to introduce information from other sources.

[0007] Multimodal models can make up for the shortcomings and noise of a single modality by integrating different types of information, thereby improving the model's resistance to environmental changes and signal interference; at the same time, data from different modalities express different semantic information in different feature spaces, and the model can learn more general feature representations, thereby enhancing the generalization performance of the model; in addition, if the information of a single modality is unreliable or unavailable in some cases, the information of the remaining modalities can provide the necessary redundant information to ensure the reliability of the model. However, multimodal signals have different characteristics and scales, and integrating them into a unified representation faces a series of challenges and difficulties.

[0008] Therefore, in the neural signal decoding based on multimodal models, how to select modal information, how to align multiple modal information, and how to fuse multimodal information to train a stable decoding model are important issues in the current field of intracortical neural signal decoding. Summary of the invention

[0009] The present invention provides a speech brain-computer interface decoding method based on multimodal feature fusion, which can solve the influence of neural signal instability and noise interference on decoding performance in the process of decoding text from intracortical neural signals, and enhance the robustness of the decoding model.

[0010] A speech brain-computer interface decoding method based on multimodal feature fusion, characterized in that it comprises the following steps:

[0011] (1) Obtain the original neural signals recorded by the hardware and the facial electromyographic signals recorded by the sensor, synchronously obtain the text sequence corresponding to the signals, and preprocess the neural signals and electromyographic signals;

[0012] (2) For neural signals and electromyographic signals, neural signal features and electromyographic signal features are extracted respectively; for text sequences, the phoneme sequence is extracted and then text features and phoneme features are further extracted;

[0013] (3) constructing a decoding model based on multimodal feature fusion, wherein the decoding model includes an encoder and a decoder;

[0014] (4) For the extracted neural signal features, electromyographic signal features, and phoneme features, after passing through the embedding layer in the encoder, the feature similarity is calculated in pairs, and the encoder is trained based on the contrast loss function;

[0015] (5) Training the decoder based on the CTC loss function; the working process of the decoder is as follows: performing feature-level fusion on the feature vectors output by the encoder of the neural signal features and the electromyographic signal features to generate fused features, decoding the fused features using the decoder to obtain predicted phoneme vectors, and converting the predicted phoneme vectors into text sequences to obtain the final predicted text sequence;

[0016] (6) During the application process, the neural signals and electromyographic signals to be decoded are input into the decoding model, and finally the predicted text is obtained.

[0017] The method of the present invention introduces two modes, electromyographic signals and phoneme sequences, based on the correlation between neural signals, facial electromyographic signals and vocalization. During the decoding process, feature-level fusion is used to reduce the influence of neural signal instability and noise interference on the model decoding performance, thereby improving the robustness of the decoding model.

[0018] In step (1), the process of preprocessing the neural signal is as follows:

[0019] The neural signal is time-binned, the neuronal firing rate is calculated, and the firing rate data is standardized and smoothed. The firing rate calculation formula is:

[0020] ;

[0021] in, represents the size of the time bin, and the number of neuron firings in each time bin is , Is the release rate.

[0022] The specific process of step (2) is as follows:

[0023] For neural signals and electromyographic signals, linear transformation is used to enhance the feature dimension of the signals, and then a LSTM encoder is used to extract the deep features of the signals, namely the neural signal features and the electromyographic signal features. For text sequences, the text-to-phoneme sequence pre-training model G2P is used to generate the phoneme sequence corresponding to the text sequence, and then the phoneme features and text features are generated through feature embedding and position encoding methods.

[0024] In step (4), a linear model is used to construct the embedding layer, and the feature similarity is calculated as:

[0025] ;

[0026] in, and represents the feature vector obtained by the encoder, represents the cosine similarity, Represents the temperature parameter.

[0027] In step (4), the calculation formula of the contrast loss function is:

[0028] ;

[0029] ;

[0030] in represents the number of samples in a mini-batch, Representation characteristics and Features No. group of positive sample pairs, Representation characteristics and Features No. group of negative sample pairs, Represents the embedded features of neural signals, represents the embedded features of the EMG signal, Embedding features representing phoneme sequences, represents the control coefficient, Representation characteristics With features The contrast loss between Represents the loss function, The contrast loss between two features is calculated by weighting.

[0031] In step (5), the decoder uses a Transformer network. During the training process, the real text sequence is converted into a phoneme vector as the real label.

[0032] In step (5), the calculation formula of the CTC loss function is:

[0033] ;

[0034] in, is the correct phoneme sequence, is the input feature sequence, For a given input Under the condition of label sequence The probability of , which is obtained by adding the probabilities of all possible mappings of the input sequence to the label sequence.

[0035] After the encoder and decoder are trained, the entire decoding model needs to be tested and evaluated. The indicator for evaluating the model performance is the word error rate, which represents the percentage of the total number of errors to the total number of words in the text.

[0036] Based on contrastive learning and feature-level fusion technology, this paper proposes a speech brain-computer interface decoding method based on multimodal feature fusion, which solves the impact of neural signal instability and noise interference on decoding performance to a certain extent. In the test, it is better than the algorithm based on recurrent neural network, which proves the effectiveness of this method. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a paradigm flow chart of a speech brain-computer interface decoding method based on multimodal feature fusion of the present invention.

[0038] Figure 2 This is a graph showing the word error rate results of decoding using the method of the present invention in a real data set. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0040] The data set used in this embodiment is a BCI data set collected and published by Stanford University. The clinical trial volunteer T12 is a 67-year-old woman with amyotrophic lateral sclerosis. During the experiment, four 64-channel, 1.5 mm long silicon microelectrode arrays were implanted in the left hemisphere of T12, which were placed in the brain area related to language production. During data collection, T12 sat in a chair that supported the head and neck, with a computer in front of him to display the instructions of the experimental task. The experimental tasks include sentence reading, monophone phonation, and oral facial movements, etc., and each task has a specific waiting period and execution period. All tasks are performed on a computer monitor, and participants are guided to perform corresponding language or motor tasks through visual prompts. In addition, the public electromyography data set of Gaddy&Klein is used, which contains electromyography signals and read text prompt information pairs.

[0041] like Figure 1 As shown, a speech brain-computer interface decoding method based on multimodal feature fusion, the specific implementation steps are as follows:

[0042] Step (1), preprocessing of neural signals and electromyographic signals. Obtain the original neural signals recorded by the hardware, obtain the facial electromyographic signals recorded by the sensor, obtain the text sequence corresponding to the signals, and divide the signals into training set, validation set and test set after preprocessing.

[0043] Specifically, the preprocessing process of neural signals is as follows: neuronal firing is converted into firing rate with a window length of 20 ms by binning, normalized using the z-scoring method (minus the mean and divided by the standard deviation), smoothed using a Gaussian kernel with a window length of 400 ms, and the neural signal delay is set to 160 ms. The firing rate calculation formula is:

[0044] ;

[0045] in, represents the size of the time bin, is the number of firings of the neuron in each time bin, Is the release rate.

[0046] The preprocessing process of the electromyographic signal is as follows: use a Butterworth high-pass filter to remove the low-frequency drift in the electromyographic signal, set the cutoff frequency to 1000 Hz, downsample the signal to reduce the amount of subsequent calculations, normalize the signal, and obtain the processed electromyographic signal data.

[0047] Step (2) constructs a decoding model based on feature fusion. The decoding model mainly includes an encoder and a decoder, and the encoder contains an embedding layer.

[0048] For neural signals and electromyographic signals, the LSTM model is used to extract deep signal features. For text sequences, the G2P text-to-phoneme conversion model published by Carnegie Mellon University is used to obtain phoneme sequences, which are then encoded into phoneme features using feature embedding and position encoding.

[0049] For the extracted neural signal features, electromyographic features and phoneme features, a linear model is used to construct a feature embedding layer. The output of the embedding layer is used to calculate the loss function based on contrastive learning and cross entropy methods to train the encoder. The feature similarity calculation formula is:

[0050] ;

[0051] in, and represents the feature vector obtained by the encoder, represents the cosine similarity, Represents the temperature parameter.

[0052] The calculation formula of contrast loss function is:

[0053] ;

[0054] ;

[0055] in, represents the number of samples in a mini-batch, Representation characteristics and Features No. group of positive sample pairs, Representation characteristics and Features No. group of negative sample pairs, Represents the embedded features of neural signals, represents the embedded features of the EMG signal, Embedding features representing phoneme sequences, represents the control coefficient, Representation characteristics With features The contrast loss between Represents the loss function, The contrast loss between two features is calculated by weighting.

[0056] In this embodiment, the LSTM model adds a dropout layer to reduce the model complexity and prevent overfitting, and uses the Adam algorithm to optimize the model parameters and the temperature parameters. Set to 0.05, the control coefficient Set to 0.8, The learning rate is set to 0.02, the dropout coefficient is set to 0.4, the number of input features is set to 512, and the number of stacking layers is 5.

[0057] The main task of the encoder is to convert the input multimodal signal into a fixed-size fused feature vector that can capture the key information of the input data. After the encoder training is completed, its model parameters will be fixed and used to provide consistent feature input for the decoder. The features output by the encoder are fused at the feature level to generate fused features, and the G2P pre-trained model and other pre-processing methods are used to convert the real text sequence into a phoneme vector for training the Transformer decoder. The fusion formula of the neural signal features and the electromyographic signal features output by the embedding layer is:

[0058] ;

[0059] in, represents the fusion feature, Represent the characteristics of neural signals and electromyographic signals respectively.

[0060] The Transformer decoder consists of multiple basic decoder blocks. Each block contains a masked multi-head self-attention layer, a cross-attention layer, and a feedforward neural network. The input and output residuals of each layer are added and normalized before being input to the next layer. The self-attention expression is:

[0061] ;

[0062] in, are query vector, key vector and value vector respectively, for The multi-head self-attention mechanism defines multiple sets of different query vectors, key vectors, and value vectors, and divides the input into multiple parts. Each attention head can learn a unique representation space and focus on different parts of the input sequence. Finally, the attention vectors of each head are connected as the output.

[0063] In the training phase of the Transformer decoder, the phoneme vector is used as the real label input into the masked multi-head self-attention layer. After feature extraction, it is input into the cross-attention layer with the fused feature vector. Finally, the output of the feedforward neural network is transformed by the linear layer and continues to be the input of the masked multi-head self-attention layer. The decoder is trained based on the autoregressive mechanism. In the inference phase of the Transformer decoder, a special vector is set to represent the starting phoneme vector. After passing through the masked multi-head self-attention layer, it is input into the cross-attention layer as the query vector. The fused feature vector is used as the key vector and value vector to perform interactive attention calculation with the phoneme vector. Finally, the output of the feedforward neural network is transformed by the linear layer and continues to be the decoder input until the decoder outputs the vector representing the termination. The input fused features are passed through the decoder to obtain the predicted phoneme vector, and then the phoneme vector is converted into a text sequence through a language model to obtain the final model predicted text.

[0064] The decoder is trained based on the CTC loss function, and the calculation formula is:

[0065] ;

[0066] in, is the correct phoneme sequence, is the input feature sequence, For a given input Under the condition of label sequence The probability of , which is obtained by adding the probabilities of all possible mappings of the input sequence to the label sequence.

[0067] Step (3): Use the training set and validation set to train and evaluate the entire decoding model to obtain the trained encoder and decoder.

[0068] Step (4) uses the test set to test the decoding model, analyze the model performance, and apply neural signal decoding. The main indicator for evaluating model performance is the word error rate (WER), which represents the percentage of total errors to the total number of words in the reference text, where the total number of errors is the total number of operations required to convert the system output text into the reference text through insertion, deletion, and substitution operations.

[0069] In order to illustrate the feasibility of the decoding method proposed in the present invention, the word error rate index of the data of clinical trial subject T12 is statistically analyzed and obtained. Figure 2 , where experiments 1, 2, and 3 represent data collected on different days. Compared with the recurrent neural network, the multimodal feature fusion model of the present invention has significantly improved the indicators in multiple experiments, and the decoding performance of the model is more stable.

[0070] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A speech brain-computer interface decoding method based on multimodal feature fusion, characterized in that: The following steps are involved: (1) Obtain the original neural signals and facial electromyographic signals, synchronously obtain the text sequence corresponding to the signals, and preprocess the neural signals and electromyographic signals; (2) For neural signals and electromyographic signals, neural signal features and electromyographic signal features are extracted respectively; for text sequences, the phoneme sequence is extracted and then text features and phoneme features are further extracted; (3) constructing a decoding model based on multimodal feature fusion, wherein the decoding model includes an encoder and a decoder; (4) For the extracted neural signal features, electromyographic signal features, and phoneme features, after passing through the embedding layer in the encoder, the feature similarity is calculated in pairs, and the encoder is trained based on the contrast loss function; The linear model is used to construct the embedding layer, and the feature similarity is calculated as: ； in, and represents the feature vector obtained by the encoder, represents the cosine similarity, represents the temperature parameter; The calculation formula of the contrast loss function is: ； ； in, represents the number of samples in a mini-batch, Representation characteristics and Features No. group of positive sample pairs, Representation characteristics and Features No. group of negative sample pairs, Represents the embedded features of neural signals, represents the embedded features of the EMG signal, Embedding features representing phoneme sequences, represents the control coefficient, Representation characteristics With features The contrast loss between Represents the loss function, The contrast loss between the two features is calculated by weighted calculation; (5) Training the decoder based on the CTC loss function; the working process of the decoder is as follows: performing feature-level fusion on the feature vectors output by the encoder of the neural signal features and the electromyographic signal features to generate fused features, decoding the fused features using the decoder to obtain predicted phoneme vectors, and converting the predicted phoneme vectors into text sequences to obtain the final predicted text sequence; (6) During the application process, the neural signals and electromyographic signals to be decoded are input into the decoding model, and finally the predicted text is obtained.

2. The speech brain-computer interface decoding method based on multimodal feature fusion according to claim 1 is characterized in that: In step (1), the process of preprocessing the neural signal is as follows: The neural signal is time-binned, the neuronal firing rate is calculated, and the firing rate data is standardized and smoothed. The firing rate calculation formula is: ； in, represents the size of the time bin, and the number of neuron firings in each time bin is , Is the release rate.

3. The speech brain-computer interface decoding method based on multimodal feature fusion according to claim 1 is characterized in that: The specific process of step (2) is as follows: For neural signals and electromyographic signals, linear transformation is used to enhance the feature dimension of the signals, and then a LSTM encoder is used to extract the deep features of the signals, namely the neural signal features and the electromyographic signal features. For text sequences, the text-to-phoneme sequence pre-training model G2P is used to generate the phoneme sequence corresponding to the text sequence, and then the phoneme features and text features are generated through feature embedding and position encoding methods.

4. The speech brain-computer interface decoding method based on multimodal feature fusion according to claim 1 is characterized in that: In step (5), the decoder uses a Transformer network. During the training process, the real text sequence is converted into a phoneme vector as the real label.

5. The speech brain-computer interface decoding method based on multimodal feature fusion according to claim 1 is characterized in that: In step (5), the calculation formula of the CTC loss function is: ； in, is the correct phoneme sequence, is the input feature sequence, For a given input Under the condition of label sequence The probability of , which is obtained by adding the probabilities of all possible mappings of the input sequence to the label sequence.

6. The speech brain-computer interface decoding method based on multimodal feature fusion according to claim 1 is characterized in that: After the encoder and decoder are trained, the entire decoding model needs to be tested and evaluated. The indicator for evaluating the model performance is the word error rate, which represents the percentage of the total number of errors to the total number of words in the text.

Images