A discrete decoding method guided by continuous decoding in a speech brain-computer interface
By introducing a discrete decoding method guided by continuous decoding into the speech brain-computer interface, and using multi-channel CNN and LSTM networks to extract brain signal features and perform transfer training, the technical limitations in the signal acquisition and decoding process are solved, achieving more efficient speech output and improved quality of life.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2024-03-26
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies are insufficient to efficiently restore speech communication in aphasia patients, especially due to problems such as low signal-to-noise ratio, insufficient resolution, and unreasonable electrode placement during signal acquisition and decoding, resulting in poor discrete decoding performance.
We employ a discrete decoding method guided by continuous decoding. We extract the spatial, frequency, and temporal features of brain signals through multi-channel 2D-CNN and LSTM networks, combine them with 1D-CNN networks for dimensionality reduction and classification, and use the continuous decoding model to perform transfer training on the discrete decoding, dynamically adjusting the loss function to optimize the decoding model.
It improves the performance of discrete decoding, enhances the effect of speech synthesis, provides more natural and direct speech output, and improves the quality of life for aphasia patients.
Smart Images
Figure CN118571237B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech brain-computer interface decoding methods, and more specifically to a discrete decoding method guided by continuous decoding in a speech brain-computer interface. Background Technology
[0002] Many diseases can cause aphasia, affecting approximately 20 million people. Their quality of life is significantly impacted, and some may even develop depression. Therefore, it is crucial to utilize specific techniques to restore their speaking abilities and improve their quality of life.
[0003] We can extract relevant physiological parameters at various levels of speech production and then achieve a certain degree of verbal communication through mapping modeling. Examples include electromyography, lip reading, and tongue ultrasound. A speech brain-computer interface is a device that generates some form of speech output from the user's brain signals.
[0004] Brain-computer interface speech decoding falls into two categories: discrete decoding, which classifies brain signals and converts them into text, and speech synthesis, which directly synthesizes speech from brain signals. Discrete decoding reduces the dimensionality of target features and allows for later correction using advanced language models. However, it accumulates errors, takes longer, and lacks information such as rhythm, stress, and emotion in the speech itself. In contrast, continuous feature decoding produces a more natural and direct output.
[0005] Currently, research and applications of speech-based brain-computer interfaces (BCIs) face numerous limitations. 1. Ordinary subjects can only undergo non-invasive signal extraction methods such as EEG and fMRI, which have low spatiotemporal resolution. 2. The acquisition of some intracranial EEG signals depends on the patient's treatment needs, rather than the needs of speech decoding research or applications, resulting in poor decoding performance. Therefore, discrete decoding methods are more commonly used than continuous decoding. We believe that continuous decoding provides a rich mapping of speech features. Even under technical limitations, such as signal-to-noise ratio, resolution, or electrode placement, which prevent fully continuous decoding from being achieved, this mapping can still help improve the performance of discrete decoding. Based on this hypothesis, we designed and implemented a solution. Unlike directly performing discrete decoding, we used continuous decoding to conduct guided transfer training on discrete decoding and compared and analyzed the results. The results prove that our hypothesis is correct and reasonable. Summary of the Invention
[0006] This invention provides a discrete decoding method guided by continuous decoding in a speech brain-computer interface.
[0007] A discrete decoding method guided by continuous decoding in a speech brain-computer interface includes the following steps:
[0008] (1) After preprocessing the brain activity signal, a time series set C*F*T with different channels and different frequency bands is obtained and input into the decoding model. The decoding model includes a continuous decoding module and a discrete decoding module set in sequence.
[0009] Where C represents the number of brain signal channels, typically the number of electrodes; F represents different frequency bands; and T represents the duration.
[0010] (2) The continuous decoding module uses a multi-channel 2D-CNN network to extract the spatial and frequency features of the input time series set C*F*T, and then uses LSTM to extract the temporal features, mapping the spatial features, frequency features and temporal features to the acoustic features of the audio.
[0011] (3) The discrete decoding module then uses a 1D-CNN network to reduce the dimensionality of the spatial features, frequency features and temporal features extracted in step (2), and then maps the dimensionality-reduced features to the classification categories to obtain the classification results;
[0012] (4) Calculate the difference Vc between the audio acoustic features output in step (2) and the actual audio acoustic features, and the difference Vd between the classification result output in step (3) and the actual classification. Set the loss function L of the decoding model. L is a linear combination of Vc and Vd. The weight coefficients λc and λd of Vc and Vd are dynamically changed. Adjust the loss function, update the network parameters of the decoding model, and obtain the optimized decoding model.
[0013] (5) Input brain activity signals into the optimized decoding model and output language text.
[0014] In step (1), the preprocessing of brain activity signals specifically includes:
[0015] Brain activity signals acquired from multiple locations include spatial and temporal information. Wavelet decomposition is used to expand the single time series of brain activity signals from each location into a multi-frequency time series, adding frequency information to the spatial and temporal information of the brain activity signals.
[0016] The expansion to multiple frequency bands improves the signal-to-noise ratio of signals in different frequency bands, making it easier for convolutional modules to extract features of the corresponding frequency bands, thereby improving the feature extraction capability of the entire model.
[0017] In step (2), the acoustic features of the audio include Mel spectrum features, fundamental frequency features, and aperiodic signal features. Mel spectrum features, fundamental frequency features, and aperiodic signal features represent the basic acoustic features of speech: the Mel spectrum expresses the changes of different frequency signals of sound over time in the time-frequency domain; the fundamental frequency expresses pitch and contains prosodic and structural information of speech; and the aperiodic signal reflects the degree of voicing. These three features basically characterize audio, providing a relatively comprehensive but generalized representation of audio, and can also conveniently provide input for sound synthesis in some vocoders, such as the WORLD vocoder.
[0018] In step (3), the 1D-CNN network includes three sequentially connected one-dimensional convolutional modules, each of which includes a convolutional layer and a pooling layer connected in sequence. Because speech decoding generally uses relatively small amounts of data, considering model complexity and overfitting issues, an overly complex structure is not adopted. Instead, three sequentially connected one-dimensional convolutional modules are used to reduce and simplify the features in the time dimension.
[0019] In step (4), the weight coefficients λc and λd of Vc and Vd are dynamically changed to adjust the loss function and update the network parameters of the decoding model. Specifically, this includes:
[0020] In the first stage, the weight coefficient λd of Vd is taken from the first segment of the sigmoid saturation function, and its size is controlled to be less than 0.02. The weight coefficient λc of Vc is 1-λd, and its size is controlled to be greater than 0.98. The continuous decoding module adjusts its own module parameters through the loss function L to achieve the maximum extraction of spatial features, frequency features and temporal features in the first stage.
[0021] In the second stage, the Vd weight coefficient λd is taken as the second segment of the sigmoid saturation function. λd increases and λc decreases. The spatial features, frequency features and temporal features obtained by the continuous decoding module guide the training of the discrete decoding module.
[0022] In the third stage, the weight coefficient λd of Vd is taken as the third segment of the sigmoid saturation function, with λd greater than 0.98 and λc less than 0.02. The discrete decoding module self-trains to complete the fine-tuning of the entire model and updates the network parameters of the entire decoding model through the loss function.
[0023] The three-stage implementation achieves three objectives: 1) enabling continuous decoding to fully learn feature representations; 2) providing guidance for discrete decoding, with a smooth transition between guidance and self-learning; and 3) allowing discrete decoding to fully absorb learning outcomes and transform learning experience through self-learning. The three-stage implementation is automatically achieved through an sigmoid saturation function, requiring no manual intervention.
[0024] The sigmoid function mentioned above is the sigmoid function.
[0025] Compared with the prior art, the present invention has the following advantages:
[0026] This invention employs a deep learning model that combines continuous and discrete decoding systems. First, it trains the continuous decoding system on the Mel spectrum, fundamental frequency, and non-periodic signals from the acoustic information. Then, it uses the features extracted from the continuous decoding to guide and transfer-based training of the discrete decoding system, ultimately completing the training of the discrete decoding (i.e., the classification task). This discrete decoding performance is significantly higher than that of pure discrete decoding. Furthermore, the fundamental frequency and other signals output from the trained continuous decoding system can be used as adjustment inputs for subsequent speech synthesis systems based on the discrete decoding results, enhancing the speech synthesis effect. Attached Figure Description
[0027] Figure 1 This is a discrete decoding model framework guided by continuous decoding.
[0028] Figure 2 The graphs show the original sigmoid function and the graphs of the function after parameter modification.
[0029] Figure 3 For the statistical analysis of the prediction results, Figure A shows the confusion matrix of the classification results for the discrete module trained and decoded individually (represented by "D"); Figure B shows the confusion matrix of the classification results for the discrete decoding module (represented by "C") under the guidance of continuous decoding; the circles on the left of Figure C represent the number of correctly decoded samples for C, the circles on the right represent the number of correctly decoded samples for D, and the middle represents the number of samples that are both correctly decoded. (In the figures, C represents discrete decoding guided by continuous decoding, and D represents discrete decoding alone).
[0030] Figure 4 For the comparison of predicted probabilities and correct class distributions (in the figure, C represents discrete decoding guided by continuous decoding, and D represents individual discrete decoding). Detailed Implementation
[0031] The discrete decoding method guided by continuous decoding in speech brain-computer interfaces is implemented through the following steps:
[0032] (1) After preprocessing the brain activity signal, the time series set C*F*T (C is the number of brain signal channels, generally the number of electrodes; F is the different frequency bands; T is the duration) of different channels and different frequency bands is input into the decoding model;
[0033] (2) The continuous decoding module uses a multi-channel 2D-CNN network to extract the spatial and frequency features of the input brain activity signal, then uses LSTM to extract the temporal features, and then maps the extracted features to the acoustic features of the audio (Mel spectrum features, fundamental frequency features, non-periodic signal features, etc.).
[0034] (3) The discrete decoding module then uses a 1D-CNN network (including pooling layers) to reduce and dimensionality down the features extracted in step (2), and then maps the reduced features to the classification categories.
[0035] (4) First, train the continuous decoding module to ensure that the feature extraction module extracts acoustic features to the maximum extent. Then, start the training of discrete decoding and gradually increase the weight of discrete decoding while reducing the training weight of continuous decoding accordingly. This process is the process of continuous decoding guiding discrete decoding training, and it is also the process of model transfer from continuous decoding to discrete decoding. Finally, the discrete decoding is self-trained to complete the fine-tuning of the entire model.
[0036] (5) Input brain activity signals into the optimized decoding model and output language text.
[0037] The preprocessing step (1) expands a single time series into a multi-frequency time series through wavelet decomposition, adding frequency information to the original spatial and temporal information of the brain activity signal, thus enriching the information dimensions of the input signal. In addition to wavelet decomposition, Hilbert transform, multitaper method (MTM), and other methods can also be used to expand the frequency information of the signal.
[0038] In step (2), multi-dimensional features of the input signal are extracted by multi-channel 2D-CNN plus LSTM.
[0039] In step (2), acoustic features, including Mel spectrum, fundamental frequency, and non-periodic signals, are continuously decoded. Diverse acoustic features facilitate feature extraction of multidimensional brain activity signals in step (2). Usable acoustic features also include envelope, vocalization mode (plosives, sidestream sounds, etc.), vocalization location (lip, labiodental, dental, etc.), tone, etc.
[0040] In step (3), the features are reduced and dimensionality is reduced by using 1D-CNN+pooling layers.
[0041] In step (4), the continuous decoding task is trained first, then the discrete decoding task is trained in a guided manner, and the task is gradually transferred. Finally, the discrete decoding task is trained to complete the classification task. This is a coexisting, guided, and transfer-oriented approach and training method. The entire process can be designed and planned using a sigmoid (but not limited to) gradient function.
[0042] The model form described in steps (2), (3), (4), and (5) describes the coexistence of discrete and continuous decoding (the feature reduction module of discrete decoding can also be after the output of 2D-CNN). The coexistence model not only allows continuous decoding to guide discrete decoding to complete classification, but also retains the capabilities of continuous decoding. Its output can also serve as supplementary information for subsequent speech synthesis based on discrete decoding results.
[0043] 1. Figure 1 As shown, a model framework for the coexistence of continuous and discrete decoding is constructed. It consists of three parts: 1) basic feature extraction, including spatial feature extraction, frequency feature extraction, and temporal feature extraction; 2) mapping of acoustic features; and 3) discrete decoding feature extraction (feature reduction) and mapping to text.
[0044] 2. In specific implementation, the acoustic features are first fitted (continuous decoding).
[0045] 2.1 Spatial and frequency features are extracted from the preprocessed brain activity signal input (the signal is a multidimensional time series, each dimension corresponds to a spatial location). In our implementation, we used 2D convolution. Depending on the spatial relative position characteristics, other forms of spatial feature extraction can also be used, such as 3D convolution or 1D convolution.
[0046] 2.2 Temporal features are extracted from the sequences after spatial and frequency feature extraction. In our implementation, LSTM is used.
[0047] 2.3 Acoustic Feature Mapping
[0048] 3. Then, when the continuous decoding training is basically finished, the discrete decoding training is started. In our implementation, we use the sigmoid function to plan the entire training process (see Formula 1 for the sigmoid function).
[0049]
[0050] Change the parameters in Formula 1 to become
[0051]
[0052] The original sigmoid function and the function after parameter modification are as follows: Figure 2 As shown.
[0053] Let x be the training epoch, and f(x) be the weight coefficient λd of the difference Vd between the predicted classification result and the actual classification in the model loss function L. The total number of training epochs is 100. Around epoch x = 40, we initiate guided training of the discrete decoding from the continuous decoding. After approximately 20 epochs, it essentially becomes a fully discrete decoding system. The programming function can also be other similar functions, such as Logistic, Tanh, and Arctan functions.
[0054] 4. The guiding and transferable nature of learning depends on the intermediate process of change. Shortening the process of change means shortening the guiding process, which is closer to the ordinary sense of transfer learning. The purpose of the guiding process is to preserve the feature space of the "teacher"—continuous decoding—to a greater extent, providing more detailed feature selection for the "student"—discrete decoding.
[0055] 5. Within this framework, continuous training can also be conducted together with discrete training.
[0056] 6. From our implementation example, we can see that guided training for continuous decoding greatly aids discrete decoding, resulting in significantly better performance than discrete decoding alone. Our implementation example involves predicting 15 alphanumeric characters, with a total of 550 samples. (Example:...) Figure 3 As shown, Figure A is the confusion matrix of the classification results for the discrete module training and decoding alone (represented by "D"); Figure B is the confusion matrix of the classification results for the discrete decoding module (represented by "C") under the guidance of continuous decoding; the circles on the left of Figure C represent the number of correctly decoded samples in C, the circles on the right represent the number of correctly decoded samples in D, and the middle represents the number of samples that are correctly decoded by both. Quantitatively, the number of samples T has increased.
[0057] like Figure 4 As shown in Figure A), the leftmost graph represents the predicted probability distributions of all correctly predicted samples by C and all correctly predicted samples by D in a 5-fold test, and their comparison. The middle graph represents the comparison of the individual predicted probability distributions of samples correctly predicted by both C and D. The right graph represents the comparison of the individual predicted probability distributions of samples correctly predicted by each of C and D. Figure B) is a summary of the 5-fold results from Figure A). Figure C) shows the distribution of samples in their respective categories as represented in Figure B). From the case-by-case discussion of predicted probabilities, in the samples shared by C and D, C's predicted probability is generally higher than D's; in the samples correctly predicted by each of C and D, C's predicted probability is also generally higher than D's, and the number of C's correctly predicted samples is also greater than D's. Therefore, comparatively, C's final discrete decoding performance is better than D's.
Claims
1. A discrete decoding method guided by continuous decoding in a voice brain-computer interface, characterized in that, Includes the following steps: (1) After preprocessing the brain activity signal, a time series set C*F*T with different channels and different frequency bands is obtained and input into the decoding model. The decoding model includes a continuous decoding module and a discrete decoding module set in sequence. Where C represents the number of brain signal channels; F represents different frequency bands; and T represents the duration. (2) The continuous decoding module uses a multi-channel 2D-CNN network to extract the spatial and frequency features of the input time series set C*F*T, and then uses LSTM to extract the temporal features, mapping the spatial features, frequency features and temporal features to the acoustic features of the audio. (3) The discrete decoding module then uses a 1D-CNN network to reduce the dimensionality of the spatial features, frequency features and temporal features extracted in step (2), and then maps the dimensionality-reduced features to the classification categories to obtain the classification results; (4) Calculate the difference Vc between the audio acoustic features output in step (2) and the actual audio acoustic features, and the difference Vd between the classification result output in step (3) and the actual classification. Set the loss function L of the decoding model. L is a linear combination of Vc and Vd. The weight coefficients λc and λd of Vc and Vd are dynamically changed. Adjust the loss function, update the network parameters of the decoding model, and obtain the optimized decoding model. (5) Input brain activity signals into the optimized decoding model and output language text.
2. The discrete decoding method under continuous decoding guidance in the speech brain-computer interface according to claim 1, characterized in that, In step (1), the preprocessing of brain activity signals specifically includes: Brain activity signals obtained from multiple locations include spatial and temporal information. Wavelet decomposition is used to expand the single time series of brain activity signals at each location into a multi-frequency time series, adding frequency information to the spatial and temporal information of the brain activity signals.
3. The discrete decoding method under continuous decoding guidance in the speech brain-computer interface according to claim 1, characterized in that, In step (2), the acoustic features of the audio include Mel spectrum features, fundamental frequency features, and aperiodic signal features.
4. The discrete decoding method under continuous decoding guidance in the speech brain-computer interface according to claim 1, characterized in that, In step (3), the 1D-CNN network includes three sequentially connected one-dimensional convolutional modules, each of which includes a convolutional layer and a pooling layer connected in sequence.
5. The discrete decoding method under continuous decoding guidance in the speech brain-computer interface according to claim 1, characterized in that, In step (4), the weight coefficients λc and λd of Vc and Vd are dynamically changed to adjust the loss function and update the network parameters of the decoding model. Specifically, this includes: In the first stage, the weight coefficient λd of Vd is taken from the first segment of the sigmoid saturation function, and its size is controlled to be less than 0.
02. The weight coefficient λc of Vc is 1-λd, and its size is controlled to be greater than 0.
98. The continuous decoding module adjusts its own module parameters through the loss function L to achieve the maximum extraction of spatial features, frequency features and temporal features in the first stage. In the second stage, the Vd weight coefficient λd is taken as the second segment of the sigmoid saturation function. λd increases and λc decreases. The spatial features, frequency features and temporal features obtained by the continuous decoding module guide the training of the discrete decoding module. In the third stage, the weight coefficient λd of Vd is taken as the third segment of the sigmoid saturation function, with λd greater than 0.98 and λc less than 0.
02. The discrete decoding module self-trains to complete the fine-tuning of the entire model and updates the network parameters of the entire decoding model through the loss function.
6. The discrete decoding method under continuous decoding guidance in the speech brain-computer interface according to claim 5, characterized in that, The sigmoid function mentioned above is the sigmoid function.