A bidirectional brain-computer interface system based on hearing

Abstract

The present invention discloses a bidirectional brain-computer interface system based on hearing, comprising the following steps: collecting electroencephalogram (EEG) signals generated by a user after making sounds through motor imagery, preprocessing the EEG signals and constructing an EEG signal data set; constructing a matching long-short-term memory (LSTM) network model, training the matching LSTM network model, obtaining the trained matching LSTM network model, and converting the EEG signal data set into corresponding text data; constructing a vocoder model, inputting text data into the vocoder to generate a corresponding speech signal; preprocessing the speech signal and EEG signal, and constructing an EEG-speech data set; classifying and recognizing the EEG-speech data set, and constructing a speech data set, an EEG-speech data set, and an EEG data set; constructing a bilateral dual generative network model, training the bilateral dual generative network model, and obtaining the trained bilateral dual generative network model; and converting the speech signal to obtain the corresponding EEG signal.

Description

Technical Field

[0001] The present invention belongs to the technical field of brain-computer interaction, and in particular relates to a bidirectional brain-computer interface system based on hearing. Background Art

[0002] In 2021, the World Health Organization estimated that neurological disorders may affect up to 25% of patients worldwide, leading to symptoms including confusion, altered consciousness, and communication difficulties. Take stroke, for example. The economic, psychological, and social impacts of stroke are increasing, leaving patients unable to communicate with the outside world. This condition cuts people off from others and severely reduces their quality of life. It is now widely believed that recovery after stroke relies on neural mechanisms that allow unaffected brain areas to take over the functions of damaged tissue. This is where brain-computer interface technology comes in. Brain-computer interfaces (BCIs) are a new type of communication control technology between the human or animal brain and computers or other electronic devices. They do not rely on traditional brain information output circuits, providing patients with the opportunity to communicate with the outside world.

[0003] Based on the direction of signal transmission, brain-computer interfaces can be divided into unidirectional brain-computer interfaces (brain to computer or computer to brain) and bidirectional brain-computer interfaces. A bidirectional brain-computer interface refers to a system that can communicate bidirectionally between the brain and the computer. In other words, it is a brain-computer interface that can read and write data from the brain. Compared with unidirectional brain-computer interfaces, bidirectional brain-computer interfaces can provide a real-time feedback mechanism, allowing users to perceive and adjust their brain activity. This is very helpful for neurofeedback training and rehabilitation processes. Users can obtain real-time information through the brain-computer interface, which helps them improve the control and regulation of brain activity; at the same time, it makes the interaction more natural and intuitive. Not only can users send commands to external devices through the brain-computer interface, but external devices can also send information to users through the brain-computer interface. This two-way interaction mode is closer to the natural human-computer interaction method in daily life, improving user experience and interaction efficiency. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a bidirectional brain-computer interface system based on hearing, which improves user experience and interaction efficiency.

[0005] To achieve the above objectives, the present invention provides an auditory-based bidirectional brain-computer interface system, comprising:

[0006] Collecting EEG signals generated by the user after motor imagery to make sounds, preprocessing the EEG signals and constructing an EEG signal dataset;

[0007] Constructing a matching long short-term memory network model, training the matching long short-term memory network model using the EEG signal dataset, obtaining a trained matching long short-term memory network model, and converting the EEG signal dataset into corresponding text data using the trained matching long short-term memory network model;

[0008] Constructing a vocoder model, inputting the text data into the vocoder model to generate a corresponding speech signal, and constructing a speech signal dataset;

[0009] Preprocessing the speech signal dataset and the EEG signal dataset, and constructing an EEG-speech dataset;

[0010] Classify and recognize the EEG-speech dataset to construct a speech dataset, an EEG-speech dataset, and an EEG dataset;

[0011] Constructing a bilateral dual generative network model, training the bilateral dual generative network model using the speech dataset, the EEG-speech dataset, and the EEG dataset to obtain a trained bilateral dual generative network model;

[0012] The trained bilateral dual generative network model is used to convert the speech signal to obtain the corresponding EEG signal.

[0013] Optionally, the process of collecting EEG signals generated by the user after motor imagery of a sound includes:

[0014] The user is required to spontaneously imagine making a sound or to actively imagine hearing a sound, so as to induce a bioelectric reaction in the user's central nervous system through imagination, generate an electrical signal and then obtain the user's EEG signal.

[0015] Optionally, preprocessing the EEG signal and constructing an EEG signal dataset includes:

[0016] The EEG signals are filtered and denoised, and the pre-processed EEG signals are divided into several segments in chronological order. The segmented EEG signals are classified and labeled according to the segmentation-recombination method to construct an EEG signal dataset.

[0017] Optionally, the matching long short-term memory network model includes input gate, forget gate and output gate;

[0018] The input gate and the forget gate jointly determine the updated cell state C;

[0019] The input gate controls the process of key features of the input EEG signal entering the cell state C;

[0020] The forget gate determines whether to retain or discard the key features of the EEG signal in the cell state;

[0021] The output gate determines whether the key features of the EEG signal based on the current input and the hidden state of the cell state C will be output to the next moment.

[0022] Optionally, the output gate matching the long short-term memory network model includes a loss function, and the loss function is provided with a penalty term, and the weight of the output gate is penalized by the penalty term.

[0023] Optionally, the process of inputting the text data into a vocoder model to generate a corresponding speech signal includes:

[0024] The text data is input into a vocoder model and a high-fidelity adversarial generative network with weight parameters to obtain a corresponding speech signal.

[0025] Optionally, converting the EEG signal dataset into corresponding text data using the trained matching long short-term memory network model includes:

[0026] The EEG signal dataset is input into the trained matching long short-term memory network model, and the model weight parameters are saved by adjusting the parameters to the optimal state. The matching module defines the labels of the segmented data and calculates the similarity between the output sequence and the original characters, obtains the result with the highest similarity, and saves the result with the highest similarity as the corresponding text data.

[0027] Optionally, classifying and identifying the EEG-speech dataset, and constructing a speech dataset, an EEG-speech dataset and an EEG dataset include: mixing the EEG signal dataset and the speech signal dataset to form a multidimensional mixing matrix as the EEG-speech dataset; using the preprocessed EEG signal as the EEG dataset; and using the preprocessed speech signal as the speech dataset.

[0028] Technical effect of the present invention: The present invention discloses a bidirectional brain-computer interface system based on hearing. By combining deep learning, auditory induction, motor imagery and brain-computer interface technologies, a "human brain-computer-human brain" bidirectional brain-computer interface system design is realized, which can realize bidirectional conversion between EEG signals and voice signals, that is, the present invention can not only generate EEG signals into corresponding voice signals, but also reconstruct voice signals into corresponding EEG signals. The accuracy of the generated voice signal compared with the real voice signal can reach 0.96, the average opinion score of the generated voice signal in the voice quality index can reach 4.36, and the similarity between the reconstructed EEG signal and the real EEG signal can reach 0.95. The results show that the final effect achieved by the bidirectional brain-computer interface system based on hearing is that it can obtain the corresponding voice signal based on the EEG signal generated by the user's motor imagery. In other words, it can inform the EEG to know what the user wants to say. At the same time, it can also reconstruct the EEG signal generated by the user when thinking about this sentence through this voice signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0030] Figure 1 This is a flow chart of an auditory-based bidirectional brain-computer interface system according to an embodiment of the present invention;

[0031] Figure 2 This is a forward processing flow chart of an embodiment of the present invention;

[0032] Figure 3 Matching the long short-term memory network structure diagram for the embodiment of the present invention;

[0033] Figure 4 This is a reverse processing flow chart of an embodiment of the present invention;

[0034] Figure 5 This is a diagram of the EEG-speech mapping network structure according to an embodiment of the present invention;

[0035] Figure 6 This is a picture of the data acquisition equipment and the data acquisition site according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] like Figure 1 As shown, this embodiment provides a bidirectional brain-computer interface system based on hearing, including:

[0039] The acquisition process: First, the user is asked to spontaneously imagine making a sound, or to actively imagine hearing a sound, and the imagination is used to induce a bioelectric reaction in the user's central nervous system, generating an electrical signal. Figure 6 The acquisition device shown obtains the user's EEG signal.

[0040] The forward process is as follows Figure 2 As shown in the figure, the overall idea is to collect the EEG generated by the user through motor imagery, convert the EEG into text using a matching long short-term memory network, and then use a vocoder to convert the text into a speech signal;

[0041] Collect the EEG signals generated by the user through motor imagery. Motor imagery here means that the user actively imagines a sentence he wants to say or hears in his mind.

[0042] Preprocess the collected EEG signals and construct a related EEG signal dataset. (Preprocessing process: filtering the collected EEG signals)

[0043] Construct a matching long short-term memory network, the network structure is as follows Figure 3 As shown in the figure, based on the constructed EEG dataset, the matching long-term and short-term network is trained to obtain the trained matching long-term and short-term network, and the user's EEG information is converted to obtain the corresponding text data.

[0044] Construct a vocoder. Here, the vocoder uses a publicly available and trained vocoder high-fidelity generative adversarial network model. Input the aforementioned text data into the vocoder to generate a speech signal.

[0045] Detailed process:

[0046] Data preprocessing: First, the EEG signals are filtered and denoised, and evenly divided into five segments in chronological order. Then, the segmented EEG signals are classified and labeled according to the segmentation-recombination method to construct a dataset.

[0047] The segmentation-recombination method uses the temporal differences between characters during imagined movement to generate corresponding EEG data. Following the principle of musical notation, the numerical matrix corresponding to each character is evenly divided into n segments in chronological order. The starting position of the pen tip in each segment is used as the label for that segment. Similar segments of different characters are recombined to form a new dataset, with each n labels corresponding to a complete character.

[0048] Construct a matching long-term short-term network, which includes an input gate, a forget gate, and an output gate. The input gate and the forget gate jointly determine how the cell state C is updated.

[0049] The forget gate determines whether to retain or discard key features of the EEG signal in the cell state. It uses a sigmoid activation function to output a value between 0 and 1, where 0 indicates complete forgetfulness and 1 indicates complete retention. This gate determines whether to retain information about the previous cell state based on the input and the previous hidden state.

[0050] The input gate controls how key features of the new input EEG signal enter the cell state. First, a sigmoid activation function determines which parts of the cell state to update, and then a tanh activation function generates candidate values ​​that may be added to the cell state.

[0051] The output gate determines which key features of the EEG signal will be output to the next moment based on the current input and the hidden state of the cell state. The output gate uses the sigmoid activation function and the tanh activation function to produce the output value.

[0052] The output value of the output gate at the last moment is extracted and input into the fully connected layer to obtain the classification result of the output value.

[0053] The output gate matching the long short-term memory includes a loss function, and the loss function has a penalty term, and the weight of the output gate is penalized by the penalty term.

[0054] The dataset is fed into a matching long-short-term memory network, where parameters are adjusted to achieve the optimal state, and the model weights are saved. This yields the EEG signal classification results. Since the matching module defines the labels for the segmented data, the classification results can be evaluated by calculating the similarity between them and any segmented data labels determined by the segmentation-recombination method. A high similarity indicates that the classification result is correlated with the corresponding segmented data label. This establishes a mapping between the classification results and characters, allowing the original characters to be output, achieving EEG signal-to-text conversion.

[0055] The obtained text data is input into a high-fidelity adversarial generative network of a vocoder model with known weight parameters, and finally a reconstructed speech signal is obtained.

[0056] The reverse process is as follows Figure 4 As shown, the overall idea is:

[0057] The generated speech signal is used as an auditory stimulus signal and reconstructed into an EEG signal using a bilateral dual generative adversarial network.

[0058] The speech signal generated by the forward process and the collected EEG signal are preprocessed.

[0059] The preprocessed data is classified and identified, and then constructed into a speech dataset, an EEG / speech dataset, and an EEG dataset. The specific classification and identification method is as follows: the preprocessed EEG data and the decoded speech data are mixed to form a multidimensional mixing matrix as the EEG / speech dataset; the preprocessed EEG signal is used as the EEG dataset; and the preprocessed speech signal is used as the speech dataset.

[0060] Construct a bilateral dual generative adversarial network model, whose network structure is as follows Figure 5 As shown, the bilateral dual generative adversarial network model is trained based on the speech dataset, EEG / speech dataset, and EEG dataset to obtain a trained model, and the generated speech signal is converted to obtain the corresponding user EEG signal.

[0061] The training process of the bilateral dual generative adversarial network in the reverse process:

[0062] The bilateral dual GAN ​​involves three domains: domain V, constructed from the speech signal vectors generated by the forward process described above; domain E, constructed from the EEG signal vectors collected by the acquisition device described above; and domain T, constructed from the transition signal vectors formed by concatenating speech and EEG signals in a certain ratio. With the introduction of domain O and leveraging the cross-domain mapping properties of the bilateral dual GAN, the training process of the bilateral dual GAN ​​is divided into three loops: the speech-transition loop, the EEG-transition loop, and the large loop, all of which are trained simultaneously. Speech signal vectors u and EEG signal vectors v are sampled from domains V and E, respectively. By concatenating the corresponding speech and EEG signals in a certain ratio, a set of transition signals o, sampled from domain T, is obtained. The primary task of the bilateral dual GAN ​​is to convert speech signals u∈V into EEG signals v∈E. The speech-transition loop aims to learn the mapping between speech signals u∈V, while the EEG-transition loop learns the mapping between EEG signals v∈E. Different from the above operation mode, the large loop aims to learn the mapping between the transition signal o∈T and the speech signal u∈V, and then learn the mapping between the transition signal o∈T and the EEG signal v∈E. By training the bilateral dual generative adversarial network, the transition signal o∈T can be regarded as a shared label of the speech and EEG signals without containing their corresponding features. The details are described below.

[0063] Speech-Transition Loop. For speech signals, generator G1 implements cross-domain mapping from domain V to domain T. That is, the real speech signal u is mapped to domain T, generating the corresponding signal G1(u,z). Then, the inverse generator G2 implements cross-domain mapping from domain T to domain V, converting signal G1(u,z) back to domain V. The output signal G2(G1(u,z),z') is the reconstructed version of the real speech signal u.

[0064] EEG-transition loop. For EEG signals, generator G3 implements cross-domain mapping from domain E to domain T, mapping the real EEG signal v to domain T and generating the corresponding signal G3(v, z'). Then, inverse generator G4 implements cross-domain mapping from domain T to domain E, converting signal G3(v, z') back to domain V and outputting the reconstructed signal G4(G3(v, z'), z) as the reconstructed version of the real EEG signal v.

[0065] Big loop. For the transition signal, four steps are required to form a big loop. First, by using generator G2, the true transition signal o is mapped to the domain V, generating the transition signal G2(o,z'). Second, by using generator G1, the transition signal G2(o,z') is converted back to the domain O, and the output signal G1(G2(o,z'),z). Third, by using generator G4, the signal G1(G2(o,z'),z) is converted to the domain E, generating G4(G1(G2(o,z'),z),z). Fourth, by using generator G3, G4(G1(G2(o,z'),z),z) is converted back to the domain O, and the reconstructed signal G3(G4(G1(G2(o,z'),z),z),z') is output as the reconstructed version of o.

[0066] Discriminator D1 learns by distinguishing the real transition signal o of domain O from the spurious output of generator G1, while discriminator D2 learns by distinguishing the real EEG signal u of domain V from the spurious output of generator G2. Similarly, discriminator D4 learns by distinguishing the real speech signal v of domain E from the spurious output of generator G4, while discriminator D3 learns by distinguishing the generated transition signal G1(G2(o,z'),z) of domain O from the spurious output of generator G3. Generators G1, G2, G3, and G4 are optimized to imitate fake outputs to fool the corresponding discriminators D1, D2, D3, and D4 while minimizing the following reconstruction losses ||u-G2(G1(u,z),z')||, ||o-G1(G2(o,z'),z)||, ||v-G4(G3(v,z'),z)||, and ||G1(G2(o,z'),z)-G3(G4(G1(G2(o,z'),z),z),z')||.

[0067] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A bidirectional brain-computer interface system based on hearing, characterized in that: include: Collecting EEG signals generated by the user after motor imagery to make sounds, preprocessing the EEG signals and constructing an EEG signal dataset; Constructing a matching long short-term memory network model, training the matching long short-term memory network model using the EEG signal dataset, obtaining a trained matching long short-term memory network model, and converting the EEG signal dataset into corresponding text data using the trained matching long short-term memory network model; Constructing a vocoder model, inputting the text data into the vocoder model to generate a corresponding speech signal, and constructing a speech signal dataset; Preprocessing the speech signal dataset and the EEG signal dataset, and constructing an EEG-speech dataset; Classify and recognize the EEG-speech dataset to construct a speech dataset, an EEG-speech dataset, and an EEG dataset; Mixing the EEG signal dataset and the speech signal dataset to form a multi-dimensional mixing matrix as the EEG-speech dataset; The preprocessed EEG signals are used as EEG datasets; The preprocessed speech signal is used as a speech dataset; Constructing a bilateral dual generative network model, training the bilateral dual generative network model using the speech dataset, the EEG-speech dataset, and the EEG dataset to obtain a trained bilateral dual generative network model; Using the trained bilateral dual generative network model, the speech signal is converted to obtain a corresponding electroencephalogram signal; The training process of the bilateral dual generation network model in the reverse process: The bilateral dual generative adversarial network involves three domains: the domain V constructed by the speech signal vector generated by the forward process, the domain E constructed by the EEG signal vector collected by the acquisition device, and the domain T constructed by the transition signal vector composed of the speech signal and the EEG signal cascaded in a certain proportion; due to the introduction of domain O and the use of the cross-domain mapping characteristics of the bilateral dual generative adversarial network, the training process of the bilateral dual generative adversarial network is divided into three cycles: speech-transition cycle, EEG-transition cycle and large cycle, and these three cycles are trained simultaneously; the speech signal vector u and the EEG signal vector v are sampled from the domains V and E respectively; by cascading the corresponding Speech and EEG signals, a set of transition signals o sampled from domain T is obtained; the main task of the bilateral dual generative adversarial network is to convert the speech signal u∈V into the EEG signal v∈E; among them, the speech-transition cycle aims to learn the mapping between speech signals u∈V, while the EEG-transition cycle learns the mapping between EEG signals v∈E; the large cycle aims to learn the mapping between the transition signal o∈T and the speech signal u∈V, and the mapping between the transition signal o∈T and the EEG signal v∈E; by training the bilateral dual generative network model, the transition signal o∈T is the shared label of the speech and EEG signals, without containing their corresponding features. The specific details are as follows: Speech-transition loop: For the speech signal, the generator G1 is used to realize the cross-domain mapping from domain V to domain T. That is, the real speech signal u is mapped to domain T and the corresponding signal G1(u,z) is generated. Then, the inverse generator G2 is used to realize the cross-domain mapping from domain T to domain V, and the signal G1(u,z) is converted back to domain V. The output signal G2(G1(u,z),z′) is used as the reconstructed version of the real speech signal u. EEG-transition loop: For EEG signals, the generator G3 is used to realize cross-domain mapping from domain E to domain t, and the real EEG signal v is mapped to domain T to generate the corresponding signal G3(v, z′); then, the inverse generator G4 is used to realize cross-domain mapping from domain T to domain E, and the signal G3(v, z′) is converted back to domain V, and the reconstructed signal G4(G3(v, z′), z) is output as the reconstructed version of the real EEG signal v; Big loop: For the transition signal, four steps are required to form a big loop; first, by using the generator G2, the real transition signal o is mapped to the domain V, and the transition signal G2(o, z′) is generated; second, by using the generator G1, the transition signal G2(o, z′) is converted back to the domain O, and the signal G1(G2(o, z′), z) is output; third, by using the generator G4, the signal G1(G2(o, z′), z) is converted to the domain E, and G4(G1(G2(o, z′), z), z) is generated; fourth, by using the generator G3, G4(G1(G2(o, z′), z), z) is converted back to the domain O, and the reconstructed signal G3(G4(G1(G2(o, z′), z), z), z′) is output as the reconstructed version of o; Discriminator D1 learns by distinguishing the real transition signal o of domain O from the false output of generator G1, while discriminator D2 learns by distinguishing the real EEG signal u of domain V from the false output of generator G2; similarly, discriminator D4 learns by distinguishing the real speech signal v of domain E from the false output of generator G4, while discriminator D3 learns by distinguishing the generated transition signal G1(G2(o,z′),z) of domain O from the false output of generator G3; Generator G1 , G2, G3, and G4 are optimized to imitate fake outputs to fool the corresponding discriminators D1, D2, D3, and D4 while minimizing the following reconstruction losses ||u-G2(G1(u,z),z′)||, ||o-G1(G2(o,z′),z)||, ||v-G4(G3(v,z′),z)||, and ||G1(G2(o,z′),z)-G3(G4(G1(G2(o,z′),z),z),z′)||.

2. The auditory-based bidirectional brain-computer interface system according to claim 1, characterized in that: The process of collecting EEG signals generated by the user through motor imagery includes: The user is required to spontaneously imagine making a sound or to actively imagine hearing a sound, so as to induce a bioelectric reaction in the user's central nervous system through imagination, generate an electrical signal and then obtain the user's EEG signal.

3. The auditory-based bidirectional brain-computer interface system according to claim 1, characterized in that: Preprocessing the EEG signal and constructing an EEG signal dataset includes: The EEG signals are filtered and denoised, and the pre-processed EEG signals are divided into several segments in chronological order. The segmented EEG signals are classified and labeled according to the segmentation-recombination method to construct an EEG signal dataset.

4. The auditory-based bidirectional brain-computer interface system according to claim 1, wherein: The matching long short-term memory network model includes input gate, forget gate and output gate; The input gate and the forget gate jointly determine the updated cell state C; The input gate controls the process of key features of the input EEG signal entering the cell state C; The forget gate determines whether to retain or discard the key features of the EEG signal in the cell state; The output gate determines whether the key features of the EEG signal based on the current input and the hidden state of the cell state C will be output to the next moment.

5. The auditory-based bidirectional brain-computer interface system according to claim 3, characterized in that: The output gate of the matching long short-term memory network model includes a loss function, wherein the loss function is provided with a penalty term, and the weight of the output gate is penalized by the penalty term.

6. The auditory-based bidirectional brain-computer interface system according to claim 1, characterized in that: The process of inputting the text data into the vocoder model to generate the corresponding speech signal includes: The text data is input into a vocoder model and a high-fidelity adversarial generative network with weight parameters to obtain a corresponding speech signal.

7. The auditory-based bidirectional brain-computer interface system according to claim 1, characterized in that: Converting the EEG signal dataset into corresponding text data using the trained matching long short-term memory network model includes: The EEG signal dataset is input into the trained matching long short-term memory network model, and the model weight parameters are saved by adjusting the parameters to the optimal state. The matching module defines the labels of the segmented data and calculates the similarity between the output sequence and the original characters, obtains the result with the highest similarity, and saves the result with the highest similarity as the corresponding text data.

Images