A visually induced bidirectional brain-computer interface system and a brain-computer bidirectional communication method

Abstract

The present invention discloses a visually induced bidirectional brain-computer interface system and a bidirectional brain-computer communication method. The system comprises a forward processing device for converting the electroencephalogram (EEG) signals generated by a user while visualizing letters; and a reverse processing device for converting the EEG signals into corresponding EEG signals after the brain is stimulated by the letters. The present invention allows for bidirectional information transfer between the brain and external devices. This system not only captures signals from the brain and decodes them for controlling external devices, but also transmits information from external devices back to the brain, enabling interaction between the human brain and computer systems.

Description

Technical Field

[0001] The present invention belongs to the technical field of brain-computer interfaces, and in particular relates to a visually induced bidirectional brain-computer interface system and a brain-computer bidirectional communication method. Background Art

[0002] A brain-computer interface (BCI) is a novel communication system that uses EEG signals generated by human neural activity to enable communication between humans and the external environment. It bypasses the muscles, spinal cord, and peripheral nerves, instead collecting EEG signals directly from the brain through EEG acquisition equipment. Using classification algorithms, it processes these signals to understand the subject's intended meaning. This is undoubtedly valuable for patients with serious illnesses such as stroke and spinal cord injury who have lost the ability to move and speak. However, there are currently no BCIs that can transmit information about a person's mental, emotional, and health status, enabling interaction between the human brain and computer systems. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a visually induced bidirectional brain-computer interface system and a brain-computer bidirectional communication method, which allow bidirectional information transmission between the brain and external devices. It can not only obtain signals from the brain and decode them for controlling external devices, but also transmit information from external devices back to the brain, thereby realizing interaction between the human brain and computer systems.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A visually induced bidirectional brain-computer interface system, comprising:

[0006] A forward processing device for converting the real-time EEG signals generated by the user when he / she imagines letters into corresponding real-time letter graphic signals;

[0007] The reverse processing device is used to convert the real-time letter graphic signal into a corresponding real-time EEG signal after the brain is stimulated by the letter graphic.

[0008] Preferably, the forward processing device comprises:

[0009] The first acquisition module is used to collect historical EEG signals generated by the user when he or she is mentally imagining letters;

[0010] The first training module is used to train the BiLSTM network model based on historical EEG signals;

[0011] The forward processing module is used to convert the real-time EEG signals into their corresponding real-time letter graphic signals based on the trained BiLSTM network model.

[0012] Preferably, the reverse processing device comprises:

[0013] A second acquisition module is used to acquire a historical letter pattern signal corresponding to the historical EEG signal after the brain is stimulated by the letter pattern;

[0014] The second training module is used to train the EEG-graphic mapping model based on historical letter-graphic signals;

[0015] The reverse processing module is used to convert the real-time letter graphic signal into a corresponding real-time EEG signal according to the trained EEG-graphic mapping model.

[0016] Preferably, the BiLSTM network model is composed of a forward LSTM and a backward LSTM; the forward LSTM is trained from the starting end of the EEG signal after data processing, and the backward LSTM is trained from the end of the EEG signal after data processing.

[0017] Preferably, the EEG-graphics mapping model is a Quad-GAN network model based on optical flow constraints.

[0018] The present invention also provides a brain-computer two-way communication method, comprising the following steps:

[0019] Convert the real-time EEG signal generated by the user when he / she imagines letters into the corresponding real-time letter graphic signal;

[0020] When the brain is stimulated by the letter pattern, the real-time letter pattern signal is converted into a corresponding real-time electroencephalogram signal.

[0021] Preferably, converting the real-time EEG signal generated by the user when the user imagines letters in motion into the corresponding real-time letter graphic signal includes:

[0022] Collect historical EEG signals generated by the user when he or she is motorically imagining letters;

[0023] Train the BiLSTM network model based on historical EEG signals;

[0024] According to the trained BiLSTM network model, the real-time EEG signals are converted into their corresponding real-time letter graphic signals.

[0025] Preferably, when the brain is stimulated by the letter pattern, converting the real-time letter pattern signal into a corresponding real-time electroencephalogram signal comprises:

[0026] When the brain is stimulated by the letter pattern, a historical letter pattern signal corresponding to the historical EEG signal is obtained;

[0027] Training the EEG-graphic mapping model based on historical letter-graphic signals;

[0028] According to the trained EEG-graphic mapping model, the real-time letter graphic signal is converted into a corresponding real-time EEG signal.

[0029] Preferably, the BiLSTM network model is composed of a forward LSTM and a backward LSTM; the forward LSTM is trained from the starting end of the EEG signal after data processing, and the backward LSTM is trained from the end of the EEG signal after data processing.

[0030] Preferably, the EEG-graphics mapping model is a Quad-GAN network model based on optical flow constraints.

[0031] The present invention realizes the design of a "human brain-computer-human brain" bidirectional brain-computer interface system by combining deep learning, visual induction, motor imagery and brain-computer interface technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0033] Figure 1 2 is a schematic structural diagram of a visually induced bidirectional brain-computer interface system according to an embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of the working principle of the forward processing device;

[0035] Figure 3 It is a working schematic diagram of the reverse processing device;

[0036] Figure 4 This is a schematic diagram of the Quad-GAN network model based on optical flow constraints;

[0037] Figure 5 It is a flowchart of the optical flow constraint process. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Example 1:

[0041] like Figure 1 As shown, an embodiment of the present invention provides a visually induced bidirectional brain-computer interface system, comprising:

[0042] A forward processing device for converting the real-time EEG signals generated by the user when he / she imagines letters into corresponding real-time letter graphic signals;

[0043] The reverse processing device is used to convert the real-time letter graphic signal into a corresponding real-time EEG signal after the brain is stimulated by the letter graphic.

[0044] The embodiment of the present invention first requires the user to imagine a piece of letter graphic information, or requires the user to actively imagine letter information, and through the imagination, a bioelectric reaction of the user's central nervous system is caused, an electrical signal is generated, and thus the user's brain electrical signal is obtained. Motor imagery (MI), as the name suggests, when a person imagines his or her limbs (or muscles) moving but there is no actual movement output, the person's specific brain area will still be activated. By analyzing the brain electrical signals and detecting and identifying the activation effects of different brain areas, the user's intention is judged, thereby achieving direct communication and control between the human brain and external devices.

[0045] The overall idea of ​​the forward processing device is to collect the brain waves generated by the user's motor imagination of letters and graphics, pre-process the EEG signals, construct a data set with the processed data, build a bidirectional long short-term memory network (BiLSTM network) to train the constructed data set, and use the trained network to convert the EEG signals into letter and graphic signals. At this time, the letter and graphic signals are in text form.

[0046] 1. Collect the user's brain waves generated by motor imagery. The brain waves are generated by requiring the user to imagine a letter pattern, or requiring the user to actively imagine a letter pattern. The imagination causes a bioelectric reaction in the user's central nervous system, generating electrical signals to obtain the user's brain wave signals.

[0047] 2. Preprocess the collected EEG signals and construct a related EEG signal dataset. (Preprocessing process: filtering the collected EEG signals)

[0048] 3. Build a BiLSTM model and train it based on the EEG signal dataset built in the previous step. (The BiLSTM model is a bidirectional LSTM, with both the forward LSTM and the backward LSTM working together.)

[0049] The overall idea of ​​the reverse processing device is to use the generated letter graphic signal as a visual stimulus signal and reconstruct it into an EEG signal using the Quad-GAN network (combined with optical flow).

[0050] 1. Preprocess the letter graphic signals generated by the forward process and the collected EEG signals.

[0051] 2. Classify and identify the preprocessed data, and construct it into a letter and pattern dataset, an EEG / letter and pattern dataset, and an EEG dataset. (The classification and identification method is: the preprocessed EEG data and the decoded letter and pattern data are mixed to form a multidimensional mixed matrix, which is used as the EEG / letter and pattern dataset; the preprocessed EEG signal is used as the EEG dataset; and the preprocessed pattern signal is used as the pattern dataset.)

[0052] 3. Construct a Quad-GAN network model, train the Quad-GAN network model based on the letter graphic dataset, EEG / graphic dataset, and EEG dataset to obtain a trained model, convert the generated letter graphic signal, and obtain the corresponding user EEG signal.

[0053] As an implementation manner of an embodiment of the present invention, a forward processing device includes:

[0054] The first acquisition module is used to collect historical EEG signals generated by the user when he or she is mentally imagining letters;

[0055] The first training module is used to train the BiLSTM network model based on historical EEG signals;

[0056] The forward processing module is used to convert the real-time EEG signals into their corresponding real-time letter graphic signals based on the trained BiLSTM network model.

[0057] Further, if Figure 2 As shown, the forward processing process of the forward processing device is as follows:

[0058] Step 11: Collect the user’s brain waves generated by motor imagery – using a non-invasive brain-computer interface

[0059] Common EEG signal acquisition can be divided into two categories: invasive (also known as invasive) and non-invasive (also known as non-invasive). Correspondingly, according to the different ways of acquiring EEG signals in the brain-computer interface, the brain-computer interface system can be divided into invasive, non-invasive and semi-invasive brain-computer interface systems. The non-invasive method is to place electrodes on the surface of the scalp to avoid the harm to the subject caused by the invasive EEG acquisition method. The subject imagines the letter graphics in the brain in a static state, and completes the EEG data acquisition at the same time, and then completes the preprocessing of the data.

[0060] Step 12: Data processing

[0061] The network model in this embodiment requires input in text format, while EEG data collected by specialized EEG acquisition equipment is in .NED format, making the two data formats incompatible. Therefore, the .NED format must first be converted to a text format type. The resulting text format is then used to construct a BiLSTM network. Data processing primarily filters out artifacts from the raw EEG signals, converts them into a suitable data format, and constructs the corresponding training and test datasets (EEG datasets).

[0062] Step 13: Build a BiLSTM network

[0063] BiLSTM is composed of a forward LSTM and a backward LSTM. A single-layer BiLSTM is composed of two LSTMs: one processes the input sequence in the forward direction, and the other processes it in the reverse direction. After processing, the outputs of the two LSTMs are concatenated. The forward LSTM is trained from the beginning of the processed EEG signal, while the backward LSTM is trained from the end of the processed EEG signal. The concatenated data is then converted into text format.

[0064] The four function layers of the LSTM network are a forget gate, two and three memory gates, and a four-layer output gate.

[0065] Forget gate: Each time a new input C is input t , LSTM will first t and the output L at the previous moment t-1 Decide which previous memories to forget - Enter C t And the output L of the previous step t-1 It is integrated into a single vector, then passed through the sigmoid neural layer (the input is compressed to the (0,1) interval), and finally multiplied point-to-point on the unit state.

[0066] Memory gate: A control unit used to control whether the data at time t (now) is incorporated into the cell state.

[0067] Output layer: will first input the current value C t and the output value L at the previous moment t-1 The information in the integrated vector is extracted using the sigmoid function. Then, the current unit state is compressed and mapped to the interval (-1, 1) using the tanh function. The output of LSTM at time t is obtained by multiplying the unit state processed by the tanh function and the integrated vector processed by the sigmoid function point-to-point.

[0068] Step 14: Input the processed data set in step 12 into the BiLSTM network. After adjusting the parameters to the optimal state, save the model. The matching module defines the labels of the segmented data and calculates the similarity between the output sequence and the original characters. The result with the highest similarity is output and saved as the corresponding text data.

[0069] As an implementation manner of an embodiment of the present invention, a reverse processing device includes:

[0070] A second acquisition module is used to acquire a historical letter pattern signal corresponding to the historical EEG signal after the brain is stimulated by the letter pattern;

[0071] The second training module is used to train the EEG-graphic mapping model based on historical letter-graphic signals;

[0072] The reverse processing module is used to convert the real-time letter graphic signal into a corresponding real-time EEG signal according to the trained EEG-graphic mapping model.

[0073] Further, if Figure 3 As shown, the reverse processing process of the reverse processing device is as follows:

[0074] Step 21: Reverse data processing

[0075] The brain activity reconstruction model uses the letter graphics obtained in the forward process as input and ultimately achieves a one-to-one mapping of the letter graphics signal to the EEG signal through an end-to-end mapping network. However, EEG signals and graphic signals belong to different signal domains. Directly mapping these two signals without processing will hardly achieve the desired results. Therefore, an intermediate transition domain is added between the EEG domain and the text domain to receive the letter graphics and EEG signals, establishing a connection between the EEG domain and the text domain, thereby achieving cross-domain mapping of EEG signals to text signals.

[0076] Step 22: Build a network model to achieve reconstruction

[0077] A network is proposed to achieve end-to-end mapping from EEG signals to letter graphics by introducing an intermediate domain. In order to enable effective conversion between different signal domains, a transition domain O is added for signal transition. The intermediate mixed domain signal extracts a portion of data from the EEG signal and the letter graphic signal in the EEG domain, and then cascades the extracted data according to a preset ratio. The cascaded signal is the transition signal of the transition domain. Different from the letter graphics and EEG signals, the transition domain signal will serve as a bridge throughout the learning process of the entire recurrent network, perform a large loop between the two different signal domains, connect the letter graphics and EEG signals built by the training generation network, and complete the point-to-point mapping from the letter graphic signal to the EEG signal, such as Figure 4 As shown, the details are as follows:

[0078] (1) Quad-GAN involves three domains U, V, and O, where O is the transition domain. The transition domain O is introduced into Quad-GAN, and a left small loop (loop one), a right small loop (loop two), and a large loop (loop three) are constructed, which are trained simultaneously. The letter graphic signal set u and the EEG signal set v are sampled from the domains U and V respectively. The corresponding letter graphic signals and EEG signals are cascaded in a certain ratio, and a set of transition signals o are sampled from the transition domain O. The transition signal o is obtained by sampling the signal after the above cascade. The main task of Quad-GAN is to convert the mapping u∈U into the mapping v∈V, that is, to complete the conversion from letter graphic signals to EEG signals.

[0079] Among them, loop one aims to learn the mapping between letter pattern signals and u∈U, while loop two learns the mapping between EEG signals v∈V. Loop three, which operates differently from the above mode, aims to learn the mapping between transition signals o∈O and letter patterns u∈U, and then learn the mapping between transition signals o∈O and EEG signals v∈V. By training Quad-GAN, the transition signal o∈O can be regarded as a shared label of letter patterns u∈U and EEG signals v∈V, without including their corresponding features. The letter pattern signals and EEG signals constructed by the training generative network are connected to complete the point-to-point mapping of EEG signals to letter pattern signals. The specific details are described below.

[0080] (2) Loop 1 (left small loop): For the letter graphic signal, the real letter graphic signal u in the U domain is mapped to the transition domain O through the generator G1:U→O, and the signal G1(u,s) is generated accordingly. Then, the signal passes through the inverse generator G2:O→U, and the signal G1(u,s) is converted back to the domain U, and the output signal As the reconstructed signal of the real letter pattern signal.

[0081] 3. Loop 2 (right small loop): For EEG signals, the real EEG signal v in V is mapped to the transition domain O through the generator G3:V→O, and the signal is generated accordingly Then, through the reverse generator G4:O→V, the signal Converted back to domain V, the output signal As the reconstructed signal of the real EEG signal v.

[0082] 4. Loop 3 (Big Loop): For the transition signal o, the big loop consists of four steps. First, by using the generator G2, the real transition signal o is mapped to the domain U to generate the transition signal Secondly, by using generator G1, the transition signal Reverse conversion back to domain U, the output signal Third, by using generator G4, the signal Convert to domain V, generating Fourth, by using generator G3, the signal Convert back to the transition domain O and output the reconstructed signal The output signal is the reconstructed version of o.

[0083] The learning of the discriminator D1 is completed by distinguishing the real transition signal o of domain O from the false output of generator G1, while the discriminator D2 is learned by distinguishing the real EEG signal u of domain U from the false output of generator G2; similarly, the discriminator D4 is learned by distinguishing the real letter graphic signal v of domain V from the false output of generator G4, while the discriminator D3 is learned by distinguishing the generated transition signal The generators G1, G2, G3, and G4 are optimized to imitate the fake outputs to fool the corresponding discriminators D1, D2, D3, and D4 while minimizing the following reconstruction loss as well as

[0084] 1. Graphic Video EEG Reconstruction - Optical Flow Constraints

[0085] To address the relationship between video frames composed of multiple letter shapes, the concept of optical flow in video processing is more suitable for solving current research problems. The original video frame sequence is processed by the FlowNet optical flow calculation network to extract the original optical flow information. Simultaneously, the video frame sequence is passed through the corresponding generator in the network to produce a series of low-definition video frames. The original video frame sequence is then combined with the optical flow information to generate a predicted warped image. Finally, the warped image and the low-definition image are combined to obtain the final high-definition image, and the next frame image is constructed.

[0086] The motion field in space is transferred to an image and represented as an optical flow field. The optical flow field is a two-dimensional vector field that reflects the grayscale variation trend of each point in the image. It can be viewed as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. The information it contains is the instantaneous velocity vector information of each pixel.

[0087] Consider the light intensity of a pixel I(x,y,t) in the first frame (where t represents the time dimension). It moves (d x ,d y ) to the next frame, using d t Time. Because it is the same pixel, according to the first assumption mentioned above, we believe that the light intensity of the pixel before and after the movement is unchanged, that is:

[0088] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0089] Perform Taylor expansion on the right side of the above equation to obtain:

[0090]

[0091] The above two equations can be simplified together to get:

[0092]

[0093] Let u and v be the velocity vectors of the optical flow along the X-axis and Y-axis respectively, and we can get:

[0094]

[0095] make Represent the partial derivatives of the grayscale of the pixel in the image along the X, Y, and T directions respectively.

[0096] Yes: I x u+I y v+I t =0, where I x , I y and I t Both can be obtained from image data, and (u,v) is the required optical flow vector.

[0097] The above formula is the optical flow constraint equation.

[0098] The present invention uses the FlowNet2.0 optical flow estimation network to perform optical flow calculations. This is the general idea of ​​the FlowNet neural network, which optimizes the optical flow effect through network superposition, warps the input letter graphic signal, and introduces a branch network to specifically handle small movements of objects. Among them, the network input is the image of the t-th frame and the t+1-th frame, which first passes through a contraction part composed of a convolutional layer to extract their respective feature maps, but this will cause the image to become smaller, so it must pass through an amplification layer to expand it to the size of the original image to predict the optical flow. The optical flow constraints used in the present invention are as follows: Figure 5 As shown in the figure, the original video frame sequence is processed by the FlowNet optical flow computation network to extract the original optical flow information. Simultaneously, the video frame sequence passes through the corresponding generator in the network to generate a series of low-definition video frames. The original video frame sequence is then combined with the optical flow information to generate a predicted warped image. Finally, the warped image and the low-definition image are combined to produce the final high-definition image, which is then used to construct the next frame.

[0099] Example 2:

[0100] An embodiment of the present invention further provides a brain-computer bidirectional communication method, comprising the following steps:

[0101] Convert the real-time EEG signal generated by the user when he / she imagines letters into the corresponding real-time letter graphic signal;

[0102] When the brain is stimulated by the letter pattern, the real-time letter pattern signal is converted into a corresponding real-time electroencephalogram signal.

[0103] As an implementation method of an embodiment of the present invention, converting the real-time EEG signal generated by the user when visualizing letters in motion into the corresponding real-time letter graphic signal includes:

[0104] Collect historical EEG signals generated by the user when he or she is motorically imagining letters;

[0105] Train the BiLSTM network model based on historical EEG signals;

[0106] According to the trained BiLSTM network model, the real-time EEG signals are converted into their corresponding real-time letter graphic signals.

[0107] As an implementation method of an embodiment of the present invention, when the brain is stimulated by a letter pattern, converting the real-time letter pattern signal into a corresponding real-time EEG signal includes:

[0108] When the brain is stimulated by the letter pattern, a historical letter pattern signal corresponding to the historical EEG signal is obtained;

[0109] Training the EEG-graphic mapping model based on historical letter-graphic signals;

[0110] According to the trained EEG-graphic mapping model, the real-time letter graphic signal is converted into a corresponding real-time EEG signal.

[0111] As an implementation method of an embodiment of the present invention, the BiLSTM network model is composed of a forward LSTM and a backward LSTM; the forward LSTM is trained from the starting end of the EEG signal after data processing, and the backward LSTM is trained from the end of the EEG signal after data processing.

[0112] As an implementation method of an embodiment of the present invention, the EEG-graphics mapping model is a Quad-GAN network model based on optical flow constraints.

[0113] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A visually induced bidirectional brain-computer interface system, characterized in that: include: A forward processing device for converting the real-time EEG signals generated by the user when he / she imagines letters into corresponding real-time letter graphic signals; A reverse processing device, configured to convert the real-time letter pattern signal into a corresponding real-time electroencephalogram signal when the brain is stimulated by the letter pattern; A forward processing device comprising: The first acquisition module is used to collect historical EEG signals generated by the user when he or she is mentally imagining letters; The first training module is used to train the BiLSTM network model based on historical EEG signals; The forward processing module is used to convert the real-time EEG signals into their corresponding real-time letter and graphic signals based on the trained BiLSTM network model; The reverse processing device comprises: A second acquisition module is used to acquire a historical letter pattern signal corresponding to the historical EEG signal after the brain is stimulated by the letter pattern; The second training module is used to train the EEG-graphic mapping model based on historical letter-graphic signals; A reverse processing module, configured to convert the real-time letter graphic signal into a corresponding real-time EEG signal according to a trained EEG-graphic mapping model; The BiLSTM network model is composed of a forward LSTM and a backward LSTM. The forward LSTM is trained from the beginning of the EEG signal after data processing, and the backward LSTM is trained from the end of the EEG signal after data processing. The EEG-graphics mapping model is a Quad-GAN network model based on optical flow constraints.

2. A brain-computer two-way communication method, characterized in that: The following steps are involved: Convert the real-time EEG signal generated by the user when he / she imagines letters into the corresponding real-time letter graphic signal; When the brain is stimulated by the letter pattern, the real-time letter pattern signal is converted into a corresponding real-time electroencephalogram signal; The real-time EEG signals generated by the user when imagining letters are converted into corresponding real-time letter graphic signals, including: Collect historical EEG signals generated by the user when he or she is motorically imagining letters; Train the BiLSTM network model based on historical EEG signals; According to the trained BiLSTM network model, the real-time EEG signals are converted into their corresponding real-time letter graphic signals; When the brain is stimulated by the letter pattern, the real-time letter pattern signal is converted into a corresponding real-time EEG signal, including: When the brain is stimulated by the letter pattern, a historical letter pattern signal corresponding to the historical EEG signal is obtained; Training the EEG-graphic mapping model based on historical letter-graphic signals; Converting the real-time letter graphic signal into a corresponding real-time EEG signal according to a trained EEG-graphic mapping model; The BiLSTM network model is composed of a forward LSTM and a backward LSTM. The forward LSTM is trained from the beginning of the EEG signal after data processing, and the backward LSTM is trained from the end of the EEG signal after data processing. The EEG-graphics mapping model is a Quad-GAN network model based on optical flow constraints.

Images