A Chinese continuous language text reconstruction method based on EEG signals

Abstract

The present invention relates to the field of natural language processing technology, and specifically discloses a method for reconstructing Chinese continuous language text based on electroencephalogram (EEG) signals, comprising the following steps: selecting a diverse Chinese corpus and preprocessing it as audio stimulation material; based on the audio stimulation material, collecting EEG data of a subject performing an auditory attention experiment and performing data preprocessing; proposing a time-scale-driven word-level EEG synchronization method to extract word-level text features and word-level EEG sequences; utilizing a spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model to extract spatiotemporal-frequency fusion EEG features and predict their word-level text features; and realizing Chinese continuous language text reconstruction based on a beam search Chinese continuous language dynamic decoding model guided by a pre-trained model and combining the predicted word-level text features. The method of the present invention fully reveals the connection between EEG signals and text features, thereby generating more natural and fluent text information, and ultimately improving the accuracy of Chinese continuous language reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for reconstructing Chinese continuous language text based on electroencephalogram signals. Background Art

[0002] Language is not only a tool for human communication, but also the foundation of thinking and intelligence. How the brain decodes and processes language information is the key to revealing the essence of human intelligence. With the rapid development of technologies such as Brain-Computer Interface (BCI), it has become possible to decode natural language from brain activity. When the brain is stimulated by language, the human brain encodes semantic information through neural activity. By analyzing these neural activities, the semantic encoding mechanism in the brain can be revealed. This research direction is not only crucial to the development of cognitive science and neuroscience, but also provides new hope for people who have lost the ability to communicate through language due to neurodegenerative diseases and trauma. The development of this direction will greatly expand our understanding of how the human brain processes language and may open up new ways of communication.

[0003] Most research on the decoding of semantic information involves the reconstruction of continuous language text in Western languages ​​such as English, and research on Chinese is still lagging behind. The brain's response to Chinese is specific, and Chinese is highly complex, with a variety of tones, characters, and contextual meanings, which differ greatly from Western languages. Directly applying existing decoding frameworks and models faces challenges. Furthermore, the lack of data on Chinese corpus stimulation further limits the development of Chinese brain signal decoding technology. Currently, research on continuous language reconstruction in Chinese is insufficient. Summary of the Invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a Chinese continuous language text reconstruction method based on EEG signals. Through this method, the connection between EEG signals and text features can be fully characterized, more fluent text information can be generated, and high-precision Chinese continuous language reconstruction can be achieved, thus solving the problems mentioned in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for reconstructing Chinese continuous language text based on EEG signals, comprising the following steps:

[0006] S1. Select a diverse Chinese corpus and pre-process it as audio stimulus material;

[0007] S2. Based on the audio stimulation materials, the subjects collected EEG data for the auditory attention experiment and performed data preprocessing on the collected EEG data;

[0008] S3. A time-scale-driven word-level-EEG synchronization method is proposed to extract word-level text features and word-level EEG sequences.

[0009] S4. Based on word-level EEG sequence and word-level text features, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is used to extract spatiotemporal-frequency fusion EEG features and predict their word-level text features;

[0010] S5. Based on the beam search Chinese continuous language dynamic decoding model guided by the pre-trained model, combined with the predicted word-level text features, Chinese continuous language text reconstruction is realized.

[0011] Preferably, in step S2, the collected EEG data is preprocessed, including channel removal, notch filtering, bandpass filtering, and independent principal component analysis; specifically, the channels related to the electrocardiogram and electrooculogram are first removed, and then a 50Hz notch filter is performed to eliminate power line interference, and then a 1-50Hz bandpass filter is used to filter out DC components and high-frequency noise, and finally an independent component analysis is performed to remove artifacts to obtain preprocessed EEG data.

[0012] Preferably, in step S3, a time-scale driven word-level-EEG synchronization method is proposed to extract word-level text features, specifically including: first, based on the audio stimulus material, using the iFlytek audio-to-text interface and manual calibration to complete the audio-to-text conversion, then using the ninth hidden layer of the pre-trained large model GPT-2 to extract the text features corresponding to each word in the text, that is, the 768-dimensional text embedding vector, and finally normalizing the text embedding vector to obtain the word-level text feature w i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

[0013] Preferably, in step S3, a time-scale driven word-level EEG synchronization method is proposed to extract word-level EEG sequences, specifically comprising: firstly, automatically aligning the audio with the extracted text using the MFA forced alignment tool; then, manually checking and correcting the alignment of the audio and text using the Praat tool to ensure complete alignment; then, obtaining the timestamp corresponding to each word when it appears in the audio; then, based on the timestamp corresponding to each word, dividing the EEG data using a fixed window method, specifically, obtaining the center time point of each word according to the start time and end time of each word, and then dividing the EEG time window based on the average time range of the center time point of each word, the EEG time window is the word-level EEG sequence e i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

[0014] Preferably, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model of step S4 includes four steps: spatiotemporal-frequency fusion EEG feature extraction, word-level text feature prediction, cosine similarity loss function calculation and self-supervised learning; the specific process is as follows: using a spatiotemporal-frequency dual-branch fusion network to extract the spatiotemporal-frequency fusion EEG feature vector of the word-level EEG sequence, and then using a fully connected layer to predict the word-level text features; then using the cosine similarity loss as the loss function, calculating the cosine similarity of the predicted word-level text features and the word-level text features to supervise the update of the parameters of the spatiotemporal-frequency dual-branch fusion network; until the loss function is fully converged, a trained spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is obtained.

[0015] Preferably, the spatiotemporal-frequency dual-branch fusion network includes a spatiotemporal domain branch and a time-frequency domain residual branch; the spatiotemporal domain branch is composed of a channel space attention mechanism CBAM module, a one-dimensional convolutional layer, a multi-layer transformer encoder, and a residual block, and the spatiotemporal features of the word-level EEG sequence are extracted through the spatiotemporal domain branch. The time-frequency domain residual branch consists of a two-dimensional convolutional layer, a bottleneck layer, and a maximum pooling layer. The time-frequency features of the word-level EEG sequence are extracted through the time-frequency domain residual branch.

[0016] Preferably, in the spatiotemporal domain branch, the spatiotemporal features of the word-level EEG sequence are extracted. The specific steps include:

[0017] First, use the CBAM module to enhance the i The signal-to-noise ratio of the CBAM module is as follows:

[0018] M C (e i )=σ(MLP(AvgPool(e i ))+MLP(MaxPool(e i )))

[0019] Among them, e i To input EEG word-level data, AvgPool and MaxPool are global average pooling and global maximum pooling respectively, MLP is a multi-layer perceptron, σ is a Sigmoid activation function, M C (e i ) is the channel attention weight;

[0020] For the spatial attention module, the specific formula is as follows:

[0021] M S (e i)=σ(Conv2D([AvgPool(e i );MaxPool(e i )]))

[0022] Among them, [AvgPool(e i );MaxPool(e i )] represents the concatenation of channel dimensions, Conv2D is a two-dimensional convolution operation, M S (e i ) is the spatial attention weight;

[0023] Then perform weighted output, the specific formula is as follows:

[0024]

[0025] This is the EEG data after passing through the CBAM module;

[0026] Then, perform convolution operation to The data is input into a one-dimensional convolutional layer and then into a multi-layer transformer encoder to dynamically assign weights to the data and encode it. The specific formula is as follows:

[0027]

[0028] Among them, Conv1D represents a one-dimensional convolutional layer, TransformerEncoder represents a multi-layer transformer encoder, Represents EEG data after being encoded by a multi-layer transformer encoder;

[0029] Then, the residual block is introduced into the network structure. The specific formula is as follows:

[0030]

[0031] Among them, ResidualBlocks represents multi-layer residual blocks, Conv1D represents one-dimensional convolutional layer, For data after passing through multiple layers of residual blocks, AdaptiveAvgPool represents the adaptive average pooling layer. is the set of eigenvalue means;

[0032] Finally, a fully connected layer is used to extract the spatiotemporal features of the word-level EEG sequence. The specific formula is as follows:

[0033]

[0034] Where w is the weight parameter of the fully connected layer, b is the bias of the fully connected layer, is the spatiotemporal feature of the word-level EEG sequence, and the feature dimension is the same as the word-level text feature w i Dimensions remain consistent.

[0035] Preferably, in the time-frequency domain residual branch, the time-frequency features of the word-level EEG sequence are extracted The specific steps include:

[0036] First, continuous wavelet transform (CWT) is applied to extract the spectrogram Fg of the word-level EEG sequence. Based on the spectrogram Fg, a two-dimensional convolutional layer and a parallel bottleneck layer are used to extract the deep spectral features of the word-level EEG sequence. The specific formula is as follows:

[0037]

[0038] Fg represents the word-level EEG sequence e i Spectrum diagram, Conv2D represents the two-dimensional convolution layer, Bottleneck represents the bottleneck layer, The kth hidden output of the spectrogram of the word-level EEG sequence after passing through the kth Bottleneck layer; The first hidden output of the spectrogram representing the word-level EEG sequence after passing through the first Bottleneck layer; The k-1th hidden output of the spectrogram of the word-level EEG sequence after passing through the k-1th Bottleneck layer;

[0039] Finally, the time-frequency features of the word-level EEG sequence are extracted through the output of the adaptive average pooling layer, the flattening layer, and the fully connected layer. The specific formula is as follows:

[0040]

[0041] Represents the output data after the Bottleneck layer after N layers, AdaptiveAvgPool is the adaptive average pooling layer, Flatten is the flattening layer, w is the weight parameter of the fully connected layer, and b is the bias of the fully connected layer. It is the time-frequency feature of the word-level EEG sequence, and the feature dimension is consistent with the spatiotemporal feature of the word-level EEG sequence and the word-level text feature dimension.

[0042] Preferably, the spatiotemporal-frequency fusion EEG feature vector is obtained by fusing the spatiotemporal features of the word-level EEG sequence and the time-frequency features of the word-level EEG sequence extracted by the spatiotemporal-frequency dual-branch fusion network, specifically by By linear splicing

[0043] Preferably, in step S5, in combination with the predicted character-level text features, a beam search Chinese continuous language dynamic decoding model guided by a pre-trained model is used to realize Chinese continuous language text reconstruction, specifically comprising: first, based on the candidate sequence at the i-th time point, introducing the pre-trained model GPT-2 to predict the probability distribution of the next character; then using the beam search method, setting the beam width to 1, selecting the l characters with the highest probability distribution as the extended characters predicted at the i+1-th time point, and calculating the character-level text features corresponding to each character in the extended characters; then calculating the predicted character-level text features corresponding to the l-th time point, and calculating the cosine similarity between the two; finally, the m extended characters with higher cosine similarity scores are combined into a candidate sequence, and the above steps are iterated to predict the next time point until the number of characters in the candidate sequence is consistent with the number of actual continuous language characters, i.e., the continuous language text reconstruction is completed.

[0044] The beneficial effects of the present invention are as follows: Compared with the existing technology, the present invention overcomes the problems of insufficient alignment accuracy between EEG data and language text, inadequate feature extraction, and unsmooth language reconstruction by adopting technologies such as a time-scale-driven word-level EEG synchronization method, a spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model, and a pre-trained model-guided dynamic decoding method. It achieves high-precision continuous language reconstruction in Chinese and significantly improves the accuracy and fluency of language reconstruction. The present invention can deeply mine language information from EEG signals, providing new ideas for the development of brain-computer interfaces, and has important application value and potential prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a method for reconstructing Chinese continuous language text based on EEG signals according to an embodiment of the present invention;

[0046] Figure 2 Schematic diagram comparing the method of the present invention and the traditional EEG text reconstruction method. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] This implementation uses the example of collecting EEG data while listening to audio stimulation. The EEG data were self-collected. All participants were monitored using a Boruikang 64-electrode EEG acquisition device. During data collection, participants were required to maintain a comfortable and relaxed state and focus their attention while listening to the audio stimulation, minimizing blinking and swallowing.

[0049] The present invention provides a technical solution: a method for reconstructing Chinese continuous language text based on EEG signals, such as Figure 1 As shown, the following steps are included:

[0050] S1. Select a diverse Chinese corpus and preprocess it as audio stimulus material.

[0051] Furthermore, the Chinese corpus used was sourced from Tonight's 80s Talk Show, Story FM, and Shengdongjixi. All audio stimuli were in Chinese. Tonight's 80s Talk Show was a stand-up talk show, while Story FM and Shengdongjixi were interview-style dialogue programs, all using Mandarin.

[0052] Based on the screened audio stimulus materials, preprocessing is performed, each audio segment is cut into 3-5 minutes, and the audio sampling frequency is unified to 44.1kHz.

[0053] S2. Based on the audio stimulation materials, the subjects collected EEG data for the auditory attention experiment and performed data preprocessing on the collected EEG data.

[0054] EEG data from the participants were collected and preprocessed for an auditory attention experiment. The auditory attention experiment consisted of multiple acquisition sessions. In each session, participants were instructed to focus on listening to audio through headphones. Each session lasted approximately 23 minutes. Each session involved pre-experiment preparation, data collection, questioning, and rest periods. Pre-experiment preparation primarily involved the experimenter fitting and adjusting the equipment for the participants, including correctly fitting the EEG cap and headphones, adjusting electrode positions, and inquiring about their status. Data collection began with a one-minute resting period to ensure the EEG signal had returned to normal levels. The experimenter then controlled audio playback and began collecting EEG data from the audio stimulus material. After the audio playback concluded, the experimenter asked the participants to answer two questions related to the audio content. Based on the correctness of the questions, the experimenter decided whether to re-acquire data for the audio. After the questions were completed, the participants rested for three minutes to avoid fatigue.

[0055] To improve the signal-to-noise ratio (SNR) of the EEG data, the collected EEG data were preprocessed, including channel removal, notch filtering, bandpass filtering, and independent principal component analysis. Specifically, channels related to the electrocardiogram (ECG) and electrooculogram (EOG) were removed. A 50Hz notch filter was then applied to eliminate power line interference. A 1-50Hz bandpass filter was then applied to remove DC components and higher-frequency noise. Finally, independent component analysis was performed to remove artifacts, resulting in the preprocessed EEG data. All EEG data described below are preprocessed EEG data.

[0056] S3. A time-scale-driven word-level EEG synchronization method is proposed to extract word-level text features and word-level EEG sequences. The time-scale-driven word-level EEG synchronization method mainly includes two parts: word-level text feature extraction and word-level EEG sequence extraction.

[0057] Furthermore, in step S3, a time-scale driven word-level-EEG synchronization method is proposed to extract word-level text features, which specifically includes: first, based on the audio stimulus material, the audio-to-text interface of iFlytek and manual calibration are used to complete the audio-to-text conversion, and then the ninth hidden layer of the pre-trained large model GPT-2 is used to extract the text features corresponding to each word in the text, that is, the 768-dimensional text embedding vector, and finally the text embedding vector is normalized to obtain the word-level text feature w i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

[0058] Furthermore, in step S3, a time-scale-driven word-level-EEG synchronization method is proposed to extract word-level EEG sequences, which specifically includes: first, automatically aligning the audio with the extracted text using the MFA forced alignment tool; then, manually checking and correcting the alignment of the audio and text using the Praat tool to ensure complete alignment; then, obtaining the timestamp corresponding to each word when it appears in the audio; then, based on the timestamp corresponding to each word, dividing the EEG data using a fixed window method, specifically, obtaining the center time point of each word according to the start time and end time of each word, and then dividing the EEG time window based on the average time range of the center time point of each word, which is the word-level EEG sequence e i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

[0059] S4. Based on word-level EEG sequence and word-level text features, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is used to extract spatiotemporal-frequency fusion EEG features and predict their word-level text features.

[0060] Furthermore, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model of step S4 includes four steps: spatiotemporal-frequency fusion EEG feature extraction, word-level text feature prediction, cosine similarity loss function calculation and self-supervised learning; the specific process is as follows: using the spatiotemporal-frequency dual-branch fusion network to extract the spatiotemporal-frequency fusion EEG feature vector of the word-level EEG sequence, and then using the fully connected layer to predict the word-level text features; then using the cosine similarity loss as the loss function, the cosine similarity of the predicted word-level text features and the word-level text features is calculated to supervise the update of the parameters of the spatiotemporal-frequency dual-branch fusion network; until the loss function is fully converged, a trained spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is obtained.

[0061] Furthermore, the spatiotemporal-frequency dual-branch fusion network can complete feature extraction and feature fusion. The two branches include a spatiotemporal domain branch and a time-frequency domain residual branch. The spatiotemporal domain branch is composed of a channel space attention mechanism CBAM (Convolutional Block Attention Module) module, a one-dimensional convolutional layer, a multi-layer transformer encoder, and a residual block. The spatiotemporal features of the word-level EEG sequence are extracted through the spatiotemporal domain branch. The time-frequency domain residual branch consists of a two-dimensional convolutional layer, a bottleneck layer, and a maximum pooling layer. The time-frequency features of the word-level EEG sequence are extracted through the time-frequency domain residual branch.

[0062] Furthermore, in the spatiotemporal domain branch, the spatiotemporal features of the word-level EEG sequence are extracted. The specific steps include:

[0063] First, use the CBAM module to enhance the i The signal-to-noise ratio of the CBAM module is as follows:

[0064] M C (e i )=σ(MLP(AvgPool(e i ))+MLP(MaxPool(e i )))

[0065] Among them, e i To input EEG word-level data, AvgPool and MaxPool are global average pooling and global maximum pooling respectively, MLP is a multi-layer perceptron, σ is a Sigmoid activation function, M C (e i ) is the channel attention weight;

[0066] For the spatial attention module, the specific formula is as follows:

[0067] M S (e i )=σ(Conv2D([AvgPool(e i );MaxPool(e i )]))

[0068] Among them, [AvgPool(e i );MaxPool(e i )] represents the concatenation of channel dimensions, Conv2D is a two-dimensional convolution operation, M S (e i ) is the spatial attention weight;

[0069] Then perform weighted output, the specific formula is as follows:

[0070]

[0071] This is the EEG data after passing through the CBAM module; through the CBAM module, the spatiotemporal domain branch can effectively improve the signal-to-noise ratio of the EEG signal, thereby extracting key spatiotemporal features and providing a purer input signal for subsequent one-dimensional convolution and Transformer encoder.

[0072] Then, perform convolution operation to The data is input into a one-dimensional convolutional layer and then into a multi-layer transformer encoder to dynamically assign weights to the data and encode it. The specific formula is as follows:

[0073]

[0074] Among them, Conv1D represents a one-dimensional convolutional layer, TransformerEncoder represents a multi-layer transformer encoder, Represents EEG data after being encoded by a multi-layer transformer encoder;

[0075] In addition, in order to improve the feature extraction ability and overall robustness of the model, the residual block is introduced into the network structure. The specific formula is as follows:

[0076]

[0077] Among them, ResidualBlocks represents multi-layer residual blocks, Conv1D represents one-dimensional convolutional layer, For data after passing through multiple layers of residual blocks, AdaptiveAvgPool represents the adaptive average pooling layer. is the set of eigenvalue means;

[0078] Finally, a fully connected layer is used to extract the spatiotemporal features of the word-level EEG sequence. The specific formula is as follows:

[0079]

[0080] Where w is the weight parameter of the fully connected layer, b is the bias of the fully connected layer, is the spatiotemporal feature of the word-level EEG sequence, and the feature dimension is the same as the word-level text feature w i Dimensions remain consistent.

[0081] Furthermore, in the time-frequency domain residual branch, the time-frequency features of the word-level EEG sequence are extracted. The specific steps include:

[0082] First, based on word-level EEG sequences e i , apply continuous wavelet transform (CWT) to extract the spectrogram Fg of the word-level EEG sequence; then, based on the spectrogram Fg, use a two-dimensional convolutional layer and a parallel bottleneck layer to extract the deep spectral features of the word-level EEG sequence. The specific formula is as follows:

[0083]

[0084] Fg represents the word-level EEG sequence e i Spectrum diagram, Conv2D represents the two-dimensional convolution layer, Bottleneck represents the bottleneck layer, The kth hidden output of the spectrogram of the word-level EEG sequence after passing through the kth Bottleneck layer; The first hidden output of the spectrogram representing the word-level EEG sequence after passing through the first Bottleneck layer; Represents the k-1th hidden output of the spectrogram of the word-level EEG sequence after passing through the k-1th Bottleneck layer.

[0085] Finally, the time-frequency features of the word-level EEG sequence are extracted through the output of the adaptive average pooling layer, the flattening layer, and the fully connected layer. The specific formula is as follows:

[0086]

[0087] Represents the output data after the Bottleneck layer after N layers, AdaptiveAvgPool is the adaptive average pooling layer, Flatten is the flattening layer, w is the weight parameter of the fully connected layer, b is the bias of the fc layer, It is the time-frequency feature of the word-level EEG sequence, and the feature dimension is consistent with the spatiotemporal feature of the word-level EEG sequence and the word-level text feature dimension.

[0088] The spatiotemporal-frequency fusion EEG feature vector is obtained by fusing the spatiotemporal features of the word-level EEG sequence and the time-frequency features of the word-level EEG sequence extracted by the spatiotemporal-frequency dual-branch fusion network. Specifically, By linear splicing

[0089] EEG feature vector fusion based on time, space and frequency Word-level text feature prediction via fully connected layers Then based on the predicted word-level text features and word-level text features w iUsing cosine similarity loss as the loss function, the cosine similarity between the two is calculated to update the parameters of the spatiotemporal-frequency dual-branch fusion network until the loss function fully converges, completing the training of the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model. In the subsequent decoding process, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model can be used to generate predicted word-level text features.

[0090] S5. Based on the beam search Chinese continuous language dynamic decoding model guided by the pre-trained model, combined with the predicted word-level text features, Chinese continuous language text reconstruction is realized.

[0091] Furthermore, in step S5, combined with the predicted word-level text features, a beam search of the Chinese continuous language dynamic decoding model guided by a pre-trained model is used to achieve Chinese continuous language text reconstruction, specifically including: first, based on the candidate sequence at the i-th time point (the candidate sequence of the initial time period is generated by beam initialization), the pre-trained model GPT-2 is introduced to predict the probability distribution of the next word; then, using the beam search method, the beam width is set to 1, and the l words with the highest probability distribution are selected as the extended words predicted at the i+1-th time point (corresponding to the predicted second word), and the word-level text features corresponding to each word in the extended words are calculated; then, the predicted word-level text features corresponding to the l-th time point are calculated, and the cosine similarity between the two is calculated; finally, the m extended words with higher cosine similarity scores are combined into a candidate sequence, and the above steps are iterated to predict the next time point until the number of words in the candidate sequence (predicted continuous language) is consistent with the number of words in the actual continuous language, that is, the continuous language text reconstruction is completed. Note that the number of words contained in the predicted continuous language is the same as the number of time points.

[0092] The continuous language decoding of the present invention is a two-stage non-end-to-end process. First, based on the audio stimulus material and the corresponding collected EEG data, the time-scale-driven word-level-EEG synchronization method is used to extract the word-level text features and word-level EEG sequences corresponding to each word. Then, based on all word-level text features and word-level EEG sequences, a spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is trained. Then, using a section of EEG data that has not participated in the training, a beam search Chinese continuous language dynamic decoding model guided by a pre-trained model is used to predict Chinese continuous language text, and the BERTScore and BLEU-N indicators are used to evaluate the results of the real audio stimulus material corresponding to the EEG data. Figure 2 As shown in Figure 3, the results show that the proposed method outperforms randomly generated Chinese continuous language and the traditional decoding model.

[0093] The present invention first proposes an EEG data acquisition paradigm based on Chinese audio stimulation to fill the gap in this field for Chinese audio stimulation EEG data. Next, the present invention proposes a time-space-frequency dual-branch fusion network to perform comprehensive feature extraction of EEG data from the time-space domain and the time-frequency domain. Finally, in the continuous text generation stage, the present invention utilizes the advantages of large language models in the field of text generation and adopts a beam search Chinese continuous language dynamic decoding algorithm to generate more accurate and smoother continuous Chinese text. From the results, the Chinese continuous language reconstructed by the present invention based on EEG data has restored semantics to a certain extent and is effective. The present invention has made certain contributions to the development of brain-computer interfaces and Chinese natural language processing.

[0094] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0095] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0096] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0097] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0098] The references to "first" and "second" in the embodiments merely distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0099] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for reconstructing Chinese continuous language text based on EEG signals, characterized in that: The steps include: S1. Select a diverse Chinese corpus and pre-process it as audio stimulus material; S2. Based on the audio stimulation materials, the subjects collected EEG data for the auditory attention experiment and performed data preprocessing on the collected EEG data; S3. A time-scale-driven word-level-EEG synchronization method is proposed to extract word-level text features and word-level EEG sequences. S4. Based on word-level EEG sequence and word-level text features, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is used to extract spatiotemporal-frequency fusion EEG features and predict their word-level text features; S5. Based on the beam search Chinese continuous language dynamic decoding model guided by the pre-trained model, combined with the predicted word-level text features, Chinese continuous language text reconstruction is realized.

2. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 1, characterized in that: In step S2, the collected EEG data are preprocessed, including channel removal, notch filtering, bandpass filtering, and independent principal component analysis; specifically, the channels related to the electrocardiogram and electrooculogram are first removed, followed by a 50 Hz notch filter to eliminate power line interference, and then a 1-50 Hz bandpass filter is used to filter out DC components and high-frequency noise, and finally an independent component analysis is performed to remove artifacts to obtain the preprocessed EEG data.

3. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 1, characterized in that: In step S3, a time-scale driven word-level EEG synchronization method is proposed to extract word-level text features. Specifically, the method includes: first, based on the audio stimulus material, audio-to-text interface and manual calibration are used to complete the audio-to-text conversion. Then, the ninth hidden layer of the pre-trained large model GPT-2 is used to extract the text features corresponding to each word in the text, that is, the 768-dimensional text embedding vector. Finally, the text embedding vector is normalized to obtain the word-level text feature w i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

4. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 1, characterized in that: In step S3, a time-scale driven word-level EEG synchronization method is proposed to extract word-level EEG sequences, which specifically includes: firstly, using the MFA forced alignment tool to automatically align the audio with the extracted text; then, using the Praat tool to manually check and correct the alignment of the audio and text to ensure complete alignment; then, obtaining the timestamp corresponding to each word when it appears in the audio; then, based on the timestamp corresponding to each word, using the fixed window method to divide the EEG data, specifically, according to the start time and end time of each word, the center time point of the word is obtained; secondly, the EEG time window is divided based on the average time range of the center time point of each word. The EEG time window is the word-level EEG sequence e i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

5. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 1, characterized in that: The spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model of step S4 includes four steps: spatiotemporal-frequency fusion EEG feature extraction, word-level text feature prediction, cosine similarity loss function calculation and self-supervised learning; the specific process is as follows: using the spatiotemporal-frequency dual-branch fusion network to extract the spatiotemporal-frequency fusion EEG feature vector of the word-level EEG sequence, and then using the fully connected layer to predict the word-level text features; then using the cosine similarity loss as the loss function, the cosine similarity of the predicted word-level text features and the word-level text features is calculated to supervise the update of the parameters of the spatiotemporal-frequency dual-branch fusion network; until the loss function is fully converged, the trained spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is obtained.

6. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 5, characterized in that: The spatiotemporal-frequency dual-branch fusion network includes a spatiotemporal domain branch and a time-frequency domain residual branch; the spatiotemporal domain branch is composed of a channel space attention mechanism CBAM module, a one-dimensional convolutional layer, a multi-layer transformer encoder, and a residual block. The spatiotemporal features of the word-level EEG sequence are extracted through the spatiotemporal domain branch. The time-frequency domain residual branch consists of a two-dimensional convolutional layer, a bottleneck layer, and a maximum pooling layer. The time-frequency features of the word-level EEG sequence are extracted through the time-frequency domain residual branch.

7. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 6, characterized in that: In the spatiotemporal domain branch, the spatiotemporal features of word-level EEG sequences are extracted. The specific steps include: First, use the CBAM module to enhance the i The signal-to-noise ratio of the CBAM module is as follows: M C (e i )=σ(MLP(AvgPool(e i ))+MLP(MaxPool(e i ))) Among them, e i To input EEG word-level data, AvgPool and MaxPool are global average pooling and global maximum pooling respectively, MLP is a multi-layer perceptron, σ is a Sigmoid activation function, M C (e i ) is the channel attention weight; For the spatial attention module, the specific formula is as follows: M S (And i )=σ(Conv2D([AvgPool(e i );MaxPool(e i )])) Among them, [AvgPool(e i );MaxPool(e i )] represents the concatenation of channel dimensions, Conv2D is a two-dimensional convolution operation, M S (e i ) is the spatial attention weight; Then perform weighted output, the specific formula is as follows: This is the EEG data after passing through the CBAM module; Then, perform convolution operation to The data is input into a one-dimensional convolutional layer and then into a multi-layer transformer encoder to dynamically assign weights to the data and encode it. The specific formula is as follows: Among them, Conv1D represents a one-dimensional convolutional layer, TransformerEncoder represents a multi-layer transformer encoder, Represents EEG data after being encoded by a multi-layer transformer encoder; Then, the residual block is introduced into the network structure. The specific formula is as follows: Among them, ResidualBlocks represents multi-layer residual blocks, Conv1D represents one-dimensional convolutional layer, For data after passing through multiple layers of residual blocks, AdaptiveAvgPool represents the adaptive average pooling layer. is the set of eigenvalue means; Finally, a fully connected layer is used to extract the spatiotemporal features of the word-level EEG sequence. The specific formula is as follows: Where w is the weight parameter of the fully connected layer, b is the bias of the fully connected layer, is the spatiotemporal feature of the word-level EEG sequence, and the feature dimension is the same as the word-level text feature w i Dimensions remain consistent.

8. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 6, characterized in that: In the time-frequency domain residual branch, the time-frequency features of the word-level EEG sequence are extracted The specific steps include: First, continuous wavelet transform (CWT) is applied to extract the spectrogram Fg of the word-level EEG sequence. Based on the spectrogram Fg, a two-dimensional convolutional layer and a parallel bottleneck layer are used to extract the deep spectral features of the word-level EEG sequence. The specific formula is as follows: Fg represents the word-level EEG sequence e i Spectrum diagram, Conv2D represents the two-dimensional convolution layer, Bottleneck represents the bottleneck layer, The kth hidden output of the spectrogram of the word-level EEG sequence after passing through the kth Bottleneck layer; The first hidden output of the spectrogram representing the word-level EEG sequence after passing through the first Bottleneck layer; The k-1th hidden output of the spectrogram of the word-level EEG sequence after passing through the k-1th Bottleneck layer; Finally, the time-frequency features of the word-level EEG sequence are extracted through the output of the adaptive average pooling layer, the flattening layer, and the fully connected layer. The specific formula is as follows: Represents the output data after the Bottleneck layer after N layers, AdaptiveAvgPool is the adaptive average pooling layer, Flatten is the flattening layer, w is the weight parameter of the fully connected layer, and b is the bias of the fully connected layer. It is the time-frequency feature of the word-level EEG sequence, and the feature dimension is consistent with the spatiotemporal feature of the word-level EEG sequence and the word-level text feature dimension.

9. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 5, characterized in that: The spatiotemporal-frequency fusion EEG feature vector is obtained by fusing the spatiotemporal features of the word-level EEG sequence and the time-frequency features of the word-level EEG sequence extracted by the spatiotemporal-frequency dual-branch fusion network. Specifically, By linear splicing 10. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 1, characterized in that: In step S5, combined with the predicted word-level text features, the Chinese continuous language dynamic decoding model guided by the pre-trained model is used to search the beam to realize the reconstruction of the Chinese continuous language text, specifically including: first, based on the candidate sequence at the i-th time point, the pre-trained model GPT-2 is introduced to predict the probability distribution of the next word; then, the beam search method is used to set the beam width to 1, and the l words with the highest probability distribution are selected as the extended words predicted at the i+1-th time point, and the word-level text features corresponding to each word in the extended words are calculated; then, the predicted word-level text features corresponding to the l-th time point are calculated, and the cosine similarity between the two is calculated; finally, the m extended words with higher cosine similarity scores are combined into candidate sequences, and the above steps are iterated to predict the next time point until the number of words in the candidate sequence is consistent with the number of real continuous language words, that is, the reconstruction of the continuous language text is completed.

Images