Chinese continuous language text reconstruction method based on electroencephalogram signals

Abstract

The invention relates to the technical field of natural language processing, and particularly discloses a Chinese continuous language text reconstruction method based on electroencephalogram signals, which comprises the following steps of: selecting a diversified Chinese corpus and preprocessing the Chinese corpus as an audio stimulation material; based on the audio stimulation material, collecting electroencephalogram data of an auditory attention experiment of the subject, and preprocessing the data; providing a time scale driven word level-EEG synchronization method, and extracting word level text features and a word level EEG sequence; extracting space-time-frequency fusion EEG features by using the space-time-frequency fusion EEG-text self-supervised learning coding model, and predicting word-level text features of the space-time-frequency fusion EEG features; and based on a cluster search Chinese continuous language dynamic decoding model guided by a pre-training model, combining the predicted word-level text features to realize Chinese continuous language text reconstruction. According to the method, the relation between the electroencephalogram signals and the text features is fully revealed, so that more natural and smoother text information is generated, and finally the accuracy of Chinese continuous language reconstruction is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a Chinese continuous language text reconstruction method based on electroencephalogram signals. Background Art

[0002] Language is not only a tool for human communication, but also the basis of thinking and intelligence. How the brain decodes and processes language information is the key to revealing the essence of human intelligence. With the rapid development of technologies such as brain-computer interface (BCI), it has become possible to decode natural language from brain activity. When the brain is stimulated by language, the human brain encodes semantic information through neural activity. By analyzing these neural activities, the semantic encoding mechanism in the brain can be revealed. This research direction is not only crucial to the development of cognitive science and neuroscience, but also provides new hope for people who have lost the ability to communicate in language due to neurodegenerative diseases and trauma. The development of this direction will greatly expand the understanding of how the human brain processes language and may open up new ways of communication.

[0003] Regarding the decoding of semantic information, most of the reconstruction of continuous language text is done in Western languages ​​such as English, and related research in Chinese is still lagging behind. The brain's response to Chinese is specific, and Chinese is highly complex, including the diversity of tones, characters, and contextual meanings, which is very different from Western languages. Direct application of existing decoding frameworks and models faces challenges. In addition, the lack of data on Chinese corpus stimulation further limits the development of Chinese brain signal decoding technology. At present, research on continuous language reconstruction in Chinese is still insufficient. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a Chinese continuous language text reconstruction method based on EEG signals. This method can fully characterize the connection between EEG signals and text features, generate more fluent text information, and achieve high-precision Chinese continuous language reconstruction, thus solving the problems mentioned in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for reconstructing Chinese continuous language text based on EEG signals, comprising the following steps:

[0006] S1. Select a diverse Chinese corpus and pre-process it as audio stimulus material;

[0007] S2. Based on the audio stimulation materials, the subjects collected EEG data of the auditory attention experiment and performed data preprocessing on the collected EEG data;

[0008] S3. A time-scale driven word-level-EEG synchronization method is proposed to extract word-level text features and word-level EEG sequences.

[0009] S4, based on word-level EEG sequence and word-level text features, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is used to extract spatiotemporal-frequency fusion EEG features and predict their word-level text features;

[0010] S5. Based on the beam search Chinese continuous language dynamic decoding model guided by the pre-trained model, combined with the predicted word-level text features, Chinese continuous language text reconstruction is achieved.

[0011] Preferably, in step S2, the collected EEG data is preprocessed, including channel removal, notch filtering, bandpass filtering, and independent principal component analysis; specifically, the channels related to the electrocardiogram and electrooculogram are first removed, and then a 50Hz notch filter is performed to eliminate power line interference, and then a 1-50Hz bandpass filter is used to filter out DC components and high-frequency noise, and finally an independent component analysis is performed to remove artifacts to obtain preprocessed EEG data.

[0012] Preferably, in step S3, a time-scale driven word-level-EEG synchronization method is proposed to extract word-level text features, specifically comprising: firstly, based on the audio stimulus material, audio-to-text conversion is completed using the iFlytek audio-to-text interface and manual calibration, then the ninth hidden layer of the pre-trained large model GPT-2 is used to extract the text features corresponding to each word in the text, that is, a 768-dimensional text embedding vector, and finally the text embedding vector is normalized to obtain the word-level text feature w i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

[0013] Preferably, in step S3, a time-scale driven word-level-EEG synchronization method is proposed to extract word-level EEG sequences, specifically comprising: firstly, using the MFA forced alignment tool to automatically align the audio with the extracted text; then, using the Praat tool to manually check and correct the alignment of the audio and the text to ensure complete alignment; then, obtaining the timestamp corresponding to each word when it appears in the audio; then, based on the timestamp corresponding to each word, using the fixed window method to divide the EEG data, specifically, obtaining the center time point of each word according to the start time and the end time of each word, and then dividing the EEG time window based on the average time range of the center time point of each word, the EEG time window is the word-level EEG sequence e i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

[0014] Preferably, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model of step S4 includes four steps: spatiotemporal-frequency fusion EEG feature extraction, word-level text feature prediction, cosine similarity loss function calculation and self-supervised learning; the specific process is as follows: using a spatiotemporal-frequency dual-branch fusion network to extract the spatiotemporal-frequency fusion EEG feature vector of the word-level EEG sequence, and then using a fully connected layer to predict the word-level text features; then using the cosine similarity loss as the loss function, calculating the predicted word-level text features and the cosine similarity of the word-level text features to self-supervise and update the parameters of the spatiotemporal-frequency dual-branch fusion network; until the loss function is fully converged, a trained spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is obtained.

[0015] Preferably, the spatiotemporal-frequency dual-branch fusion network includes a spatiotemporal domain branch and a time-frequency domain residual branch; the spatiotemporal domain branch is composed of a channel space attention mechanism CBAM module, a one-dimensional convolutional layer, a multi-layer transformer encoder, and a residual block, and the spatiotemporal features of the word-level EEG sequence are extracted through the spatiotemporal domain branch. The time-frequency domain residual branch consists of a two-dimensional convolutional layer, a bottleneck layer, and a maximum pooling layer. The time-frequency features of the word-level EEG sequence are extracted through the time-frequency domain residual branch.

[0016] Preferably, in the spatiotemporal domain branch, the spatiotemporal features of the word-level EEG sequence are extracted The specific steps include:

[0017] First, use the CBAM module to enhance the i The signal-to-noise ratio of the CBAM module is as follows:

[0018] M C (e i )=σ(MLP(AvgPool(e i ))+MLP(MaxPool(e i )))

[0019] Among them, e i is the input EEG word-level data, AvgPool and MaxPool are global average pooling and global maximum pooling respectively, MLP is a multi-layer perceptron, σ is a Sigmoid activation function, M C (e i ) is the channel attention weight;

[0020] For the spatial attention module, the specific formula is as follows:

[0021] M S (e i)=σ(Conv2D([AvgPool(e i );MaxPool(e i )]))

[0022] Among them, [;] represents the concatenation of channel dimensions, Conv2D is a two-dimensional convolution operation, and M S (e i ) is the spatial attention weight;

[0023] Then the weighted output is performed, the specific formula is as follows:

[0024]

[0025] This is the EEG data after passing through the CBAM module;

[0026] Next, we perform a convolution operation to The data is input into a one-dimensional convolutional layer and then into a multi-layer transformer encoder to dynamically weight and encode the data. The specific formula is as follows:

[0027]

[0028] Among them, Conv1D represents a one-dimensional convolutional layer, TransformerEncoder represents a multi-layer transformer encoder, Represents EEG data after being encoded by a multi-layer transformer encoder;

[0029] Then, the residual block is introduced into the network structure. The specific formula is as follows:

[0030]

[0031]

[0032] Among them, Conv1D represents a one-dimensional convolutional layer. is the data after passing through multiple layers of residual blocks, AdaptiveAvgPool represents the adaptive average pooling layer, is the set of eigenvalue means;

[0033] Finally, a fully connected layer is used to extract the spatiotemporal features of the word-level EEG sequence. The specific formula is as follows:

[0034]

[0035] Where w is the weight parameter of the fully connected layer, b is the bias of the fully connected layer, is the spatiotemporal feature of the word-level EEG sequence, and the feature dimension is the same as the word-level text feature wi The dimensions remain consistent.

[0036] Preferably, in the time-frequency domain residual branch, the time-frequency features of the word-level EEG sequence are extracted The specific steps include:

[0037] First, continuous wavelet transform (CWT) is applied to extract the spectrogram Fg of the word-level EEG sequence. Based on the spectrogram Fg, a two-dimensional convolutional layer and a parallel bottleneck layer are used to extract the deep spectral features of the word-level EEG sequence. The specific formula is as follows:

[0038]

[0039] Fg represents the word-level EEG sequence e i Spectrum diagram, Conv2D represents a two-dimensional convolutional layer, Bottleneck represents a bottleneck layer, The kth hidden output of the spectrogram representing the word-level EEG sequence after passing through the kth Bottleneck layer; The first hidden output of the spectrogram representing the word-level EEG sequence after passing through the first Bottleneck layer; The k-1th hidden output of the spectrogram representing the word-level EEG sequence after passing through the k-1th Bottleneck layer;

[0040] Finally, the time-frequency features of the word-level EEG sequence are extracted through the output of the adaptive average pooling layer, the flattening layer, and the fully connected layer. The specific formula is as follows:

[0041]

[0042] It represents the output data after the Bottleneck layer after N layers, AdaptiveAvgPool is the adaptive average pooling layer, Flatten is the flattening layer, w is the weight parameter of the fully connected layer, b is the bias of the fc layer, It is the time-frequency feature of the word-level EEG sequence, and the feature dimension is consistent with the spatiotemporal feature of the word-level EEG sequence and the word-level text feature dimension.

[0043] Preferably, the spatiotemporal-frequency fusion EEG feature vector is obtained by fusing the spatiotemporal features of the word-level EEG sequence extracted by the spatiotemporal-frequency dual-branch fusion network and the time-frequency features of the word-level EEG sequence. Through linear splicing

[0044] Preferably, in step S5, in combination with the predicted character-level text features, a beam search Chinese continuous language dynamic decoding model guided by a pre-trained model is used to realize Chinese continuous language text reconstruction, specifically comprising: first, based on the candidate sequence at the i-th time point, the pre-trained model GPT-2 is introduced to predict the probability distribution of the next character; then, the beam search method is used, the beam width is set to 1, and the l characters with the highest probability distribution are selected as the extended characters predicted at the i+1-th time point, and the character-level text features corresponding to each character in the extended characters are calculated; then, the predicted character-level text features corresponding to the l-th time point are calculated, and the cosine similarity between the two is calculated; finally, the m extended characters with higher cosine similarity scores are combined into a candidate sequence, and the above steps are iterated to predict the next time point until the number of characters in the candidate sequence is consistent with the number of actual continuous language characters, that is, the continuous language text reconstruction is completed.

[0045] The beneficial effects of the present invention are as follows: compared with the prior art, the present invention adopts technologies such as a time-scale driven word-level EEG synchronization method, a spatiotemporal frequency fusion EEG-text self-supervised learning encoding model, and a pre-trained model-guided dynamic decoding method, which makes up for the problems of insufficient alignment accuracy between EEG data and language text, insufficient feature extraction, and unsmooth language reconstruction, and achieves high-precision Chinese continuous language reconstruction, significantly improving the accuracy and fluency of language reconstruction. The present invention can deeply mine language information in EEG signals, provides new ideas for the development of brain-computer interfaces, and has important application value and potential prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a method for reconstructing Chinese continuous language text based on EEG signals according to an embodiment of the present invention;

[0047] Figure 2 Schematic diagram for comparing the method of the present invention with the traditional EEG text reconstruction method. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] This implementation uses the example of collecting EEG data while listening to audio stimulation, and the EEG data is self-collected data. All subjects collected EEG data while listening to audio using the Bo Rui Kang 64-wet electrode EEG acquisition device. During the data collection process, the subjects maintained a comfortable and relaxed state, and when listening to audio stimulation, they needed to concentrate and reduce actions such as blinking and swallowing saliva.

[0050] The present invention provides a technical solution: a method for reconstructing Chinese continuous language text based on EEG signals, such as Figure 1 As shown, the following steps are included:

[0051] S1. Select a diverse Chinese corpus and preprocess it as audio stimulus material.

[0052] Furthermore, the Chinese corpus used comes from: Tonight's 80s Talk Show, Story FM, and Shengdongjixi. All audio stimuli are in Chinese, among which Tonight's 80s Talk Show is a one-person talk show, and Story FM and Shengdongjixi are interview-type dialogue programs, all of which use Mandarin.

[0053] Based on the screened audio stimulus materials, preprocessing was performed, each audio segment was cut into 3-5 minutes, and the audio sampling frequency was unified to 44.1kHz.

[0054] S2. Based on the audio stimulation materials, the subjects collected EEG data of the auditory attention experiment and performed data preprocessing on the collected EEG data.

[0055] The subjects collected EEG data for the auditory attention experiment and performed EEG data preprocessing. The auditory attention experiment includes multiple collection sessions. In each session, each subject is required to concentrate on listening to the audio in the headphones. Each session is about 23 minutes. In each session, pre-experimental preparation, data collection, questioning and subject rest steps are carried out respectively. The pre-experimental preparation mainly involves the experimenter wearing and adjusting the equipment for the subjects, mainly including correctly wearing the EEG cap, wearing headphones, adjusting the electrode position and asking the subject about the status. Data collection is to first collect the resting state of the subject for 1 minute to ensure that the EEG signal returns to normal level. Then the experimenter controls the audio playback and starts to collect EEG data of listening to audio stimulation materials. After the audio playback ends, the experimenter asks the subject to answer 2 questions related to the audio content. According to the correctness of the questions, the experimenter decides whether to re-collect the data of the audio; after the questions are over, the subject rests for 3 minutes to avoid brain fatigue effect.

[0056] In order to improve the signal-to-noise ratio of EEG data, the collected EEG data were preprocessed, including channel removal, notch filtering, bandpass filtering, and independent principal component analysis. Specifically, the channels related to the electrocardiogram and electrooculogram were first removed, and then a 50Hz notch filter was used to eliminate power line interference, and a 1-50Hz bandpass filter was used to filter out DC components and higher-frequency noise. Finally, independent component analysis was performed to remove artifacts to obtain preprocessed EEG data. The EEG data described below are all preprocessed EEG data.

[0057] S3. A time-scale driven word-level-EEG synchronization method is proposed to extract word-level text features and word-level EEG sequences. The time-scale driven word-level-EEG synchronization method mainly includes two parts: word-level text feature extraction and word-level EEG sequence extraction.

[0058] Furthermore, in step S3, a time-scale driven word-level-EEG synchronization method is proposed to extract word-level text features, which specifically includes: firstly, based on the audio stimulus material, the audio-to-text interface of iFlytek and manual calibration are used to complete the audio-to-text conversion, and then the ninth hidden layer of the pre-trained large model GPT-2 is used to extract the text features corresponding to each word in the text, that is, the 768-dimensional text embedding vector, and finally the text embedding vector is normalized to obtain the word-level text feature w i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

[0059] Furthermore, in step S3, a time-scale driven word-level-EEG synchronization method is proposed to extract word-level EEG sequences, which specifically includes: firstly, the audio is automatically aligned with the extracted text using the MFA forced alignment tool; then, the alignment of the audio and the text is manually checked and corrected using the Praat tool to ensure complete alignment; then, the timestamp corresponding to each word when it appears in the audio is obtained; then, based on the timestamp corresponding to each word, the EEG data is divided using a fixed window method. Specifically, the center time point of each word is obtained according to the start time and the end time of each word, and then the EEG time window is divided based on the average time range of the center time point of each word. The EEG time window is the word-level EEG sequence e i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

[0060] S4. Based on word-level EEG sequence and word-level text features, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is used to extract spatiotemporal-frequency fusion EEG features and predict their word-level text features.

[0061] Furthermore, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model of step S4 includes four steps: spatiotemporal-frequency fusion EEG feature extraction, word-level text feature prediction, cosine similarity loss function calculation and self-supervised learning; the specific process is as follows: using a spatiotemporal-frequency dual-branch fusion network to extract the spatiotemporal-frequency fusion EEG feature vector of the word-level EEG sequence, and then using a fully connected layer to predict the word-level text features; then using the cosine similarity loss as the loss function, calculating the predicted word-level text features and the cosine similarity of the word-level text features to self-supervise the update of the parameters of the spatiotemporal-frequency dual-branch fusion network; until the loss function is fully converged, a trained spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is obtained.

[0062] Furthermore, the spatiotemporal-frequency dual-branch fusion network can complete feature extraction and feature fusion. The two branches include a spatiotemporal domain branch and a time-frequency domain residual branch. The spatiotemporal domain branch is composed of a channel space attention mechanism CBAM (Convolutional Block Attention Module) module, a one-dimensional convolutional layer, a multi-layer transformer encoder, and a residual block. The spatiotemporal features of the word-level EEG sequence are extracted through the spatiotemporal domain branch. The time-frequency domain residual branch consists of a two-dimensional convolutional layer, a bottleneck layer, and a maximum pooling layer. The time-frequency features of the word-level EEG sequence are extracted through the time-frequency domain residual branch.

[0063] Furthermore, in the spatiotemporal branch, the spatiotemporal features of the word-level EEG sequence are extracted. The specific steps include:

[0064] First, use the CBAM module to enhance the i The signal-to-noise ratio of the CBAM module is as follows:

[0065] M C (e i )=σ(MLP(AvgPool(e i ))+MLP(MaxPool(e i )))

[0066] Among them, e i is the input EEG word-level data, AvgPool and MaxPool are global average pooling and global maximum pooling respectively, MLP is a multi-layer perceptron, σ is a Sigmoid activation function, M C (e i ) is the channel attention weight;

[0067] For the spatial attention module, the specific formula is as follows:

[0068] M S (e i )=σ(Conv2D([AvgPool(e i );MaxPool(e i )]))

[0069] Among them, [;] represents the concatenation of channel dimensions, Conv2D is a two-dimensional convolution operation, and M S (e i ) is the spatial attention weight;

[0070] Then the weighted output is performed, the specific formula is as follows:

[0071]

[0072] This is the EEG data after passing through the CBAM module. Through the CBAM module, the spatiotemporal domain branch can effectively improve the signal-to-noise ratio of the EEG signal, thereby extracting key spatiotemporal features and providing a purer input signal for subsequent one-dimensional convolution and Transformer encoders.

[0073] Next, we perform a convolution operation to The data is input into a one-dimensional convolutional layer and then into a multi-layer transformer encoder to dynamically weight and encode the data. The specific formula is as follows:

[0074]

[0075] Among them, Conv1D represents a one-dimensional convolutional layer, TransformerEncoder represents a multi-layer transformer encoder, Represents EEG data after being encoded by a multi-layer transformer encoder;

[0076] In addition, in order to improve the feature extraction ability and overall robustness of the model, the residual block is introduced into the network structure. The specific formula is as follows:

[0077]

[0078] Among them, Conv1D represents a one-dimensional convolutional layer. is the data after passing through multiple layers of residual blocks, AdaptiveAvgPool represents the adaptive average pooling layer, is the set of eigenvalue means;

[0079] Finally, a fully connected layer is used to extract the spatiotemporal features of the word-level EEG sequence. The specific formula is as follows:

[0080]

[0081] Where w is the weight parameter of the fully connected layer, b is the bias of the fully connected layer, is the spatiotemporal feature of the word-level EEG sequence, and the feature dimension is the same as the word-level text feature w i The dimensions remain consistent.

[0082] Furthermore, in the time-frequency domain residual branch, the time-frequency features of the word-level EEG sequence are extracted The specific steps include:

[0083] First, based on word-level EEG sequences i, apply continuous wavelet transform CWT (Continuous WaveletTransform) to extract the spectrum Fg of the word-level EEG sequence; then, based on the spectrum Fg, use the two-dimensional convolutional layer and the parallel bottleneck layer to extract the deep spectrum features of the word-level EEG sequence. The specific formula is as follows:

[0084]

[0085] Fg represents the word-level EEG sequence e i Spectrum diagram, Conv2D represents a two-dimensional convolutional layer, Bottleneck represents a bottleneck layer, The kth hidden output of the spectrogram representing the word-level EEG sequence after passing through the kth Bottleneck layer; The first hidden output of the spectrogram representing the word-level EEG sequence after passing through the first Bottleneck layer; Represents the k-1th hidden output of the spectrogram of the word-level EEG sequence after passing through the k-1th Bottleneck layer.

[0086] Finally, the time-frequency features of the word-level EEG sequence are extracted through the output of the adaptive average pooling layer, the flattening layer, and the fully connected layer. The specific formula is as follows:

[0087]

[0088] It represents the output data after the Bottleneck layer after N layers, AdaptiveAvgPool is the adaptive average pooling layer, Flatten is the flattening layer, w is the weight parameter of the fully connected layer, b is the bias of the fc layer, It is the time-frequency feature of the word-level EEG sequence, and the feature dimension is consistent with the spatiotemporal feature of the word-level EEG sequence and the word-level text feature dimension.

[0089] The spatiotemporal-frequency fusion EEG feature vector is obtained by fusing the spatiotemporal features of the word-level EEG sequence extracted by the spatiotemporal-frequency dual-branch fusion network and the time-frequency features of the word-level EEG sequence. Through linear splicing

[0090] EEG feature vector based on spatiotemporal-frequency fusion Word-level text feature prediction via fully connected layers Then based on the predicted word-level text features and word-level text features w i, taking the cosine similarity loss as the loss function, calculating the cosine similarity of the two, and updating the parameters of the spatiotemporal-frequency dual-branch fusion network, until the loss function is fully converged, and the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model training is completed. In the subsequent decoding process, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model can be used to generate predicted word-level text features.

[0091] S5. Based on the beam search Chinese continuous language dynamic decoding model guided by the pre-trained model, combined with the predicted word-level text features, Chinese continuous language text reconstruction is achieved.

[0092] Further, in step S5, in combination with the predicted character-level text features, the Chinese continuous language dynamic decoding model is searched by the pre-trained model to realize the reconstruction of the Chinese continuous language text, which specifically includes: firstly, based on the candidate sequence at the i-th time point (the candidate sequence of the initial time period is generated by beam initialization), the pre-trained model GPT-2 is introduced to predict the probability distribution of the next character; then, the beam search method is used to set the beam width to l, and the l characters with the highest probability distribution are selected as the extended characters predicted at the i+1-th time point (corresponding to the predicted second character), and the character-level text features corresponding to each character in the extended characters are calculated; then, the predicted character-level text features corresponding to the l-th time point are calculated, and the cosine similarity between the two is calculated; finally, the m extended characters with higher cosine similarity scores are combined into candidate sequences, and the above steps are iterated to predict the next time point until the number of characters in the candidate sequence (predicted continuous language) is consistent with the number of real continuous language characters, that is, the reconstruction of the continuous language text is completed. Note that the number of characters contained in the predicted continuous language is the same as the number of time points.

[0093] The continuous language decoding of the present invention is carried out in two stages, non-end-to-end. First, based on the audio stimulation material and the corresponding collected EEG data, the time-scale-driven word-level-EEG synchronization method is used to extract the word-level text features and word-level EEG sequences corresponding to each word; then, based on all word-level text features and word-level EEG sequences, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is trained; then, a section of EEG data that has not participated in the training is used to predict the Chinese continuous language text based on the beam search Chinese continuous language dynamic decoding model guided by the pre-training model, and the BERTScore and BLEU-N indicators are used to evaluate the results of the real audio stimulation material corresponding to the EEG data. Figure 2 As shown, the results show that the proposed method is better than randomly generated Chinese continuous language and also better than the traditional decoding model.

[0094] The present invention first proposes an EEG data acquisition paradigm based on Chinese audio stimulation to make up for the gap in Chinese audio stimulation EEG data in this field. Next, the present invention proposes a time-space-frequency dual-branch fusion network to perform comprehensive feature extraction of EEG data from the time-space domain and the time-frequency domain. Finally, in the continuous text generation stage, the present invention utilizes the advantages of large language models in the field of text generation and adopts a beam search Chinese continuous language dynamic decoding algorithm to generate more accurate and smoother continuous Chinese text. From the results, the Chinese continuous language reconstructed by the present invention based on EEG data has restored semantics to a certain extent and is effective. The present invention has made certain contributions to the development of brain-computer interfaces and Chinese natural language processing.

[0095] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0096] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0097] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0098] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0099] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0100] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for reconstructing Chinese continuous language text based on EEG signals, characterized in that: The steps include: S1. Select a diverse Chinese corpus and pre-process it as audio stimulus material; S2. Based on the audio stimulation materials, the subjects collected EEG data of the auditory attention experiment and performed data preprocessing on the collected EEG data; S3. A time-scale driven word-level-EEG synchronization method is proposed to extract word-level text features and word-level EEG sequences. S4. Based on word-level EEG sequence and word-level text features, the spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is used to extract spatiotemporal-frequency fusion EEG features and predict their word-level text features; S5. Based on the beam search Chinese continuous language dynamic decoding model guided by the pre-trained model and combined with the predicted word-level text features, Chinese continuous language text reconstruction is realized.

2. The Chinese continuous language text reconstruction method based on EEG signals according to claim 1 is characterized in that: In step S2, the collected EEG data is preprocessed, including channel removal, notch filtering, bandpass filtering, and independent principal component analysis; specifically, the channels related to the electrocardiogram and electrooculogram are first removed, and then a 50Hz notch filter is performed to eliminate power line interference, and then a 1-50Hz bandpass filter is used to filter out DC components and high-frequency noise, and finally an independent component analysis is performed to remove artifacts to obtain the preprocessed EEG data.

3. The Chinese continuous language text reconstruction method based on EEG signals according to claim 1 is characterized in that: In step S3, a time-scale driven word-level-EEG synchronization method is proposed to extract word-level text features, which specifically includes: firstly, based on the audio stimulus material, the audio-to-text interface of iFlytek and manual calibration are used to complete the audio-to-text conversion, and then the ninth hidden layer of the pre-trained large model GPT-2 is used to extract the text features corresponding to each word in the text, that is, the 768-dimensional text embedding vector, and finally the text embedding vector is normalized to obtain the word-level text feature w i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

4. The Chinese continuous language text reconstruction method based on EEG signals according to claim 1 is characterized in that: In step S3, a time-scale driven word-level EEG synchronization method is proposed to extract word-level EEG sequences, which specifically includes: firstly, the audio is automatically aligned with the extracted text using the MFA forced alignment tool; then, the alignment of the audio and text is manually checked and corrected using the Praat tool to ensure complete alignment; then, the timestamp corresponding to each word when it appears in the audio is obtained; then, based on the timestamp corresponding to each word, the EEG data is divided using a fixed window method. Specifically, the center time point of each word is obtained according to the start time and the end time of each word, and then the EEG time window is divided based on the average time range of the center time point of each word. The EEG time window is the word-level EEG sequence e i (i=1,2,...,N), i represents the i-th word, and N represents the total number of words.

5. The Chinese continuous language text reconstruction method based on EEG signals according to claim 1 is characterized in that: The spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model of step S4 includes four steps: spatiotemporal-frequency fusion EEG feature extraction, word-level text feature prediction, cosine similarity loss function calculation and self-supervised learning; the specific process is as follows: using a spatiotemporal-frequency dual-branch fusion network to extract the spatiotemporal-frequency fusion EEG feature vector of the word-level EEG sequence, and then using a fully connected layer to predict the word-level text feature; then using the cosine similarity loss as a loss function, calculating the predicted word-level text feature and the cosine similarity of the word-level text feature to self-supervise and update the parameters of the spatiotemporal-frequency dual-branch fusion network; until the loss function is completely converged, a trained spatiotemporal-frequency fusion EEG-text self-supervised learning encoding model is obtained.

6. The Chinese continuous language text reconstruction method based on EEG signals according to claim 5 is characterized in that: The spatiotemporal dual-branch fusion network includes a spatiotemporal domain branch and a time-frequency domain residual branch; the spatiotemporal domain branch is composed of a channel space attention mechanism CBAM module, a one-dimensional convolutional layer, a multi-layer transformer encoder, and a residual block, and the spatiotemporal features of the word-level EEG sequence are extracted through the spatiotemporal domain branch The time-frequency domain residual branch consists of a two-dimensional convolutional layer, a bottleneck layer, and a maximum pooling layer. The time-frequency features of the word-level EEG sequence are extracted through the time-frequency domain residual branch.

7. The method for reconstructing Chinese continuous language text based on EEG signals according to claim 6, characterized in that: In the spatiotemporal branch, the spatiotemporal features of word-level EEG sequences are extracted. The specific steps include: First, use the CBAM module to enhance the i The signal-to-noise ratio of the CBAM module is as follows: M C (e i )=σ(MLP(AvgPool(e i ))+MLP(MaxPool(e i ))) Among them, e i is the input EEG word-level data, AvgPool and MaxPool are global average pooling and global maximum pooling respectively, MLP is a multi-layer perceptron, σ is a Sigmoid activation function, M C (e i ) is the channel attention weight; For the spatial attention module, the specific formula is as follows: M S (And i )=σ(Conv2D([AvgPool(e i );MaxPool(e i )])) Among them, [;] represents the concatenation of channel dimensions, Conv2D is a two-dimensional convolution operation, and M S (e i ) is the spatial attention weight; Then the weighted output is performed, the specific formula is as follows: This is the EEG data after passing through the CBAM module; Next, we perform a convolution operation to The data is input into a one-dimensional convolutional layer and then into a multi-layer transformer encoder to dynamically weight and encode the data. The specific formula is as follows: Among them, Conv1D represents a one-dimensional convolutional layer, TransformerEncoder represents a multi-layer transformer encoder, Represents EEG data after being encoded by a multi-layer transformer encoder; Then, the residual block is introduced into the network structure. The specific formula is as follows: Among them, Conv1D represents a one-dimensional convolutional layer. is the data after passing through multiple layers of residual blocks, AdaptiveAvgPool represents the adaptive average pooling layer, is the set of eigenvalue means; Finally, a fully connected layer is used to extract the spatiotemporal features of the word-level EEG sequence. The specific formula is as follows: Where w is the weight parameter of the fully connected layer, b is the bias of the fully connected layer, is the spatiotemporal feature of the word-level EEG sequence, and the feature dimension is the same as the word-level text feature w i The dimensions remain consistent.

8. The Chinese continuous language text reconstruction method based on EEG signals according to claim 6 is characterized by: In the time-frequency domain residual branch, the time-frequency features of the word-level EEG sequence are extracted The specific steps include: First, continuous wavelet transform (CWT) is applied to extract the spectrogram Fg of the word-level EEG sequence. Based on the spectrogram Fg, a two-dimensional convolutional layer and a parallel bottleneck layer are used to extract the deep spectral features of the word-level EEG sequence. The specific formula is as follows: Fg represents the word-level EEG sequence e i Spectrum diagram, Conv2D represents a two-dimensional convolutional layer, Bottleneck represents a bottleneck layer, The kth hidden output of the spectrogram representing the word-level EEG sequence after passing through the kth Bottleneck layer; The first hidden output of the spectrogram representing the word-level EEG sequence after passing through the first Bottleneck layer; The k-1th hidden output of the spectrogram representing the word-level EEG sequence after passing through the k-1th Bottleneck layer; Finally, the time-frequency features of the word-level EEG sequence are extracted through the output of the adaptive average pooling layer, the flattening layer, and the fully connected layer. The specific formula is as follows: It represents the output data after the Bottleneck layer after N layers, AdaptiveAvgPool is the adaptive average pooling layer, Flatten is the flattening layer, w is the weight parameter of the fully connected layer, b is the bias of the fc layer, It is the time-frequency feature of the word-level EEG sequence, and the feature dimension is consistent with the spatiotemporal feature of the word-level EEG sequence and the word-level text feature dimension.

9. The Chinese continuous language text reconstruction method based on EEG signals according to claim 5 is characterized in that: The spatiotemporal-frequency fusion EEG feature vector is obtained by fusing the spatiotemporal features of the word-level EEG sequence extracted by the spatiotemporal-frequency dual-branch fusion network and the time-frequency features of the word-level EEG sequence. Through linear splicing 10. The Chinese continuous language text reconstruction method based on EEG signals according to claim 1 is characterized in that: In step S5, in combination with the predicted character-level text features, the beam search Chinese continuous language dynamic decoding model guided by the pre-trained model is used to realize the reconstruction of Chinese continuous language text, which specifically includes: first, based on the candidate sequence at the i-th time point, the pre-trained model GPT-2 is introduced to predict the probability distribution of the next character; then, the beam search method is used to set the beam width to l, and the l characters with the highest probability distribution are selected as the extended characters predicted at the i+1-th time point, and the character-level text features corresponding to each character in the extended characters are calculated; then, the predicted character-level text features corresponding to the l-th time point are calculated, and the cosine similarity between the two is calculated; finally, the m extended characters with higher cosine similarity scores are combined into candidate sequences, and the above steps are iterated to predict the next time point until the number of characters in the candidate sequence is consistent with the number of real continuous language characters, that is, the reconstruction of the continuous language text is completed.

Images