Eeg signal text decoding method based on joint loss framework and post-edit optimization

Abstract

The application discloses a kind of electroencephalogram signal text decoding methods based on joint loss framework and post-editing optimization, belong to brain-computer interface and natural language processing field, including: obtaining electroencephalogram signal, based on electroencephalogram feature extraction network construction classification model, the electroencephalogram signal is input into the classification model, obtains predetermined semantic category;Based on mixed decoding framework construction decoding model, the electroencephalogram signal is input into the decoding model, obtains output feature text;Based on pre-training language model construction post-editing model, based on the predetermined semantic category and the output feature text are post-edited based on the post-editing model, obtain text decoding result.The application is introduced CTC / Attention mixed framework, so that decoding process can be automatically completed without external alignment information, reduce the dependence on external signals such as eye movement, enhance the versatility of method.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of brain-computer interface and natural language processing, and particularly relates to an electroencephalogram signal text decoding method based on a joint loss framework and post-editing optimization. BACKGROUND

[0002] With the development of pre-training models, some studies have begun to attempt to model the EEG text decoding task as a translation task, using an encoder to extract high-level features of the electroencephalogram signal, and then connecting a pre-training model to realize end-to-end training. This kind of method takes advantage of the powerful generation ability of the pre-training model to improve the decoding performance to a certain extent. The model takes the word-level feature sequence of the EEG as input, first inputs it into a multi-layer Transformer encoder, and then obtains the mapped embedding representation through a single-layer feedforward network. Then, the mapped embedding representation is input into the pre-training model BART, and finally the decoded text is generated through the BART model.

[0003] However, directly regarding the EEG text decoding task as a translation task has obvious limitations. The EEG signal has significant time sequence characteristics, and the alignment relationship between its input and output sequence is closer to sequence prediction tasks such as automatic speech recognition (ASR) than to traditional natural language translation tasks. Therefore, directly regarding the EEG text decoding task as a translation task has certain inadaptability. In order to solve this problem, methods such as EEG-TO-TEXT rely on additional information such as eye movement signals to intercept the EEG sequence corresponding to each word and convert it into word-level features. However, in actual use scenarios, such additional information is often difficult to obtain, which makes the deployment and actual application of the model more complex and difficult. On the other hand, due to the limited amount of EEG data, the powerful fitting ability of large pre-training models can easily lead to overfitting, thereby affecting the generalization ability of the model and limiting its effectiveness in practical applications. In addition, existing EEG text decoding methods often have the problem of paying too much attention to low-granularity information (such as word level), resulting in suboptimal quality of the generated text, especially in capturing high-level semantic information of thought content. This is mainly because the loss function during model training usually only includes word-level cross-entropy loss, and the loss function does not explicitly consider high-level semantic information. This to some extent also leads to poor decoding results. SUMMARY

[0004] To solve the above technical problems, the application provides an electroencephalogram signal text decoding method based on a joint loss framework and post-editing optimization, comprising:

[0005] obtaining an electroencephalogram signal, constructing a classification model based on an electroencephalogram feature extraction network, inputting the electroencephalogram signal into the classification model to obtain a predetermined semantic category;

[0006] The decoding model is constructed based on a hybrid decoding framework, the electroencephalogram signal is input into the decoding model, and an output feature text is obtained.

[0007] A post-editing model is constructed based on a pre-trained language model, the predetermined semantic category and the output feature text are post-edited based on the post-editing model, and a text decoding result is obtained.

[0008] Preferably, the process of obtaining the predetermined semantic category comprises: extracting time domain and frequency domain features of the electroencephalogram signal based on the electroencephalogram feature extraction network, and classifying the time domain and frequency domain features of the electroencephalogram signal into the predetermined semantic category.

[0009] Preferably, the hybrid decoding framework comprises an encoder, a classification module and an attention mechanism module.

[0010] Preferably, the process of obtaining the output feature text comprises:

[0011] The electroencephalogram signal is feature extracted and down-sampled based on the encoder, and an extracted signal is obtained.

[0012] The extracted signal is text transcribed based on the attention mechanism module, and a target text sequence is obtained.

[0013] The extracted signal and the target text sequence are sequence aligned based on the classification module, and the output feature text is obtained.

[0014] Preferably, the encoder comprises a front-end feature extractor and an attention encoder; the attention mechanism module is a Transformer decoder; and the classification module is a connection time sequence classification module.

[0015] Preferably, the process of obtaining the text decoding result comprises:

[0016] The predetermined semantic category is taken as a prompt, the output feature text is input into the post-editing model for calculation, the decoded text is grammatically corrected and semantically optimized, and the text decoding result is obtained.

[0017] In another aspect, the present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method when executing the computer program.

[0018] In another aspect, the present application also provides a computer readable storage medium storing a computer program, characterized in that the computer program is executable by a processor to implement the method.

[0019] Compared with the prior art, the present application has the following advantages and technical effects:

[0020] The present application firstly applies a joint decoding framework of connection time classification (CTC) and attention-based decoder to the electroencephalogram (EEG) text decoding task. The CTC decoding branch solves the problem of inconsistent lengths between the input EEG signal and the output text sequence by establishing a time alignment between the EEG signal and the output text. The attention module models the context-dependent relationship of the EEG signal through the attention mechanism, thereby realizing efficient and accurate text generation. The combination of CTC and attention is pioneering in this field. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and of the related description are used to explain the application and are not intended to limit the application. In the drawings:

[0022] Fig. 1 The present application is a decoding model structure schematic diagram of the embodiment;

[0023] Fig. 2 The present application is a method overall flowchart of the embodiment. DETAILED DESCRIPTION

[0024] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0025] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0026] Embodiment one

[0027] As Figs. 1-2 shown, the present embodiment provides an electroencephalogram text decoding method based on a joint loss framework and post-editing optimization, comprising:

[0028] Obtaining an electroencephalogram signal, constructing a classification model based on an electroencephalogram feature extraction network, inputting the electroencephalogram signal into the classification model, and obtaining a predetermined semantic category;

[0029] Constructing a decoding model based on a hybrid decoding framework, inputting the electroencephalogram signal into the decoding model, and obtaining an output feature text;

[0030] A post-editing model is constructed based on a pre-trained language model, and the predetermined semantic categories and the output feature text are post-edited based on the post-editing model to obtain a text decoding result.

[0031] The specific implementation is:

[0032] The post-editing optimization method based on semantic category prediction and pre-trained language model includes a classifier module for predicting the semantic categories of EEG signals and inputting the semantic categories as prompts into a pre-trained language model to combine with decoded text for syntax and semantic optimization, thereby improving the accuracy and fluency of the decoding result.

[0033] The classification model is used to extract semantic category information from the electroencephalogram signal. The model extracts time domain and frequency domain features of the electroencephalogram signal through an electroencephalogram feature extraction network EEGNet and classifies them into predetermined semantic categories. These semantic category information will be input into the post-editing model to guide the post-editing process.

[0034] The decoding model adopts a hybrid decoding framework of CTC (Connectionist Temporal Classification) and Attention. The CTC decoding branch aligns the input electroencephalogram signal with the output text sequence dynamically, allowing sequence generation without explicit alignment. The Attention decoding branch further strengthens context modeling and semantic capture through the attention mechanism, improving the flexibility and accuracy of decoding. The two decoding branches are trained end-to-end, which can improve the feature extraction capability of the encoder, effectively handle the time sequence dependence of the electroencephalogram signal, and ultimately improve the accuracy and fluency of the text generated by the two decoding branches. The decoding model specifically includes:

[0035] Encoder: The encoder is composed of a front-end feature extractor and an attention encoder. The front-end feature extractor is usually composed of several layers of convolutional networks, aiming to extract features and downsample the input signal, reducing computational overhead; in this invention, EEGNet is used as the front-end feature extractor to process the original electroencephalogram signal to adapt to the decoding requirements of the electroencephalogram task. The attention encoder is composed of several stacked Conformer blocks to capture long and short term dependencies and deeply model the input signal.

[0036] Attention decoding branch: This branch uses a Transformer decoder for decoding and uses a cross-entropy loss for training. The attention mechanism of the Transformer allows the model to accurately focus on specific parts of the entire input sequence at each step, extracting only the information from key positions, thereby achieving accurate text transcription; while the cross-entropy loss function models the mapping relationship between the input and output sequences in a non-displayed alignment manner without introducing any conditional independence assumption, ensuring effective end-to-end learning.

[0037] CTC decoding branch: This branch converts the output features of the encoder into a probability distribution of each symbol through a fully connected layer and is constrained by the CTC loss. The CTC loss utilizes the frame-level label information and effectively solves the sequence alignment problem between the input and output by means of the conditional independence assumption.

[0038] The CTC loss function is used to perform timing alignment between the feature sequence of the EEG signal and the target text sequence, and the output intermediate state provides alignment constraints.

[0039] The attention mechanism-based encoder-decoder structure is used to capture the context information of the EEG signal. The signal is feature-extracted by the encoder, and the target text sequence is generated by combining the decoder.

[0040] The CTC module and the Attention module are integrated into a hybrid decoding framework, which utilizes the CTC output to guide the attention alignment process of the Attention, and the two are jointly trained to achieve the complementary advantages of the two decoding methods.

[0041] To further optimize the decoding results, the post-editing model is introduced in this embodiment, which uses the pre-trained language model BART to post-edit the output of the decoding model. The post-editing model takes the text output by the decoding model and the semantic category output by the classification model as input, and performs targeted modification and polishing on the text output by the decoding model. This step effectively improves the quality of the decoding results and ensures the semantic consistency and fluency of the decoding results.

[0042] In this embodiment, the CTC / Attention hybrid decoding framework is used in the decoding part, which performs dynamic alignment between the input EEG signal and the output text through the CTC loss, avoiding the limitations of the fixed alignment mode in traditional methods. At the same time, the Attention part enhances the modeling of the context through the attention mechanism, improving the flexibility and accuracy of the decoding. In the post-editing part, the decoding results of the decoding model are post-edited according to the predicted semantic category, which injects semantic information into them, further improving the accuracy of the decoding. As shown in Table 1, this embodiment achieves a BLEU1 value of 25.31% and a ROUGE1 value of 25.82% on the ZUCO dataset, which is significantly higher than existing methods.

[0043] Table 1

[0044]

[0045] By introducing a pre-trained language model as a post-editing model, combined with the semantic category prompt information provided by the classification model, the post-editing process can optimize the decoding results in a targeted manner, correct grammatical errors, eliminate ambiguities and enhance semantic consistency. This post-editing optimization significantly improves the fluency and semantic consistency of the generated text. Compared with the original decoding model, the BLEU-N (N=2, 3, 4) is increased by 1.94%, 2.93% and 2.90% respectively.

[0046] In another aspect, the embodiment also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method when executing the computer program.

[0047] In another aspect, the embodiment also provides a computer readable storage medium, which stores a computer program, characterized in that the computer program is executed by a processor to implement the method.

[0048] The technical effects of the embodiment are:

[0049] The embodiment adopts a CTC / Attention hybrid decoding framework in the decoding part, dynamically aligns the input EEG signal and the output text through CTC loss, avoids the limitations of the fixed alignment mode in traditional methods, and at the same time, the Attention part enhances the modeling of the context through the attention mechanism, improving the flexibility and accuracy of decoding. In the post-editing part, the decoding results of the decoding model are post-edited according to the predicted semantic categories, and semantic information is injected into them, further improving the accuracy of decoding. The present application achieves a BLEU1 value of 25.08% and a ROUGE1 value of 25.35% on the ZUCO dataset, which is significantly higher than existing methods.

[0050] By introducing a pre-trained language model as a post-editing model, combined with the semantic category prompt information provided by the classification model, the post-editing process can optimize the decoding results in a targeted manner, correct grammatical errors, eliminate ambiguities and enhance semantic consistency. This post-editing optimization significantly improves the fluency and semantic consistency of the generated text. Compared with the EEG-TO-TEXT method, the BLEU4 is increased by 1.05%.

[0051] Traditional EEG text decoding methods often rely on external auxiliary information such as eye movement signals for alignment in the decoding process. However, these information is often difficult to obtain in practical applications, limiting the applicability of the model. The embodiment introduces a CTC / Attention hybrid framework, which enables the decoding process to be completed automatically without external alignment information, reducing the dependence on external signals such as eye movement, and enhancing the generality of the method.

[0052] The above merely provides the preferred embodiment of the present application, and the protection scope of the present application is not limited thereto. Any modification or replacement within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An electroencephalogram signal text decoding method based on a joint loss framework and post-edit optimization, characterized in that, The method comprises the following steps: acquiring an electroencephalogram signal, constructing a classification model based on an electroencephalogram feature extraction network, inputting the electroencephalogram signal into the classification model, and obtaining a predetermined semantic category; constructing a decoding model based on a hybrid decoding framework, inputting the electroencephalogram signal into the decoding model, and obtaining an output feature text; constructing a post-editing model based on a pre-trained language model, post-editing the predetermined semantic category and the output feature text based on the post-editing model, and obtaining a text decoding result; the process of obtaining the predetermined semantic category comprises the following steps: extracting time domain and frequency domain features of the electroencephalogram signal based on the electroencephalogram feature extraction network, and classifying the time domain and frequency domain features of the electroencephalogram signal into the predetermined semantic category; the hybrid decoding framework comprises an encoder, a classification module and an attention mechanism module; the process of obtaining the output feature text comprises the following steps: performing feature extraction and down-sampling on the electroencephalogram signal based on the encoder, and obtaining an extracted signal; performing text transcription on the extracted signal based on the attention mechanism module, and obtaining a target text sequence; performing sequence alignment on the extracted signal and the target text sequence based on the classification module, and obtaining the output feature text; the encoder comprises a front-end feature extractor and an attention encoder; the attention mechanism module is a Transformer decoder; and the classification module is a connection time sequence classification module; the process of obtaining the text decoding result comprises the following steps:

2. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, inputting the output feature text into the post-editing model for calculation by taking the predetermined semantic category as a prompt, performing grammar correction and semantic optimization on the decoded text, and obtaining the text decoding result.

3. A computer-readable storage medium storing a computer program, the computer-readable storage medium being characterized by, The processor executes the computing program to realize the method of claim 1. The computer program is executed by the processor to realize the method of claim 1.

Images